Programming Kraftwerkflows: git-worktree

git-worktree

git-worktree enables us to manage multiple working trees attached to the same repository. Like most (all?) git-* commands, you interact with it via git worktree rather than git-worktree. I came to know git-worktree via a Google search: “working on 2 git branches at the same time”. Thanks Google!

Why use worktree?

I’m glad you asked that. Most codebases are contributed to by a team. That means plenty of pull requests. Pull requests are important even if you’re not on a team, but you’ll inevitably have more of them when a team is involved. Now PRs are great, but often your teammates cannot review your changes immediately. Does that mean you can pack up and go home to drink clamatos preparados? Maybe, but maybe not.

I want to stay productive even when my PR will be sitting there for a couple days. Sometimes the right way to fill that time is with work on another project, sometimes not. Even when you do need to stay hacking on the same project, you can usually simply checkout another branch get your work done there before moving back to the previous branch when your team starts providing feedback. There is another scenario though, and that’s the scenario in which you want to use git-worktree. In this scenario, your team’s feedback starts trickling in while you’re not at a good stopping point in another branch. You want to essentially work on two branches at the same time. Some folks will just cp -r repo repo_2 in this scenario. Those folks might wonder why that’s no good enough. The reasons are profound but quite simple to understand. There are essentially 2:

  1. Copying a large project takes a long time. In a plain cp -r execution, there’s at least two things happening that are unnecessary and take the bulk of that time.
    1. Copying all of node_modules. Using git-worktree in concert w/ a symbolic link can be better here.
    2. Copying all of your git history. git-worktree doesn’t do this.
  2. When you cp -r you’re more likely to end up with the two folders out of sync in a serious way. This isn’t merely anecdotal—git-worktree keep your remotes etc. in sync

How to use it

First, you need to have git 2.5+. Before becoming interested in using git-worktree I was using git 2.2. I simple brew upgrade git gave me git 2.9 and it’s working great! git worktree add ../ignite_002 master will create a new folder named ignite_002 and set it’s head to master, as long as you’re not already on master.

Conclusion

I’m new to git-worktree, but so far I’m finding it a useful workflow upgrade that I’ll apply in a limited set of situations. It represents the biggest improvement to my git workflow in months.
If I’ve piqued your interest, read on:

Cheers and happy Kraftwerking my fellow Gitlians!


Nightwatch Parlance

Nightwatch has its own little semantic world. It’s a world that, while not “on fleek”, makes a lot of sense to me. In a teamgramming context I add a gang of comments if introducing new tech or use thereof. In introducing a new type of UI test to a project at Netflix, I decided to explain some of the Nightwatch parlance. I’ve transposed some of that here for y’all.

Elements

Elements start with an @ and allow you to have improved semantics within a Nightwatch test context. CSS selectors are typical either more verbose or more terse than elements in Nightwatch parlance.

Commands

Commands give you the ability to extend the Nightwatch API w/ your own methods.

Tags

Tags allow you to flexibly group your tests according to your own organization principles, allowing you to execute subsets of all tests.
Within a spec it looks like this:

1
2
3
4
5
6
function createNotepadTestRunner () { }

module.exports = {
tags: [ 'sanity', 'ancillary' ],
NotepadTest: createNotepadTestRunner()
}

An example of leveraging this in concert w/ NPM scripts:

1
2
3
4
5
6
{
"scripts": {
"test:ancillary": "npm t -- --tag ancillary",
"test": "nightwatch -c ./config/nightwatch.js --env chrome"
}

}

Web driver

A web driver is a piece of software that allows you to manipulate a website using the web client’s native interface. Selenium is the web driver, written in Java, that Nightwatch provides a beautiful JavaScript API for. Nightwatch supports most everything Selenium does. It also achieves the remarkable feat of being familiar to both Selenium devs new to JavaScript and JavaScript devs unfamiliar w/ Selenium.


Hating on simplicity: A developer's passion?

It’s common to see talented engineers shit on egalitarian software solutions only because they’re not perfect for them.

That mentality is both socially and technically toxic in a larger successful company that writes enduring software.

There’s a great benefit + beauty in having a very simple system that’s easy to replicate and update—easy to propagate technically & socially—but it’s tough to love simplicity with a belief system that rewards complexity.


CORS 'R' US: CouchDB & Nginx teamwork

Many years after CouchDB debuted, we still see developer after developer—manager after manager—bypass CouchDB only to rebuild exactly what CouchDB offers using a collection of other technologies. As a Netflix employee I’m well aware of the diverse set of needs that can lead to an number of different combinations of datastores and web applications fronting them. However, the majority (a deafening majority in fact) of applications need only a somewhat performant storage mechanism paired with a JSON-HTTP transport mechanism. Why use PostgreSQL w/ an ORM and Spring to achieve the exact same thing CouchDB does by itself? I don’t know if I’ll ever understand why so many teams made the wrong decision there.

The good news is that we don’t need to be one of those teams. You can get on the rapid development train with CouchDB and even take it up a notch by letting Nginx take care of a couple little things that you might be tempted to use Node.js for—but really do not need to. For now I’ll leave you with an Nginx site config that will allow you to use cross-origin resource sharing w/ CouchDB, effectively eliminating the need for any database server programming. Without further ado:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;

root /usr/share/nginx/html;
index index.html index.htm;

# Make site accessible from http://localhost/
server_name localhost;

# http://wiki.apache.org/couchdb/Nginx_As_a_Reverse_Proxy

location / {
# https://michielkalkman.com/snippets/nginx-cors-open-configuration.html

if ($request_method = 'OPTIONS') {
#
# Om nom nom cookies
#
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PUT';
#
# Custom headers and headers various browsers *should* be OK with but aren't
#
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';
#
# Tell client that this pre-flight info is valid for 20 days
#
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 204;
}

if ($request_method = 'PUT') {
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PUT';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';
}

if ($request_method = 'POST') {
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PUT';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';
}

if ($request_method = 'GET') {
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PUT';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';
}

proxy_pass http://localhost:5984;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

location ~ ^/(.*)/_changes {
proxy_pass http://localhost:5984;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}


}

Dictionarius: Thennable

The Womack Dictionarius word of the day is Thennable (sometimes spelled thenable).

Cases

There are two primary cases of this word. I’ve provided example sentences for each case:

Case 1 ~ Adjective

Is the return value of API.getDictionariusEntries() thennable? I want to try await with it

Case 2 ~ Noun

Each of the methods of the API object are Thennables. We feel this is a good convention for any async-heavy networking object.

Duck Reasons

You may ask “why do we need the word/concept thennable given we already have the word/concept Promise?”. I’m glad you asked madam. The reason we need “thennable” is, well, JavaScript (or ducks). JavaScript, like Ruby & Objective-C, makes heavy use of duck-typing. Despite typed-functional-language pundits manufacturing successive declarations of imminent conflagration vis-à-vis dynamically typed proglangs, dynamic typing that’s deferred to runtime (duck typing) has benefits in both unit testing and general programming. The Thennable is a supreme manifestation of said benefits. When a JavaScript function designed for a Promise receives a Thennable (often in a unit testing scenario), things Just Work™. I don’t know about you madam, but I don’t want to conform to a protocol just the unit test a simple function. Example:

1
2
3
4
5
// foo.js

export function promiseMe (prollyAPromiseRight, doSomethingElse) {
return prollyAPromiseRight.then(doSomethingElse)
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// foo.test.js
import spec from 'tape'
import { spy } from 'sinon'
import { promiseMe } from './foo'

spec('foo', ({ test, end : endSpec }) => {
test('promiseMe', ({ ok, end : endTest }) => {
const thennable = { then : spy() }
const doSomethingElse = spy(function () {
ok(thennable.then.calledOnce, 'the object in the 1st position gets "thenned"')
ok(thennable.then.calledWith(doSomethingElse))
ok(doSomethingElse.calledOnce, '`then` is called with the fn in the 2nd position')
endSpec()
})

promiseMe(thennable, doSomethingElse)
})
endSpec()
})

Now I could have used a spied-on Promise in that test. But the experienced unit tester knows they should write the simplest, fastest test that will satisfy the requirements. A Thennable takes up less CPU cycles, less memory and more accurately reflects the requirements of the function than a Promise instance would.

May the “then” last forever!

Until next time in The Womack Dictionarius—keep on “thenning”.


Mass JSON Edits in a Monorepo

As mentioned in the previous post, I’ve migrated a couple key Abacus repos from Stash to Github. I’ve done this in preparation for open source as well as to leverage greater tooling and infrastructure. I like Stash, but it cannot compete with the Github ecosystem.

One of the two repos I’ve migrated is the Abacus monorepo. For context, Abacus consists of 3 main repos:

  1. Abacus Editor App, a reference implementation and future basis for a hosted offering
  2. Abacus Viz Framework, a framework for laying out performant virtual DOM viz with academic principles
  3. Abacus, a monorepo containing many small modules that other project, including the above, are made out of. Why a monorepo? To take advantage of centralized testing & tooling as well as to see what Lerna is made out of

As part of migrating #3, I used The Silver Searcher to find all references to Stash. Most of them were in the Lerna packages folder at packages/abacus-*/package.json. I’m a fan of using the most precise tool you’re effective with, so I opted to use Trent Mick’s json instead of something like sed. Using zsh in concert with json, I was able to precisely edit those package files like so:

1
2
3
4
5
6
7
#!/usr/bin/env zsh

for pkg in packages/abacus-*/package.json
do
echo "Editing $pkg with $1"
json -I -f $pkg -e $1
done
1
./pkedit 'this.repository.url="git@github.com:Netflix/abacus.git"

Hopefully that will help you out in your own massive JSON situations. Cheers y’all!


Happiness is DRY Code, or "Programming Language Stratification"

Cryptic prologue

As George Burns may or may not have said:

Happiness is DRY code in a good shell script

Your code—directed by Matthew Vaughn


In the age of devops and the full-stack developer, it’s common to have many layers of programming in your life. These layers are often treated as classes in sociological sense, where an implicit hierarchy of resource allocation and standards are applied subconciously.

To everything, there is a season


As in sociology, the hierarchy shifts, unshifts, pushes and pops over time.
JavaScript code could be said to be higher up in 2016 than in 2004.
Objective-C might be lower in 2016 than 2010.
Throughout the ages though, it’s been prevalent to see the same programmers that give incredible attention-to-detail in their code throw all caution and decorum to the wind when writing shell scripts.

Shell scripts—no longer just for 90s cyberpunk movies

As the Perl, Ruby and Node.js communities subsequently lowered the barrier to CLIs further and further, you’ve seen the types of devs willing to consume and create CLIs expand. I credit this change with a renewed interest in pure shell programs written for bash and zsh. Alongside this, even within the shell script sphere there’s been a stratification—between public (OSS) and private (proprietary or personal) shell scripts. While proprietary Java code often receives similar scrutiny to OSS Java code, the same doesn’t hold true for scripts. It doesn’t have to, and shouldn’t, be this way though.

Walk the talk

I was compelled to write this post because of the immense benefit I’ve received over time from treating my zsh scripts much like I would JS packages. I put them in version control and all that jazz. But most importantly I endeavor to make the code as readable and DRY as possible. An example of how this benefits me in the day-to-day can be seen in the following real-life example. I had a script like this that’s part of the zsh setup I sync across multiple machines. This piece is a set of aliases for moving into directories I commonly develop within.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
source $ZSH_SCRIPTS_DIR/bootstrap.zsh

alias gpom="git push origin master"

alias zscripts="c $ZSH_SCRIPTS_DIR"

# Personal / OSS
alias ghub="c $GHUB_DIR"
alias bitb="c $BITB_DIR"
alias ghjw="c $GHUB_DIR/jameswomack"
alias exti="c $GHUB_DIR/jameswomack/exploding-video-tiles"

# Netflix
alias stash="c $STASH_DIR"

# Netflix DEA
alias ignite="c $STASH_DIR/dseplat/ignite"
alias nmdash="c $STASH_DIR/nmdash/nmdash"
alias abacuse="c $STASH_DIR/abacus/abacus-editor"
alias abacusa="c $STASH_DIR/abacus/abacus-editor-app"

Now yesterday I migrated a couple of Abacus repos from Stash to Github. That affects my aliases across several machines, and also affects scripts outside the above example. But because I’ve carefully put common paths into variables and already setup syncing across each of my machines, it’s a simple matter of find-and-replace (in one file) + commit and push. If I hadn’t broken out my common paths into variables… if I hadn’t modularized my scripts… if I hadn’t setup up version control and syncing… this seemingly simple task would be rife with room for error. In the end I simply changed this

1
2
alias abacuse="c $STASH_DIR/abacus/abacus-editor"
alias abacusa="c $STASH_DIR/abacus/abacus-editor-app"

to this

1
2
alias abacuse="c $GHUB_DIR/abacus/abacus-editor"
alias abacusa="c $GHUB_DIR/abacus/abacus-editor-app"

and then

1
2
git commit -am "refactor(aka): Abacus Github migration"
ggpush

It’s the little things :)


The Reproducibility Crisis

Schooled


Ever since I was a boy, I’ve had an issue with text books. These sacred tomes filled with confident assertions offered a steadfast view of the universe I lived in, but when I peeked into the text books of others I learned we were each living in a different universe.

Different than my friends’

When I looked at the text books of my friends, who went to other schools, the science they learned was different than my science. It was as if they lived in a different universe. I say that because, the books never said “this is what we think” or “this is our guess about how this works”. No no no no. They said “this is what we know” or “these are the facts”. So at Loma Verde Elementary, the facts were quite different than at Castle Park or Wolf Canyon. Nevermind that the facts were in Spanglish at my Loma Verde—that wasn’t the big deal—it’s that they were as if from another universe than at CP or WC.

Different than my parents


Most of us have had the feeling that our parents are from a different universe. Well I can say with assuredness that my parents truly are from another. How do I know this? I know because I’ve read my mother’s 6th grade science tome. In it, dinosaurs are lizards—not in any way related to birds. In my mom’s universe, the 1990s were filled with flying cars. My 1990s were filled with false promises of the type of bullet train Nihon seemingly had for decades, and a Britney Spears video that sent horomones into overdrive.

School’s Out

The localized reality distortion fields didn’t cease to exist outside of the educational sphere. I could barely turn the corner toward the local market or turn on the TV without another scientific “fact” being blurted out at me.

There is no truth, only sugary refreshment


When Mulder said the truth is out there, he should have also told me it takes as many forms as there are blog posts and subtweets. While it always struck me as odd that scientists always know the truth the whole truth and nothing but the truth—and yet change their mind every 5 minutes—that doesn’t seem to strike anyone else as anything other than awesome. Take The Coca-Cola Company for instance! They spend millions of pounds each year on British scientists that use their exceptional Anglo-Saxon brain power to invent whatever universal truths Coca-Cola asks them to. That’s the Union Jack promise! And while we’re all grateful for CC proving that Mexican Coke is healthy, there’s a greater yarn spinning factory out yonder. That factory is the medical research establishment.

Curing cancer one irreproducible study at a time

Here’s where I get serious—serious enough to stop telling a stupid story and get to the cold hard facts. Essentially, next time you hear there’s a new cure or big breakthrough in cancer reasearch, that might not mean much beyond a scientist getting an award and a few that-a-boys stuffed with Canadian dollars and Prosecco popsicles. The reason?

“…Clinical trials in oncology have the highest failure rate…barriers to clinical development [are] lower than for other disease areas, and a larger number of drugs with suboptimal preclinical validation will enter oncology trials.”

Translation? You don’t need to prove your oncological results to make money or gain respect from them. But don’t listen to my irreproducible blog post. Read the facts:


The everlasting benefit of naming conventions

Take a look at this example .tern-project file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"libs": [
"browser",
"ecma5",
"ecma6"
],

"loadEagerly": [
"./node_modules/abacus-notepad-component/dist/*.js",
"./node_modules/activity-component/lib/*.js",
"./node_modules/component-popup/src/popup.jsx",
"./server/**/*.js",
"./server/*.js",
"./client/src/js/**/*.js"
],

"plugins": {
"node": {}
}

}

We’re using TernJS and it’s loadEagerly option to have intelligent & dynamic autocomplete available in our JavaScript editor setup. It works anywhere from vim to Visual Studio Code. But I digress.

As out project grows, we add more entries. As our number of projects grow, it will likely get copied all over the place, including onto other developers’ machines that you do not control. Even if you did automate (and even control) the propogation and maintenance of this file, you’d have a problem: the module names, file paths and file names aren’t normalized. A file path is an address. Addresses are normalized in society because they serve a purpose that is not reached if humans cannot make assumptions about them. When identifying structures within the United States of America, addresses are generally normalized to meet the following assumptions:

  • The first piece of the identifier is real number, with the vast majority being integers (a small percentage have a vulgar fraction appended). Most importantly, the overwhelming majority of US streets have the odd numbers on one side of the street and the even numbers on the other. Anecdotal evidence shows this to be even more important than chromatic sequence when locating a structure
  • The second piece is almost always the name of the thoroughfare touching the land nearest the official entrance to the structure
  • The (optional) third piece is a sub-identifier representing that identity of the unit within the structure identified by the preceding and succeeding pieces
  • The fourth piece is the city name
  • The fifth piece is the state name
  • The sixth piece is the Zone Improvement Code or ZIP code. It consist of five- and four-digit integer separated by a hypen (or “dash”). This is, in my opinion, one of the weaker parts of the address system as most folks do not know the 4 digit appendage that was introduced in 1983, nor do they usually know many ZIP codes other than their own

Just as the home address system in the USA does, a file path convention will have stronger and weaker aspects to it. All the same, we all know having an address system is better than none, so why would you not have one for files? In retrospect it becomes an obvious choice.

Take a look at an improved .tern-project file, taking into account the lessons of the address system

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
"libs": [
"browser",
"ecma5",
"ecma6"
],

"loadEagerly": [
"./node_modules/abacus-*-component/lib/**/*.js",
"./lib/**/*.js"
],

"plugins": {
"node": {}
}

}

What changes did we make?

  • Name all our team’s custom components using the format teamname-modulename-component
  • Always put our transpiled/consumable JavaScript files in a folder named lib, organized into appropriate subfolders& always using the extension .js. This is the convention, whether it’s a small package or an application

That’s it! We went from 6 entries to 2 just like that.

Happy filemaking!


Automating the hiring process

A representative of an organization called TestDome Ltd contacted me a couple days ago. Today, they pinged me again, asking “Any thoughts on this?”. What they wanted my thoughts on was whether I could use their service of automating using programming quizzes to filter out folks in the (often long-winding) hiring process. My answer was as follows:

My thought is that this type of automation cannot solve the type of hiring problems we have at Netflix. Programming tests embody an extremely poor evaluation of senior programmers. Senior programmers leverage their past experience to effectively combine the best existing solutions in such a way that they’ve created some new, maintainable and sustainable. Senior programmers are good at working with others. They conform to, while incrementally improving, coding style and standards. The only way to pre-suppose about these things is looking through open source contributions, Stack Overflow answers, behavior on Twitter and then pair programming with them. An automated programming quiz merely tests how long they’ve spent doing automated programming quizzes.