The next great frontier in ML: dogs in hats

I’ve been messing around with Keras, a ML library that makes it pretty easy to do AI experiments. I decided to try out image recognition, so I found a picture of Domino in a fez:

Then I wrote the following Python program which uses the existing ResNet50 image recognition library.

import numpy as np
from keras.preprocessing import image
from keras.applications import resnet50

# Load the image
img = image.load_img('fez.jpg', target_size=(224, 224))
input = image.img_to_array(img)
# Keras can process an array of images, but we're only passing one, so turn it into an array with one element.
input = np.expand_dims(input, axis=0)
# Normalize the RGB values into a 0-1 range, since ML algorithms get thrown off by big ranges.
input = resnet50.preprocess_input(input)

# Predict what's in the photo.
model = resnet50.ResNet50()
predictions = model.predict(input)
predicted_classes = resnet50.decode_predictions(predictions, top=10)

for _, name, liklihood in predicted_classes[0]:
    print("this is an image of {} {}".format(name, liklihood))

In addition to the obvious prereqs, you have to sudo apt-get install libhdf5-dev; pip install pillow certifi h5py.

This prints:

this is an image of Bouvier_des_Flandres 0.4082675576210022
this is an image of briard 0.3710797429084778
this is an image of Newfoundland 0.10781265050172806
this is an image of giant_schnauzer 0.04042242094874382
this is an image of Scotch_terrier 0.038422249257564545
this is an image of komondor 0.012891216203570366
this is an image of Tibetan_terrier 0.0026010528672486544
this is an image of affenpinscher 0.0024157813750207424
this is an image of standard_poodle 0.0021669857669621706
this is an image of Kerry_blue_terrier 0.002110496861860156

It’s pretty solid on “it’s a dog.” I’m disappointed in the lack of fez-related IDs. However, I like some of these as possible breeds for Domino:

Tibetan terrier is pretty close, they are relatives.
Newfies are adorable, but the size is a little off.
I have no idea where Komondor came from. They’re amazing looking, but pretty distinct.

So, still needs some work. But not bad for less than 30 lines of code.

GitHub notification… notifier

Here is what my inbox look like each morning:

All those pink-tagged messages are emails from GitHub. Gmail cannot figure out which ones are important and GitHub’s notification stream, in my experience, is useless. It’s noisy and doesn’t clear the way I’d expect. The problem is, people actually do mention me on bugs. And I often have no idea.

So, I made my own notification system. It’s a Chrome extension that is a little ear that sits next to your location bar. If you have an unread GitHub notification, it turns red. If you’re all caught up, it’s green.

If this would be useful to you, please give it a try!

Download the Chrome extension here.

Feedback and suggestions welcome!

Stamping your builds


By default, Bazel tries not to include anything about the system state in build outputs. However, released binaries and libraries often want to include something like the version they were built at or the branch or tag they came from.

To reconcile this, Bazel has an option called the workspace status command. This command is run outside of any sandboxes on the local machine, so it can access anything about your source control, OS, or anything else you might want to include. It then dumps its output into bazel-out/volatile-status.txt, which you can use (and certain language rulesets provide support for accessing from code).

For our example, let’s suppose we’re creating fortune cookie binaries. We want each binary to be “stamped” with a different fortune, so we’ll use the fortune command as our status. Then, our build will add “in bed” to the end of the fortune, since we’re all super mature here.

Let’s create a genrule:

    name = "cookie",
    srcs = [],
    outs = ["fortune-cookie"],
    cmd = "cat bazel-out/volatile-status.txt | grep -v BUILD_TIMESTAMP > $@; echo ' bed.' >> $@",
    stamp = True,

The most important part here is the stamp attribute: this tells the genrule that it depends on the volatile-status.txt file. Without this attribute, the volatile-status.txt file is not a dependency of this rule, so it might not exist when the genrule is run.

cmd prints out this status. The status has a default “BUILD_TIMESTAMP” field as well, so we strip that out.

Now create a script (and make it executable):


/usr/games/fortune -s | tr 'n' ' '

This generates short-ish fortunes and removes all of the newlines (Bazel status files are line-based, each new line is independent and written to volatile-status.txt in any order the status generator feels like).

Now we can build our fortune cookie by supplying the fortune teller to the build:

$ bazel build --stamp --workspace_status_command=$PWD/ //:cookie
INFO: Found 1 target...
Target //:cookie up-to-date:
INFO: Elapsed time: 0.467s, Critical Path: 0.09s

Things to note:

  • You must enable stamping on the command line.
  • You also must pass the full path to the script, otherwise Bazel won’t find it.

Now if your take a look at bazel-genfiles/fortune-cookie:

$ cat bazel-genfiles/fortune-cookie
Truth is the most valuable thing we have -- so let us economize it. 		-- Mark Twain bed.


Snail Spam

When I started blogging, I called my blog “Snail in a Turtleneck,” a cute image that Andrew & I came up with. I drew up my mascot:

A bemused snail, wearing a turtleneck.
A bemused snail, wearing a turtleneck.

and I began posting cartoons I had drawn. I quickly became bored of doing cartoons, and found I was more motivated to put up technical blog posts. Most of my initial readers were coworkers and MongoDB users. When Andrew and I got married, I told my teammates the day before that I’d be out the next day, as I was getting married (we got married at the city clerk’s, so it wasn’t a big production). When I got back to work, I found this at my desk:

A stuffed snail, a snail tape dispenser, and a very lovely bouquet (that was entirely free of snails).
A stuffed snail, a snail tape dispenser, and a very lovely bouquet (that was entirely free of snails).

I was very touched by their thoughtfulness: Andrew and I still have the stuffed snail and I brought the tape dispenser along to Google (where, unfortunately, it was later lost during an intra-office move).

However, as MongoDB gained popularity, some of my posts became very popular and I began to regret the name: customers seemed a little embarrassed to mention they had gotten advice from it and a lot of people didn’t realize that I was actually behind it. I purchased, set up permanent redirects, and basically stopped referencing “Snail in a Turtleneck.” After a couple of years, I let the domain name lapse.

Last week, someone told me that my site had been hacked. I was confused, until they told me it was I took a look and, bizarrely, someone seems to have taken a dump of my site circa 2011, put spam on the index, and put it up at, complete with my artwork, cartoons, etc. The domain was registered through a privacy protection service, so I guess the next step is sending a DCMA takedown notice to the registrar.

Who does this? (I mean, spammers, but… so annoying.)

Four alternative debugging techniques

I’ve recently been working on a side project that uses WebGL and a physics engine that was transpiled from C++ into JavaScript so… printing variable to the console and using the debugger just weren’t cutting it. I started thinking about the other ways I debug things:

  1. Ship of Theseus debugging: the ship of Theseus is a thought experiment: if you have something and you gradually replace every part, at which point is it a different thing? This is debugging via finding a working example, then gradually mutating it (without breaking it, source control is helpful here) into what you actually want to do. There are a couple of problems: this often leaves some cruft around from the original program and I often never figure out why it wasn’t working in the first place. Which brings us to…
  2. Homeland security debugging: if you see a suspicious variable name or method call, don’t keep it to yourself. Tell a coder or library maintainer. This is where you go back through all of the sketchy parts of your code and make sure they are actually doing what you expect. After working on a program for a while, I’ll usually end up with parts of codebase that are a bit questionable. Why am I passing a literal “1” as an argument in here? Why do I have two names for this variable? Basically, I’m going through my program, line by line, checking all of my vague suspicions.
  3. Thunderdome debugging: one coder and one bug enter, one coder leaves. This is kind of a meta-technique I use for weird, difficult-to-reproduce issues in integration tests where I have to dig through logs or do a 12-step process every time I need to test a change. Basically, I get a big cup of coffee, sit down at my machine, and try everything while mainlining caffeine. Afterwards, I generally couldn’t even tell you what the bug ended up being, but it is no more. This is the kind of debugging that generally does not happen unless I’m being paid for it.
  4. Wooden nickel debugging: try testing code against the most useless possible inputs. Sometimes I have a very complex chunk of code that doesn’t work. I don’t want to spend the time to get a meaningful test running against it, but I start writing unit tests, passing in the most trivial input possible: the empty string, 0, an empty array. And often, after an input or two, I’ve figured out what it’s doing wrong.

Anyone else have any non-traditional ways that they debug?

Compilation à la mode

Sundae from Black Tap, incidentally around the corner from the NYC Google office.
Sundae from Black Tap, incidentally around the corner from the NYC Google office.

Bazel lets you set up various “modes” of compilation. There are several built-in (fast, optimized, debug) and you can define your own. The built in ones are:

  • Fast: build your program as quickly as possible. This is generally best for development (when you want a tight compile/edit loop) and is the default, when you don’t specify anything. Your build’s output is generated in bazel-out/local-fastbuild.
  • Optimized: code is compiled to run fast, but may take longer to build. You can get this by running bazel build -c opt //your:target and it’s output will be generated in bazel-out/local-opt. This mode is the best for code that will be deployed.
  • Debug: this leaves in symbols and generally optimizes code for running through a debugger. You can get this by running bazel build -c dbg //your:target and it’s output will be generated in bazel-out/local-dbg

You don’t have to actually know where the build’s output is stored, bazel will update the bazel-bin/bazel-genfiles symlinks for you automatically at the end of your build, so they’ll always point to the right bazel-out subdirectory.

Because each flavor’s output is stored in a different directory, each mode effectively has an entirely separate set of build artifacts, so you can get incremental builds when switching between modes. On the downside: when you build in a new mode for the first time, it’s essentially a clean build.

Note: -c is short for --compilation_mode, but everyone just says -c.

Defining a new mode

Okay, you can’t really define your own mode without tons of work, but you can create named sets of options (which is probably what you wanted anyway unless you’re writing your own toolchain).

For example, let’s say I generally have several flags I want to run with when I’m trying to debug a failing test. I can create a “gahhh” config in my ~/.bazelrc as follows:

test:gahhh --test_output=all
test:gahhh --nocache_test_results
test:gahhh --verbose_failures
test:gahhh -c dbg

The “test” part indicates what command this applies to (“build”, “query” or “startup” are other useful ones). The “:gahhh” names the config, and then I give the option. Now, when I get frustrated, I can run:

$ bazel test --config gahhh //my:target

and I don’t have to remember the four options that I want to use.

Startup idea #6ec4e42a-28cc-4425-9ebc-61ac8e224580: Adventurer’s gear for geeky hikers

I’m going to start “calling” my startup ideas in the same way Andy Dwyer calls band names.


So, first up: it’s like REI for D&D players.

We’d sell a “basic adventurer’s kit” that came with iron rations, wineskin, torches, 50 feet of rope, etc.

Then you could get “class specialization” kits, for example:

  • Rogue: contains lockpicks, a pack of cards, and invisible ink.
  • Wizard: parchment, ink, and a dozen small vials of reagents, orb.
  • Cleric: bandages, salves, holy symbol.

We could also offer Tolkien-esque maps of hiking areas and fancy medieval-looking bags/knives/hiking boots. See what carrying 40lbs of gear into the woods actually feels like! Then get it as a gift for a friend.

Gotta get the gear.
Gotta get the gear.

Star Trek invades our timeline

I was at Kennedy Space Center yesterday and they have an exhibit with all of the Apollo mission flags. Having mission flags is a great idea, more software launches should have flags, too. I noticed one in particular:


(Please excuse the poor image quality, I have a technology-defying ability to take crappy photographs.)

Those symbols on the flag look vaguely familiar…


Compare to:


What’s hilarious to me is that Star Trek: The Original Series started airing in 1966. Apollo 15 didn’t launch until 1971, so it must have been pretty blatant that they were copying that. I couldn’t find anything about it with a brief Google search.

Another exhibit about the space shuttle confirmed the intermingling between NASA and Star Trek:


Live long and prosper!

Non-technical advice for startups and open source projects

A former coworker recently asked me about what had worked well (and not) at MongoDB. I realized that I actually know a bunch of things about running an open source project/startup, some of which may not be common knowledge, so I figured I’d share some here.

Things changed dramatically as the company grew and the product matured, but this post is about the project’s infancy (when the company was less than, say, 20 people). At that point, our #1 job was pleasing users. This meant everything from getting back to mailing list questions within a few minutes to shifting long-term priorities based on what users wanted. We’d regularly have a user complain about an issue and a couple of hours later we’d tell them “Please try again from head, we’ve fixed the issue.”

Unfortunately, fast turnaround meant bugs. We weren’t using code review at this point (give us a break, it was 2008), so we could get things out as fast as we could code them, but they might not… exactly… work. Or might interact badly with other parts of the system. I’m not sure what the solution to this would be: I think fast iteration was part of what made us successful. People got stuff done.

(via Poorly Drawn Lines.)

Users loved how fast we fixed things, but the support load was insane (at times) and basically could not be handled by non-engineers (i.e., we could not hire a support staff). The only reason this worked was that the founders were twice as active as anyone else on the list, setting the example. We actually had a contest running for a while that if anyone had more messages than Eliot in a month, they would get an iPad. No one ever won that iPad.

If a bug couldn’t be fixed immediately, it was still incredibly important to get back to people fast. Users didn’t actually care if you couldn’t solve their problem, they wanted a response. (I mean, they obviously preferred it if we had a solution, but we got 100% goodwill if we responded with a solution, 95% goodwill if we responded with “crap, I’ve filed a bug” within 15 minutes, 50% goodwill within a day, and 0% after that.) I’m still not as good at this as I’d like to be, but I recommend being as responsive as possible, even if you can’t actually help.

Because our users were developers (and at this point, early adopters), they generally detested “business speak.” Occasionally we had a new non-technical person join who wan’t familiar with developer culture, which resulted in some really negative initial perceptions on Reddit (that was the most memorable one, but we also had some business-speak issues on Twitter and other channels).

In fact, when we first hired a non-technical person, I was really skeptical. I didn’t understand why an engineering company making a product for other engineers would need a non-engineer. Now I’d credit her with 50% of MongoDB’s success and a good deal of the developers’ happiness. Meghan would reach out to and contact users and contributors, get to know meetup and conference organizers, and was generally a “router” for all incoming requests that we didn’t know what to do with (“Can we get someone to come talk at our conference in Timbuktu?” “Could I have an internship?” “Can you send me 20 MongoDB mugs to give to my coworkers?”). In general, there was a surprising amount of non-technical stuff that was important for success. We had a piece of software that people definitely wanted and needed, but I don’t think it would have been nearly as successful without the non-technical parts.