The Haunted Homesteader

Andrew and I have always loved old places. Optimally, we’d like to live in a wizard’s tower on top of a mountain. However, we’d be willing to settle for a castle with a thousand acres of land. More realistically, we’d like an old place with a couple of acres. That you can reach without a car from NYC (I said more realistically, not actually realistically).

So, sometimes I browse the real estate listings, especially near the train stations along Metro North and one day I noticed something unusual. There was a place that was being sold right next to the train station. It was 10 acres of property. They were asking less than $1 million.

Okay, those were the good things. There were… a couple downsides. It was old (good) and had obviously been abandoned for years (not so good). It was missing certain crucial elements like windows. And a roof. It needed a completely new septic system and well replacement (well water in combination with septic tank problems: eww), needed new wiring, and god knows what else. It was a “historical property,” so any repairs we made had to be okay-ed by a historical accuracy board and materials would probably be exorbitantly expensive: no off-the-shelf windows from Home Depot. On top of all that, the walls were literally made of asbestos, so either we pay an exorbitant amount to have all of the asbestos removed, or pay an exorbitant amount for each repair we did because everyone would have to wear spacesuits and take crazy precautions.

So, I said, “Let’s just go up on the train an take a look. So I can get it out of my system.”

Andrew pretended to believe me and off we went. We got off the train and… the property was right there. Like, 5-minute stroll up the hill from where we got off the train. Note the “up-the-hill” part: this thing was basically invisible from every angle. We’d gone hiking from this train stop a hundred times and never seen it before. It was on a bluff overlooking the Hudson. We carefully circled the property, the ridiculously large property, trying to get a clear view of the place. There was no actual road leading to it, it was just… abandoned by time, on a bluff overlooking the Hudson. I was done for.

We continued to circle it and the “backyard” (the part not overlooking the Hudson, did I mention it fucking OVERLOOKS THE HUDSON?!) melts into reserved state land, so it’ll never be developed. We’d have a forest in our backyard for perpetuity. We walked down one of the trails through the woods in the “backyard” and sat down on a big rock overlooking a waterfall. We discussed, and negotiated, and fantasized. We started off agreeing that we’d both be comfy offering 1/5 the asking price. A few hours later, our butts were freezing, Domino had crawled into Andrew’s lap, and we had negotiated ourselves up to, “well, the asking price is sort of reasonable…” We’re terrible negotiators.

So, we contacted the real-estate agent so we could see the inside of the house. She took us on a tour and, in some ways, the place was amazing: floor-to-ceiling bay windows with views of the river, fireplaces in every room, and more rooms that we knew what to do with. In other ways, it was… not so amazing. For example, it was built before indoor plumbing, so the original floor plan had not accounted for things like bathrooms. Nor closets, apparently that wasn’t a thing. It was built for rich people in the 1800s, so the kitchen was in the basement (keep the help out of the way). There was obviously no electricity, so none of the ceilings have wiring for light fixtures. The owners built an addition with plumbing & electric in the 20s, but given its size and shape, it’s a bit quirky. For instance, they added an awesome “secret door” bookcase that swings open to reveal… a bathtub in a closet.

After seeing the inside, we were in love, but the love was tempered by the desire to not have to work until we died restoring the place. We talked to a some contractors. We talked to our friends. We talked to our parents.

And… we decided against it. I’m a bit heartsick over it, but it’s just 10 years too early for us to be able to dedicate that kind of time to fixing up a place. Doing the mature, responsible thing sucks.

Now we’re mainlining Ask This Old House. When the time comes, we’ll be so ready.

The living room.
The living room.

Recruiting review

Just got this message from a recruiter:

I know I recently reached out to you, but seriously, you’re the cat’s meow and I can see you being the perfect fit for [company]. Your current experience at Google is spot on with what our Talent Team is looking for.

[Description of company]

If I am way off base in my analysis of your profile, please let me know. However, if there is a small chance that I may have hit the nail on the head – I’d love to discuss the opportunity to join their team.

[Sign off]

deTECHtive | Talent Acquisition Manager

Points for actually describing what the company does. Points off for:

  • Using more analogies than I could swing a dead cat at.
  • deTECHtive
  • Being a Googler == what they’re looking for.

All in all, I rate it two resumes out of five: nothing egregious, but nothing appealing, either.

[star rating=”2″]

Using AutoValue with Bazel

AutoValue is a really handy library to eliminate boilerplate in your Java code. Basically, if you have a “plain old Java object” with some fields, there are all sorts of things you need to do to make it work “good,” e.g., implement equals and hashCode to use it in collections, make all of its fields final (and optimally immutable), make the fields private and accessed through getters, etc. AutoValue generates all of that for you.

To get AutoValue work with Bazel, I ended up modifying cushon’s example. There were a couple of things I didn’t like about it, since I didn’t want the AutoValue library to live in my project’s BUILD files. I set it up so it was defined in the AutoValue project, so I figured I’d share what I came up with.

In your WORKSPACE file, add a new_http_archive for the AutoValue jar. I’m using the one in Maven, but not using maven_jar because I want to override the BUILD file to provide AutoValue as both a Java library and a Java plugin:

    name = "auto_value",
    url = "",
    build_file_content = """
    name = "jar",
    jars = ["auto-value-1.3.jar"],

    name = "autovalue-plugin",
    generates_api = 1,
    processor_class = "",
    deps = [":jar"],

    name = "processor",
    exported_plugins = [":autovalue-plugin"],
    exports = [":jar"],
    visibility = ["//visibility:public"],

Then you can depend on @auto_value//:processor in any java_library target:

    name = "project",
    srcs = [""],
    deps = ["@auto_value//:processor"],

…and Bob’s your uncle.

You do you

I’m a little tired and depressed this week. However, this was a very inspiring speech Neil Gaiman gave to new grads of an art school:

Neil Gaiman Addresses the University of the Arts Class of 2012 from The University of the Arts (Phl) on Vimeo.

I think that his point about doing what you love, regardless of the money, holds doubly true for programmers. We are extremely lucky in that, unlike artists, we can make an okay salary nearly anywhere. We might as well work on things that make us happy.

Snail Spam

When I started blogging, I called my blog “Snail in a Turtleneck,” a cute image that Andrew & I came up with. I drew up my mascot:

A bemused snail, wearing a turtleneck.
A bemused snail, wearing a turtleneck.

and I began posting cartoons I had drawn. I quickly became bored of doing cartoons, and found I was more motivated to put up technical blog posts. Most of my initial readers were coworkers and MongoDB users. When Andrew and I got married, I told my teammates the day before that I’d be out the next day, as I was getting married (we got married at the city clerk’s, so it wasn’t a big production). When I got back to work, I found this at my desk:

A stuffed snail, a snail tape dispenser, and a very lovely bouquet (that was entirely free of snails).
A stuffed snail, a snail tape dispenser, and a very lovely bouquet (that was entirely free of snails).

I was very touched by their thoughtfulness: Andrew and I still have the stuffed snail and I brought the tape dispenser along to Google (where, unfortunately, it was later lost during an intra-office move).

However, as MongoDB gained popularity, some of my posts became very popular and I began to regret the name: customers seemed a little embarrassed to mention they had gotten advice from it and a lot of people didn’t realize that I was actually behind it. I purchased, set up permanent redirects, and basically stopped referencing “Snail in a Turtleneck.” After a couple of years, I let the domain name lapse.

Last week, someone told me that my site had been hacked. I was confused, until they told me it was I took a look and, bizarrely, someone seems to have taken a dump of my site circa 2011, put spam on the index, and put it up at, complete with my artwork, cartoons, etc. The domain was registered through a privacy protection service, so I guess the next step is sending a DCMA takedown notice to the registrar.

Who does this? (I mean, spammers, but… so annoying.)

Four alternative debugging techniques

I’ve recently been working on a side project that uses WebGL and a physics engine that was transpiled from C++ into JavaScript so… printing variable to the console and using the debugger just weren’t cutting it. I started thinking about the other ways I debug things:

  1. Ship of Theseus debugging: the ship of Theseus is a thought experiment: if you have something and you gradually replace every part, at which point is it a different thing? This is debugging via finding a working example, then gradually mutating it (without breaking it, source control is helpful here) into what you actually want to do. There are a couple of problems: this often leaves some cruft around from the original program and I often never figure out why it wasn’t working in the first place. Which brings us to…
  2. Homeland security debugging: if you see a suspicious variable name or method call, don’t keep it to yourself. Tell a coder or library maintainer. This is where you go back through all of the sketchy parts of your code and make sure they are actually doing what you expect. After working on a program for a while, I’ll usually end up with parts of codebase that are a bit questionable. Why am I passing a literal “1” as an argument in here? Why do I have two names for this variable? Basically, I’m going through my program, line by line, checking all of my vague suspicions.
  3. Thunderdome debugging: one coder and one bug enter, one coder leaves. This is kind of a meta-technique I use for weird, difficult-to-reproduce issues in integration tests where I have to dig through logs or do a 12-step process every time I need to test a change. Basically, I get a big cup of coffee, sit down at my machine, and try everything while mainlining caffeine. Afterwards, I generally couldn’t even tell you what the bug ended up being, but it is no more. This is the kind of debugging that generally does not happen unless I’m being paid for it.
  4. Wooden nickel debugging: try testing code against the most useless possible inputs. Sometimes I have a very complex chunk of code that doesn’t work. I don’t want to spend the time to get a meaningful test running against it, but I start writing unit tests, passing in the most trivial input possible: the empty string, 0, an empty array. And often, after an input or two, I’ve figured out what it’s doing wrong.

Anyone else have any non-traditional ways that they debug?

Compilation à la mode

Sundae from Black Tap, incidentally around the corner from the NYC Google office.
Sundae from Black Tap, incidentally around the corner from the NYC Google office.

Bazel lets you set up various “modes” of compilation. There are several built-in (fast, optimized, debug) and you can define your own. The built in ones are:

  • Fast: build your program as quickly as possible. This is generally best for development (when you want a tight compile/edit loop) and is the default, when you don’t specify anything. Your build’s output is generated in bazel-out/local-fastbuild.
  • Optimized: code is compiled to run fast, but may take longer to build. You can get this by running bazel build -c opt //your:target and it’s output will be generated in bazel-out/local-opt. This mode is the best for code that will be deployed.
  • Debug: this leaves in symbols and generally optimizes code for running through a debugger. You can get this by running bazel build -c dbg //your:target and it’s output will be generated in bazel-out/local-dbg

You don’t have to actually know where the build’s output is stored, bazel will update the bazel-bin/bazel-genfiles symlinks for you automatically at the end of your build, so they’ll always point to the right bazel-out subdirectory.

Because each flavor’s output is stored in a different directory, each mode effectively has an entirely separate set of build artifacts, so you can get incremental builds when switching between modes. On the downside: when you build in a new mode for the first time, it’s essentially a clean build.

Note: -c is short for --compilation_mode, but everyone just says -c.

Defining a new mode

Okay, you can’t really define your own mode without tons of work, but you can create named sets of options (which is probably what you wanted anyway unless you’re writing your own toolchain).

For example, let’s say I generally have several flags I want to run with when I’m trying to debug a failing test. I can create a “gahhh” config in my ~/.bazelrc as follows:

test:gahhh --test_output=all
test:gahhh --nocache_test_results
test:gahhh --verbose_failures
test:gahhh -c dbg

The “test” part indicates what command this applies to (“build”, “query” or “startup” are other useful ones). The “:gahhh” names the config, and then I give the option. Now, when I get frustrated, I can run:

$ bazel test --config gahhh //my:target

and I don’t have to remember the four options that I want to use.

The Mixed-Up Directories of Mrs. Bazel E. Frankweiler

Bazel has several directory trees that it uses during a build.


The most obvious directory is the source tree where your code lives and where you run your builds. This is, by default, what Bazel uses for source files.

However, you can combine several source trees by using the --package_path option. This basically overlays them, from Bazel’s point of view. For instance, if you had:


Then if you ran bazel build --package_path=/home/user/gitroot/my-project:/usr/local/other-proj //..., Bazel would “see” the directory tree:


If a package is defined in multiple package paths, the first path “wins” (e.g., the top-level package containing foo/ above).

Finally, source code for external repositories is tucked away in Bazel’s output base. You can see it by running:

$ ls $(bazel info output_base)/external

Execution root

Once bazel figures out what packages a build is going to use, it creates a symlink tree called the execution root, where the build actually happens.* It basically traverses all of the packages it found and comes up with the most efficient way it can to symlink them together. For example, in the directory tree above, you’d end up with:

    WORKSPACE -> /home/user/gitroot/my-project/WORKSPACE
    BUILD -> /home/user/gitroot/my-project/BUILD
    foo/ -> /home/user/gitroot/my-project/foo
    bar/ -> /usr/local/other-proj/bar
    ... # Tools built into the Bazel binary
    ... # C++ compiler tools

You can check out the execution root by running:

$ ls $(bazel info execution_root)


Here’s the * from the execution root section above! If you’re on Linux (and, hopefully, OS X soon), your build actually takes place in a sandbox based on the execution root. All of the files your build needs (and hopefully none it doesn’t) are mounted into their own namespace and executed in a hermetically sealed environment: no network or filesystem access (other than what you specified).

You can see what’s being mounted and where by running Bazel with a couple extra flags:

$ bazel build --sandbox_debug --verbose_failures //...

Derived roots

We’re working on improving how configurable Bazel is, but for years the main configuration options have been the platform you’re building for and what your compiler options were. If you run an optimized build, then a debug build, and then an optimized build again, you’d like your results to be cached from the first run, not overwritten each time. In a somewhat questionable design move, Bazel uses a special set of output directories that are named based on the configuration. So if you build an optimized binary, it’ll create it under execroot/my-project/bazel-out/local-opt/bin/my-binary. If you then build it as a debug binary, it’ll put it under execroot/my-project/bazel-out/local-dbg/bin/my-binary. Then, if you build it optimized again, it’ll be able to switch back to using the local-opt directory. (However, bazel uses symlinks out the wazoo, so I don’t know why it doesn’t use symlinks to track which configuration is being used. Seems like it’d be a lot easier to have the outputs directly under execroot/my-project.)

Also, Bazel distinguishes between files created by genrule and… everything else. Almost all output is under bazel-out/config/bin, but genrules are under bazel-out/config/genfiles.

(I’m kind of bitter about these files, I’ve been working on a change for what feels like months because of this stupid directory structure.)

Note that Bazel symlinks execroot/ws/bazel-out/config/bin to bazel-bin and execroot/ws/bazel-out/config/genfiles to bazel-genfiles. These “convenience symlinks” are the outputs shown at the end of your build.


Suppose you build a binary in your favorite language. When you run that binary, it tries to load a file during runtime. Bazel encourages you to declare these files as runfiles, runtime dependencies of your build. If they change, your binary won’t be recompiled (because they’re runtime, not compile-time dependencies) but they will cause tests to be re-run.

Bazel create a directory for these files as a sibling to your binary: if you build //foo:my-binary, the runfiles will be under bazel-bin/foo/my-binary.runfiles. You can explore the directory or see a list of them all in the runfiles manifest, also a sibling of the binary:

$ cat bazel-bin/foo/my-project.runfiles_manifest

Note that a binary can run files from anywhere on the filesystem (they’re binaries, after all). We just recommend using runfiles so that you can keep them together and express them as build dependencies.

Custom, locally-sourced output filenames

Skylark lets you use templates in your output file name, e.g., this would create a file called target.timestamp:

touch = rule(
    outputs = {"date_and_time": "%{name}.timestamp"},
    implementation = _impl,

So if you had touch(name = "foo") in a BUILD file and built :foo, you’d get foo.timestamp.

I’d always used %{name}, but I found out the other day that you can actually use other attributes, too. For example, you could have:

greet = rule(
    attrs = {"my_name": attr.string()},
    outputs = {"greeting": "hi-there-%{my_name}"},
    implementation = _impl,

Then if you have greet(name = "a-greeting", my_name = "kristina") and build :a-greeting, you’ll get “hi-there-kristina” as an output file.

The entire source for this example is available as a GitHub gist (all four lines of implementation function not shown above).

Using environment variables in Skylark repository rules

If you’ve every used the AppEngine rules, you know the pain that is wait for all 200 stupid megabytes of API to be downloaded. The pain is doubled because I already have a copy of these rules on my workstation.

To use the local rules, all I have to do is override the @com_google_appengine_java repository in my WORKSPACE file, like so:

load("//appengine:appengine.bzl", "APPENGINE_BUILD_FILE")
    name = "com_google_appengine_java",
    path = "/Users/kchodorow/Downloads",
    build_file_content = APPENGINE_BUILD_FILE,

However, this is still imperfect: I don’t really want to maintain changes that basically amount to a performance optimization in my local client.

By using environment variables in the appengine_repository rule, we can do even better. I’m going to create a new rule that checks if the APPENGINE_SDK_PATH environment variable is set. If it is, it will use a local_repository to pull in AppEngine, otherwise it will fall back on downloading the .zip.

So, to start, let’s take a look at the existing rule that pulls in the AppEngine SDK. As of this writing, it looks like this:

      name = "com_google_appengine_java",
      sha256 = "189ec08943f6d09e4a30c6f86382a9d15b61226f042ee4b7c066b2466fd980c4",
      build_file_content = APPENGINE_BUILD_FILE,

First, let’s modify this to use a repository rule instead of native.maven_jar:

def _find_locally_or_download_impl(repository_ctx):
     ".", "189ec08943f6d09e4a30c6f86382a9d15b61226f042ee4b7c066b2466fd980c4", "", "")
  repository_ctx.file("BUILD", APPENGINE_BUILD_FILE)

_find_locally_or_download = repository_rule(
  implementation = _find_locally_or_download_impl,
  local = False,

def appengine_repositories():
  _find_locally_or_download(name = "com_google_appengine_java")

This code functions identically (basically) to the original code, so now let’s add an option for using a local path. Modify in the implementation function to check the environment:

def _find_locally_or_download_impl(repository_ctx):
  if 'APPENGINE_SDK_PATH' in repository_ctx.os.environ:
    path = repository_ctx.os.environ['APPENGINE_SDK_PATH']
    if path == "":
      fail("APPENGINE_SDK_PATH set, but empty")
    repository_ctx.symlink(path, APPENGINE_DIR)
     ".", "189ec08943f6d09e4a30c6f86382a9d15b61226f042ee4b7c066b2466fd980c4", "", "")
  repository_ctx.file("BUILD", APPENGINE_BUILD_FILE)

Now we can download a copy of the SDK and try our rule (feel free to use an existing copy, if you have one on your system).

APPENGINE_SDK_PATH=/path/to/your/sdk/download bazel build //your/appengine/app

Problems with this:

  • You can’t actually set APPENGINE_SDK_PATH to where Bazel downlaoded the SDK the first time around ($(bazel info output_base)/external/com_google_appengine_java), which is suuuuper tempting to do. If you do, Bazel will delete the downloaded copy (because you changed the repository def) and then symlink the empty directory to itself. Never what you want.
  • It caches the environment variable, so if you change your mind you have to run bazel clean to use a different APPENGINE_SDK_PATH. I think this is a bug, although there’s some debate about that.