I’ve started writing a series of dog-related short stories at mr-domino.kchodorow.com, if you’re interested. This blog (kchodorow.com/blog) will still be a “proper blog,” where I blog about my life & tech stuff. The Domino-related blog is more for creative writing.
configure: error: lib_i_don’t_care_about.so not found.
I work on Bazel, so I don’t usually get to see it from a user’s point of view. However, yesterday I added seven new projects to Bazel’s continuous integration, all of which promptly broke. I started cloning them and trying to fix them.
These projects were various user-contributed rules for Bazel: rules for building Go, .NET, AppEngine, and more. I had never built any code in most of these languages (Rust? Sass? D?) However, using Bazel, I could build and debug each of the projects without installing any prerequisites or copying libraries to certain places on my machine. Bazel handled downloading and configuring all of the libraries and tools that I needed. And now my patches are starting to get merged and the go rules, at least, are green! One down, six to go.
JoCo Cruise 2016: a misanthrope’s perspective
I was on the JoCo cruise last week. I’ve read quite a few pieces on it and everyone is gushing over it, so I figured I’d put up my perspective.
Basically, I’m not a fan. The point of a cruise seems to be to eat and drink as much as possible, pay as much money as possible, and have zero mental stimulation.
The freedom, or lack thereof
They search and X-ray everything you bring on the boat to make sure you’re not bringing more than two bottles of wine. Everything is very tightly controlled to make sure you end up parting with the maximum amount of money during your cruise, so there are a million tiny inconveniences like not being able to get a drink for a friend or leave the boat with a coffee.
The food
Is not very good. I love starch, but it was like a week of eating bread mush with lots of cheese. There’s a “secret” Indian menu I ordered from a lot, which was a slight improvement, but was basically like the worst of Indian Row in NYC.
The coffee was terrible. On of my friends actually brought beans, a grinder, and Aeropress after their experience last year.
The staff
The workers were either cringe-inducingly obsequient (memorizing our names and asking us repeatedly if there was anything else at all they could get us) or sullen and ignored us for as long as possible.
I originally wanted to report how I’d go up to a crewmember to ask a question and they’d ignore me for a few minutes, fiddling with their phone before saying “what?” but then I heard that Royal Caribbean fires people who don’t get 10s on their customer satisfaction surveys (which explains the other type of worker). I don’t really want to get them fired, I just want to be able to find out if I’m in the right place to go kayaking or whatever.
Also, almost all of passengers and almost none of the crew appeared to be white. That felt really weird.
The boat
Have you ever lived in a mall for a week? I have, and let me tell you, it’s depressing. It was like being trapped in an old McDonald’s with creepy injection-molded plastic everything bolted to the floor. It literally gave me nightmares.
The ports of call
Every port where we stopped had a giant fence around the cruise ship area. Want to go out and see what the island is like? No problem, either pay for a taxi or walk for a half-hour through Gucci stores and Ye Olde Touriste bars. I’m not sure whether they have these Area-51-type fences to keep the locals out or the tourists in, but either way, I hate it.
The on-ship entertainment
This is less relevant for next year, but the things the Royal Caribbean offered as “entertainment” were obnoxious. Talks on “how to buy an expensive watch” or “procedures you can have to look younger.” With the casino, the constant greatest-hits karaoke, the the bars every 30 feet, everything was about not thinking.
Minor, but annoying
You had to use hand sanitizer all the time, which was disgusting. Preferable to the alternative, but it always felt like someone drooling on my hands.
In conclusion…
My friends all loved it and are going back next year, but the whole experience made me want to go camping for the rest of my life.
Star Trek invades our timeline
I was at Kennedy Space Center yesterday and they have an exhibit with all of the Apollo mission flags. Having mission flags is a great idea, more software launches should have flags, too. I noticed one in particular:
(Please excuse the poor image quality, I have a technology-defying ability to take crappy photographs.)
Those symbols on the flag look vaguely familiar…
Compare to:
What’s hilarious to me is that Star Trek: The Original Series started airing in 1966. Apollo 15 didn’t launch until 1971, so it must have been pretty blatant that they were copying that. I couldn’t find anything about it with a brief Google search.
Another exhibit about the space shuttle confirmed the intermingling between NASA and Star Trek:
Live long and prosper!
Flag-Friday: debugging tests with –java_debug
To step through a Java test that you’re running with bazel test, use the --java_test flag:
$ bazel test --java_debug //src/test/java/com/example:hello-test WARNING: Streamed test output requested so all tests will be run locally, without sharding, one at a time. INFO: Found 1 test target... Listening for transport dt_socket at address: 5005
At this point, switch over to your IDE and fire up a remote debugging configuration, with the host localhost:5005. The debugger will attach to the running process, your test will start, and the debugger will stop at your first breakpoint.
Saving the (prod) environment
You can create different environments (e.g., testing, prod, mobile, rainforest) with Bazel, then use them to make sure that targets only build with the right environment. This is a cool feature that’s undocumented (because it’s still in development, shhhhh, don’t tell anyone I told you about it).
Let’s say you have a prod SSH key that you don’t want used in development or test builds. You could restrict it to only be used in prod builds by defining the following:
environment(name = "dev")
environment(name = "prod")
environment(name = "testing")
environment_group(
name = "buildenv",
defaults = ["dev"],
environments = [
"dev",
"prod",
"testing",
],
)
filegroup(
name = "ssh-key",
restricted_to = [":prod"],
srcs = ["key"],
)
Now whenever we use :ssh-key, it has to be in a prod-environment rule. For example, this works:
cc_binary(
name = "prod-job",
srcs = ["job.cc"],
restricted_to = [":prod"],
data = ["ssh-key"],
)
This doesn’t:
cc_test(
name = "job-test",
srcs = ["job_test.cc"],
data = [":ssh-key"],
)
Building the second one gives:
$ bazel build :job-test ERROR: /Users/kchodorow/test/a/BUILD:34:1: in cc_test rule //:job-test: dependency //:ssh-key doesn't support expected environment: //:dev. ERROR: Analysis of target '//:job-test' failed; build aborted. INFO: Elapsed time: 0.167s
Hopefully, if someone tried to add restricted_to = [":prod"] to a test, it’d “look wrong” and be easier to catch.
Note that you must set your defaults sanely: when I first tried this, I made the environment_group‘s defaults = ["prod"] and then was confused that I wasn’t getting any errors. Everything is built for the default environments unless specified otherwise!
This lets us say: “If a depends on b and b is restricted to a certain environment, then a must be restricted to the environment.” However, there is another direction to look at this from: if a is restricted to an environment, b must be compatible with that environment. To express this, you can use “compatible_with“:
filegroup(
name = "dev-key",
srcs = ["key.dev"],
compatible_with = [
":dev",
":testing"
],
)
Now anything that’s restricted to “:dev” or “:testing” environments can depend on “:dev-key”. For example, these work:
cc_binary(
name = "dev-job",
srcs = ["job.cc"],
data = [":dev-key"],
)
cc_test(
name = "job-test",
srcs = ["job_test.cc"],
restricted_to = [":testing"],
data = [":dev-key"],
)
This does not:
cc_binary(
name = "prod-job",
srcs = ["job.cc"],
restricted_to = [":prod"],
data = [":dev-key"],
)
The full matrix (assuming env is an environment) is:
b |
b restricted to env |
b compatible with env |
|
|---|---|---|---|
a |
✓ | ✗ | ✓ |
a restricted to env |
✗ | ✓ | ✓ |
a compatible with env |
✗ | ✗ | ✓ |
Remember that environments are targets themselves, so avoid proliferating environments that aren’t global to the global scope (don’t make them publicly visible and keep them as private as possible).
Combining projects without converting to a monorepo
Bazel allows you to combine multiple directories from across your filesystem and pretend all of the sources are part of your project. This is a little hard to picture, so let’s use a concrete example.
Let’s say you have two projects you’re working on, checked out at ~/gitroot/spaghetti-stable and ~/gitroot/meatballs-master. You don’t want to combine them into one repository, but you have integration tests that run your meatballs service on top of your spaghetti platform and end-to-end tests that make requests (it forks them).
That is, the filesystem might look something like this:
gitroot/
spaghetti-stable/
WORKSPACE
spaghetti/
BUILD
plate_of_spaghetti.cc
meatballs-master/
WORKSPACE
meatballs/
BUILD
pile_of_meatballs.cc
end_to_end_test.cc
I’ve made up these directories on GitHub (spaghetti and meatballs), if you want to take a look.
The spaghetti/BUILD file can be pretty simple:
cc_library(
name = "spaghetti",
srcs = ["plate_of_spaghetti.cc"],
visibility = ["//visibility:public"],
)
The meatballs/BUILD file is similar, but you also have an end-to-end test that depends on both //spaghetti and //meatballs:
cc_library(
name = "meatballs",
srcs = ["pile_of_meatballs.cc"],
)
cc_test(
name = "end_to_end_test",
srcs = ["end_to_end_test.cc"],
deps = [
":meatballs",
"//spaghetti",
],
)
Note that we’re depending on //spaghetti, even though it’s not under meatballs-master/. We can combine the two directories during the build by running bazel with the --package_path argument:
$ bazel test --package_path %workspace%:/home/k/gitroot/spaghetti-stable:/usr/local/lib/bazel/base_workspace
//meatballs:end_to_end_test
This means: when you’re looking for package, first check ~/gitroot/meatballs-master (%workspace% is the current directory). Then check ~/gitroot/spaghetti-stable. Finally, check Bazel’s binary installer location (for internal tools Bazel needs during the build).
When your test finishes, take a look at ~/gitroot/meatballs-master/bazel-meatballs-master. This is called the execution root and it’s where Bazel actually runs build commands:
$ ls -l bazel-meatballs-master/ total 36 drwxr-x--- 2 k k 20480 Dec 8 14:08 _bin drwxr-x--- 3 k k 4096 Dec 8 14:08 bazel-out drwxr-x--- 2 k k 4096 Dec 8 14:08 external lrwxrwxrwx 1 k k 64 Dec 8 14:08 meatballs -> home/k/test/meatballs-master/meatballs lrwxrwxrwx 1 k k 64 Dec 8 14:08 spaghetti -> /home/k/test/spaghetti-stable/spaghetti lrwxrwxrwx 1 k k 41 Dec 8 14:08 tools -> /usr/local/lib/bazel/base_workspace/tools
You can see that Bazel has combined the directories on the package path to create a single directory that contains both meatballs/ and spaghetti/ subdirectories. The source directory (~/gitroot/meatballs-master) is left unchanged.
If we were going to do this regularly, we could add the package path option to our .bazelrc file and then we don’t have to specify it every build.
To try this out, you can download the sources with:
$ git clone https://github.com/kchodorow/spaghetti-stable.git $ git clone https://github.com/kchodorow/meatballs-master.git
Then cd into meatballs-master/ and run!
Non-technical advice for startups and open source projects
A former coworker recently asked me about what had worked well (and not) at MongoDB. I realized that I actually know a bunch of things about running an open source project/startup, some of which may not be common knowledge, so I figured I’d share some here.
Things changed dramatically as the company grew and the product matured, but this post is about the project’s infancy (when the company was less than, say, 20 people). At that point, our #1 job was pleasing users. This meant everything from getting back to mailing list questions within a few minutes to shifting long-term priorities based on what users wanted. We’d regularly have a user complain about an issue and a couple of hours later we’d tell them “Please try again from head, we’ve fixed the issue.”
Unfortunately, fast turnaround meant bugs. We weren’t using code review at this point (give us a break, it was 2008), so we could get things out as fast as we could code them, but they might not… exactly… work. Or might interact badly with other parts of the system. I’m not sure what the solution to this would be: I think fast iteration was part of what made us successful. People got stuff done.
(via Poorly Drawn Lines.)
Users loved how fast we fixed things, but the support load was insane (at times) and basically could not be handled by non-engineers (i.e., we could not hire a support staff). The only reason this worked was that the founders were twice as active as anyone else on the list, setting the example. We actually had a contest running for a while that if anyone had more messages than Eliot in a month, they would get an iPad. No one ever won that iPad.
If a bug couldn’t be fixed immediately, it was still incredibly important to get back to people fast. Users didn’t actually care if you couldn’t solve their problem, they wanted a response. (I mean, they obviously preferred it if we had a solution, but we got 100% goodwill if we responded with a solution, 95% goodwill if we responded with “crap, I’ve filed a bug” within 15 minutes, 50% goodwill within a day, and 0% after that.) I’m still not as good at this as I’d like to be, but I recommend being as responsive as possible, even if you can’t actually help.
Because our users were developers (and at this point, early adopters), they generally detested “business speak.” Occasionally we had a new non-technical person join who wan’t familiar with developer culture, which resulted in some really negative initial perceptions on Reddit (that was the most memorable one, but we also had some business-speak issues on Twitter and other channels).
In fact, when we first hired a non-technical person, I was really skeptical. I didn’t understand why an engineering company making a product for other engineers would need a non-engineer. Now I’d credit her with 50% of MongoDB’s success and a good deal of the developers’ happiness. Meghan would reach out to and contact users and contributors, get to know meetup and conference organizers, and was generally a “router” for all incoming requests that we didn’t know what to do with (“Can we get someone to come talk at our conference in Timbuktu?” “Could I have an internship?” “Can you send me 20 MongoDB mugs to give to my coworkers?”). In general, there was a surprising amount of non-technical stuff that was important for success. We had a piece of software that people definitely wanted and needed, but I don’t think it would have been nearly as successful without the non-technical parts.
One weird trick for fast CI
Compilers hate this! (Just kidding, compilers are easy-going.)
For many build systems, you have to do a clean build to be sure you’re getting the correct result, so your CI has to always do a clean build. On the other hand, Bazel is designed so that you never have to do a bazel clean (if you ever run bazel clean and get a different answer, please file a bug!).
Thus, to cater to most build tools, CI systems generally run a loop like this:
However, for Bazel, doing a clean build every time wastes a lot of time and resources. For Bazel, the continuous integration system should look like this:
Of course, most build systems actually make this a bit difficult to set up, as you wouldn’t want this kind of configuration for a non-Bazel continuous build! However, Jenkins lets you set it up fairly simply. I’ll go through setting up a Jenkins instance that will build a local codebase every 5 minutes.
First, I’ll make my “source directory:”
$ mkdir jenkins-workspace $ cd jenkins-workspace $ touch WORKSPACE
I’m on OS X, so I installed the Jenkins pkg. When it is done installing, it pops up a web page prompts you to create your first job.
Click on the link and put in a name and select “Freestyle Build”, then hit “OK”.
On the next page, select “Advanced”. Select “Use custom workspace” and put in the directory from above.
Set it up to run every 5 minutes:
Now add the following command as the build step:
This will build all targets in your workspace and then run all tests (so it will find any compile errors as well as test failures). //tools and //third_party contain a bunch of targets that won’t build out-of-the-box, so they’re filtered out using the -//target syntax (see the “Subtractive Patterns” section under Target Patterns).
Note that you don’t have to do any cleanup after the build, since Bazel never pollutes your source directory with build artifacts. You just want to pull, build, pull, build.
Now hit the “Save” button. It’ll take you back to your dashboard and begin the countdown to the first build. Click on the play button in the rightmost column to test it out (or just wait five minutes).
If you make changes, Jenkins will just build/test those changes:
If you haven’t made any changes, Bazel won’t have to do any work at all:
This can be a huge time saver for large projects.
Creating a javadoc rule for Bazel
A couple of users have asked about how to generate javadoc with Bazel. There’s no built-in way, but I figured it might be useful to whip together a new rule to do so.
If you’d like to use this rule, download it to your workspace, load it in your build file, and give it a list of sources:
load("/javadoc", "javadoc")
javadoc(
name = "app-doc",
srcs = glob(["**/*.java"]),
)
Then build:
$ bazel build :app-doc INFO: Found 1 target... Target //:app-doc up-to-date: bazel-bin/app-doc.zip INFO: Elapsed time: 0.141s, Critical Path: 0.00s
Now, if you look at bazel-bin/app-doc.zip, you can see that it contains the HTML tree generated by javadoc:
$ unzip -l bazel-bin/app-doc.zip
Archive: bazel-bin/app-doc.zip
Length Date Time Name
--------- ---------- ----- ----
6990 2015-09-28 15:06 app-doc/SomeClass.html
568 2015-09-28 15:06 app-doc/allclasses-frame.html
548 2015-09-28 15:06 app-doc/allclasses-noframe.html
...
You can then unzip it wherever you want to view your docs.
Note that this is an extremely limited implementation: I just dashed this off in 20 minutes. It doesn’t support any javadoc options and probably doesn’t handle dependencies correctly. Let me know if you have any issues with it and I can implement improvements as needed.
