Collecting transitive runfiles with skylark

Bazel has a concept it calls runfiles for files that a binary uses during execution. For example, a binary might need to read in a CSV, an ssh key, or a .json file. These files are generally specified separately from your sources for a couple of reasons:

  • Bazel can understand that it is a runtime, not compile dependency (so if the runfiel changes, the binary does not need to be rebuilt).
  • The type is less restrictive: most rules have restrictions on what its sources can “look like” (e.g., Java sources end in .java or .jar, Go sources end in .go, Python sources end in .py or .pyc, etc.).

Thus, these runfiles are often specified in a separate data attribute.

If you’re writing a skylark rule that combines several executables, you will probably want the skylark rule to also combine the runfiles for all of them. Let’s create a rule that can combine several executables and include all of their runfiles. As a toy example, I created a rule below that creates an executable. The rule has one attribute, data, that can be other files or rules. The executable, when run, will just list all of the runfiles it has available.

For example, suppose you had the following BUILD file:

list_runfiles(
    name = 'main-course', 
    data = ['lasagna.txt']
)

If we ran bazel run :main-course, it would print:

./my_workspace/main-course
./my_workspace/lasagna.txt
./MANIFEST

However, we can also provide list_runfiles targets as data. For example, our BUILD file could say:

list_runfiles(
    name = 'main-course', 
    data = [
        'lasagna.txt',
        ':side-dishes',
        ':drink',
    ]
)

list_runfiles(
    name = 'side-dishes', 
    data = [
        ':soup',
        ':salad',
    ]
)

list_runfiles(
    name = 'soup', 
    data = ['gazpacho.txt']
)

list_runfiles(
    name = 'salad', 
    data = ['waldorf.txt']
)

list_runfiles(
    name = 'drink', 
    data = ['milk.txt']
)

Then running bazel run :main-course will print:

.
./my_workspace
./my_workspace/side-dishes
./my_workspace/waldorf.txt
./my_workspace/salad
./my_workspace/milk.txt
./my_workspace/main-course
./my_workspace/lasagna.txt
./my_workspace/gazpacho.txt
./my_workspace/soup
./my_workspace/drink
./MANIFEST

:main_course has collected all of its transitive runfiles in its runfiles tree.

Here’s the list_runfiles rule definition:

def _list_runfiles_impl(ctx): 
  ctx.file_action(
    output = ctx.outputs.executable,
    content = 'n'.join([
        "#!/bin/bash",   
        "cd $0.runfiles",
        "find"
    ]),
    executable = True)
  return struct(runfiles = ctx.runfiles(collect_data = True))

list_runfiles = rule(
    attrs = {
        "data": attr.label_list(
            allow_files = True,
            cfg = DATA_CFG,
        ),
    },
    executable = True,
    implementation = _list_runfiles_impl,
)

The key is the line is runfiles = ctx.runfiles(collect_data = True). collect_data will automatically “harvest” the runfiles from data, srcs, and deps attributes. We’ve only defined data for this rule, so that’s what it will use.

Note that this won’t work if you change "data": attr.label_list( to something not covered by collect_data, e.g., "stuff": attr.label_list(. (Theoretically this should be covered by transitive_files, but I couldn’t actually get that working.)

Runfiles

Startup idea #6ec4e42a-28cc-4425-9ebc-61ac8e224580: Adventurer’s gear for geeky hikers

I’m going to start “calling” my startup ideas in the same way Andy Dwyer calls band names.

newbandname

So, first up: it’s like REI for D&D players.

We’d sell a “basic adventurer’s kit” that came with iron rations, wineskin, torches, 50 feet of rope, etc.

Then you could get “class specialization” kits, for example:

  • Rogue: contains lockpicks, a pack of cards, and invisible ink.
  • Wizard: parchment, ink, and a dozen small vials of reagents, orb.
  • Cleric: bandages, salves, holy symbol.

We could also offer Tolkien-esque maps of hiking areas and fancy medieval-looking bags/knives/hiking boots. See what carrying 40lbs of gear into the woods actually feels like! Then get it as a gift for a friend.

Gotta get the gear.
Gotta get the gear.

Using a generated header file as a dependecy

Someone asked me today about how to use a generated header as a C++ dependency in Bazel, so I figured I’d write up a quick example.

Create a BUILD file with a genrule that generates the header and a cc_library that wraps it, say, foo/BUILD:

genrule(
    name = "header-gen",
    outs = ["my-header.h"],
    # This command would probably actually call whatever tool was generat
    cmd = "echo 'int x();' > $@",
)

cc_library(
    name = "lib",
    hdrs = ["my-header.h"],
    srcs = ["lib.cc"],
    visibility = ["//visibility:public"]
)

Now you can depend on //foo:lib as you would a “normal” cc_library:

cc_binary(
    name = "bin",
    srcs = ["bin.cc"],
    deps = ["//foo:lib"],
)

And bin.cc would look like:

#include "foo/my-header.h"

// ...
int main() {
   x();  // Uses x defined in my-header.h.
}

Operation: Crappy Sewing Machine commences

This weekend, I went to a bra-making workshop and won a sewing machine in a raffle. It isn’t really crappy, but I spent a couple of hours un-jamming it, so I’m bitter.

The interesting thing about this machine is that it has a built-in camera, so you can take photos and video of exactly what you’re sewing and see them on the app. You can also buy new stitches from your phone and transfer them to your machine, so I started getting curious: what protocol is my sewing machine using? Could I write my own client?

I took a look at the manual, but there was nothing more technical than how to install the app in there. I checked the website, no other documentation there. I debated contacting customer service, but if I liked talking to people I wouldn’t be a programmer, so I fired up Wireshark and took a look at the network. I’m not too proficient with Wireshark, though, so I couldn’t figure out how to make it capture anything useful.

After a couple hours of fighting with it, I gave in and emailed customer support. I figured maybe they’d just forward me to a developer who would be happy to tell me about their protocol. Nope:

As far as the communication from the sewing machine to the app goes, I don’t know all the nuts and bolts but I do know it is proprietary information and is one of the features that makes the Spiegel 60609 so unique!

Bleh.

I realized that it would probably be easier for me to decompile the app, rather than sniffing the network, so I downloaded the APK using a sketchy service (I’m not sure if this is the best one out there, but it’s the least-offensive one I found) and undexed it using dex2jar:

$ chmod +x *.sh
$ ./d2j-dex2jar.sh path/to/Spiegel Social Sewing App_v1.0.4_apkpure.com.apk
dex2jar ../sewing-machine/Spiegel Social Sewing App_v1.0.4_apkpure.com.apk -> ./Spiegel Social Sewing App_v1.0.4_apkpure.com-dex2jar.jar

I opened it up in Intellij and boom, source code. Unfortunately, Intellij’s built-in decompiler choked on the one most interesting class (com.spiegel.android.spiegel.app.ui.settings.SpiegelMachineFacadeImpl). I tried to debug why (it could open every other .class file), but realized it would probably be easier to try another decompiler. I fired up JD-Gui and out popped the source!

Screen Shot 2016-04-12 at 9.13.51 PM

Turned out my sewing machine is running a PHP server, which is easy enough to communicate with. I think there are ~20 of these machines in the wild, so this is unlikely to be of any use to anyone, ever, but I look forward to writing my own client.

Here’s a video from it of it jamming the first time I attempted to use it:

configure: error: lib_i_don’t_care_about.so not found.

I work on Bazel, so I don’t usually get to see it from a user’s point of view. However, yesterday I added seven new projects to Bazel’s continuous integration, all of which promptly broke. I started cloning them and trying to fix them.

These projects were various user-contributed rules for Bazel: rules for building Go, .NET, AppEngine, and more. I had never built any code in most of these languages (Rust? Sass? D?) However, using Bazel, I could build and debug each of the projects without installing any prerequisites or copying libraries to certain places on my machine. Bazel handled downloading and configuring all of the libraries and tools that I needed. And now my patches are starting to get merged and the go rules, at least, are green! One down, six to go.

JoCo Cruise 2016: a misanthrope’s perspective

I was on the JoCo cruise last week. I’ve read quite a few pieces on it and everyone is gushing over it, so I figured I’d put up my perspective.

Basically, I’m not a fan. The point of a cruise seems to be to eat and drink as much as possible, pay as much money as possible, and have zero mental stimulation.

The freedom, or lack thereof

They search and X-ray everything you bring on the boat to make sure you’re not bringing more than two bottles of wine. Everything is very tightly controlled to make sure you end up parting with the maximum amount of money during your cruise, so there are a million tiny inconveniences like not being able to get a drink for a friend or leave the boat with a coffee.

The food

Is not very good. I love starch, but it was like a week of eating bread mush with lots of cheese. There’s a “secret” Indian menu I ordered from a lot, which was a slight improvement, but was basically like the worst of Indian Row in NYC.

The coffee was terrible. On of my friends actually brought beans, a grinder, and Aeropress after their experience last year.

The staff

The workers were either cringe-inducingly obsequient (memorizing our names and asking us repeatedly if there was anything else at all they could get us) or sullen and ignored us for as long as possible.

I originally wanted to report how I’d go up to a crewmember to ask a question and they’d ignore me for a few minutes, fiddling with their phone before saying “what?” but then I heard that Royal Caribbean fires people who don’t get 10s on their customer satisfaction surveys (which explains the other type of worker). I don’t really want to get them fired, I just want to be able to find out if I’m in the right place to go kayaking or whatever.

Also, almost all of passengers and almost none of the crew appeared to be white. That felt really weird.

The boat

Have you ever lived in a mall for a week? I have, and let me tell you, it’s depressing. It was like being trapped in an old McDonald’s with creepy injection-molded plastic everything bolted to the floor. It literally gave me nightmares.

The ports of call

Every port where we stopped had a giant fence around the cruise ship area. Want to go out and see what the island is like? No problem, either pay for a taxi or walk for a half-hour through Gucci stores and Ye Olde Touriste bars. I’m not sure whether they have these Area-51-type fences to keep the locals out or the tourists in, but either way, I hate it.

The on-ship entertainment

This is less relevant for next year, but the things the Royal Caribbean offered as “entertainment” were obnoxious. Talks on “how to buy an expensive watch” or “procedures you can have to look younger.” With the casino, the constant greatest-hits karaoke, the the bars every 30 feet, everything was about not thinking.

Minor, but annoying

You had to use hand sanitizer all the time, which was disgusting. Preferable to the alternative, but it always felt like someone drooling on my hands.

In conclusion…

My friends all loved it and are going back next year, but the whole experience made me want to go camping for the rest of my life.

Star Trek invades our timeline

I was at Kennedy Space Center yesterday and they have an exhibit with all of the Apollo mission flags. Having mission flags is a great idea, more software launches should have flags, too. I noticed one in particular:

apollo_flag

(Please excuse the poor image quality, I have a technology-defying ability to take crappy photographs.)

Those symbols on the flag look vaguely familiar…

apollo_rotated

Compare to:

starfleet_insignia

What’s hilarious to me is that Star Trek: The Original Series started airing in 1966. Apollo 15 didn’t launch until 1971, so it must have been pretty blatant that they were copying that. I couldn’t find anything about it with a brief Google search.

Another exhibit about the space shuttle confirmed the intermingling between NASA and Star Trek:

enterprise_shuttle

Live long and prosper!

Flag-Friday: debugging tests with –java_debug

To step through a Java test that you’re running with bazel test, use the --java_test flag:

$ bazel test --java_debug //src/test/java/com/example:hello-test
WARNING: Streamed test output requested so all tests will be run locally, without sharding, one at a time.
INFO: Found 1 test target...
Listening for transport dt_socket at address: 5005

At this point, switch over to your IDE and fire up a remote debugging configuration, with the host localhost:5005. The debugger will attach to the running process, your test will start, and the debugger will stop at your first breakpoint.

Saving the (prod) environment

You can create different environments (e.g., testing, prod, mobile, rainforest) with Bazel, then use them to make sure that targets only build with the right environment. This is a cool feature that’s undocumented (because it’s still in development, shhhhh, don’t tell anyone I told you about it).

Let’s say you have a prod SSH key that you don’t want used in development or test builds. You could restrict it to only be used in prod builds by defining the following:

environment(name = "dev")
environment(name = "prod")
environment(name = "testing")

environment_group(
    name = "buildenv",
    defaults = ["dev"],
    environments = [
        "dev",
        "prod",
        "testing",
    ],
)

filegroup(
    name = "ssh-key",
    restricted_to = [":prod"],
    srcs = ["key"],
)

Now whenever we use :ssh-key, it has to be in a prod-environment rule. For example, this works:

cc_binary(
    name = "prod-job",
    srcs = ["job.cc"],
    restricted_to = [":prod"],
    data = ["ssh-key"],
)

This doesn’t:

cc_test(
    name = "job-test",
    srcs = ["job_test.cc"],
    data = [":ssh-key"],
)

Building the second one gives:

$ bazel build :job-test
ERROR: /Users/kchodorow/test/a/BUILD:34:1: in cc_test rule //:job-test: dependency //:ssh-key doesn't support expected environment: //:dev.
ERROR: Analysis of target '//:job-test' failed; build aborted.
INFO: Elapsed time: 0.167s

Hopefully, if someone tried to add restricted_to = [":prod"] to a test, it’d “look wrong” and be easier to catch.

Note that you must set your defaults sanely: when I first tried this, I made the environment_group‘s defaults = ["prod"] and then was confused that I wasn’t getting any errors. Everything is built for the default environments unless specified otherwise!

This lets us say: “If a depends on b and b is restricted to a certain environment, then a must be restricted to the environment.” However, there is another direction to look at this from: if a is restricted to an environment, b must be compatible with that environment. To express this, you can use “compatible_with“:

filegroup(
    name = "dev-key",
    srcs = ["key.dev"],
    compatible_with = [
        ":dev",
        ":testing"
    ],
)

Now anything that’s restricted to “:dev” or “:testing” environments can depend on “:dev-key”. For example, these work:

cc_binary(
    name = "dev-job",
    srcs = ["job.cc"],
    data = [":dev-key"],
)

cc_test(
    name = "job-test",
    srcs = ["job_test.cc"],
    restricted_to = [":testing"],
    data = [":dev-key"],
)

This does not:

cc_binary(
    name = "prod-job",
    srcs = ["job.cc"],
    restricted_to = [":prod"],
    data = [":dev-key"],
)

The full matrix (assuming env is an environment) is:

b b restricted to env b compatible with env
a
a restricted to env
a compatible with env

Remember that environments are targets themselves, so avoid proliferating environments that aren’t global to the global scope (don’t make them publicly visible and keep them as private as possible).