Using environment variables in Skylark repository rules

If you’ve every used the AppEngine rules, you know the pain that is wait for all 200 stupid megabytes of API to be downloaded. The pain is doubled because I already have a copy of these rules on my workstation.

To use the local rules, all I have to do is override the @com_google_appengine_java repository in my WORKSPACE file, like so:

load("//appengine:appengine.bzl", "APPENGINE_BUILD_FILE")
new_local_repository(
    name = "com_google_appengine_java",
    path = "/Users/kchodorow/Downloads",
    build_file_content = APPENGINE_BUILD_FILE,
)

However, this is still imperfect: I don’t really want to maintain changes that basically amount to a performance optimization in my local client.

By using environment variables in the appengine_repository rule, we can do even better. I’m going to create a new rule that checks if the APPENGINE_SDK_PATH environment variable is set. If it is, it will use a local_repository to pull in AppEngine, otherwise it will fall back on downloading the .zip.

So, to start, let’s take a look at the existing rule that pulls in the AppEngine SDK. As of this writing, it looks like this:

  native.new_http_archive(
      name = "com_google_appengine_java",
      url = "http://central.maven.org/maven2/com/google/appengine/appengine-java-sdk/%s/%s.zip" % (APPENGINE_VERSION, APPENGINE_DIR),
      sha256 = "189ec08943f6d09e4a30c6f86382a9d15b61226f042ee4b7c066b2466fd980c4",
      build_file_content = APPENGINE_BUILD_FILE,
  )

First, let’s modify this to use a repository rule instead of native.maven_jar:

def _find_locally_or_download_impl(repository_ctx):
   repository_ctx.download_and_extract(
     "http://central.maven.org/maven2/com/google/appengine/appengine-java-sdk/%s/%s.zip" % (APPENGINE_VERSION, APPENGINE_DIR),
     ".", "189ec08943f6d09e4a30c6f86382a9d15b61226f042ee4b7c066b2466fd980c4", "", "")
  repository_ctx.file("BUILD", APPENGINE_BUILD_FILE)

_find_locally_or_download = repository_rule(
  implementation = _find_locally_or_download_impl,
  local = False,
)

def appengine_repositories():
  _find_locally_or_download(name = "com_google_appengine_java")

This code functions identically (basically) to the original code, so now let’s add an option for using a local path. Modify in the implementation function to check the environment:

def _find_locally_or_download_impl(repository_ctx):
  if 'APPENGINE_SDK_PATH' in repository_ctx.os.environ:
    path = repository_ctx.os.environ['APPENGINE_SDK_PATH']
    if path == "":
      fail("APPENGINE_SDK_PATH set, but empty")
    repository_ctx.symlink(path, APPENGINE_DIR)
  else:
   repository_ctx.download_and_extract(
     "http://central.maven.org/maven2/com/google/appengine/appengine-java-sdk/%s/%s.zip" % (APPENGINE_VERSION, APPENGINE_DIR),
     ".", "189ec08943f6d09e4a30c6f86382a9d15b61226f042ee4b7c066b2466fd980c4", "", "")
  repository_ctx.file("BUILD", APPENGINE_BUILD_FILE)

Now we can download a copy of the SDK and try our rule (feel free to use an existing copy, if you have one on your system).

APPENGINE_SDK_PATH=/path/to/your/sdk/download bazel build //your/appengine/app

Problems with this:

• You can’t actually set APPENGINE_SDK_PATH to where Bazel downlaoded the SDK the first time around ($(bazel info output_base)/external/com_google_appengine_java), which is suuuuper tempting to do. If you do, Bazel will delete the downloaded copy (because you changed the repository def) and then symlink the empty directory to itself. Never what you want.
• It caches the environment variable, so if you change your mind you have to run bazel clean to use a different APPENGINE_SDK_PATH. I think this is a bug, although there’s some debate about that.