Using secrets with Google AppEngine

For side project #4323194 (implement a chrome extension that looks like this: 👂 and turns red when someone mentions you on GitHub), I needed to implement oauth from AppEngine to GitHub. As I’ve mentioned before, oauth is my nemesis, but for this project there didn’t seem to be a great way around it. It actually wasn’t as bad as I remember… maybe I know more about HTTP now? Either way, I only messed up ~16 times before I got it authenticating properly.

When you want an app to work with the GitHub API, you go to GitHub and set up a new application, tell it what URL it should send people to after login, and it gives you a “secret key” that no one else should know. Then you simply implement oauth’s easy, intuitive flow:

  1. Redirect a user to https://github.com/login/oauth/authorize when you want them to log in.
  2. GitHub will ask the person to log in, then redirect back to the URL you gave it when you set up your app.
  3. You POST the secret key you got to https://github.com/login/oauth/access_token.
  4. GitHub replies with an access token, which you can then use in the header of subsequent requests to access the API.

The problem here is #3: the secret key. In ye olde world of “I have a server box, I shall SSH into it and poke things,” I would simply set an environment variable, SOOPER_SECRET=<shhh>, then get that from my Java code. However, AppEngine prevents that sort of (convenient) nonsense.

So, I poked around and the thing I saw people recommending was to store the value in the database. This is an interesting idea. On the downside, it’s approximately a zillion times slower than accessing an environment variable. On the other hand, this is a login flow that makes three separate HTTP requests, I don’t think a database lookup is going to make a huge difference. On the plus side, it “automatically propagates” to new machines as you scale.

So I began working on a class to store the secret. Requirements:

  • Easy to set: I want to be able to visit a URL (e.g., /secrets) to set the value.
  • Difficult for others to set: I don’t want users to be able to override keys I’ve created, or create their own keys.
  • Perhaps most importantly: difficult for me to unintentionally commit to a public GitHub repo. I am super bad at this, so I need a completely brain-dead way to never, ever have this touch my local code, otherwise it will end up on GitHub.

What I decided on:

Create a servlet (/secrets) that takes the key/value to set as a query parameter. The servlet will only set the secret key for keys I’ve defined in code (so visitors can’t set up their own secret keys) and will only set the secret key if it doesn’t exist in the database, yet. Thus, after the first time I visit /secrets, it’ll be a no-op (and actually can be disabled entirely in production). Because the secret is given as a query parameter, it never hits my code base. It will appear in request logs, but I’m willing to live with that.

What this looks like in an AppEngine app:


    
        secrets
        com.meab.oauth.SecretDatastore
    
    
        secrets
        /secrets
    

And the Java code does some URI parsing and then:

  private void findOrInsert(String key, String value) {
    Entity entity = getEntity(key);
    if (entity != null) {
      // No need to insert.
      return;
    }

    entity = new Entity(ENTITY_TYPE);
    entity.setProperty("key", key);
    entity.setProperty("value", value);
    datastore.put(entity);
  }

And the nice thing about using Google’s AppEngine datastore is that it’s easy (relatively) to write tests for all this.

You can check out the sources & tests at my git repo. (Note that the extension doesn’t actually work yet, right now it just logs in. I’ll write a followup post once it’s functional, since I think this might be relevant to some of my readers’ interests.)

That’s senior programmer to you, buddy

After about a decade of professional programming, I have finally gotten promoted. For the first time. This is a weird industry.

Regardless, I am now a “Senior Software Engineer.” Woo!

Thinking about it, this has been a goal of mine for a long time. Now that I’ve achieved it, I’m not sure what’s next.

“…and Alexander wept, for there were no more worlds to conquer.”

Pain-free OAuth with AppEngine

meme-5828337892065280

I do a lot of side projects (or at least start them) and implementing authentication is always like chewing on razor blades. I started another project recently using AppEngine and, bracing myself with a lot of caffeine and “suck it up, princess” attitude, I started doing “oauth appengine” searches.

After digging through some documentation, I realized that AppEngine actually does OAuth right. All you have to do is add the following to your src/main/webapp/WEB-INF/web.xml file:

    
        
           my-thing
           /members-only/*
        
        
           *
        
    

The <url-pattern>/members-only/*</url-pattern> means that when someone goes to any page under members-only/, they’ll have to go through the “login with Google” flow.

On the server side, there’s no annoying URL encodings to get right or tokens to keep track of. You can just access the logged-in person’s username, e.g., in Java:

public class DemoServlet extends HttpServlet {
    @Override
    public void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
    	String account = req.getUserPrincipal().getName();

        ...
    }
}

That’s it! This is such a killer feature to me, I can’t believe I never knew about about it before.

(One other thing that I can’t believe I never knew about is SimpleHTTPServer. python -m SimpleHTTPServer will serve static files from the current directory. I’m pretty sure everyone already knows about this, but just in case there’s someone else out there…)

Makin’ Mazes

After my previous post on the subtleties of CSS subpixel rendering, Andrew pointed out that readers might be more interested in how to dynamically generate mazes. It sounded crazy, but here we are.

First of all, if you’re interested in this stuff, there’s a great slideshow on maze generation here and more resources on the procedural content generation wiki.

Basically, your maze is made up of boxes:

single_square

You create the maze by making a grid of these boxes, and then Kool-Aid-maning your way through certain walls.

Ohhh yeah.
Ohhh yeah.

How do you choose which walls? That’s where the algorithm comes in.

First, you choose a square to start the maze at. You add this square to an “active set,” which is the set of squares that you’re currently enmazing:

this.active_set_.push(startNode);

Then you choose one of the square’s neighbors at random. If that square isn’t part of the maze yet, you connect it to the starting square and add it to the active set. If it is already part of the maze, you choose a different neighbor. If all of the neighbors are already part of the maze, you remove the starting square from the active set.

var new_node = startNode.getNeighbor();
if (new_node) {
    this.active_set_.push(new_node);
} else {
    goog.array.remove(this.active_set_, startNode);
}

Now choose a member from the active set and repeat until the active set is empty.

Putting this all together, this looks like the following:

Maze.prototype.generate = function() {
    this.active_set_.push(this.startNode_);
    while (this.active_set_.length > 0) {
        this.run_();
    }
};

Maze.prototype.run_ = function() {
    var node = this.pickActiveNode();
    var new_node = node.getNeighbor();
    if (new_node) {
        this.active_set_.push(new_node);
    } else {
        goog.array.remove(this.active_set_, node);
    }
};

The actual full code for this is a couple hundred lines long and you can get it from my Github repo. However, the two functions above are the “meat,” the rest is just details.

Screen Shot 2014-01-18 at 12.22.27 PM

The most interesting part of the code above is the var node = this.pickActiveNode(); line. Depending on how you pick nodes from the active set, you get different “shaped” mazes. For example, here’s random:

random

30% choosing the first square in the active set:

some_pop_first

60% choosing the first square in the active set:

60p_pop_first

Each maze has a different “character.” Pretty cool!

Fixing CSS antialiasing on Chrome

I’ve been working on creating mazes with LimeJs and I ran into a problem:

Screen Shot 2014-01-18 at 12.42.13 PM

As you can see, this maze looks like it was rendered by a dying printer: there are horizontal white lines all over the place. I was going crazy trying to track down the cause: the squares were perfectly sized/placed and the padding and margins were all zero pixels. It looked fine in Firefox, just Chrome had the issue. Finally, Andrew figured out that it was antialiasing on pixel borders and, if I offset the images by less than a pixel, they wouldn’t be “rounded off.”

I tried offsetting everything by .5 pixels:

Screen Shot 2014-01-18 at 12.41.57 PM

Better, but not perfect. However, with .1 pixels:

Screen Shot 2014-01-18 at 12.41.34 PM

Good enough!

I made a patch for this but I haven’t made it work for masks so I’m not going to do a pull request for it (yet). However, you can cherry-pick it from my fork of LimeJs and use it by adding the following line to your start() function:

lime.Renderer.DOM.ANTIALIAS_OFFSET.y = .1;

If anyone knows anything about why Chrome antialiases stuff like this, I’d be interested in knowing (I found lots of sites complaining that it was weird, but none explaining it).

Happy hacking!

Smart Pointers and Heirloom Underpants

batman

I’ve seen a lot of descriptions of scoped_ptr and unique_ptr, but often they don’t give clear examples of when to use one vs. the other. Hopefully this will clear things up.

unique_ptr is like an heirloom ring: you might purchase it, have it all your life, and end up being buried with it. For example, something like:

void Human::Life() {
  std::unique_ptr ring(new Ring());
  // Leaving the scope is "dying" and the ring is destroyed
}

However, you could also pass it on to a child. For example:

void Human::Life() {
  std::unique_ptr ring(new Ring());
  // Allocates a new child on the heap that will outlive this
  Kid *kid = CreateChild(this, GetSpouse());
  kid->Inherit(ring.release());
}

Now the parent’s life can end and the ring will survive.

On the other hand, scoped_ptr is more like “my underpants will die with me.” There is no passing them on. The intended use is something like:

void Human::Life() {
  scoped_ptr underpants(new Underpants());
  for (int i = 0; i < 1000; i++) {
      SendToCleaners(underpants.get());
  }
}

You might lend them out, but you retain ownership. Say you try to pass it on anyway with something like:

void Human::Life() {
  scoped_ptr underpants(new Underpants());
  Kid *kid = CreateChild(this, GetSpouse());
  kid->Inherit(underpants.get());
}

At the end of Life(), the underpants will be destroyed and any pointers to them will now point to garbage. Which makes sense, since they’re pointing to a dead guy’s underpants.

Finally, there’s shared_ptr, which is more like your award-winning chili recipe. If you tell a bunch of people about it then you’re sharing ownership of the chili recipe. Eventually all of you will die and the knowledge will be gone but you don’t know who’s going to the big chili pot in the sky in what order, so the data isn’t destroyed until you’re all gone.

Upcoming Hackathons

A couple of interesting hackathons are coming up:

  • October 4-7th: Ludum Dare October Challenge – The challenge is to monetize a game. Win the challenge by earning $1. You can monetize a game you’ve already made (or make a new one).
  • October 11-12th: Tizen Hack, an in-person jam in NYC for making an app for the Tizen app store. I’ve never heard of it before, but it was done by the Linux Foundation and supports HTML5 or C++ for app development, so it sounds interesting.
  • October 11-13th: Asylum Jam, where the goal is to create a horror game that does not use mental illness or psychiatry as a source of fear. Nice Halloween-themed game jam for a good cause.

Busy month!

Hello, Lumpy World!

Overview: how to generate 3D terrain for iOS. This assumes basic knowledge of iOS programming (how to create a project and add files to it.)

I’ve been working on an iOS app that uses cocos3d and Robert’s hill-generating algorithm, which make nice hills that remind me of early Mario levels. For example:

Nice starter world
My lovely verdant lumps.

You can download the hill generation library from Github. I’ve converted Rob’s code from C++ to Objective-C and added support rectangular terrain (instead of just square terrain).

To generate your own terrain with it, download and install cocos3d and create a new cocos3d2 project called “TerrainGenerator”. It’ll create a 3D “Hello, world!” application. Hit the “Play” button to make sure it runs:

hello, world
Absolutely gorgeous.

To add the hill generation code, download the library from Github and add HillTerrain.m and HillTerrain.h to your project.

Replace the contents of TerrainGeneratorScene.m with:

#import "TerrainGeneratorScene.h"
#import "CC3PODResourceNode.h"
#import "CC3ActionInterval.h"
#import "CC3MeshNode.h"
#import "CC3Camera.h"
#import "CC3Light.h"
#import "CC3ParametricMeshNodes.h"
#import "HillTerrain.h"


@implementation TerrainGeneratorScene

-(void) dealloc {
  [super dealloc];
}

-(void) initializeScene {
  // Lights
  CC3Light* lamp = [CC3Light nodeWithName: @"Lamp"];
  lamp.location = cc3v(-2.0, 0.0, 0.0);
  lamp.isDirectionalOnly = NO;

  // Camera
  CC3Camera* cam = [CC3Camera nodeWithName: @"Camera"];
  cam.location = cc3v(128.0, 0.0, 300.0);
  [cam rotateByAngle:15 aroundAxis:cc3v(1.0, 0.0, 0.0)];
  [cam addChild: lamp];
  [self addChild: cam];

  // Action! (Well, okay, just terrain.)
  [self createTerrain];

  // OpenGL fun.
  [self createGLBuffers];
  [self releaseRedundantContent];
  [self selectShaderPrograms];
}

-(void) createTerrain {
  HillTerrain *terrain = [[HillTerrain alloc] init];
  [terrain generate];

  CC3MeshNode *mesh = [[[CC3MeshNode alloc] init] retain];
  mesh.vertexContentTypes =
    kCC3VertexContentLocation | kCC3VertexContentColor | kCC3VertexContentNormal;
  [mesh populateAsRectangleWithSize:CGSizeMake(terrain.size.width, terrain.size.height)
                  andRelativeOrigin:CGPointZero
                    andTessellation:CC3TessellationMake(terrain.size.width-1,
                                                        terrain.size.height-1)];

  int count = 0;
  for (int j = 0; j < terrain.size.height; ++j) {
    for (int i = 0; i < terrain.size.width; ++i) {
      float height = [terrain getCell:cc3p(i,j)];
      [mesh setVertexLocation:CC3VectorMake(i, j, height * 128) at:count];
      [mesh setVertexColor4F:ccc4f(0.0, height, 1.0-height, 1.0) at:count];
      ++count;
    }
  }

  [mesh setShouldUseLighting:YES];
  [self addChild:mesh];
  [terrain release];
}

@end

Save and run, and you should see something like this:

Default terrain
Hello, lumpy world!

The createTerrain method is where all of the magic happens, so let’s take it one section at a time:

Generating the terrain

Here’s the part that actually generates the terrain:

  HillTerrain *terrain = [[HillTerrain alloc] init];
  [terrain generate];

You can change any of the parameters you want here, for example, let’s try a different seed:

  HillTerrain *terrain = [[HillTerrain alloc] init];
  [terrain setSeed:123]; // Because I'm so creative.
  [terrain generate];

This generates a nice range of mountains:

A blobby range of mountains
The majestic Blob Range

Creating something OpenGL can use

The next part is mapping this terrain onto an array of vertexes. First, we create a mesh, aka the surface we want to display. You can picture it like a fishing net you’ll be draping over the landscape.

  CC3MeshNode *mesh = [[[CC3MeshNode alloc] init] retain];

Specifying the types of storage we’ll need

Next, we have to let Cocos3d know what type of info we want to store about this mesh. For us, this includes:

  1. The location of each vertex, because that’s the point of all this.
  2. The color at each vertex, because it’ll vary based on height.
  3. The normal of each vertex, which is the direction light will bounce off of it. This is automatically populated by Cocos3d, so you don’t have to worry about it for now. However, it’s interesting to try removing this one and seeing what the terrain looks like.
  mesh.vertexContentTypes =
    kCC3VertexContentLocation | kCC3VertexContentColor | kCC3VertexContentNormal;

Allocating the mesh

Now that we’ve told Cocos3d the types of storage we need, we tell it the shape and size to allocate: a rectangular grid the same size as our terrain.

  [mesh populateAsRectangleWithSize:CGSizeMake(terrain.size.width, terrain.size.height)
                  andRelativeOrigin:CGPointZero
                    andTessellation:CC3TessellationMake(terrain.size.width-1,
                                                        terrain.size.height-1)];

andTessellation specifies how many squares across the terrain will be, so you want that number to be one fewer than the number of vertexes you have. For example, if you had a tessellation of 1 square by 1 square, you’d need 4 vertexes (one for each corner of the square). Thus, the -1s.

Mapping the terrain onto the mesh

First, we get the height at one of the map coordinates. This will be a number between 0 and 1, so we’ll scale it up to something reasonable, given our scale (height*128 in this case).

      float height = [terrain getCell:cc3p(i,j)];
      [mesh setVertexLocation:CC3VectorMake(i, j, height * 128) at:count];

Then we set the color, making the high locations greenest and the lowest locations bluest:

      [mesh setVertexColor4F:ccc4f(0.0, height, 1.0-height, 1.0) at:count];

The at:count indicates the index of the vertex we’re setting. Cocos3d keeps all of the vertexes in a big array and this is the index into it.

Turning on the lights

Finally, we tell OpenGL to actually apply light to the mesh. (Try removing this line and see what it looks like.)

  [mesh setShouldUseLighting:YES];

And we add the mesh as a child to our scene and free the terrain memory:

  [self addChild:mesh];
  [terrain release];

That’s it! If anyone has any suggestions or improvements, please feel free to file a pull request. I’m not too happy with island generation yet: I haven’t figured out the right combination of options so my islands aren’t just lumps.

If you’re interested, Rob’s algorithm is pretty interesting and easy-to-follow. I recommend reading through his description if you plan on using it in a project.

Achievement Unlocked: Found bug in LLVM debugger (maybe)

Also, on an unrelated note, I seem to have found a bug in LLDB (the LLVM debugger) in the making of this blog post, which makes me inordinately proud.

Programming a State Machine

A monster modeled after a dog in my neighborhood.
A monster modeled after a dog in my neighborhood.

My attempts at game programming usually turn into impenetrable spaghetti code: “If the player walks through this door, then have him talk to the princess, unless he’s killed a guard, in which case the guards attack, or if he comes out of the secret passage…”

The game I’m working on now is pretty simple, but I’ve kept it really clean (so far) by using a state machine to keep track of what should happen when. Basically, each scene in the game is a state. There’s an overarching state machine which runs the current state on each tap. A state can either return itself or choose a new state to run next time.

In Objective C (+cocos2d), a state looks like this:

@interface State : NSObject {
    CCLayer *ui;
}

-(id) init:(CCLayer*)layer;
-(Class) processTouch:(UITouch*)touch withEvent:(UIEvent*)event;

@end

The processTouch function either returns nil, which means “run me again next time” or the next state to run. The other half is a “machine” to run the states:

// -----------------
// Interface
// -----------------

@interface StateMachine : NSObject {
    State *currentState;
    CCLayer *ui;
}

-(id) init:(CCLayer*)layer;
-(BOOL) processTouch:(UITouch*)touch withEvent:(UIEvent*)event;

@end

// -----------------
// Implementation
// -----------------

@implementation StateMachine

// Initialize the state machine by setting currentState to the first state
-(id) init:(CCLayer*)layer {
    self = [super init];

    if (self) {
        ui = layer;
        currentState = [[FirstState alloc] init:ui];
    }

    return self;
}

// Run this from the UI's touch dispatcher: it runs the current state's processing code
-(BOOL) processTouch:(UITouch*)touch withEvent:(UIEvent*)event {
    Class nextState = [currentState processTouch:touch withEvent:event];

    if (nextState != nil) {
        currentState = [(State*)[nextState alloc] init:ui];
    }
}

@end

Then, you might have an implementation like this for a swords & sorcery RPG:

@interface PalaceDungeonState : State {
    Guard *guard;
}

@implementation PalaceDungeonState

-(id) init:(CCLayer*)layer {
    // Use ui to render a dungeon
}

-(State*) processTouch:(UITouch*)touch withEvent:(UIEvent*)event {
    if (guard.alive) {
        [guard updatePosition];
    }

    CGPoint touched = [ui convertTouchToNodeSpace:touch];

    switch (touched) {
    case GUARD:
         [guard dies];
         break;
    case STAIRWAY:
         return PalaceStairwayState;
    case SECRET_PASSAGE:
         return SecretPassageState;
    }

    return nil;
}

@end

I’m not thrilled with doing so much work in the init, so for this type of game I’d probably move that to a start method that would be called by StateMachine on state changes.

Regardless, I’ve found this makes it a lot easier to make a complicated sequence of events while keeping my code readable.

––thursday #7: git-new-workdir

Often I’ll fix a bug (call it “bug A”), kick off some tests, and then get stuck. I’d like to start working on bug B, but I can’t because the tests are running and I don’t want to change the repo while they’re going. Luckily, there’s a git tool for that: git-new-workdir. It basically creates a copy of your repo somewhere else on the filesystem, with all of your local branches and commits.

git-new-workdir doesn’t actually come with git-core, but you should have a copy of the git source anyway, right?

$ git clone https://github.com/git/git.git

Copy the git-new-workdir script from contrib/workdir to somewhere on your $PATH. (There are some other gems in the contrib directory, so poke around.)

Now go back to your repository and do:

$ git-new-workdir ./ ../bug-b

This creates a directory one level up called bug-b, with a copy of your repo.

Thanks to Andrew for telling me about this.

Edited to add: Justin rightly asks, what’s the difference between this and local clone? The difference is that git-new-workdir creates softlinks everything in your .git directory, so your commits in bug-b appear in your original repository.