Simulating Network Paritions with mongobridge

Note: mongobridge does not come with the MongoDB binaries, but you can build it by getting the source code and running scons mongobridge.

Let’s say we have a replica set with members (M1, M2, and M3) and we want to see what happens when M1 and M3 cannot reach each other (or any other sort of network partition). How do we do this?

We can’t shut down M3 because that would test something different. If we block incoming connections to M3, we end up blocking M2’s connections, too, which we don’t want.

We want M3 to only block M1’s connection, not M2’s. This is where mongobridge comes in.

mongobridge allows you to accept connections on one port and send them on to a MongoDB process listening on another port. So, for the M1↔M3 connection, we can set up a bridge. This bridge, like most bridges, needs to go two directions (M1→M3 and M3→M1), so we actually need two instances of mongobridge.

So, we’ll make the “onramps” for our bridge M1::10013 (for the M1→M3 connection) and M3:10031 (for the M3→M1 connection). To set this up, we run:

$ ssh M1
M1$ mongobridge --port 10013 --dest M3:27017
M1$ exit
$ ssh M3
M3$ mongobridge --port 10031 --dest M1:27017
M3$ exit

This means, “Take all traffic heading to M1:10013 and send it to M3:27017. Take all traffic heading to M3:10031 and send it to M1:27017.

However, M1:27017 doesn’t want to send its traffic to M1:10013, its configuration says to send M3-bound traffic to M3:27017. Same with M3:27017, it doesn’t want to send its traffic to M3:10031, it wants to send it to M1:27017. And we can’t reconfigure the set so that different members have different configurations… officially.

Unofficially, we can change each config to anything we want.

Warning: never, ever do what I’m about to do in production, this is just for trying out funky networking tricks.

Shut down M1 and M3, and start each back up without the --replSet option on a different port. This way it won’t connect to the rest of the replica set and you can edit the local.system.replset configurations to point at the bridges, instead of the other members.

> // supposing we started them on port 27018 instead of 27017...
> db = (new Mongo("M1:27018")).getDB("local")
> db.system.replset.update({}, {$set : {"members.2.host" : "M1:10013"}})
> db = (new Mongo("M3:27018")).getDB("local")
> db.system.replset.update({}, {$set : {"members.0.host" : "M3:10031"}})

Now, restart M1 and M3 with their proper ports and --replSet. They will connect to each other through mongobridge and our replica set will be ready to use.

To simulate a network outage, kill the mongobridges between two servers.

Suppose M1 is the primary and you have knocked out the network between M1 and M3. If you do a write on M1, M2 will sync to M1 and then M3 will sync to M2, so you can see that the write will still make it to M3.

And that’s the dramatic payoff.