Testing MongoDB Replica Pairs with Perl

I was just fixing this, and it’s a pain to setup/test, so I figured I’d put up what I did here, so at least I’d never have to figure it out again.  So, start two db servers and an arbiter:

$ mkdir ~/data1
$ mkdir ~/data2
$
$ ./mongod --pairwith localhost:27018 --arbiter localhost:27019 --dbpath ~/data1
$ ./mongod --port 27018 --pairwith localhost:27017 --arbiter localhost:27019 --dbpath ~/data2
$ ./mongod --port 27019 #arbiter

Then kick off the following Perl script:

use strict;
use warnings;
use Carp;
use Data::Dumper;

use MongoDB;

# order of left_host/right_host is unimportant
my $conn = MongoDB::Connection->new('right_host' => 'localhost', 'left_host' => 'localhost', 'left_port' => 27018);

my $db = $conn->get_database('x');
my $coll = $db->get_collection('y');

$coll->drop;

for (my $i=0; $ifind_one;
    };

    print $@."n";

    sleep 1;
}

Now, you’ll start getting stuff like:

finding...
finding...
finding...

That means everything is okay. Now, kill the first mongod process that you started. (The first one you started is probably master. If you don’t see a “pair: setting master=1 was 0” on the other one after a couple seconds, restart the first one and kill the second one.) You’ll notice your output changes to:

finding...couldn't find master at /usr/local/lib/perl5/site_perl/5.10.0/i686-linux/MongoDB/Connection.pm line 177.
finding...couldn't find master at /usr/local/lib/perl5/site_perl/5.10.0/i686-linux/MongoDB/Connection.pm line 177.
finding...couldn't find master at /usr/local/lib/perl5/site_perl/5.10.0/i686-linux/MongoDB/Connection.pm line 177.

The slave hasn’t become master yet, so there’s nothing the db can use (there will probably be a 2-10 second delay before the slave becomes master). Once you see “pair: setting master=1 was 0” scroll by on your db logs, you’ll notice the Perl script’s output change back to:

finding...
finding...
finding...

You can go back and forth, killing off master dbs and restarting them.

Final Days in CA

I was reading my new book about the actress in rehab, and I was vaguely wondering why the writer’s name sounded familiar.  I couldn’t quite place it, so I flipped to the back to see what her bio said.  Suddenly I realized where I knew her from.

“Hey Andrew, guess who wrote this book?”  I showed him the front cover.

“No!  No way!  THE Carrie Fisher?”

Yup.  My book was written by Princess Leia.  It wasn’t great, but it wasn’t terrible, either.  Now I’m wondering if I should get another book.  We’re flying home tomorrow, but I feel vaguely uncomfortable going even 24 hours without something to read.

Speaking of going home, I mailed all of our dirty clothes home today so that we’ll have almost nothing to carry tomorrow.  I asked the front desk, and they told me the nearest post office was in the back of Macy’s.  I walked down the Macy’s block and back again.  It was a whole block of Macy’s, nothing else.  Finally, I asked at a restaurant.

“The post office is inside Macy’s.”

Weird.  I went into Macy’s, where there was no sign of a post office.  I asked an over-collagened makeup hawker where the post office was.  “Oh, in the basement.”

I went down the escalator, wandered around for a bit, and finally gave up and asked another sales lady.  “If you follow the red wall, it’s down there.”  Oh.

So, to get to the post office, you go into Macy’s, go down to the basement, make a U-turn, and it’s in sort of a back room.  I’ve been to speakeasies that were easier to find.

Reading in San Francisco

Andrew and I are relaxing in San Francisco this weekend.  I gave a talk at the Perl Mongers meetup on Thursday, which was awesome, but stressful.

I’m almost done with The Clown, by Heinrich Boll, which is one of the most depressing books I’ve ever read.  “Didn’t you just read a depressing book before that, too?” Andrew asked me.

“Oh yeah, Cry the Beloved Country.  This is more specific depressing, Cry made me despair of mankind in general.”

“So, it’s a pointed depressing.”

Anyway, I’ve been looking forward to getting it over with.  It’s really good, but it is so sad.  The book is very old, too, so it is sort of falling apart as I read it.  It cries these little broken-off bits of paper all over me as I read it, like it was sad, too.

Anyway, this morning we went to the San Francisco Library (because we’re big dorks).  They had a $1 book shelf, and I picked up a new, light, fun looking book.

“What’s it about?” Andrew asked.

“This movie star in drug rehab.”

He looked at me squiggly-eyed.  “Sounds depressing.”

“The first sentence is, ‘I probably shouldn’t have given my phone number to the guy who pumped my stomach.’  I think it’ll be alright.”

Adventures in Ramen

Last night, I was exhausted.  When Andrew and I got home, he wasn’t hungry, so I made some ramen.  I have this special way I make ramen where I microwave it until the noodles are soft, then stir fry them.  It’s really good.

We were sitting around, reading the internets, when Andrew said, “Do you smell smoke?”

We looked towards the kitchen, where there was smoke billowing from the microwave.

“Oh crap!” I said.

He leaped up, rushed into the kitchen, hit the stop button, and started choking on the smoke.   “Did you put in any water?” he coughed.

“Oh crap!”  I went a few steps towards the kitchen, inhaled a bunch of smoke, and decided he had the situation under control.

When he opened the microwave door, smoke poured out.  Sitting at the bottom of the bowl was a briquette of ramen that was pitch black and smoking.  Andrew filled the bowl with water in the sink.  It hissed and steamed, eventually turning into a brown sludge around the burned ramen.

The crisis was over, but the apartment stunk.  There was so much smoke it looked foggy.  We opened all the windows, turned on the fan, and still could barely breathe.

“Shall we go out for dinner then?” I asked.

“Let’s.”

Luckily, we live on a street with 14 different ramen places.  It was delicious.

“You know, you almost set the apartment on fire cooking noodles,” Andrew pointed out.

It’s going to be a while until I live this one down.

I’m going to Brazil!

I have been invited to speak at Latinoware 2009!  It’s October 22-24th and I’m giving a talk on non-relational databases in general, MongoDB in particular.  I’ll be giving it in English, but it’s going to be simulcast in Spanish and Portugese.

I’ve never been to South America, and I’m so excited.  The conference is right near these famous waterfalls that look freakin gorgeous.  It looks like we might be able to check out the rainforest a bit, too (it’s right nearby).  So.  Cool.

I made the travel reservations today and it’s going to take me 15 hours to get there, 19 hours to get back.  The way back has a seven hour stopover, followed by a 10 hour flight.  Ugh… not looking forward to that part of it.  At least it’s at the end of the trip so we can get home and collapse.

Anyway, I am really looking forward to seeing Brazil.  I’m going to try to learn some Portuguese before I go, but I dunno… I hope people speak some English.  I’ve never been very good at picking up (non-programming) languages.

OSCON

I’m at OSCON, which is really fun (and exhausting).

Last night, when my company’s venture capitalist found out I had MongoDB stickers, he asked me to put them on the tables scattered around the conference area, where everyone sat to talk and surf the web.

For some reason, doing so made me feel like a total tool, but this morning, I sucked it up and put a little pile on each table.  As I was walking away from one table, a guy sitting there loudly said, “I bet she doesn’t even know what ‘Mongo’ is from.”  Jerk.  What geek isn’t a Mel Brooks fan?  Even if I hadn’t seen Blazing Saddles, I work on the freakin project.

When I put the last pile down on the table at which I was eating breakfast, the guy sitting next to me immediately picked one up and stuck it on his laptop.

“Do you use MongoDB?” I asked, surprised at the chances, but gratified.

“No, I’ve never heard of it, but it’s a lot nicer than the HP logo.”

So, good job Jason!  (The graphic designer for MongoDB.)  I mean, not only with the spiffy business cards and stickers, but he actually made our Confluence wiki pretty, which is no mean feat.

Soccer: No Slimy GirlS Allowed!

Now, I love playing soccer, which I haven’t gotten to do much since high school. Recently, someone created a pickup soccer game meetup for IT professionals (http://www.meetup.com/soccer-nyc/).  What with it being geeks+soccer, this sounded awesome, so I requested to join.  Denied!

Your request to join NYC Turquoise IT – Soccer For IT Professionals has been declined

The person who declined your request, Xxxxxxxxxx, said:
—————————————————————-
sorry not for girls
—————————————————————-

and he added the “Male only” bullet point to the membership requirements.

WTF? It’s a freakin pickup game! Are they afraid women will get hurt? Sit around buffing our nails? What?

Edit: I just told my cowoker Mike about it and he linked me to http://groups.google.com/group/nyc-pickup-soccer, which is a group he started and thus, he assures me, is coed.  Everyone (interested in soccer) should join this group!

How do you make the web reliable?

Andrew and I were talking this morning about how we can count on the New York Times to be accurate, but an article linked to by Reddit is often horribly biased.  We began thinking about how to hold the web accountable, and we came up with a nifty idea.  Make it a wiki/competition/social network.  Here’s our plan:

You download a Firefox extension that puts  a little “+/- comments” in the corner of your screen.  If you’re on a page you like, you click the “+”.  If you don’t like the page, you click the “-“.  If you have more to say, click “comments” and type in a comment about the page, and anyone visiting the site after you will be able to see it.  Clicking “comments” will show you all the comments other people have made about this page, too.

Let’s take an example.  Recently, Microsoft put up a page comparing IE8 to other browsers.  It was… a bit biased.  But there’s no comments section!  So, using my handy-dandy extension, I click on “-” because I think it’s a dumb page.  Then I click on “comments” and type “biased much?” (clever, eh?).  Then I see that someone else made a truly intelligent comment and linked to an impartial comparison.  I click on the “+” next to their name, upvoting their comment.  Oh, also, I see that the page has been rated -923 by users overall.

Now, I’m totally riffing off of Stack Overflow here, which had the brilliant idea of attaching karma to users. When I upvote a user’s comment, they get 10 karma points.  When I downvote, they lose a karma point, and so do I.  More karma gives you more privileges.  

I’m really psyched about this.  It seems like it would be easy to develop the basic idea (basically a user system and a blog system) and there are a zillion features we can add later. Benefits for users:

  • Ability to comment on web sites with no comments section (cool ones I’ve thought of are Twitter pages, ftp:// (downloads), and pay sites’ logins (yes, I’m a jerk))
  • The satisfaction of gaining karma to gain more power
  • Access to other visitors to the site, whom you’d never normally be able to interact with

So, I’m going to try to implement it.

CouchDB vs. MongoDB Benchmark

Edit (9/1/10): this benchmark is old, silly, and should probably be ignored in favor of more recent and representative ones. I don’t want to take it down for historical purposes, but seriously people, it was never a good benchmark, it’s over a year old at this point, and both databases have changed a lot.

Edit (12/6/09): this is the #1 Google result for “mongodb benchmark”, so I figure I’ll do some community service: if you’re interested in benchmarks, you might want to look at the 3rd party ones listed on the mongodb.org website.


Felix Geisendörfer did a benchmark in PHP that was super-easy for me to port into MongoDB. You can see his post on his blog.

And now… comparing his results for CouchDB with mine for MongoDB’s (I did the graph in Open Office, which is why the quality sucks):

As you can see, MongoDB does, uh, slightly better.  Here are the numbers:

# of Inserts Couch Total Time (sec) Couch Time/Doc (ms) Mongo Total Time (sec) Mongo Time/Doc (ms)
1 .0015 1.46 .0005 .5
2 .0015 .75 .0004 .2096
3 .0017 .56 .0005 .1604
4 .0017 .44 .0005 .1190
5 .0018 .36 .0005 .1060
6 .0019 .32 .0006 .0931
7 .0021 .3 .0006 .0847
8 .0022 .27 .0007 .0789
9 .0023 .25 .0007 .0734
10 .0025 .25 .0007 .0721
50 .0072 .14 .0024 .0476
100 .0136 .14 .0044 .0442
500 .0687 .14 .0253 .0505
1000 .1361 .14 .0372 .0372
2500 .4686 .19 .0278 .0111
5000 .9165 .18 .0488 .0098
7500 1.5116 .2 .0835 .0111
10000 2.3111 .23 .1065 .0107
25000 6.8684 .27 .2711 .0108
50000 15.8227 .32 .5430 .0109
100000 35.3071 .35 .1.7697 .0177
250000 104.0009 .42 6.4533 .0258
500000 230.6021 .46 11.7684 .0235
750000 352.7959 .47 17.0473 .0227
1000000 487.3284 .49 18.4376 .0184

Please let me know if I made any mistakes, all the values were hand-copied.

I ran these tests using the PHP driver on Ubuntu 9.04 on my MacBook Pro.  You can see the test script I forked on Github.

A little analysis: Both DBs start with some overhead, but by 1000 inserts CouchDB seems to be chugging along nicely.  MongoDB takes slightly longer to hit its groove, hitting its peak around 10000.  They both slow a little near the end, as MongoDB starts spending most of its time allocating files and, although I know almost nothing about CouchDB’s structure, I’d guess it’s doing something similar.

Because we are the Mutha Flippin Win

I’ve been keeping records of funny and nice things people have said about MongoDB. A lot of them can be found at http://www.mongodb.org/display/DOCS/User+Feedback, but there are some good ones that weren’t quite… “right” for the official page. So, slightly off-color, biased, or weird quotes ahoy:

Mailing list:

“Guys at Redmond should get a long course from you about what is the software development and support :-)”
-kunthar

Twitter:

@mully It was a well orchestrated effort for @mongodb world domination. I was the shock, @jnunemaker was the awe.
-pengwynn

the @mongodb guys are great! They figure out segfaults, fix bugs and push new builds in practically real time!
-brondsem

Mongo looks like the Mutha Flippin Win. I need to carve some time out to play. http://www.mongodb.org/display/DOCS/Home
-thejohnny

IRC:

“limit=-10 is my hero”
-claymation

Blogs

“It actually has a kind of quirky, yet lovable syntax for defining criteria.”
-http://railstips.org/2009/6/3/what-if-a-key-value-store-mated-with-a-relational-database-system