Rodger and I at BiggerBoat got Hadoop and HBase up and running on Amazon EC2 today. We initially set up a cluster of 1 master and 10 slaves. After a quick calculation of how much this costs to keep running 24/7, we started trying to figure out how to scale the thing DOWN as well as UP, and to be able to do so dynamically. Seems like the tricky piece is the Hadoop storage, not so much the compute power available. Amazon just launched their Elastic Block Store a few days ago, so we’re seeing how that fits in. Seems like the EBS I/O is pretty good given our Bonnie++ tests.
Tom White has some architecture scenarios for building this kind of stuff.
More thoughts on EC2 / EBS / Hadoop | ♥data♥ | 26-Aug-08 at 1:34 pm | Permalink
[...] made some comments yesterday about wanting to be able to scale the Hadoop cluster down as well as up, in particular being able [...]