♥data♥

I’m still getting up to speed on running Hadoop on EC2. Found this AWS post today describing how to easily port data into a hadoop cluster from S3, as well as easily create new Hadoop slaves using the AMI system images, start up clusters, and tear down clusters.

I made some comments yesterday about wanting to be able to scale the Hadoop cluster down as well as up, in particular being able to disable cores, which are the really expensive part of running a cluster on AWS.

Now we need to look into the AWS scripts and AMI images that are available to see how feasible it is to just maintain more data volumes than images. What I’m (roughly) thinking is that we might set up M data volumes for the DFS, but might want to run 1 <= N <= data/task nodes. In the case that N < M, you just load some of the N nodes with more than 1 EBS volume.

Also need to look into how HDFS deals with adding new volumes, i.e. will it just start replicating data onto nodes as they are added into the system? Is there a way to hot-add rather than restarting the data master? Hot-adding EBS volumes onto existing data nodes?

Looks like Darren has starting a blog. Awesome. Adding it to my link roll as soon as I figure out how.

I’m looking forward to that post comparing monitoring systems!

Rodger and I at BiggerBoat got Hadoop and HBase up and running on Amazon EC2 today. We initially set up a cluster of 1 master and 10 slaves. After a quick calculation of how much this costs to keep running 24/7, we started trying to figure out how to scale the thing DOWN as well as UP, and to be able to do so dynamically. Seems like the tricky piece is the Hadoop storage, not so much the compute power available. Amazon just launched their Elastic Block Store a few days ago, so we’re seeing how that fits in. Seems like the EBS I/O is pretty good given our Bonnie++ tests.

Tom White has some architecture scenarios for building this kind of stuff.

I’m in the process of setting up an analytic workflow at BiggerBoat. It’s looking like the main theme in data structures around here will be the sparse matrix. So I’ve been playing with opensource technologies for sparse matrices. Apache Hadoop’s HBase is looking like a good choice for now, maybe Hive later.

Right now I’m getting familiar with the former. As part of this, I’m improving the docs on the wiki to make them more user- (as opposed to core developer-) friendly. My documentation goal right now is to add some data transformation example code. There are already lots of hadoop examples for doing text -&gt text mapping, e.g. grep, cat, etc. For HBase not so much. I.e.

• text to text (done, many examples
• flatfile to HBase table (Bulk loader in the HBase wiki, I haven’t tried it yet)
• HBase table to flatfile
• HBase table to HBase table

I’ll be adding updated, complete, and simple code for the latter two (three?) in the next few days to the HBase/MapReduce page.

I’ve been thinking about how to track unique users today. These are my so-far-unorganized thoughts. Please comment!

You can’t track users by cookie alone for a couple of reasons: 1) they might use multiple computers, 2) they might delete cookies, 3) multiple users might share the same computer (same account=same cookie)

You also can’t track users by IP address alone for some more reasons: 1) they might be using a mobile device or portable computer that moves from IP to IP, 2) there could be multiple machines passing through a single gateway IP (i.e. LAN NAT).

However, if you combine the cookie/IP information together, you can start to address some of these issues. Let’s assume you have some webserver logs that minimally contain <IP address "A">, <cookie ID "C">, <timestamp T> triples.

            === time ===>
PATTERN 1:
C1-A1 ---      ---
C1-A2    ---         ---
C1-A3       ---   ---

PATTERN 2:
C1-A1 ------
C2-A1       ------
C1-A2             ------

PATTERN 3:
C1-A1 -----!
C2-A1       -----!
C3-A1             ------

PATTERN 4:
C1-A1 ------      ------
C2-A1     ----------

PATTERN 5:
C1-A1 -----     -----
C2-A1      -----     ---

This matrix indicates the compatibility of each of the patterns (P1-P5)
with several different classes of cookie/IP address combination that we
might want to detect.

                                     patterns
                               P1  P2  P3  P4  P5
profiles                      --------------------
multiple users per IP        | -   +   -   +   +
multiple users per cookie    | -   -   -   -   -
multiple IPs per user        | +   +   -   -   -
multiple cookies per user    | -   -   +   -   +
cookie deletion              | -   -   +   -   -
"permanent" IP change        | -   +   -   -   -

Note that none of these patterns gives any indication for the “multiple
users per cookie” profile. To assess if there is more than one
user/cookie, you might want to look at the context in which you’re
observing the cookie. Consider attributes like (timezone corrected)
time-of-day, day-of-week, type of content being viewed.

I’m an old hand with SGE and a more user of Hadoop / Pig.  Good to see that there is interest in making these technologies interoperate.

LATIMER the world champion of magic www.latimeronline.com

Man, I really want a Magic Castle invite…

Hopefully the keywords I’m using here will help the next poor soul who has to learn this part of the Mozilla API. I needed to find where in the window some elements were. This information is not contained in the usual DOM referred to via the document variable. There are actually two parallel DOMs: the Content DOM (the usual one) and the Layout DOM. The Layout DOM’s structure contains all the elements in the Content DOM, but has positional information available to you as BoxObject (Firefox 2.*) or BoundingClientRect (Firefox 3.*) objects. Read the XUL Box Object Tutorial

Here’s how you get the boxes. For the sake of example, we’ll refer to the first

Firefox 2.*:

tab = document.getElementsByTagName("table").item(0);
tabBox = document.getBoxObjectFor(tab);
//tabBox.x
//tabBox.y
//tabBox.width
//tabBox.height

the x, y attributes give the location of the upper-left corner of the element, relative to the browser window. width, height tell you the size of the element.

Firefox 3.*:

tab = document.getElementsByTagName("table").item(0);
tabBox = document.getBoundingClientRect(tab);
//tabBox.left
//tabBox.top
//tabBox.width
//tabBox.height

The left, top attributes are equivalent to the Firefox 2.* x, y attributes, and width, height have identical meaning.

These docs helped me piece this together:
http://developer.mozilla.org/en/docs/DOM:element.getBoundingClientRect
http://developer.mozilla.org/en/docs/XUL_Tutorial:Box_Objects

Some good stuff on MouseEvents I’m stashing here for my own reference:
http://developer.mozilla.org/samples/domref/dispatchEvent.html
http://blog.stchur.com/blogcode/event-rerouting/

Image borrowed from Mozilla’s Layout Engine by L. David Baron