♥data♥

I’m in the process of setting up an analytic workflow at BiggerBoat. It’s looking like the main theme in data structures around here will be the sparse matrix. So I’ve been playing with opensource technologies for sparse matrices. Apache Hadoop’s HBase is looking like a good choice for now, maybe Hive later.

Right now I’m getting familiar with the former. As part of this, I’m improving the docs on the wiki to make them more user- (as opposed to core developer-) friendly. My documentation goal right now is to add some data transformation example code. There are already lots of hadoop examples for doing text -&gt text mapping, e.g. grep, cat, etc. For HBase not so much. I.e.

• text to text (done, many examples
• flatfile to HBase table (Bulk loader in the HBase wiki, I haven’t tried it yet)
• HBase table to flatfile
• HBase table to HBase table

I’ll be adding updated, complete, and simple code for the latter two (three?) in the next few days to the HBase/MapReduce page.

I’ve been thinking about how to track unique users today. These are my so-far-unorganized thoughts. Please comment!

You can’t track users by cookie alone for a couple of reasons: 1) they might use multiple computers, 2) they might delete cookies, 3) multiple users might share the same computer (same account=same cookie)

You also can’t track users by IP address alone for some more reasons: 1) they might be using a mobile device or portable computer that moves from IP to IP, 2) there could be multiple machines passing through a single gateway IP (i.e. LAN NAT).

However, if you combine the cookie/IP information together, you can start to address some of these issues. Let’s assume you have some webserver logs that minimally contain <IP address "A">, <cookie ID "C">, <timestamp T> triples.

            === time ===>
PATTERN 1:
C1-A1 ---      ---
C1-A2    ---         ---
C1-A3       ---   ---

PATTERN 2:
C1-A1 ------
C2-A1       ------
C1-A2             ------

PATTERN 3:
C1-A1 -----!
C2-A1       -----!
C3-A1             ------

PATTERN 4:
C1-A1 ------      ------
C2-A1     ----------

PATTERN 5:
C1-A1 -----     -----
C2-A1      -----     ---

This matrix indicates the compatibility of each of the patterns (P1-P5)
with several different classes of cookie/IP address combination that we
might want to detect.

                                     patterns
                               P1  P2  P3  P4  P5
profiles                      --------------------
multiple users per IP        | -   +   -   +   +
multiple users per cookie    | -   -   -   -   -
multiple IPs per user        | +   +   -   -   -
multiple cookies per user    | -   -   +   -   +
cookie deletion              | -   -   +   -   -
"permanent" IP change        | -   +   -   -   -

Note that none of these patterns gives any indication for the “multiple
users per cookie” profile. To assess if there is more than one
user/cookie, you might want to look at the context in which you’re
observing the cookie. Consider attributes like (timezone corrected)
time-of-day, day-of-week, type of content being viewed.

I’m an old hand with SGE and a more user of Hadoop / Pig.  Good to see that there is interest in making these technologies interoperate.

I’m frequently monitoring webservers, cache servers, database servers, etc by tailing their log files, e.g.

tail -f /etc/httpd/logs/access_log

I like the –color option provided by grep, but found it to be too limited (only one allowed, no wildcard support). After a bit of searching to see if a tool existed for doing arbitrary colorizing, I found
acoc, the Arbitrary Command Output Colourer.

…which almost did what I needed, but couldn’t read from a pipe. So I wrote pcoc, the Piped Command Output Colorizer. I’m only publishing this because I’ve been using it for about 1 1/2 years, and still find it useful.

Source code at the end of this post. Here’s an example that highlights iPhone/iPod user agents and requests with a 500/400/404 HTTP response:

tail -f ./logs/access_log | pcoc -f '(iPod)=bold cyan' -f '(iPhone)=bold magenta' -f '\b(500|404|400)\b=red on_black'

Sorry, no screenshots :(.

pcoc source:

#!/usr/bin/perl
use strict;
use Getopt::Long;
use Term::ANSIColor qw(colored);
$|++;
 
my %format = ();
GetOptions( "format|f=s" => \%format);
 
if ( ! keys %format ) {
  print <<"EOF";
Synopsis:
        pcoc - Piped Command Output Colorizer.  Inspired by acoc.
 
Usage:
 
        $0 -f '<regex1>=<color1>' -f '<regex2>=<color2>'
 
$0 reads from a pipe and colorizes each line based on format (-f) parameters.
 
Arguments:
 
-f '<regex>=<color>'  Required, multiple values okay. 
 
        <regex>: A regular expression from which \$1 will be colorized
 
        <color>: One or more colorization keywords, see perldoc
        Term::ANSIColor, but briefly they are:
 
        boldness:
                bold
        foreground:
                red yellow green blue magenta cyan black white
        background:
                on_red on_yellow on_green on_blue on_magenta on_cyan
                on_black on_white
 
Examples:
 
        #highlight the account's shell in bold green
        cat /etc/passwd | $0 -f '.+:([^:]+)\$=bold green'
 
        #... and the username in red with black background
        cat /etc/passwd | $0 -f '([^:]+)=red on_black' -f '.+:([^:]+)\$=bold green'
 
Copyright/License:
 
        Allen Day <allenday\@ucla.edu>, licensed under GPL 2006-2008
 
EOF
  exit(1);
}
 
while ( my $line = <> ) {
  chomp( $line );
  foreach my $f ( keys %format ) {
    my @c = split ',', $format{ $f };
 
    if ( $line =~ qr/$f/ ) {
      while ( my ( $s, $t ) = $f =~ m/^(.*?)\(+(.+?)\)+/ ) {
        my $c = pop @c || last;
        $line =~ s/($s)($t)/$1.colored($2,$c)/e;
        $f =~ s/^(.*?)\((.+?)\)/$1$2/;
      }
    }
  }
  print "$line\n";
}

I was preparing a report on iPhone locales from some web server logs, and noticed a few oddities. Some of the hits appear to be coming from the new 3G iPhone 2.0, check out the User-Agent strings:

# observed from 1 metrocast.net (NY) IP
Mozilla/5.0 (iPod; U; iPhone OS 2_0 like Mac OS X; en-us) AppleWebKit/525.17 (KHTML, like Gecko) Version/3.1 Mobile/5A240d Safari/5525.7
# observed from 1 optonline.net (NY) IP
Mozilla/5.0 (iPhone Simulator; U; CPU iPhone OS 2_0 like Mac OS X; en-us) AppleWebKit/525.18.1 (KHTML, like Gecko) Version/3.1.1 Mobile/5A345 Safari/525.20

The former is confirmed to be an iPhone 2.0 User-Agent string on the MacRumors Forums.

Other unusual/rare iPhone/iPod User-Agent/UA strings:

Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/4A102 Safari/419 (United States)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543 Safari/419.3
Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Cydia/1.0.2460-59

Update July 11. iPhone 2.0 is out, and the UA is (note the Safari revision increment from the earlier pre-launch UA):

Mozilla/5.0 (iPhone; U; CPU iPhone OS 2_0 like Mac OS X; en-us) AppleWebKit/525.18.1 (KHTML, like Gecko) Version/3.1.1 Mobile/5A345 Safari/525.20

While the iPod with iPhone 2.0 software update UA is:

Mozilla/5.0 (iPod; U; CPU iPhone OS 2_0 like Mac OS X; en-us) AppleWebKit/525.18.1 (KHTML, like Gecko) Version/3.1.1 Mobile/5A347 Safari/525.20

Note that both the upgraded iPod and the iPhone UAs both contain the string “iPhone” in them, so you may need to update your device-detection logic if you care about discriminating between iPods and iPhones. Not yet clear to me how to discriminate between an upgraded iPhone 1.0 w/ 2.0 software, and a bona fide 3G iPhone 2.0. Will post more when I figure this out.

Know anything else about these? Leave me a comment!

Setting up Taste v1.7.2 on a CentOS 4 x86_64 box.

Taste has merged with Mahout now, but I still want to do this standalone b/c I’m having trouble getting the JUnit tests to pass for Mahout. With that out of the way…

These are the shell commands I assembled after following the Taste Demo guide.

#make sure you have ant, and the JDK.  I don't recommend the CentOS stock, get them from Sun/Apache
#download necessary .jar files, sources, data files.  unpack/move them to correct locations.
wget http://internap.dl.sourceforge.net/sourceforge/taste/taste-1.7.2.zip
wget http://internap.dl.sourceforge.net/sourceforge/proguard/proguard4.2.zip
wget http://www.grouplens.org/system/files/million-ml-data.tar__0.gz
wget http://www.hightechimpact.com/Apache/tomcat/tomcat-5/v5.5.26/bin/apache-tomcat-5.5.26.tar.gz
unzip taste-1.7.2.zip
unzip proguard4.2.zip
tar -xvzf million-ml-data.tar__0.gz
tar -xvzf apache-tomcat-5.5.26.tar.gz
cp proguard4.2/lib/proguard.jar lib/
mv [mr]*.dat src/example/com/planetj/taste/example/grouplens/
#start up tomcat on port 8080 (default)
JAVA_OPTS="-server -da -dsa -Xms1024m -Xmx1024m" JAVA_HOME=/usr/java/jdk1.6.0_02 sh apache-tomcat-5.5.26/bin/startup.sh
#build taste.war, and inject it into tomcat
JDK_HOME=/usr/java/jdk1.6.0_02 JAVA_HOME=/usr/java/jdk1.6.0_02 ant build-grouplens-example
cp taste.war apache-tomcat-5.5.26/webapps/
#test the app.  may take a minute or two on the first query.
wget -O - -S 'http://localhost:8080/taste/RecommenderServlet?userID=1&debug=true'

Once you get that working, you can tweak the demo slightly to work on another data set. You just need to know the grouplens file format. ratings.dat is of the format:

UserID::MovieID::Rating::Timestamp

e.g.

1::1193::5::978300760

and movies.dat is of the format:

MovieID::Title::Genres

e.g.

1::Toy Story (1995)::Animation|Children's|Comedy

I wrote a script, let’s call it load_taste.pl, that can generate new movies.dat and ratings.dat files from an alternate data source. If I make these new files, I can drop them in place of the grouplens data, rebuild the .war files, and make recommendations on this other data set. Here’s how to do it:

#generate ratings.dat and movies.dat.  move them to replace the grouplens data files.
perl ./load_taste.pl
mv [mr]*.dat src/example/com/planetj/taste/example/grouplens/
#get rid of stale .war and .jar files
rm taste.war grouplens.jar
#build the "quick" version of the example.  see below for build.xml patch
JDK_HOME=/usr/java/jdk1.6.0_02 JAVA_HOME=/usr/java/jdk1.6.0_02 ant build-grouplens-example-quick
#inject the re-built .war file into tomcat.
cp taste.war apache-tomcat-5.5.26/webapps/
#get rid of stale tomcat caches
rm -rf apache-tomcat-5.5.26/webapps/taste apache-tomcat-5.5.26/temp/taste.*.txt

Note that I’ve defined a new ant build target called “build-grouplens-example-quick”. The purpose of this is that we only want to rebuild grouplens.jar and taste.war, not reoptimize/reverify/rebuild taste.jar, etc. The “build-grouplens-example” target takes ~55 seconds to complete on my machine, whereas the “build-grouplens-example-quick” target takes ~2 seconds. Here’s a diff to the original build.xml file:

--- /tmp/build.xml      2008-03-21 21:18:20.000000000 -0700
+++ ./build.xml 2008-06-30 11:46:18.000000000 -0700
@@ -161,6 +161,58 @@
      <delete file="${my-web.xml}"/>
   </target>
 
+  <target depends="" name="build-taste-server-quick" description="Builds deployable web-based Taste server">
+     <fail unless="my-recommender.jar" message="Please set -Dmy-recommender.jar=XXX"/>
+     <fail unless="my-recommender-class" message="Please set -Dmy-recommender-class=XXX"/>
+     <tempfile property="my-web.xml"/>
+     <copy file="src/main/com/planetj/taste/web/web.xml" tofile="${my-web.xml}">
+       <filterset>
+               <filter token="RECOMMENDER_CLASS" value="${my-recommender-class}"/>
+       </filterset>
+     </copy>
+     <war destfile="${release-war}" webxml="${my-web.xml}">
+       <lib dir=".">
+               <include name="${release-jar}"/>
+               <include name="${my-recommender.jar}"/>
+       </lib>
+       <lib dir="lib/axis"/>
+       <classes dir="build">
+               <include name="com/planetj/taste/web/**"/>
+       </classes>
+       <fileset dir="src/main/com/planetj/taste/web">
+               <include name="RecommenderService.jws"/>
+       </fileset>
+     </war>
+     <delete file="${my-web.xml}"/>
+  </target>
+  <target depends="" name="build-grouplens-example-quick" description="Builds deployable GroupLens example">
+     <javac source="1.5"
+            target="1.5"
+            deprecation="true"
+          debug="true"
+          optimize="false"
+            destdir="build"
+            srcdir="src/example">
+       <compilerarg value="-Xlint:all"/>
+       <classpath>
+               <pathelement location="${release-jar}"/>
+               <pathelement location="${annotations.jar}"/>
+       </classpath>
+     </javac>
+     <jar jarfile="grouplens.jar">
+       <fileset dir="src/example">
+               <include name="com/planetj/taste/example/grouplens/ratings.dat"/>
+               <include name="com/planetj/taste/example/grouplens/movies.dat"/>
+       </fileset>
+       <fileset dir="build">
+               <include name="com/planetj/taste/example/grouplens/**"/>
+       </fileset>
+     </jar>
+     <property name="my-recommender.jar" value="grouplens.jar"/>
+     <property name="my-recommender-class" value="com.planetj.taste.example.grouplens.GroupLensRecommender"/>
+     <antcall target="build-taste-server-quick"/>
+  </target>
+
   <target depends="build,optimize" name="build-grouplens-example" description="Builds deployable GroupLens example">
      <javac source="1.5"
             target="1.5"

Overview

This documents my experience setting up hadoop on three 2-cpu hyperthreaded Centos 5 x86 boxes.

The machines being used are called:

• hadoop-0-0
• hadoop-0-1
• hadoop-0-2

System Setup

Unless otherwise stated, each task was executed on all three machines.

Set up hadoop user

I added a “hadoop” user up on each node, and untarred the hadoop source directly in to the /home/hadoop directory.  So there was a /home/hadoop/conf, /home/hadoop/bin, etc.

useradd hadoop
passwd hadoop
cd /tmp
wget http://apache.siamwebhosting.com/hadoop/core/hadoop-0.17.0/hadoop-0.17.0.tar.gz
su hadoop
cd ~
tar -xvzf /tmp/hadoop*gz
mv hadoop*/* ./
rmdir hadoop*
exit

Next, I set up passphrase-less ssh. On hadoop-0-0, I set up a DSA key:

su hadoop
ssh-keygen -t dsa
#[...use blank passphrase...]
scp ~/.ssh/id_dsa.pub hadoop-0-1:/tmp/
scp ~/.ssh/id_dsa.pub hadoop-0-2:/tmp/

Then, on hadoop-0-1 and hadoop-0-2:

su hadoop
mkdir ~/.ssh
chmod 700 ~/.ssh
cp /tmp/id_dsa.pub ~/.ssh/authorized_keys
chmod 644 ~/.ssh/authorized_keys

This allows login from hadoop-0-0 to both hadoop-0-1 and hadoop-0-2 without typing a password. It’s necessary.

Install prerequisite software

ssh and rsync are required. You can install them like so:

yum -y install ssh rsync

I installed Java 6 from Sun (it’s important to mention here that the CentOS yum/JPackage RPMs for gcc-java, etc did not work for Mahout, so it had to be the Sun Java).

rpm -Uvh j*rpm

Setup system services

I turned off iptables and ip6tables, and disabled them on startup. You could configure them, but I just turned them off. They get in the way of the nodes communicating.

/etc/init.d/iptables stop
/etc/init.d/ip6tables stop
/usr/sbin/ntsysv
#[...]

Then, I edited the hosts file. I actually did this later in the process after a bunch of debugging, but it makes sense to do it here. You need to make sure that the name by which you refer to a host is not associated with the loopback IP address (127.0.0.1). So your /etc/hosts file should look something like this on hadoop-0-0:

127.0.0.1    localhost localhost.localdomain
::1    localhost6.localdomain6 localhost6
10.0.0.1    hadoop-0-0

where the 10.0.0.1 entry is optional if you have some other way to resolve it, e.g. DNS. The key point is that the hadoop-0-0 name not be associated with 127.0.0.1.

Configuring hadoop in standalone mode

I tried using the Hadoop Quick Start guide first. It was trivial to get working in standalone mode. The documentation is sufficient, I won’t discuss it further.

Configuring hadoop in cluster mode

I tried using the Hadoop Cluster Setup guide next. It describes there are four types of hadoop services, split across three types of machines:

• NameNode machine, runs NameNode service
• JobTracker machine, runs JobTracker service
• Slave machine, runs TaskTracker and DataNode services

I wanted to have 2 slave nodes.  So I decided to split it up like:

• hadoop-0-0 - NameNode and JobTracker
• hadoop-0-1 - DataNode and TaskTracker
• hadoop-0-2 - DataNode and TaskTracker

Here’s what my config files look like on all three machines:

/home/hadoop/conf/hadoop-env.sh: added one line.

export JAVA_HOME=/usr/java/jdk1.6.0_06

/home/hadoop/conf/masters:

hadoop-0-0

/home/hadoop/conf/slaves:

hadoop-0-1
hadoop-0-2

/home/hadoop/conf/hadoop-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop-0-0:9000/</value>
<final>true</final>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://hadoop-0-0:9001/</value>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/data</value>
<final>true</final>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/hadoop/mapred/system</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/hadoop/mapred/local</value>
<final>true</final>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>2</value>
<final>true</final>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>2</value>
<final>true</final>
</property>
</configuration>

This defines some services on hadoop-0-0 ports 9000 and 9001, and says that the slaves should get 2 map and 2 reduce tasks each (one for each of my 4 CPUs [remember, I'm hyperthreading]).

Make sure the configuration files are the same on all machines.

Starting hadoop services

From here I pretty much followed the tutorial.

First, set up the filesystem for the NameNode service on the NameNode (hadoop-0-0 for me):

/home/hadoop/bin/hadoop namenode -format

Next start up HDFS, the distributed filesystem:

/home/hadoop/bin/start-dfs.sh

Then start up the JobTracker service on the JobTracker node (hadoop-0-0 for me):

/home/hadoop/bin/start-mapred.sh

This also starts up the TaskTracker and DataNode services on all the nodes specified in the /home/hadoop/conf/slaves file.

At this point, doing a:

ps x

as the hadoop user on the hadoop-0-{0,1,2} machines will show some java daemons running. If you run into trouble, there is diagnostic information in the /home/hadoop/logs/*.log files. These were indispensible in debugging my setup.

Testing hadoop setup

Testing HDFS

To test HDFS, the distributed file system. On hadoop-0-0:

[hadoop@hadoop-0-0 ~]$ ./bin/hadoop dfs -ls /
Found 1 items
/home   <dir>           2008-06-20 18:58        rwxr-xr-x       hadoop  supergroup
[hadoop@hadoop-0-0 ~]$ ./bin/hadoop dfs -touchz /foo
[hadoop@hadoop-0-0 ~]$ ./bin/hadoop dfs -ls /
Found 2 items
/foo    <r 3>   0       2008-06-20 23:49        rw-r–r–       hadoop  supergroup
/home   <dir>           2008-06-20 18:58        rwxr-xr-x       hadoop  supergroup
[hadoop@hadoop-0-0 ~]$ ./bin/hadoop dfs -rm /foo
Deleted /foo
[hadoop@hadoop-0-0 ~]$

Testing MapReduce

Haven’t done this yet, except in standalone mode.

Introduction

I’ve been learning about some of the techniques used by the so-called “Black Hat SEO” community for boosting their rankings in search engine results. Intriguing stuff. I’m by no means an expert in this area, but the theory underlying building black-hat pages and networks sure looks like it has a lot to do with my primary areas of interest.

Generating Unique Content

One “Black Hat SEO” application area is automatically generating HTML pages to improve search engine rankings. This technique uses a Markov process to generate text. The idea is to build one or more web pages that contain the keywords the SEO is targeting. The method basically works like this:

1. Assemble a corpus of text to train the model. For example, Project Gutenberg
2. Build an order-N (typically N=2) Markov model that captures the state changes in the corpus
3. Generate text from the model, periodically throwing in some keywords
4. Link the generated page to some other page to which you want to send traffic
5. Repeat again from Step 1

One problem with this approach — aside from the fact that the keywords don’t really fitting in with the flow of the model — is that the model is trained on inappropriate text. For instance, suppose you were trying to optimize for keywords:

• keywords
• statistics
• Search engine optimization
• SEO
• Automatic content generation
• Automatic content extraction
• HTML content extraction
• Markov Model

… then you probably wouldn’t want to train your model on, say, Jane Austen’s Pride and Prejudice.

Improve Generated Text: Use Niche Corpora

A better thing to do would be to find some nice web pages containing keywords, statistics, seo, Markov model, and so on. That way you’ll pick up related keywords that you didn’t initially think of (or weren’t suggested by your keyword expansion tool), too.

But let’s face it. The corpora are going to be in HTML format. So the question now becomes, How do I automate the transformation of HTML into plain text for input to the model? A few strawman ideas, followed by my remarks:

• Get an HTML document, and remove all <element/>s. Won’t work very well. You end up training on page navigation, footers, headers, etc.
• Build a site- or software-specific parser (e.g. for Wikipedia, or for Wordpress) to extract the main content. Scalability and maintenance nightmare. This is not generalizable to general text extraction. You’ll be constantly fixing broken parsers, too.
• Devise a scoring system that can identify the main content of the page. Exactly!

I did find some methods for scoring page fragments, such as the Perl modules HTML::Content::Extractor and HTML::Extract, and another method described by Nooks. There are also a few intersting ideas in Gupta’s WWW2003 paper.

None of that Perl code linked above actually works, but Nooks and Jean Tavernier generally had the right idea. Basically, they look “down” the DOM to find the sub-DOM with the highest text/tag ratio.

The main problem with this approach is that it biases for DOM leaves, or “twigs” that are very close to leaves. You end up having to write special rules for accomodating the idiosyncrosies of each particular page dealt with, and it basically turns back into an HTML parsing exercise.

The other problem, and possibly more significant one from a statistician’s point of view, is that the ratio is not a well-understood metric for making decisions about what constitutes a “good” versus a “bad” sub-document. It would be better to have a p-value…

Balls and Urns

Fortunately, Fisher’s exact test can be applied to this problem. Here’s how you can apply it, explanation follows. First, let’s define some variables:

• X: the total number of words in the whole document.
• x: the number of words in a sub-document.
• Y: the total number of <element/>s in the whole document.
• y: the number of <element/>s in a sub-document.

Then, we perform the following algorithm to identify the single best sub-document:

tree; //the HTML tree's root node
minP = 1; //minimum p-value observed in the document
subD = ""; //sub-document corresponding to minimum p-value
X = calculatex(tree);
Y = calculatey(tree);
look(tree);
function look (node) {
  x = calculatex(node);
  y = calculatey(node);
  p = calculateHyperG(x,y,X,Y);
  if ( p < minP ) {
    minP = p;
    subD = node;
  }
  C = children(node);
  foreach (c in C) {
    look(c);
  }
}

Balls and Urns, Explained

The pseudocode above is examining each sub-document of the HTML document in turn and identifying the one with the smallest p-value. The p-value is calculated using the hypergeometirc distribution, where we consider that a sub-document has x words and y HTML <element/>s. This, in the context of the total document having X words and Y HTML <element/>s. It’s better than a simple ratio calculation because it does not bias for the tree’s leaves. That is, the p-value does not consider only the size of x+y.

Caveats

Bear in mind that testing so many sub-documents, especially for very large HTML documents, warrants so-called “multiple hypothesis testing correction“, such as a Bonferroni correction. It’s outside the scope of this article.

Also, the tests performed are not entirely independent. That is, if node B is a child of node A then B will have some effect on A when calculating A’s p-value and must be factored out. This is also a well-defined problem but is, alas, also outside the scope of this article. Do your homework! Hint: learn about the Gene Ontology.

Conclusion

Fine and dandy, but does it work? My conclusion: seems to work. Here’s a CGI script demonstrating the hypergeometric content extraction technique on CNN.com. It reports a text snippet at the beginning and end of the single “best” sub-document and the corresponding (uncorrected) p-value. Twiddle the u parameter to test on a page of your choice. Some pages may block the user-agent I’m using…

There is also the issue of what to consider an element and what not to… or maybe even element weighting. For instance, maybe <p/> and <i/> elements shouldn’t be penalized because they’re commonly associated with text, but <script/> elements are heavily penalized.

I’ve had a need over the last week to work with some sparse matrix data in R. This was a totally new problem for me, and I can now sympathize with anyone else having to do this and will document the experience.

It seems that the de-facto standard for moving sparse matrices is around is to use the Harwell-Boeing file format, aka “harbo”. It’s a horrible and largely undocumented fixed-width (think Fortran) file format. The best documentation I could find was in source code here, although you may be able to piece more of it together with Koders. R does include a harbo reader as part of the SparseM package.

Given that I’m more comfortable working in Perl than in R or Fortran, I decided to have a look on CPAN to see what was available. As it turns out, there is a package called Text::SenseClusters from Ted Pedersen that ships with a nifty program, mat2harbo.pl. I found the preferred sparse matrix “mat” format used by Text::SenseClusters to be more reasonable than harbo. Here’s an example.

5 5 15
2 9 4 9
1 6 2 5 3 7 4 8 5 6
1 4 2 5
1 7 2 6 3 7
1 9 2 8 3 9

. There is a header line “

5 5 15

” that defines the matrix rows, columns, and number of non null fields. Each subsequent (possibly blank) line gives index/value pairs for the non-null positions in that row. Easy!

At this point I was formulating a plan to:

1. use my matrix writer to write in “mat” format to file1.mat.
2. convert file1.mat to file2.harbo using mat2harbo.pl from Text::SenseClusters.
3. import file2.harbo into R using the read.matrix.hb() function in the SparseM package.
4. convert the SparseM matrix to an R graph (graph package).
5. get back to my original problem… analyzing the matrix in R with Boost via the RBGL package.

Well, it wasn’t that easy.

Step 1 went okay. Step 2 had problems with null columns, and had some glitches in the output format. Some of these glitches were easy to fix (e.g. matrix definition of “rra” to “RRA”), but others were very difficult due to the fact that mat2harbo.pl didn’t provide “full” harbo support, and the SparseM reader needed some of the file format features that weren’t supported.

So I wrote my own “mat” file -> R matrix.rsc object constructor myself. Here it is:

read.matrix.pair = function (file,debug=FALSE) {
mat.lines = readLines(file);
header = mat.lines[1];
F = strsplit(header,' ')[[1]];
nrow = as.integer(F[1]);
ncol = as.integer(F[2]);
nelem = as.integer(F[3]);
 
ja = vector("list",nrow);
ra = vector("list",nrow);
ia = vector("list",nrow);
ia[[1]] = c(1);
 
for ( i in 2:(nrow+1) ) { #nrow
mat.line = strsplit(mat.lines[i],' ')[[1]];
if ( length(mat.line) > 0 ) {
if ( debug ) print(paste('non-empty row',i-1));
ja[[i-1]] = mat.line[  seq(1,length(mat.line),by=2)];
ra[[i-1]] = mat.line[1+seq(1,length(mat.line),by=2)];
ia[[i]] = ia[[i-1]] + length(mat.line)/2;
if ( debug ) print(paste('  pos:',ja[[i-1]]));
if ( debug ) print(paste('  dat:',ra[[i-1]]));
} else {
if ( debug ) print(paste('    empty row',i-1));
ia[[i]] = ia[[i-1]];
}
}
ans.ja = as.integer(unlist(ja));
ans.ra = as.integer(unlist(ra));
ans.ia = as.integer(unlist(ia));
dimension = as.integer(c(nrow,ncol));
 
if ( debug ) {
print(paste('nrow',nrow));
print(paste('ncol',ncol));
print('ra');print(ans.ra);
print('ja');print(ans.ja);
print('ia');print(ans.ia);
}
rd.o = new("matrix.csr", ra = ans.ra, ja = ans.ja, ia = ans.ia, dimension = dimension)
}

This let me just read the “mat” file directly into R. After that, the conversion to a graph object seems to work okay. I say seems to because I’m still waiting for the SparseM -> graph conversion routine to finish. It’s a 50K x 50K matrix with about 2 million edges, so it’s taking a little while…

Took about as long to convert as it took me to post this. Everything is fine. Now I get back to doing all-by-all Dijkstra on the graph, or at least find a reasonably fast way to allow for one-off queries.