Allen Day's Weblog

pcoc - Piped Command Output Colorizer

allenday — Sun, 06 Jul 2008 01:37:09 +0000

I’m frequently monitoring webservers, cache servers, database servers, etc by tailing their log files, e.g.

tail -f /etc/httpd/logs/access_log

I like the –color option provided by grep, but found it to be too limited (only one allowed, no wildcard support). After a bit of searching to see if a tool existed for doing arbitrary colorizing, I found
acoc, the Arbitrary Command Output Colourer.

…which almost did what I needed, but couldn’t read from a pipe. So I wrote pcoc, the Piped Command Output Colorizer. I’m only publishing this because I’ve been using it for about 1 1/2 years, and still find it useful.

Source code at the end of this post. Here’s an example that highlights iPhone/iPod user agents and requests with a 500/400/404 HTTP response:

tail -f ./logs/access_log | pcoc -f '(iPod)=bold cyan' -f '(iPhone)=bold magenta' -f '\b(500|404|400)\b=red on_black'

Sorry, no screenshots :(.

pcoc source:

#!/usr/bin/perl
use strict;
use Getopt::Long;
use Term::ANSIColor qw(colored);
$|++;
 
my %format = ();
GetOptions( "format|f=s" => \%format);
 
if ( ! keys %format ) {
  print <<"EOF";
Synopsis:
        pcoc - Piped Command Output Colorizer.  Inspired by acoc.
 
Usage:
 
        $0 -f '=' -f '='
 
$0 reads from a pipe and colorizes each line based on format (-f) parameters.
 
Arguments:
 
-f '='  Required, multiple values okay. 
 
        : A regular expression from which \$1 will be colorized
 
        : One or more colorization keywords, see perldoc
        Term::ANSIColor, but briefly they are:
 
        boldness:
                bold
        foreground:
                red yellow green blue magenta cyan black white
        background:
                on_red on_yellow on_green on_blue on_magenta on_cyan
                on_black on_white
 
Examples:
 
        #highlight the account's shell in bold green
        cat /etc/passwd | $0 -f '.+:([^:]+)\$=bold green'
 
        #... and the username in red with black background
        cat /etc/passwd | $0 -f '([^:]+)=red on_black' -f '.+:([^:]+)\$=bold green'
 
Copyright/License:
 
        Allen Day <allenday\@ucla.edu>, licensed under GPL 2006-2008
 
EOF
  exit(1);
}
 
while ( my $line = <> ) {
  chomp( $line );
  foreach my $f ( keys %format ) {
    my @c = split ',', $format{ $f };
 
    if ( $line =~ qr/$f/ ) {
      while ( my ( $s, $t ) = $f =~ m/^(.*?)\(+(.+?)\)+/ ) {
        my $c = pop @c || last;
        $line =~ s/($s)($t)/$1.colored($2,$c)/e;
        $f =~ s/^(.*?)\((.+?)\)/$1$2/;
      }
    }
  }
  print "$line\n";
}

iPhone 2.0 User-Agent string, other iPhone/iPod data

allenday — Sat, 05 Jul 2008 01:53:10 +0000

I was preparing a report on iPhone locales from some web server logs, and noticed a few oddities. Some of the hits appear to be coming from the new 3G iPhone 2.0, check out the User-Agent strings:

# observed from 1 metrocast.net (NY) IP
Mozilla/5.0 (iPod; U; iPhone OS 2_0 like Mac OS X; en-us) AppleWebKit/525.17 (KHTML, like Gecko) Version/3.1 Mobile/5A240d Safari/5525.7
# observed from 1 optonline.net (NY) IP
Mozilla/5.0 (iPhone Simulator; U; CPU iPhone OS 2_0 like Mac OS X; en-us) AppleWebKit/525.18.1 (KHTML, like Gecko) Version/3.1.1 Mobile/5A345 Safari/525.20

The former is confirmed to be an iPhone 2.0 User-Agent string on the MacRumors Forums.

Other unusual/rare iPhone/iPod User-Agent/UA strings:

Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/4A102 Safari/419 (United States)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543 Safari/419.3
Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Cydia/1.0.2460-59

Know anything about these? Leave me a comment!

Notes on setting up Taste

allenday — Mon, 30 Jun 2008 10:49:55 +0000

Setting up Taste v1.7.2 on a CentOS 4 x86_64 box.

Taste has merged with Mahout now, but I still want to do this standalone b/c I’m having trouble getting the JUnit tests to pass for Mahout. With that out of the way…

These are the shell commands I assembled after following the Taste Demo guide.

#make sure you have ant, and the JDK.  I don't recommend the CentOS stock, get them from Sun/Apache
#download necessary .jar files, sources, data files.  unpack/move them to correct locations.
wget http://internap.dl.sourceforge.net/sourceforge/taste/taste-1.7.2.zip
wget http://internap.dl.sourceforge.net/sourceforge/proguard/proguard4.2.zip
wget http://www.grouplens.org/system/files/million-ml-data.tar__0.gz
wget http://www.hightechimpact.com/Apache/tomcat/tomcat-5/v5.5.26/bin/apache-tomcat-5.5.26.tar.gz
unzip taste-1.7.2.zip
unzip proguard4.2.zip
tar -xvzf million-ml-data.tar__0.gz
tar -xvzf apache-tomcat-5.5.26.tar.gz
cp proguard4.2/lib/proguard.jar lib/
mv [mr]*.dat src/example/com/planetj/taste/example/grouplens/
#start up tomcat on port 8080 (default)
JAVA_OPTS="-server -da -dsa -Xms1024m -Xmx1024m" JAVA_HOME=/usr/java/jdk1.6.0_02 sh apache-tomcat-5.5.26/bin/startup.sh
#build taste.war, and inject it into tomcat
JDK_HOME=/usr/java/jdk1.6.0_02 JAVA_HOME=/usr/java/jdk1.6.0_02 ant build-grouplens-example
cp taste.war apache-tomcat-5.5.26/webapps/
#test the app.  may take a minute or two on the first query.
wget -O - -S 'http://localhost:8080/taste/RecommenderServlet?userID=1&debug=true'

Once you get that working, you can tweak the demo slightly to work on another data set. You just need to know the grouplens file format. ratings.dat is of the format:

UserID::MovieID::Rating::Timestamp

e.g.

1::1193::5::978300760

and movies.dat is of the format:

MovieID::Title::Genres

e.g.

1::Toy Story (1995)::Animation|Children's|Comedy

I wrote a script, let’s call it load_taste.pl, that can generate new movies.dat and ratings.dat files from an alternate data source. If I make these new files, I can drop them in place of the grouplens data, rebuild the .war files, and make recommendations on this other data set. Here’s how to do it:

#generate ratings.dat and movies.dat.  move them to replace the grouplens data files.
perl ./load_taste.pl
mv [mr]*.dat src/example/com/planetj/taste/example/grouplens/
#get rid of stale .war and .jar files
rm taste.war grouplens.jar
#build the "quick" version of the example.  see below for build.xml patch
JDK_HOME=/usr/java/jdk1.6.0_02 JAVA_HOME=/usr/java/jdk1.6.0_02 ant build-grouplens-example-quick
#inject the re-built .war file into tomcat.
cp taste.war apache-tomcat-5.5.26/webapps/
#get rid of stale tomcat caches
rm -rf apache-tomcat-5.5.26/webapps/taste apache-tomcat-5.5.26/temp/taste.*.txt

Note that I’ve defined a new ant build target called “build-grouplens-example-quick”. The purpose of this is that we only want to rebuild grouplens.jar and taste.war, not reoptimize/reverify/rebuild taste.jar, etc. The “build-grouplens-example” target takes ~55 seconds to complete on my machine, whereas the “build-grouplens-example-quick” target takes ~2 seconds. Here’s a diff to the original build.xml file:

--- /tmp/build.xml      2008-03-21 21:18:20.000000000 -0700
+++ ./build.xml 2008-06-30 11:46:18.000000000 -0700
@@ -161,6 +161,58 @@
      {my-web.xml}"/>
   
 
+  
+     
+     
+     
+     {my-web.xml}">
+       
+               {my-recommender-class}"/>
+       
+     
+     {release-war}" webxml="${my-web.xml}">
+       
+               {release-jar}"/>
+               {my-recommender.jar}"/>
+       
+       
+       
+               
+       
+       
+               
+       
+     
+     {my-web.xml}"/>
+  
+  
+     1.5"
+            target="1.5"
+            deprecation="true"
+          debug="true"
+          optimize="false"
+            destdir="build"
+            srcdir="src/example">
+       
+       
+               {release-jar}"/>
+               {annotations.jar}"/>
+       
+     
+     
+       
+               
+               
+       
+       
+               
+       
+     
+     
+     
+     
+  
+
   
      1.5"
             target="1.5"

Notes on setting up hadoop on CentOS 5 x86

allenday — Sat, 21 Jun 2008 06:53:54 +0000

Overview

This documents my experience setting up hadoop on three 2-cpu hyperthreaded Centos 5 x86 boxes.

The machines being used are called:

hadoop-0-0
hadoop-0-1
hadoop-0-2

System Setup

Unless otherwise stated, each task was executed on all three machines.

Set up hadoop user

I added a “hadoop” user up on each node, and untarred the hadoop source directly in to the /home/hadoop directory. So there was a /home/hadoop/conf, /home/hadoop/bin, etc.

useradd hadoop
passwd hadoop
cd /tmp
wget http://apache.siamwebhosting.com/hadoop/core/hadoop-0.17.0/hadoop-0.17.0.tar.gz
su hadoop
cd ~
tar -xvzf /tmp/hadoop*gz
mv hadoop*/* ./
rmdir hadoop*
exit

Next, I set up passphrase-less ssh. On hadoop-0-0, I set up a DSA key:

su hadoop
ssh-keygen -t dsa
#[...use blank passphrase...]
scp ~/.ssh/id_dsa.pub hadoop-0-1:/tmp/
scp ~/.ssh/id_dsa.pub hadoop-0-2:/tmp/

Then, on hadoop-0-1 and hadoop-0-2:

su hadoop
mkdir ~/.ssh
chmod 700 ~/.ssh
cp /tmp/id_dsa.pub ~/.ssh/authorized_keys
chmod 644 ~/.ssh/authorized_keys

This allows login from hadoop-0-0 to both hadoop-0-1 and hadoop-0-2 without typing a password. It’s necessary.

Install prerequisite software

ssh and rsync are required. You can install them like so:

yum -y install ssh rsync

I installed Java 6 from Sun (it’s important to mention here that the CentOS yum/JPackage RPMs for gcc-java, etc did not work for Mahout, so it had to be the Sun Java).

rpm -Uvh j*rpm

Setup system services

I turned off iptables and ip6tables, and disabled them on startup. You could configure them, but I just turned them off. They get in the way of the nodes communicating.

/etc/init.d/iptables stop
/etc/init.d/ip6tables stop
/usr/sbin/ntsysv
#[...]

Then, I edited the hosts file. I actually did this later in the process after a bunch of debugging, but it makes sense to do it here. You need to make sure that the name by which you refer to a host is not associated with the loopback IP address (127.0.0.1). So your /etc/hosts file should look something like this on hadoop-0-0:

127.0.0.1    localhost localhost.localdomain
::1    localhost6.localdomain6 localhost6
10.0.0.1    hadoop-0-0

where the 10.0.0.1 entry is optional if you have some other way to resolve it, e.g. DNS. The key point is that the hadoop-0-0 name not be associated with 127.0.0.1.

Configuring hadoop in standalone mode

I tried using the Hadoop Quick Start guide first. It was trivial to get working in standalone mode. The documentation is sufficient, I won’t discuss it further.

Configuring hadoop in cluster mode

I tried using the Hadoop Cluster Setup guide next. It describes there are four types of hadoop services, split across three types of machines:

NameNode machine, runs NameNode service
JobTracker machine, runs JobTracker service
Slave machine, runs TaskTracker and DataNode services

I wanted to have 2 slave nodes. So I decided to split it up like:

hadoop-0-0 - NameNode and JobTracker
hadoop-0-1 - DataNode and TaskTracker
hadoop-0-2 - DataNode and TaskTracker

Here’s what my config files look like on all three machines:

/home/hadoop/conf/hadoop-env.sh: added one line.

export JAVA_HOME=/usr/java/jdk1.6.0_06

/home/hadoop/conf/masters:

hadoop-0-0

/home/hadoop/conf/slaves:

hadoop-0-1
hadoop-0-2

/home/hadoop/conf/hadoop-site.xml:








fs.default.name
hdfs://hadoop-0-0:9000/
true


mapred.job.tracker
hdfs://hadoop-0-0:9001/
true


dfs.data.dir
/home/hadoop/data
true


mapred.system.dir
/home/hadoop/mapred/system
true


mapred.local.dir
/home/hadoop/mapred/local
true


mapred.tasktracker.map.tasks.maximum
2
true


mapred.tasktracker.reduce.tasks.maximum
2
true

This defines some services on hadoop-0-0 ports 9000 and 9001, and says that the slaves should get 2 map and 2 reduce tasks each (one for each of my 4 CPUs [remember, I'm hyperthreading]).

Make sure the configuration files are the same on all machines.

Starting hadoop services

From here I pretty much followed the tutorial.

First, set up the filesystem for the NameNode service on the NameNode (hadoop-0-0 for me):

/home/hadoop/bin/hadoop namenode -format

Next start up HDFS, the distributed filesystem:

/home/hadoop/bin/start-dfs.sh

Then start up the JobTracker service on the JobTracker node (hadoop-0-0 for me):

/home/hadoop/bin/start-mapred.sh

This also starts up the TaskTracker and DataNode services on all the nodes specified in the /home/hadoop/conf/slaves file.

At this point, doing a:

ps x

as the hadoop user on the hadoop-0-{0,1,2} machines will show some java daemons running. If you run into trouble, there is diagnostic information in the /home/hadoop/logs/*.log files. These were indispensible in debugging my setup.

Testing hadoop setup

Testing HDFS

To test HDFS, the distributed file system. On hadoop-0-0:

[hadoop@hadoop-0-0 ~]$ ./bin/hadoop dfs -ls /
Found 1 items
/home              2008-06-20 18:58        rwxr-xr-x       hadoop  supergroup
[hadoop@hadoop-0-0 ~]$ ./bin/hadoop dfs -touchz /foo
[hadoop@hadoop-0-0 ~]$ ./bin/hadoop dfs -ls /
Found 2 items
/foo       0       2008-06-20 23:49        rw-r–r–       hadoop  supergroup
/home              2008-06-20 18:58        rwxr-xr-x       hadoop  supergroup
[hadoop@hadoop-0-0 ~]$ ./bin/hadoop dfs -rm /foo
Deleted /foo
[hadoop@hadoop-0-0 ~]$

Testing MapReduce

Haven’t done this yet, except in standalone mode.

Desktop Tower Defense

allenday — Sat, 31 May 2008 20:13:26 +0000

7872. It’s either my most recent score, or a lower bound on the number of minutes I’ve spent on Desktop Tower Defense. Take your pick, just don’t tell my dissertation committee

Play here, or check my mazes. Warning, it’s addictive if you like puzzle/realtime strategy games.

Los Angeles SoC(i)al Tech Scene

allenday — Fri, 30 May 2008 20:37:35 +0000

I’m collating here all the tech events/sites I hear about that are specific to Los Angeles / Southern California. This post is the result of several conversations I’ve had, verbally and via email, about the scattered tech scene in Los Angeles. Disclaimer: I’m not the organizer of any of these events, and most of them I’ve never even attended so I can’t vouch for the quality. For now, the events are in the order in which I hear about them.

If you know of a resource I missed leave a comment and I’ll add it in.

The Construction and Usage of a Microarray Data Warehouse

allenday — Wed, 28 May 2008 04:40:20 +0000

That’s my dissertation topic, and I’m defending it Thursday morning. You can grab the latest PDF here. I’ll update this post with my PowerPoint slides when I finish them. The crux of the work was published last year as Celsius: A community resource for affymetrix microarray data. Genome Biology, 6(8), 2007. [pdf]

Update: Powerpoint Slides are now online. Also, my oral defense is complete. Other than filing my defense with the UCLA research library, I am now a Doctor of Philosophy, Human Genetics.

Statistical HTML Content Extraction

allenday — Tue, 27 May 2008 08:48:08 +0000

Introduction

I’ve been learning about some of the techniques used by the so-called “Black Hat SEO” community for boosting their rankings in search engine results. Intriguing stuff. I’m by no means an expert in this area, but the theory underlying building black-hat pages and networks sure looks like it has a lot to do with my primary areas of interest.

Generating Unique Content

One “Black Hat SEO” application area is automatically generating HTML pages to improve search engine rankings. This technique uses a Markov process to generate text. The idea is to build one or more web pages that contain the keywords the SEO is targeting. The method basically works like this:

Assemble a corpus of text to train the model. For example, Project Gutenberg
Build an order-N (typically N=2) Markov model that captures the state changes in the corpus
Generate text from the model, periodically throwing in some keywords
Link the generated page to some other page to which you want to send traffic
Repeat again from Step 1

One problem with this approach — aside from the fact that the keywords don’t really fitting in with the flow of the model — is that the model is trained on inappropriate text. For instance, suppose you were trying to optimize for keywords:

keywords
statistics
Search engine optimization
SEO
Automatic content generation
Automatic content extraction
HTML content extraction
Markov Model

… then you probably wouldn’t want to train your model on, say, Jane Austen’s Pride and Prejudice.

Improve Generated Text: Use Niche Corpora

A better thing to do would be to find some nice web pages containing keywords, statistics, seo, Markov model, and so on. That way you’ll pick up related keywords that you didn’t initially think of (or weren’t suggested by your keyword expansion tool), too.

But let’s face it. The corpora are going to be in HTML format. So the question now becomes, How do I automate the transformation of HTML into plain text for input to the model? A few strawman ideas, followed by my remarks:

Get an HTML document, and remove all s. Won’t work very well. You end up training on page navigation, footers, headers, etc.
Build a site- or software-specific parser (e.g. for Wikipedia, or for Wordpress) to extract the main content. Scalability and maintenance nightmare. This is not generalizable to general text extraction. You’ll be constantly fixing broken parsers, too.
Devise a scoring system that can identify the main content of the page. Exactly!

I did find some methods for scoring page fragments, such as the Perl modules HTML::Content::Extractor and HTML::Extract, and another method described by Nooks. There are also a few intersting ideas in Gupta’s WWW2003 paper.

None of that Perl code linked above actually works, but Nooks and Jean Tavernier generally had the right idea. Basically, they look “down” the DOM to find the sub-DOM with the highest text/tag ratio.

The main problem with this approach is that it biases for DOM leaves, or “twigs” that are very close to leaves. You end up having to write special rules for accomodating the idiosyncrosies of each particular page dealt with, and it basically turns back into an HTML parsing exercise.

The other problem, and possibly more significant one from a statistician’s point of view, is that the ratio is not a well-understood metric for making decisions about what constitutes a “good” versus a “bad” sub-document. It would be better to have a p-value…

Balls and Urns

Fortunately, Fisher’s exact test can be applied to this problem. Here’s how you can apply it, explanation follows. First, let’s define some variables:

X: the total number of words in the whole document.
x: the number of words in a sub-document.
Y: the total number of s in the whole document.
y: the number of s in a sub-document.

Then, we perform the following algorithm to identify the single best sub-document:

tree; //the HTML tree's root node
minP = 1; //minimum p-value observed in the document
subD = ""; //sub-document corresponding to minimum p-value
X = calculatex(tree);
Y = calculatey(tree);
look(tree);
function look (node) {
  x = calculatex(node);
  y = calculatey(node);
  p = calculateHyperG(x,y,X,Y);
  if ( p < minP ) {
    minP = p;
    subD = node;
  }
  C = children(node);
  foreach (c in C) {
    look(c);
  }
}

Balls and Urns, Explained

The pseudocode above is examining each sub-document of the HTML document in turn and identifying the one with the smallest p-value. The p-value is calculated using the hypergeometirc distribution, where we consider that a sub-document has x words and y HTML s. This, in the context of the total document having X words and Y HTML s. It’s better than a simple ratio calculation because it does not bias for the tree’s leaves. That is, the p-value does not consider only the size of x+y.

Caveats

Bear in mind that testing so many sub-documents, especially for very large HTML documents, warrants so-called “multiple hypothesis testing correction“, such as a Bonferroni correction. It’s outside the scope of this article.

Also, the tests performed are not entirely independent. That is, if node B is a child of node A then B will have some effect on A when calculating A’s p-value and must be factored out. This is also a well-defined problem but is, alas, also outside the scope of this article. Do your homework! Hint: learn about the Gene Ontology.

Conclusion

Fine and dandy, but does it work? My conclusion: seems to work. Here’s a CGI script demonstrating the hypergeometric content extraction technique on CNN.com. It reports a text snippet at the beginning and end of the single “best” sub-document and the corresponding (uncorrected) p-value. Twiddle the u parameter to test on a page of your choice. Some pages may block the user-agent I’m using…

There is also the issue of what to consider an element and what not to… or maybe even element weighting. For instance, maybe

and elements shouldn’t be penalized because they’re commonly associated with text, but