I made up a test dataset of gene/mRNA/exon 3-tiered feature groups by permuting some gene model data from the knownGene annotation set of the Hg18 build of the human genome.  You can grab the data set and script used to generate it here.  There are several files mRNA.EN.txt.gz that contain gzipped gene models, where N=3..7 indicates there are 10^N models in the file, uniformly distributed across a 500-megabase reference sequence.

I’m planning to load these data into a couple of different systems and then compare performance on some of the typical Bio::DB::GFF API calls.  I can personally test on:

• Chado
• The default Bio::DB::GFF schema (does it have a name?)
• The SOLR backend I’m about to implement

I know there are other feature DBs out there.  It would be good to include them as well in a later pass or to have someone else contribute the data once I get the benchmarking script written.

	public static void itemItemRecommendations(String path, String file) {
		File f = new File(path, file);
	    try {
			DataModel model = new FileDataModel(f);
			model.refresh(null);
		    ItemSimilarity itemSimilarity = new LogLikelihoodSimilarity(model);
		    ItemBasedRecommender itemRecommender = new GenericItemBasedRecommender(model, itemSimilarity);
		    for ( Item i : model.getItems() )
			    for ( RecommendedItem j : itemRecommender.mostSimilarItems(i.getID(), 50) )
			    	if ( j.getValue() >= 0.7 )
			    		System.out.println(i.getID() + "\t" + j.getItem().getID() + "\t" + String.format("%.3f", j.getValue()));
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (TasteException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

This outputs item1 –recommends–>item2 pairs with a weight. I’m taking this and putting it into a solr document so I can display related item2s alongside item1 when it’s viewed.

Input data are comma-delimited tuples like so:

1fe7401b81eed49353d0cbeba5383848,5212,0.6
3c1832954a6e8781836fed670bb37b24,5212,1
70273e4c7c77700ee97acb8d0306c405,5213,0.8
1f057ccde135acbc881008bbf466e7e1,5213,1
51d44c7baca65ad39d11ba87bf2d438b,5213,1
adc924559b37114cd97d1f5cf7c71419,5213,1
78e254b4a11e61d76ff63cea02de4de8,5213,1
5c373ec7d9ad4a6f392c291d8ccba5ce,5213,0.2
fab8537564094fa8885f6214e6b682e1,5213,1
127f46aabcdbc2d2d04da8398a996c75,5213,1

Works great. Thanks Sean.

1.1.1.1
1.1.1.2
[...]
254.254.254.253
254.254.254.254

We can use hadoop streaming to chunk the list so we can do the DNS lookups in parallel. Easy and requires little to know thought:

./bin/hadoop jar contrib/streaming/*-streaming.jar -input /home/aday/classC.dat -output /home/aday/classC_dns.dat -mapper 'perl -ne '\''print `host $_`'\''' -numReduceTasks 0

We wrap the host call in backticks so we can trap non-zero exit codes and get an error message on stdout courtesy of perl.

Pretty much supports my sense that scheduling is not Hadoop’s strong suit. It’s really pretty shitty. Would be great to see some more cross-pollination between the Beowulf (SGE, PBS, Globus) and MapReduce (Hadoop, HBase) communities. The former have more mature scheduling, resource management and permissions models. They don’t really do a good job thought with providing a framework for distributed, parallel computing at the application level though — everything is roll-your-own. Perhaps Hadoop could be integrated as a parallel environment to consume resources from a SGE master [1, 2] rather than managing its own mapper/reducer pools.

A less ambitious scheduler improvement is to modify the way the Hadoop scheduler allocates map/reduce resources. The main itch I’m trying to scratch right now has to do with the coupling of map/reduce allocation. There are some cases where it seems this shouldn’t be done. Read the dialog with Nathan below if you care to know more.

	allenday	is it possible to decouple mapper and reducer slot allocation for jobs?
	allenday	i mean, if a job is #1 in the MR queue, but it is not yet ready to reduce, can it be prevented from consuming reducer slots?
	|<–	Smokinn has left irc.freenode.net ()
	|<–	savage- has left irc.freenode.net (Read error: 110 (Connection timed out))
	–>|	overlast (n=overlast@19.181.210.220.dy.bbexcite.jp) has joined #hadoop
	|<–	overlast has left irc.freenode.net (“Leaving…”)
	nathanmarz	allenday: i think that would be hard… reducing starts while the mapping is happening (copy stage)
	allenday	nathanmarz, i frequently find that while the reduce has “started”, it can just sit there for a long time doing nothing
	allenday	this is most common with nutch
	allenday	so there could be a bunch of other jobs further back in the queue that get starved for reduces b/c the head of the queue is squatting on the slots
	nathanmarz	it just sits there in the reduce phase?
	allenday	for sure nutch does, yeah
	allenday	during fetch, when it crawling
	|<–	cutting has left irc.freenode.net (“Leaving.”)
	nathanmarz	i see
	nathanmarz	i don’t have that much familiarity with nutch
	nathanmarz	is it possible to increase the number of reducers?
	allenday	yep, but then you can get into i/o trouble later
	nathanmarz	for the job i mean, not the cluster
	allenday	oh
	allenday	it sounds like you propose having these squatters consume minimal # of reducers (e.g. only 1)
	nathanmarz	actually, the opposite
	nathanmarz	let’s say you have 16 reduce slots
	nathanmarz	and the job i set to use 16 reducers
	nathanmarz	each one of those reducers potentially has to go over a lot of data
	nathanmarz	if the job is instead set to use a lot more reducers, like 100 or something
	nathanmarz	than an individual reducer will go a lot faster
	nathanmarz	and potentially, those freed reduce slots will go to jobs with higher priority
	allenday	ok, so you introduce priority to bump the further back ahead in the queue
	nathanmarz	yea
	allenday	is that settable in jobconf?
	nathanmarz	you can set num reducers
	–>|	tobias_au (n=opera@CPE-121-50-201-65.dsl.OntheNet.net) has joined #hadoop
	allenday	so let’s suppose the job that squats on reduce slots gets to the head of the queue. regardless of if it has 16 or 100 reducers configured
	nathanmarz	JobConf#setNumReduceTasks
	allenday	and that it it still in map phase only. has not begun reducing yet
	allenday	until one of those reduces finishes (i.e. the map has finished) all slots are still filled
	allenday	it’s only when the first reduce finishes that the job at #2 can take over a reduce slot
	nathanmarz	right
	nathanmarz	yea that’s true
	allenday	that’s bad
	nathanmarz	this scheme doesn’t help until mappers finished
	allenday	you really want this #1 job
	allenday	when it is allocating reducers
	allenday	to have low priority in acquiring the slots
	nathanmarz	right
	nathanmarz	well you don’t want it to acquire any slots until mappers finish
	allenday	so you give reduce slots to #2, #3, #4, etc. until everyone who wants slots has them. then you assign to #1
	allenday	or until #1 is ready…
	allenday	is it just me or does the queueing system in hadoop kind of suck?
	allenday	i am coming here from sun grid which puts a lot of emphasis on this aspect
	nathanmarz	well, the priority system will work if you start job #1 after the other jobs
	nathanmarz	if you start the other jobs after #1 then they will get starved of reducers
	allenday	heh, but the whole reason it is in #1 is because it was submitted first, right? isn’t hadoop FIFO wrt jobs?
	nathanmarz	if they’re the same priority
	nathanmarz	so maybe decreasing the reducers job #1 uses is the way to go
	nathanmarz	set it so it doesn’t use all the reduce slots on the cluster
	allenday	i need to do some research to see if there are jira open for improving the scheduler. or if there are some commercial plugins to improve the scheduling
	nathanmarz	definitely room for improvement, agreed
	allenday	yeah, that was what i thought you meant initially. it’s a hack too though, and breaks down when the number of jobs gets large
	allenday	i’m surprised they are coupled. do you understand how it works when the mapper hands off to the reducer?
	allenday	b/c i don’t and i need to
	nathanmarz	yes
	allenday	can i get the 2min version?
	nathanmarz	the reason the reducers start while the mappers are running is because there’s some work they can do without all the map data
	nathanmarz	each reducer needs to copy the relevant outputs from all the mappers to its machine
	nathanmarz	this is called the “copy” phase and can occur in parallel with mapping
	allenday	ok, i’ve seen that
	allenday	so what we need is a flag taht indicates there will be no data to copy until maps all finish
	nathanmarz	yea
	nathanmarz	a flag that says not to pipeline the process
	allenday	default behavior is to have the flag off and copy greedily
	allenday	which is like it does now
	allenday	turn the flag on says to wait until upstream map finishes before grabbing a reduce slot and kicking off the copy
	allenday	**all upstream maps
	nathanmarz	http://hadoop.apache.org/core/docs/current/hadoop-default.html
	nathanmarz	those are all the hadoop config parameters
	nathanmarz	you might be able to find something in there
	allenday	yeah, i fiind goodies in there every time i read that page
	allenday	i am only ~1mo into hadoop
	allenday	here’s another scheduling related question/issue i’m having
	allenday	i find that job i/o and cpu usage tend to synchronize after a while
	allenday	b/c if there is a slow moving job in the queue, all the others tend to get jammed behind it
	allenday	have you seen this?
	nathanmarz	no, i haven’t
	nathanmarz	but that’s interesting
	allenday	it comes back to resource (mis)allocation by the scheduler
	nathanmarz	how are you measuring that?
	allenday	it’s this same issue where jobs will consume all the slots
	allenday	so if you have a slow moving thing blocking all the resources
	allenday	no one else can get past
	allenday	then when the slow moving job finishes, the others all start getting processed very quickly (high cpu load during map), then as they begin to finish there is a flurry of i/o
	allenday	it’s like congestion on the freeway where one car slams on the breaks it sends this wave of traffic jam behind it
	allenday	assuming the freeway is already close to capacity (not sparse)

cat /some/input | cut -f 1 | sort | uniq > /some/output

on an HDFS file.

The basic thing you want to do to get a map working follows. The general rule of thumb is that if there is one or more lines of output for each line of input, then you don’t need to use any reducers, hence the -numReduceTasks 0 option.

$HADOOP_HOME/bin/hadoop jar contrib/streaming/*-streaming.jar -input /some/input -output /some/output -mapper 'cut -f 1' -numReduceTasks 0

In my case though, I wanted to uniqify my list. Putting uniq into the mapper chain would cause the job to fail. Instead I had to drop the -numReduceTasks 0 and do like so:

$HADOOP_HOME/bin/hadoop jar contrib/streaming/*-streaming.jar -input /some/input -output /some/output -mapper 'cut -f 1' -reducer 'uniq'

Note also that I didn’t need to include the sort from my original shell command. That’s because sorting is implicit in the MapReduce process.

As usual, I’m new to all of this, so if you have any insights leave a comment.

Step 1: modify Nutch URL filters

Okay, so first thing, modify $NUTCH_HOME/conf/crawl-urlfilter.txt .  Let’s assume you only care about JPEG images, change this line:

-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$

to this:

-\.(gif|GIF|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|bmp|BMP)$

also update the “MY.DOMAIN.NAME” section appropriately.

Step 2: set up crawl configuration

Edit $NUTCH_HOME/conf/nutch-site.xml. You want to update/add properties for http.content.limit and file.content.limit so that your big files don’t get truncated. Look at $NUTCH_HOME/conf/nutch-default.xml for examples of how to do this. You might also want to adjust Protocol.CHECK_ROBOTS

Step 3: crawl

I’m not going to go into this here as it is well-covered elsewhere. Basically you just want to make a list of seed URLs, then let nutch do its thing, e.g. like:

$NUTCH_HOME/bin/nutch crawl /home/allenday/urls -dir /home/allenday/crawled -depth 5

This is going to generate some directories under /home/allenday/crawled/segments.

Step 4: massage crawl outputs and extract images

Merge the crawl segments into one big segment. This makes the following steps easier.

$NUTCH_HOME/bin/nutch mergesegs /tmp/merged /home/allenday/crawled/segments

Now dump the segment and show the image URLs fro the crawl.

$NUTCH_HOME/bin/nutch readseg -dump /tmp/merged/* /tmp/dump
$NUTCH_HOME/bin/hadoop dfs -cat /tmp/dump/dump | grep -aE 'URL'

The grep should show something like this:

URL:: http://spicylogic.com/some-url.html
URL:: http://spicylogic.com/some-url.jpg

Obviously you’re interested in grepping for jpg, jpeg, etc. Do it.

Once you have the image list, you can use this little Java program to pull the images out of the segment one by one.

package com.spicylogic.allenday;
 
//JDK imports
import java.io.ByteArrayInputStream;
import java.io.DataInput;
import java.io.DataInputStream;
import java.io.DataOutput;
import java.io.DataOutputStream;
import java.io.IOException;
import java.util.Arrays;
import java.util.zip.InflaterInputStream;
 
//Hadoop imports
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.ArrayFile;
import org.apache.hadoop.io.DataOutputBuffer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.SequenceFile.ValueBytes;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.UTF8;
import org.apache.hadoop.io.VersionMismatchException;
import org.apache.hadoop.io.Writable;
 
//Nutch imports
import org.apache.nutch.metadata.Metadata;
import org.apache.nutch.protocol.Content;
import org.apache.nutch.util.MimeUtil;
import org.apache.nutch.util.NutchConfiguration;
 
public final class ExtractFile {
  public static void main(String argv[]) throws Exception {
 
    String usage = "Content (-local | -dfs <namenode:port>) url segment";
 
    if (argv.length < 3) {
      System.out.println("usage:" + usage);
      return;
    }
    Configuration conf = NutchConfiguration.create();
    FileSystem fs = FileSystem.parseArgs(argv, 0, conf);
    try {
      String segment = argv[1];
 
      Path file = new Path(segment, Content.DIR_NAME + "/part-00000/data");
      SequenceFile.Reader reader = new SequenceFile.Reader(fs, file, conf);
 
      Text key = new Text();
      Content content = new Content();
 
      while (reader.next(key, content)) {
        //System.err.println( key + "\t=\t" + argv[0] );
        if (key.equals(new Text(argv[0]))) {
          System.out.write( content.getContent(), 0, content.getContent().length );
          break;
        }
      }
      reader.close();
    } finally {
      fs.close();
    }
  }
}

Compile it, then you can do like so:

$NUTCH_HOME/bin/hadoop --config YOUR:CLASS:PATH com.spicylogic.allenday.ExtractFile http://spicylogic.com/some-url.jpg /tmp/merged/* > out.jpg

Hope that helps. Let me know if you have corrections/clarifications (or a complete script!) for this post and I’ll be happy to merge them in with attribution.

Thanks for this post go to Kaz Muzik, who was doing something similar to back up his blog. Also, the com.spicylogic.allenday.ExtractFile class is based on org.apache.nutch.protocol.Content class.