Comments on: Using Nutch to download large binary media and image files http://www.spicylogic.com/allenday/blog/2008/08/29/using-nutch-to-download-large-binary-media-and-image-files/ ♥data♥ Thu, 10 Jun 2010 05:18:18 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Michael Hanney http://www.spicylogic.com/allenday/blog/2008/08/29/using-nutch-to-download-large-binary-media-and-image-files/comment-page-1/#comment-401 Michael Hanney Fri, 15 Jan 2010 06:40:39 +0000 http://www.spicylogic.com/allenday/blog/?p=55#comment-401 Nice work Allen, This helped me a lot. Thanks. I followed your example, and updated it to work with Nutch 1.0 and Hadop 0.20. The changes are minor: import org.apache.commons.cli.Options; import org.apache.nutch.util.NutchConfiguration; then Options opts = new Options(); Configuration conf = NutchConfiguration.create(); GenericOptionsParser parser = new GenericOptionsParser(conf, opts, argv); FileSystem fs = FileSystem.get(conf); Mike. Nice work Allen,

This helped me a lot. Thanks.

I followed your example, and updated it to work with Nutch 1.0 and Hadop 0.20. The changes are minor:

import org.apache.commons.cli.Options;
import org.apache.nutch.util.NutchConfiguration;

then

Options opts = new Options();
Configuration conf = NutchConfiguration.create();
GenericOptionsParser parser = new GenericOptionsParser(conf, opts, argv);
FileSystem fs = FileSystem.get(conf);

Mike.

]]> By: Extracting Nutch crawl data | mhanney http://www.spicylogic.com/allenday/blog/2008/08/29/using-nutch-to-download-large-binary-media-and-image-files/comment-page-1/#comment-400 Extracting Nutch crawl data | mhanney Fri, 15 Jan 2010 06:32:22 +0000 http://www.spicylogic.com/allenday/blog/?p=55#comment-400 [...] this example – Using Nutch to download large binary media and image files by Allen Day I made a basic Java console app to simply list the keys (I also wrote the data to stdout as a test, [...] [...] this example – Using Nutch to download large binary media and image files by Allen Day I made a basic Java console app to simply list the keys (I also wrote the data to stdout as a test, [...]

]]>