Pretty much supports my sense that scheduling is not Hadoop’s strong suit. It’s really pretty shitty. Would be great to see some more cross-pollination between the Beowulf (SGE, PBS, Globus) and MapReduce (Hadoop, HBase) communities. The former have more mature scheduling, resource management and permissions models. They don’t really do a good job thought with providing a framework for distributed, parallel computing at the application level though — everything is roll-your-own. Perhaps Hadoop could be integrated as a parallel environment to consume resources from a SGE master [1, 2] rather than managing its own mapper/reducer pools.

A less ambitious scheduler improvement is to modify the way the Hadoop scheduler allocates map/reduce resources. The main itch I’m trying to scratch right now has to do with the coupling of map/reduce allocation. There are some cases where it seems this shouldn’t be done. Read the dialog with Nathan below if you care to know more.

	allenday	is it possible to decouple mapper and reducer slot allocation for jobs?
	allenday	i mean, if a job is #1 in the MR queue, but it is not yet ready to reduce, can it be prevented from consuming reducer slots?
	|<–	Smokinn has left irc.freenode.net ()
	|<–	savage- has left irc.freenode.net (Read error: 110 (Connection timed out))
	–>|	overlast (n=overlast@19.181.210.220.dy.bbexcite.jp) has joined #hadoop
	|<–	overlast has left irc.freenode.net (“Leaving…”)
	nathanmarz	allenday: i think that would be hard… reducing starts while the mapping is happening (copy stage)
	allenday	nathanmarz, i frequently find that while the reduce has “started”, it can just sit there for a long time doing nothing
	allenday	this is most common with nutch
	allenday	so there could be a bunch of other jobs further back in the queue that get starved for reduces b/c the head of the queue is squatting on the slots
	nathanmarz	it just sits there in the reduce phase?
	allenday	for sure nutch does, yeah
	allenday	during fetch, when it crawling
	|<–	cutting has left irc.freenode.net (“Leaving.”)
	nathanmarz	i see
	nathanmarz	i don’t have that much familiarity with nutch
	nathanmarz	is it possible to increase the number of reducers?
	allenday	yep, but then you can get into i/o trouble later
	nathanmarz	for the job i mean, not the cluster
	allenday	oh
	allenday	it sounds like you propose having these squatters consume minimal # of reducers (e.g. only 1)
	nathanmarz	actually, the opposite
	nathanmarz	let’s say you have 16 reduce slots
	nathanmarz	and the job i set to use 16 reducers
	nathanmarz	each one of those reducers potentially has to go over a lot of data
	nathanmarz	if the job is instead set to use a lot more reducers, like 100 or something
	nathanmarz	than an individual reducer will go a lot faster
	nathanmarz	and potentially, those freed reduce slots will go to jobs with higher priority
	allenday	ok, so you introduce priority to bump the further back ahead in the queue
	nathanmarz	yea
	allenday	is that settable in jobconf?
	nathanmarz	you can set num reducers
	–>|	tobias_au (n=opera@CPE-121-50-201-65.dsl.OntheNet.net) has joined #hadoop
	allenday	so let’s suppose the job that squats on reduce slots gets to the head of the queue. regardless of if it has 16 or 100 reducers configured
	nathanmarz	JobConf#setNumReduceTasks
	allenday	and that it it still in map phase only. has not begun reducing yet
	allenday	until one of those reduces finishes (i.e. the map has finished) all slots are still filled
	allenday	it’s only when the first reduce finishes that the job at #2 can take over a reduce slot
	nathanmarz	right
	nathanmarz	yea that’s true
	allenday	that’s bad
	nathanmarz	this scheme doesn’t help until mappers finished
	allenday	you really want this #1 job
	allenday	when it is allocating reducers
	allenday	to have low priority in acquiring the slots
	nathanmarz	right
	nathanmarz	well you don’t want it to acquire any slots until mappers finish
	allenday	so you give reduce slots to #2, #3, #4, etc. until everyone who wants slots has them. then you assign to #1
	allenday	or until #1 is ready…
	allenday	is it just me or does the queueing system in hadoop kind of suck?
	allenday	i am coming here from sun grid which puts a lot of emphasis on this aspect
	nathanmarz	well, the priority system will work if you start job #1 after the other jobs
	nathanmarz	if you start the other jobs after #1 then they will get starved of reducers
	allenday	heh, but the whole reason it is in #1 is because it was submitted first, right? isn’t hadoop FIFO wrt jobs?
	nathanmarz	if they’re the same priority
	nathanmarz	so maybe decreasing the reducers job #1 uses is the way to go
	nathanmarz	set it so it doesn’t use all the reduce slots on the cluster
	allenday	i need to do some research to see if there are jira open for improving the scheduler. or if there are some commercial plugins to improve the scheduling
	nathanmarz	definitely room for improvement, agreed
	allenday	yeah, that was what i thought you meant initially. it’s a hack too though, and breaks down when the number of jobs gets large
	allenday	i’m surprised they are coupled. do you understand how it works when the mapper hands off to the reducer?
	allenday	b/c i don’t and i need to
	nathanmarz	yes
	allenday	can i get the 2min version?
	nathanmarz	the reason the reducers start while the mappers are running is because there’s some work they can do without all the map data
	nathanmarz	each reducer needs to copy the relevant outputs from all the mappers to its machine
	nathanmarz	this is called the “copy” phase and can occur in parallel with mapping
	allenday	ok, i’ve seen that
	allenday	so what we need is a flag taht indicates there will be no data to copy until maps all finish
	nathanmarz	yea
	nathanmarz	a flag that says not to pipeline the process
	allenday	default behavior is to have the flag off and copy greedily
	allenday	which is like it does now
	allenday	turn the flag on says to wait until upstream map finishes before grabbing a reduce slot and kicking off the copy
	allenday	**all upstream maps
	nathanmarz	http://hadoop.apache.org/core/docs/current/hadoop-default.html
	nathanmarz	those are all the hadoop config parameters
	nathanmarz	you might be able to find something in there
	allenday	yeah, i fiind goodies in there every time i read that page
	allenday	i am only ~1mo into hadoop
	allenday	here’s another scheduling related question/issue i’m having
	allenday	i find that job i/o and cpu usage tend to synchronize after a while
	allenday	b/c if there is a slow moving job in the queue, all the others tend to get jammed behind it
	allenday	have you seen this?
	nathanmarz	no, i haven’t
	nathanmarz	but that’s interesting
	allenday	it comes back to resource (mis)allocation by the scheduler
	nathanmarz	how are you measuring that?
	allenday	it’s this same issue where jobs will consume all the slots
	allenday	so if you have a slow moving thing blocking all the resources
	allenday	no one else can get past
	allenday	then when the slow moving job finishes, the others all start getting processed very quickly (high cpu load during map), then as they begin to finish there is a flurry of i/o
	allenday	it’s like congestion on the freeway where one car slams on the breaks it sends this wave of traffic jam behind it
	allenday	assuming the freeway is already close to capacity (not sparse)