Question

I am using cascading 2.0 in scala, and meet a weird compilation issue

This is main parts of the code

val sojSource = createSojSource(optionMap("input"))

val sinkScheme = createSojScheme(true)
val sink = new Hfs(sinkScheme, optionMap("output"), SinkMode.REPLACE)

var pipe = new Pipe("soj")
val each = new Each(pipe, new SojSampleFilter())

val flowDef = new FlowDef().addSource(pipe, sojSource).addTailSink(each, sink)// compile error

And here is the error message for the last line of code

type mismatch; found : cascading.tap.hadoop.Hfs required: cascading.tap.Tap[_, _, _] 
Note: org.apache.hadoop.mapred.JobConf <: Any (and cascading.tap.hadoop.Hfs <: cascading.tap.Tap[org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector]), but Java-defined class Tap is invariant in type Config. You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10) 
Note: org.apache.hadoop.mapred.RecordReader <: Any (and cascading.tap.hadoop.Hfs <: cascading.tap.Tap[org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector]), but Java-defined class Tap is invariant in type Input. You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10) 
Note: org.apache.hadoop.mapred.OutputCollector <: Any (and cascading.tap.hadoop.Hfs <: cascading.tap.Tap[org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector]), but Java-defined class Tap is invariant in type Output. You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10) 

Could you, please, tell me how can I fix this and what is the reason?

Answer 1

I found this code in the scalding tree which works around this problem:

// The scala compiler has problems with the generics in Cascading
protected def castHfsTap(tap : Hfs) : Tap[JobConf, RecordReader[_,_], OutputCollector[_,_]] = {
  tap.asInstanceOf[Tap[JobConf, RecordReader[_,_], OutputCollector[_,_]]]
}