Question

I have this reduce function:

protected void reduce(Text key, Iterable<SortedMapWritable> values, Context context) throws IOException, InterruptedException {
    StringBuilder strOutput = new StringBuilder();
    double sum = 0, i = 0;
    DoubleWritable val = null;

    SortedMapWritable tmp = values.iterator().next();
    strOutput.append("[");
    Set<WritableComparable> keys = tmp.keySet();
    for (WritableComparable mapKey : keys) {                    
        val = (DoubleWritable)tmp.get(mapKey);
        sum += val.get();
        if(i > 0)
            strOutput.append(",");
        strOutput.append(val.get());
        i++;
    }
    strOutput.append("]");

    context.write(new Text(key.toString()), new Text(strOutput.toString()));
    context.write(new Text(key.toString() + "Med"), new Text(Double.toString(sum/i)));
}

As SortedMapWritable, I used a <LongWritable,DoubleWritable>, as we can see at this code

    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
    final Context ctx = context;
    Configuration conf = new Configuration();
    FileSystem hdfs = FileSystem.get(conf); 
    Path srcPath = new Path(hdfs.getWorkingDirectory() + "/" + value);  
    Path dstPath = new Path("/tmp/");       

    hdfs.copyToLocalFile(srcPath, dstPath);

    final StringBuilder errbuf = new StringBuilder();
    final Pcap pcap = Pcap.openOffline(dstPath.toString() + "/" +value, errbuf);
    if (pcap == null) {
        throw new InterruptedException("Impossible create PCAP file");
    }

    final HashMap<Integer,JxtaSocketFlow> dataFlows = new HashMap<Integer,JxtaSocketFlow>();
    final HashMap<Integer,JxtaSocketFlow> ackFlows = new HashMap<Integer,JxtaSocketFlow>();

    generateHalfSocketFlows(errbuf, pcap, dataFlows, ackFlows);
    final Text jxtaPayloadKey = new Text("JXTA_Payload");
    final Text jxtaRelyRtt = new Text("JXTA_Reliability_RTT");

    SortedMapWritable payOutput = new SortedMapWritable();
    SortedMapWritable rttOutput = new SortedMapWritable();

    for (Integer dataFlowKey : dataFlows.keySet()) {
        JxtaSocketFlow dataFlow = dataFlows.get(dataFlowKey);
        JxtaSocketStatistics stats = dataFlow.getJxtaSocketStatistics();

        payOutput.put(new LongWritable(stats.getEndTime()), new DoubleWritable((stats.getPayload())/1024));         
        HashMap<Integer,Long> rtts = stats.getRtts();
        for (Integer num : rtts.keySet()) {
            LongWritable key = new LongWritable(stats.getEndTime() + num);                                                      
            rttOutput.put(key, new DoubleWritable(rtts.get(num)));
        }
    }

    try{
        ctx.write(jxtaPayloadKey, payOutput);
        ctx.write(jxtaRelyRtt, rttOutput);
    }catch(IOException e){
        e.printStackTrace();
    }catch(InterruptedException e){
        e.printStackTrace();
    }
}

On reduce function, for each key, the value has been concatenated with the previous values.

For example, in the correct way, the keys and values should be:

key1 -> {a, b, c} key2 -> {d, e, f}

But the values has been

key1 -> {a, b, c} key2 -> {a, b, c, d, e, f}

Does anyone knows why is this happening and how can I avoid this?

Answer 1

There is an open bug with hadoop https://issues.apache.org/jira/browse/HADOOP-5454 that might explain the problem you are having.

In the following code, row.clear() is required to prevent the values appending from one iteration to the next.

@Log4j
public class StackOverFlowReducer extends Reducer
{
    public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException
    {
        for (SortedMapWritable row : values)
        {
            log.info(String.format("New Map : %s", Joiner.on(",").join(row.entrySet())));
            row.clear();//https://issues.apache.org/jira/browse/HADOOP-5454
        }
    }
}

I only tested the workaround within a single key.
I hope it helps.