Question

There are account ids, each with a timestamp grouped by username. foreach of these username groups I want all pairs of (oldest account, other account).

I have a java reducer that does that, can I rewrite it as a simple pig script?

Schema:

{group:(username),A: {(id , create_dt)}

Input:

(batman,{(id1,100), (id2,200), (id3,50)})
(lulu  ,{(id7,100), (id9,50)})

Desired output:

(batman,{(id3,id1), (id3,id2)})
(lulu  ,{(id9,id7)})

Answer 1

Not that anyone seems to care, but here goes. You have to create a UDF:

desired =   foreach my_input generate group as n, FIND_PAIRS(A) as pairs_bag;

And the UDF:

public class FindPairs extends EvalFunc<DataBag> {
@Override
    public DataBag exec(Tuple input) throws IOException {
        Long pivotCreatedDate = Long.MAX_VALUE;
        Long pivot = null;

        DataBag accountsBag = (DataBag) input.get(0);
        for (Tuple account : accountsBag){
            Long accountId = Long.parseLong(account.get(0).toString());
            Long creationDate = Long.parseLong(account.get(4).toString());
            if (creationDate < pivotCreatedDate ) {
                // pivot is the one with the minimal creation_dt
                pivot = accountId;
                pivotCreatedDate = creationDate;
            }
        }

        DataBag allPairs = BagFactory.getInstance().newDefaultBag();
        if (pivot != null){
            for (Tuple account : accountsBag){
                Long accountId = Long.parseLong(account.get(0).toString());
                Long creationDate = Long.parseLong(account.get(4).toString());
                if (!accountId.equals(pivot)) {
                    // we don't want any self-pairs
                    Tuple output = TupleFactory.getInstance().newTuple(2);
                    if (pivot < accountId){
                            output.set(0, pivot.toString());
                            output.set(1, accountId.toString());
                    }
                    else {
                  output.set(0, accountId.toString());
                    output.set(1, pivot.toString());
                    }
                allPairs.add(output);
            }
        }               
        return allPairs;
}

and if you wanna play real nicely, add this:

/**
 * Letting pig know that we emit a bag with tuples, each representing a pair of accounts
 */
@Override
public Schema outputSchema(Schema input) {
    try{
        Schema pairSchema = new Schema();
        pairSchema.add(new FieldSchema(null, DataType.BYTEARRAY));
        pairSchema.add(new FieldSchema(null, DataType.BYTEARRAY));
        return new Schema(
                new FieldSchema(null,
                new Schema(pairSchema), DataType.BAG));         
    }catch (Exception e){
            return null;
    }
}   

}