Question

I am trying to drop records that contain at least one null in any of the fields. For example, if the data has 3 fields, then:

filtered = FILTER data by ($0 is not null) AND ($1 is not null) AND ($2 is not null)

Is there any cleaner way to do this, without having to write out 3 boolean expressions?

Answer 1

If all of the fields are of numeric types, you could simply do something like

filtered = FILTER data BY $0*$1*$2 is not null;

In Pig, if any terms in an arithmetic expression are null, the result is null.

You could also write a UDF to take an arbitrary number of arguments and return null (or 0, or false, whatever you find most convenient) if any of the arguments are null.

filtered = FILTER data BY NUMBER_OF_NULLS($0, $1, $2) == 0;

where NUMBER_OF_NULLS is defined elsewhere, e.g.

public class NUMBER_OF_NULLS extends EvalFunc {
    public Integer exec(Tuple input) {
        if (input == null) { return 0; }

        int c = 0;
        for (int i = 0; i < input.size(); i++) {
            if (input.get(i) == null) c++;
        }
        return c;
    }
}

Note: I have not tested the above UDF, and I don’t claim it adheres to any best practices for writing clear, robust UDFs. You should add exception-handling code, for example.