Question

I have a set set of records that I am loading from a file and the first thing I need to do is get the max and min of a column.
In SQL I would do this with a subquery like this:

   select c.state, c.population, 
(select max(c.population) from state_info c) as max_pop, 
(select min(c.population) from state_info c) as min_pop
from state_info c

I assume there must be an easy way to do this in PIG as well but I’m having trouble finding it. It has a MAX and MIN function but when I tried doing the following it didn’t work:

records=LOAD '/Users/Winter/School/st_incm.txt'  AS (state:chararray, population:int);
with_max = FOREACH records GENERATE state, population, MAX(population);

This didn’t work. I had better luck adding an extra column with the same value to each row and then grouping them on that column. Then getting the max on that new group. This seems like a convoluted way of getting what I want so I thought I’d ask if anyone knows a simpler way.

Thanks in advance for the help.

Answer 1

As you said you need to group all the data together but no extra column is required if you use GROUP ALL.

Pig

records = LOAD 'states.txt'  AS (state:chararray, population:int);
records_group = GROUP records ALL;
with_max = FOREACH records_group 
           GENERATE
               FLATTEN(records.(state, population)), MAX(records.population);

Input

CA  10
VA  5
WI  2

Output

(CA,10,10)
(VA,5,10)
(WI,2,10)