Question

How do you process a flat file with Pig? For example, if you had a line containing a record where the first four positions were the year, the next 5 were a product code, and the last 8 contained the MSRP, how would you query this data with Pig? I’m probably missing something simple, but everything I’ve found thus far requires a delimiter to be used when loading data with Pig.

Some sample data is provided below:

1999ABCDE12234.00
2000DCEFS00020.00
2012FFEWS00005.55

Thanks in advance.

Jeremy

Answer 1

One way to split a line based on positions is to use REGEX_EXTRACT_ALL .

E.g:

A = LOAD 'flat.txt' as (line:chararray);
B = FOREACH A GENERATE FLATTEN(REGEX_EXTRACT_ALL(line, 
      '^(.{1,4})(.{1,5})(.*)$')) AS (year:int, prod_code:chararray, msrp:double);
dump B;
(1999,ABCDE,12234.00)
(2000,DCEFS,00020.00)
(2012,FFEWS,00005.55)