Question

We aim to leverage PIG for largescale log analysis of our server logs. I need to load a PIG map datatype from a file.

I tried running a sample PIG script with the following data.

A line in my CSV file, named ‘test’ (to be processed by PIG) looks like,

151364,[ref#R813,highway#secondary]

My PIG Script

a = LOAD 'test' using PigStorage(',') AS  (id:INT, m:MAP[]);
DUMP a;

The idea is to load an int and the second element as a hashmap.
However, when I dump, the int field get parsed correctly(and gets printed in the dump) but the map field is not parsed resulting in a parsing error.

Can someone please explain if I am missing something?

Answer 1

I think there is a delimiter related problem (such as field-delimiter is somehow effecting parsing of map field or it is confused with map-delimiter).

When this input data is used (notice I used semicolon as field-delimiter):

151364;[ref#R813,highway#secondary]

below is the output from my grunt shell:

grunt> a = LOAD '/tmp/temp2.txt' using PigStorage(';') AS (id:int, m:[]);
grunt> dump a;
...
(151364,[highway#secondary,ref#R813])

grunt> b = foreach a generate m#'ref'; 
grunt> dump b;
(R813)