Question

I have a csv file with following format

TRAABRX12903CC4816,1548880,2:19,4:7,...
.
.
.

My problem is that I want to interpret as

{(key:chararray,key2:int,{(id:int,cnt:int)})}

So far my code is

data = LOAD 'mxm_dataset_test_3.txt' using PigStorage(',');
data0 = foreach data generate $0 as key:chararray, {$2 ..} as bow;

For data0 this the schema data0: {key: chararray,bow: {(NULL)}}
When I try to explicit cast it to (bag{tuple(chararray)}) with

data0 = foreach data generate $0 as key:chararray, {$2 ..} as bow;

this gives the error Cannot cast bag with schema :bag{:tuple(:NULL)} to bag with schema :bag{:tuple(:chararray)}


Answer 1

Use the TOBAG built-in function to build your bag:

$ cat input
TRAABRX12903CC4816,1548880,2:19,4:7
TRAABRX12903CC4816,1548881,2:19,4:7,5:23,7:3
TRAABRX12903CC4816,1548882
TRAABRX12903CC4816,1548883,17:33
$ cat test.pig
data = LOAD 'input' USING PigStorage(',');
data0 = FOREACH data GENERATE $0 AS key:chararray, $1 AS key2:chararray, TOBAG($2 ..) AS bow:{(pair)};
DESCRIBE data0;
DUMP data0;
$ pig -x local test.pig
...
data0: {key: chararray,key2: chararray,bow: {(pair: NULL)}}
...
(TRAABRX12903CC4816,1548880,{(2:19),(4:7)})
(TRAABRX12903CC4816,1548881,{(2:19),(4:7),(5:23),(7:3)})
(TRAABRX12903CC4816,1548882,{})
(TRAABRX12903CC4816,1548883,{(17:33)})

If you want to split up the id:cnt pairs, however, this gets trickier. Because there is no way to assign a schema to an arbitrary number of elements, and TOBAG is a UDF, Pig can’t cast the bytearray to a chararray or anything else later.

I would recommend loading the entire line as a string (USING PigStorage('\n')), using STRSPLIT with a limit of 3 to get your key, key2, and comma-delimited list of strings, then iterate with STRSPLIT on comma and then on colon to get the pairs you want, using FLATTEN and TOBAG as needed. I would demonstrate this for you, but I am stuck on Pig 0.9, and judging by PIG-2311, this isn’t possible until Pig 0.10.

The simplest solution may just be to write your own UDF to interpret a string like 2:13,9:4,5:4:

data = LOAD 'input' USING PigStorage('\n') AS (line:chararray);
data0 = FOREACH data GENERATE FLATTEN(STRSPLIT(line, ',', 3)) AS (key:chararray, key2:chararray, pairs:chararray);
data1 = FOREACH data0 GENERATE key, key2, myudfs.PARSE_PAIRS(pairs);

where myudfs.PARSE_PAIRS returns a bag with the tuples you want. Good luck.