Question

I have 3 table to batch input, how to specify the parameter in pig?

for example, tmp/001.csv,tmp/002.csv,tmp/003.csv, in pig script, how could I write the -param and the LOAD statement to input these table in one time?

Some one gives me a example

pig -param nums="'001','002','003'" test.pig

in pig script,

LOAD 'tmp/{nums}.csv' AS ...

But it looks like only 001.csv is read by pig.

Answer 1

You’ll most probably need to move the /tmp to the params too:

pig -param ins=/tmp/001.csv,/tmp/002.csv,/tmp/003.csv test.pig

LOAD '${ins}' AS ...

Obviously if you have many files to list (and they all can match a simple glob) then this can be reduced even further:

pig -param ins=/tmp/*.csv test.pig