Home/ Questions/Q 8107369
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-06T00:44:52+00:00

When developing Pig scripts that use the STORE command I have to delete the

  • 0

When developing Pig scripts that use the STORE command I have to delete the output directory for every run or the script stops and offers:

2012-06-19 19:22:49,680 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: Output Location Validation Failed for: 'hdfs://[server]/user/[user]/foo/bar More info to follow:
Output directory hdfs://[server]/user/[user]/foo/bar already exists

So I’m searching for an in-Pig solution to automatically remove the directory, also one that doesn’t choke if the directory is non-existent at call time.

In the Pig Latin Reference I found the shell command invoker fs. Unfortunately the Pig script breaks whenever anything produces an error. So I can’t use

fs -rmr foo/bar

(i. e. remove recursively) since it breaks if the directory doesn’t exist. For a moment I thought I may use

fs -test -e foo/bar

which is a test and shouldn’t break or so I thought. However, Pig again interpretes test‘s return code on a non-existing directory as a failure code and breaks.

There is a JIRA ticket for the Pig project addressing my problem and suggesting an optional parameter OVERWRITE or FORCE_WRITE for the STORE command. Anyway, I’m using Pig 0.8.1 out of necessity and there is no such parameter.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    At last I found a solution on grokbase. Since finding the solution took too long I will reproduce it here and add to it.

    Suppose you want to store your output using the statement

    STORE Relation INTO 'foo/bar';

    Then, in order to delete the directory, you can call at the start of the script

    rmf foo/bar

    No “;” or quotations required since it is a shell command.

    I cannot reproduce it now but at some point in time I got an error message (something about missing files) where I can only assume that rmf interfered with map/reduce. So I recommend putting the call before any relation declaration. After SETs, REGISTERs and defaults should be fine.

    Example:

    SET mapred.fairscheduler.pool 'inhouse';
REGISTER /usr/lib/pig/contrib/piggybank/java/piggybank.jar;
%default name 'foobar'
rmf foo/bar
Rel = LOAD 'something.tsv';
STORE Rel INTO 'foo/bar';
    • 0