Question

Problem Statement:-
Below is the script that someone else wrote and he left the company so I don’t know whom should I ask about this. So that is the reason I am posting here to find the solution.

What this script does is- It gzip the data from a particular folder (/data/ds/real/EXPORT_v1x0) for a particular date (20121017) and move it back to HDFS (hdfs://ares-nn/apps/tech/ds/new/) directory.

date=20121017
groups=(0 '1[0-3]' '1[^0-3]' '[^01]')
  for shard in 0 1 2 3 4 5 6 7 8 9 10 11; do
    for piece in 0 1 2 3; do
        group=${groups[$piece]}
        if ls -l /data/ds/real/EXPORT_v1x0_${date}_${shard}_T_${group}*.dat.gz; then
          gzip -dc /data/ds/real/EXPORT_v1x0_${date}_${shard}_T_${group}*.dat.gz | \
          hadoop jar /export/home/ds/lib/HadoopUtil.jar com.host.hadoop.platform.util.WriteToHDFS -z -u \
          hdfs://ares-nn/apps/tech/ds/new/$date/EXPORT-part-$shard-$piece
          sleep 15
       fi
    done
  done

So during the migration to HDFS I found out this file has some problem in HDFS-

hdfs://ares-nn/apps/tech/ds/new/20121017/EXPORT-part-8-3

So Is there any way by doing some permutation from the above script we can find out what are the files under this directory (/data/ds/real/EXPORT_v1x0) which ultimately got converted to this hdfs://ares-nn/apps/tech/ds/new/20121017/EXPORT-part-8-3 which has the problem.

Any thoughts?

Update:-

Something like this below?

groups=(0 '1[0-3]' '1[^0-3]' '[^01]')
  for shard in 0 1 2 3 4 5 6 7 8 9 10 11; do
    for piece in 0 1 2 3; do
        group=${groups[$piece]}
        if ls -l /data/ds/real/EXPORT_v1x0_${date}_${shard}_T_${group}*.dat.gz; then
          [ "$date/EXPORT-part-$shard-$piece" == "20121017/EXPORT-part-8-3" ] && {
                echo /data/real/EXPORT_v1x0_${date}_${shard}_T_${group}*.dat.gz
            }
         fi
    done
  done

Few Sample Files Format I have in the /data/real/EXPORT folder-

/data/real/EXPORT_v1x0_20121017_4_T_115600_115800.dat.gz
/data/real/EXPORT_v1x0_20121017_4_T_235600_235800.dat.gz
/data/real/EXPORT_v1x0_20121017_4_T_115800_120000.dat.gz
/data/real/EXPORT_v1x0_20121017_4_T_235800_000000.dat.gz

And few sample output that I got after making changes-

/data/real/EXPORT_v1x0_20121017_0_T_0*.dat.gz: No such file or directory
/data/real/EXPORT_v1x0_20121017_0_T_1[0-3]*.dat.gz: No such file or directory
/data/real/EXPORT_v1x0_20121017_0_T_1[^0-3]*.dat.gz: No such file or directory
/data/real/EXPORT_v1x0_20121017_0_T_[^01]*.dat.gz: No such file or directory

Answer 1

In this case reaplce the whole gzip line to:

[ "$date/EXPORT-part-$shard-$piece" == "20121017/EXPORT-part-8-3" ] && {
    echo /data/real/EXPORT_v1x0_${date}_${shard}_T_${group}*.dat.gz
}

That should do the trick.

Edit: remove sleep to speed up the loop!