Question

I have written a shell script for automatic
1) start of hadoop services (namenode,datanode,jobtracker,tasktracker,secondary namenode),
2) dropping all tables from hive
3) again importing all tables in hive from SQL SERVER

And I am calling this shel script from java. Below is the code of Shell Script and Java Code

Shell Script:

export HADOOP_HOME=/home/hadoop/hadoop-0.20.2-cdh3u2/
export HIVE_HOME=/home/hadoop/hive-0.7.1/
export SQOOP_HOME=/home/hadoop/sqoop-1.3.0-cdh3u1/
export MSSQL_CONNECTOR_HOME=/home/hadoop/sqoop-sqlserver-1.0
export HBASE_HOME=/home/hadoop/hbase-0.90.1-cdh3u0
export ZOOKEEPER_HOME=/home/hadoop/zookeeper-3.3.1+10
export SQOOP_CONF_DIR=/home/hadoop/sqoop-1.3.0-cdh3u1/conf/

/home/hadoop/hadoop-0.20.2-cdh3u2/bin/hadoop/start-all.sh
/home/hadoop/hadoop-0.20.2-cdh3u2/bin/hadoop -rmr /user/hadoop/*

/home/hadoop/hive-0.7.1/bin/hive -e 'show tables' > TablesToDelete.txt
while read line1
do
    echo 'drop table '$line1
    /home/hadoop/hive-0.7.1/bin/hive -e 'drop table '$line1
done < TablesToDelete.txt

while read line
do
    echo $line" ------------------------------"
/home/hadoop/sqoop-1.3.0-cdh3u1/bin/sqoop-import --connect 'jdbc:sqlserver://192.168.1.1;username=abcd;password=12345;database=HadoopTest' --table line --hive-table $line  --create-hive-table --hive-import -m 1 --hive-drop-import-delims --hive-home /home/hadoop/hive-0.7.1 --verbose
done < /home/hadoop/sqoop-1.3.0-cdh3u1/bin/tables.txt

Java Code:

public class ImportTables
{

    public static void main(String arsg[])
    {
        PrintWriter pw=null;
        try
        {
            Formatter formatter = new Formatter();
            String LogFile = "Log-"+ formatter.format("%1$tm%1$td-%1$tH%1$tM%1$tS", new Date());   
            File f=new File("/home/hadoop/"+LogFile);
            FileWriter fw1=null;   
            pw=new PrintWriter(f);

            String cmd = "/home/hadoop/sqoop-1.3.0-cdh3u1/bin/TablesToImport.sh"; // this is the command to execute in the Unix shell

            // create a process for the shell
            ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
            pb.redirectErrorStream(true); // use this to capture messages sent to stderr
            Process shell = pb.start();
            InputStream shellIn = shell.getInputStream(); // this captures the output from the command
            int shellExitStatus = shell.waitFor();
            // wait for the shell to finish and get the return code
            // at this point you can process the output issued by the command

            // for instance, this reads the output and writes it to System.out:
            int c;
            while ((c = shellIn.read()) != -1)
            {
                System.out.write(c);
            }

            // close the stream
            shellIn.close();
        }
        catch(Exception e)
        {
            e.printStackTrace();
            e.printStackTrace(pw);
            pw.flush();
            System.exit(1);
        }



    }
}

But as I run the program I see nothiing on the console, and program remains in running mode.
And If I put the following code ion shell script:

/home/hadoop/hive-0.7.1/bin/hive -e 'show tables' > TablesToDelete.txt
while read line1
do
    echo 'drop table '$line1
    /home/hadoop/hive-0.7.1/bin/hive -e 'drop table '$line1
done < TablesToDelete.txt

Then the output come as:

Cannot find hadoop installation: $HADOOP_HOME must be set or hadoop must be in the path

What is the problem in my program/script? Where and How to set HADOOP_HOME and all that path in my script?

Answer 1

The call to waitFor is a blocking call, just as the name implies. It halts further execution until the process is done. But since your code is also the sink for the process’s stdout, the whole thing blocks. Just move the waitFor to after you’ve processed the script’s output.