Question

We have a HPC node that runs some of our tasks in it. I have a task in my .net project that kicks the bcp utility on the HPC node and the output of the query that I have runs into 9 Mb.

When the HPC node runs this task the output of the query is dumped into a file and then after it dumps around 5mb of data it suddenly stops dumping any more data and this happens all the time. (Please note this isnt any data issue as its not crashing on a particular row every time). this may or may not be of significance but I dump the data into a different server which has adequate permissions set.

I have run the command with the same query directly on the hpc node and on other comps and it gives the right output.

I’m running the bcp command as follows:

var processInfo = new ProcessStartInfo(“bcp.exe”, argument)
{
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
UseShellExecute = false
};

        var proc = new Process { StartInfo = processInfo, EnableRaisingEvents = true };
        proc.Exited += new EventHandler(bcp_log);
        proc.Start();
        proc.WaitForExit();

So my code actually waits for each bcp task to run before it goes ahead as I call it multiple times.

FYI to remind you again it only fails when my o/p exceeds a certain no of bytes in this case approx 5mb.

Any help is much appreciated.

P.S: I would want to add the bcp utility is installed on all the hpc nodes

Answer 1

Hey the problem here was the HPC node was not able to redirect the output of the bcp command to the standard output. The standard output has a minimum in such cases and in our case when the no of rows in our table exceeded more than 50000 it no longer was able to redirect to the standard output. on explicitly flushing the standard output at certain time intervals you can still redirect to the standard output without any problem