SOLUTION BELOW
We have an ETL system that extracts data into a CSV, uploads it to another server, and then needs to connect to the other server and call a java jar to load the csv into memcache. I’ve got a script that can perform every step of this but loses the SSH connection for the final step. The process on the remote machine continues and completes.
I’m using Net::SSH::Perl for this and it receives a “Connection failed: Connection reset by peer” error after running for a short time. I’ve boiled the script down to this and replicated the results:
#!/usr/bin/perl
use strict;
use Net::SSH::Perl;
use Log::Log4perl;
my ($stdout, $stderr, $exit, $ssh);
$ssh = Net::SSH::Perl->new('sshost',
identity_files => ['/path/to/key.rsa'],
protocol => 2,
debug => 1);
$ssh->login('user');
my $cmd = "java -Xms4096m -Xmx4096m -DetlDate=20120427 -DmemcacheHosts=host1,host2 -cp etl-0.1-SNAPSHOT.jar com.nnn.platform.service.etl";
$ssh->register_handler("stdout", sub {
my($channel, $buffer) = @_;
print "STDOUT: ", $buffer->bytes;
});
$ssh->register_handler("stderr", sub {
my($channel, $buffer) = @_;
print "STDERR: ", $buffer->bytes;
});
$ssh->cmd("cd /usr/local/loader; $cmd");
The SSH debug info I get is:
localhost: Reading configuration data /home/user/.ssh/config
localhost: Reading configuration data /etc/ssh_config
localhost: Connecting to sshost, port 22.
localhost: Remote protocol version 2.0, remote software version OpenSSH_4.3
localhost: Net::SSH::Perl Version 1.34, protocol version 2.0.
localhost: No compat match: OpenSSH_4.3.
localhost: Connection established.
localhost: Sent key-exchange init (KEXINIT), wait response.
localhost: Algorithms, c->s: 3des-cbc hmac-sha1 none
localhost: Algorithms, s->c: 3des-cbc hmac-sha1 none
localhost: Entering Diffie-Hellman Group 1 key exchange.
localhost: Sent DH public key, waiting for reply.
localhost: Received host key, type 'ssh-dss'.
localhost: Host 'sshost' is known and matches the host key.
localhost: Computing shared secret key.
localhost: Verifying server signature.
localhost: Waiting for NEWKEYS message.
localhost: Send NEWKEYS.
localhost: Enabling encryption/MAC/compression.
localhost: Sending request for user-authentication service.
localhost: Service accepted: ssh-userauth.
localhost: Trying empty user-authentication request.
localhost: Authentication methods that can continue: publickey,gssapi-with-mic.
localhost: Next method to try is publickey.
localhost: Trying pubkey authentication with key file '/path/to/key.rsa'
localhost: Login completed, opening dummy shell channel.
localhost: channel 0: new [client-session]
localhost: Requesting channel_open for channel 0.
localhost: channel 0: open confirm rwindow 0 rmax 32768
localhost: Got channel open confirmation, requesting shell.
localhost: Requesting service shell on channel 0.
localhost: channel 1: new [client-session]
localhost: Requesting channel_open for channel 1.
localhost: Entering interactive session.
localhost: Sending command: cd /usr/local/loader; java -Xms4096m -Xmx4096m -DetlDate=20120427 -DmemcacheHosts=host1,host2 -cp etl-0.1-SNAPSHOT.jar com.nnn.platform.service.etl
localhost: Sending command: cd /usr/local/loader; java -Xms4096m -Xmx4096m -DetlDate=20120427 -DmemcacheHosts=host1,host2 -cp etl-0.1-SNAPSHOT.jar com.nnn.platform.service.etl
localhost: Requesting service exec on channel 1.
localhost: channel 1: open confirm rwindow 0 rmax 32768
The jar’s output is then printed to STDERR and I see it returned. After 9 seconds it stops and I eventually get the connection reset by peer error. The STDERR handler is working as expected.
I’m not sure if this is an issue with Net::SSH::Perl handling commands that take awhile to run/return only over STDERR or something more. I’ve been considering switching to Net::SSH2 as it seems like a fuller featured library, but I’d really like to know why this is failing.
SOLUTION
The problem was with the output only going to STDERR. I edited my command to add 2>&1 and thereby redirect STDERR to STDOUT and suddenly everything worked as expected.
The problem was with the output only going to STDERR. I edited my command to add 2>&1 and thereby redirect STDERR to STDOUT and suddenly everything worked as expected.