Question

I have a long command saved in a file.
I also have a list of input files, which I want to pass to this same command,
so I have a %s specifier at two places.

$ cat https-tcp-session.txt

rm -f /tmp/A.raw /tmp/B.raw /tmp/result.raw; \
rwfilter --sport=443 --proto=6 --pass=stdout %s | \
rwsort --fields=sIP,dIP | \
rwgroup --id-fields=sIP,dIP --summarize | \
rwfilter --input-pipe=stdin --pass=/tmp/A.raw --packets=200-; \
rwfilter --dport=443 --proto=6 --pass=stdout %s | \
rwsort --fields=sIP,dIP | \
rwgroup --id-fields=sIP,dIP --summarize | \
rwfilter --input-pipe=stdin --pass=/tmp/B.raw --packets=200-; \
rwmatch --relate=1,2 --relate=2,1 \
/tmp/A.raw /tmp/B.raw /tmp/result.raw;

Next I try it on the python REPL.

>>> cmd = open('https-tcp-session.txt').read()
>>> cmd = cmd.replace('%s', 'trace.rwf.gz')
>>> time = '/usr/bin/time -f "%e"'
>>> stmt = '%s %s'%(time, cmd)
>>> os.system(stmt)
0.01 
0

That does not seem right.
For curiosity I put in an input file, and try to run it from the shell

$ /usr/bin/time -f "%e" bash https-tcp-session.txt
17.73

I have a feeling time is only timing the first command in the block (rm)

Answer 1

As you may have suspected, it seems you are only running time on the first command in your https-tcp-session.txt file. Your code in the Python REPL is sending something like the following to bash (or whichever shell you’re using):

/usr/bin/time -f "%e" rm -f /tmp/A.raw /tmp/B.raw /tmp/result.raw; \
rwfilter --sport=443 --proto=6 --pass=stdout %s | \
rwsort --fields=sIP,dIP | \
# rest omitted

The semicolon at the end of the first line ends a command, so the shell runs time on rm only.

One possible fix would be to wrap your block of commands in a group. That way, time runs on the group as a whole.

Alternatively, you could use positional parameters. You could replace the %s symbols in your https-tcp-session.txt commands with $1, which bash interprets as the first command-line argument. You can then run https-tcp-session.txt as a shell script, passing the value to use in $1 as the command line argument, for example:

$ bash https-tcp-session.txt trace.rwf.gz

(Please note that I haven’t tested either approach.)