Question

I am using a python script to make a system call to zgrep and only print the first result using the -m1 option.

Script:

#! /usr/bin/env python2.7

import subprocess

print subprocess.check_output("zgrep -m1 'a' test.txt.gz", shell=True)

Error:

When running the script on large files (+2MB), the following error is generated.

> ./broken-zgrep.py

gzip: stdout: Broken pipe
Traceback (most recent call last):
  File "./broken-zgrep.py", line 25, in <module>
    print subprocess.check_output("zgrep -m1 'a' test.txt.gz", shell=True)
  File "/usr/intel/pkgs/python/2.7/lib/python2.7/subprocess.py", line 537, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'zgrep -m1 'a' test.txt.gz' returned non-zero exit status 2

However, if I copy the command that python complains about and run it in the shell directly, it works fine.

> zgrep -m1 'a' test.txt.gz
0000000 8c82 524d 67a4 c37d 0595 a457 b110 3192

The command’s exit status is 0 after manually running in the shell, which indicates a success. Python says the command exits with error code 2.

> echo $?
0

Here is how to make an example test file to reproduce the error. It creates a 100000 line hex file of random vaules and uses gzip to compress it.

cat /dev/urandom | hexdump | head -n 100000 | gzip > test.txt.gz

---

Seemingly irrelevant changes that will prevent the error:

• Make a smaller test file

  cat /dev/urandom | hexdump | head -n 100 | gzip > test.txt.gz

• Running without the -m1 option (warning: will spam terminal)

  print subprocess.check_output("zgrep 'a' test.txt.gz", shell=True)

• Using grep instead of zgrep on an uncompressed file

  cat /dev/urandom | hexdump | head -n 100000 > test.txt

  print subprocess.check_output("grep -m1 'a' test.txt", shell=True)

• Running the equivalent command in perl

  perl -e 'print `zgrep -m1 'a' test.txt.gz`'

---

I don’t know why the combination of python, zgrep, -m option, and large files produces this error. If any of these factors is eliminated, then there is no error.

My best guess about the cause is from reading the grep man page about the -m option.

   -m NUM, --max-count=NUM
          Stop reading a file after NUM matching lines.  If the  input  is
          standard  input  from a regular file, and NUM matching lines are
          output, grep ensures that the standard input  is  positioned  to
          just  after the last matching line before exiting, regardless of
          the presence of trailing context lines.  This enables a  calling
          process  to resume a search.  When grep stops after NUM matching
          lines, it outputs any trailing context lines.

I originally assumed that the -m option would simply cause grep to exit after finding NUM matches. But maybe there is something funny going on with grep and standard input. This still doesn’t explain why the error only occours for large compressed files though.

I ended up porting my script from python to perl to get around this problem, so there is not any immediate need for a solution. But I would really like to gain a better understanding of why this perfect storm of circumstances fails.

Answer 1

zgrep is just a shell script, it is roughly equivalent to gunzip test.txt.gz | grep -m1 'a'. The gunzip just extracts chunks and passes them on to grep. Then, when grep finds the pattern, it exits.

If gunzip hasn’t finished uncompressing the file by then, future writes to the stdout of gunzip (which is connected to the stdin of grep) will fail. This is precisely what’s happening in your case:

gzip: stdout: Broken pipe