Question

I’ve been trying to convert this format:

2012-03-16 13:47:30.465 -0400   START  Running    Lab.script    19    on_the

The only way I could figure it out was using sed, however when I try to use subprocess in python, it won’t read the proper command and it will give me an error. Also, it’s not the right format, I’d like it to be similar to csv.

This is what I’m working with currently:

   f = open("newlogfile.csv", "w")
   p = subprocess.Popen(["sed","-e","'s/^[ ]*//g'","-e","'s/\([0-9a-zA-Z\.]*\)","*/\1;/g'","LogFile.txt"], stdout=f, stderr=f)

   f.close()

I’m pretty new to using subprocess and very new to using sed, any help would be appreciated.

Thank you in advance

UPDATE:

  fin = csv.reader(open('LogFile.txt', 'rb'),delimiter='\t')
  fout = csv.writer(open('newLogFile.csv', 'w'))

  for row in fin:
    fout.writerow(';'.join(row))

Answer 1

You should really look into the csv module for a cleaner method of making this conversion, I believe you will want code that looks something like this:

fin = csv.reader(open('LogFile.txt', 'rb'), delimiter='\t')
fout = open('newLogFile.csv', 'w')
for row in fin:
    fout.write(';'.join(row) + '\n')

The rest of this answer should help to explain why your sed method isn’t working and how to fix it.

Try changing your command list to the following:

["sed", "-e", "s/^[ ]*//g", "-e", "s/\([0-9a-zA-Z\.]*\)*/\1;/g", "LogFile.txt"]

Note that the second s/.../.../g was split into two entries in your original example, which is likely what was causing the problem. Also you do not need the single quotes because Popen will do whatever escaping is necessary to pass the entire entry in the argument list to sed as a single argument.

You should also replace stderr=f with stderr=subprocess.STDOUT, which is the correct way to send stderr to the same place as stdout.

Complete code:

f = open("newlogfile.csv", "w")
args = ["sed", "-e", "s/^[ ]*//g", "-e", "s/\([0-9a-zA-Z\.]*\)*/\1;/g", "LogFile.txt"]
p = subprocess.Popen(args, stdout=f, stderr=subprocess.STDOUT)
f.close()