Question

I have the following python script which takes some inputs and puts them in files depending on the sys.argv. I want to add in some duplicate entry checking… As in if a value is passed through sys.argv to be put into a file but it already exists do nothing, else print line to the file.

I was thinking of doing this with subprocess and using the systems find/grep commands (for windows/linux respectively), however I cannot get this test to work.

Any thoughts/ code welcome please.

Thanks

# Import Modules for script
import os, sys, fileinput, platform, subprocess

# Global variables
hostsFile = "hosts.txt"
hostsLookFile = "hosts.csv"
hostsURLFileLoc = "urls.conf"

# Determine platform
plat = platform.system()

if plat == "Windows":

# Define Variables based on Windows and process
    #currentDir = os.getcwd()
    currentDir = "C:\\Program Files\\Splunk\\etc\\apps\\foo\\bin"
    hostsFileLoc = currentDir + "\\" + hostsFile
    hostsLookFileLoc = currentDir + "\\..\\lookups\\" + hostsLookFile
    hostsURLFileLoc = currentDir + "\\..\\default\\" + hostsURLFileLoc
    hostIP = sys.argv[1]
    hostName = sys.argv[2]
    hostURL = sys.argv[3]
    hostMan = sys.argv[4]
    hostModel = sys.argv[5]
        hostDC = sys.argv[6]



    # Add ipAddress to the hosts file for python to process 
    with open(hostsFileLoc,'a') as hostsFilePython:

#       print "Adding ipAddress: " + hostIP + " to file for ping testing"
#       print "Adding details: " + hostIP + "," + hostName + "," + hostURL + "," + hostMan + "," + hostModel + " to file"
        hostsFilePython.write(hostIP + "\n")

    # Add all details to the lookup file for displaying on-screen and added value
    with open(hostsLookFileLoc,'a') as hostsLookFileCSV:

        hostsLookFileCSV.write(hostIP + "," + hostName + "," + hostURL + "," + hostMan + "," + hostModel + "," + hostDC +"\n")

    if hostURL != "*":  

        with open(hostsURLFileLoc,'a+') as hostsURLPython:

            hostsURLPython.write("[" + hostName + "]\n" + "ping_url = " + hostURL + "\n") 

UPDATE: I am trying a small snippet of called based on that provided by steveha, I’m having trouble with the os.rename part

>>> import os
>>> import sys
>>> in_file = "inFile.txt"
>>> out_file = "outFile.txt"
>>> dir = "C:\\Python27\\"
>>> found_in_file = False
>>> with open(in_file) as in_f, open(out_file,"w") as out_f:
...     for line in in_f:
...             if line.endswith("dax"):
...                     found_in_file = True
...     if not found_in_file:
...             out_f.write("192.168.0.199\tdax\n")
...     os.rename( os.path.join(dir, in_f), os.path.join(dir,out_f))

to which I get the following error.

Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "C:\Python27\lib\ntpath.py", line 73, in join
    elif isabs(b):
  File "C:\Python27\lib\ntpath.py", line 57, in isabs
    s = splitdrive(s)[1]
  File "C:\Python27\lib\ntpath.py", line 125, in splitdrive
    if p[1:2] == ':':
TypeError: 'file' object is not subscriptable

any thoughts?

Answer 1

It would be easier, and faster, to do the “grep” tasks in Python directly:

with open("filename") as f:
    for line in f:
        if "foo" in line:
           ... # do something to handle the case of "foo" in line

Here is a program that will add a host called “dax” to /etc/hosts:

import sys
_, in_fname, out_fname = sys.argv

found_in_file = False
with open(in_fname) as in_f, open(out_fname, "w") as out_f:
    for line in lst:
        if line.endswith("dax"):
            found_in_file = True
        out_f.write(line)
    if not found_in_file:
        out_f.write("192.168.0.199\tdax\n")

You specify two filenames, and the output filename will get a copy of /etc/hosts. If the system name “dax” is already found in /etc/hosts, the copy will be exact; otherwise a line will be appended.

You can extend this idea by using a regular expression to detect a particular line, and you can edit a line by writing a different line instead of the original.

This program uses a regular expression to find all entries in the /etc/hosts file that are in the range from 192.168.0.10 to 192.168.0.59 inclusive. Those lines are rewritten to move them to 192.168.1.* where the * is the original address, unchanged.

import re
import sys
_, in_fname, out_fname = sys.argv

pat = re.compile(r'^192.168.0.(\d+)\s+(\S+)')

with open(in_fname) as in_f, open(out_fname, "w") as out_f:
    for line in in_f:
        m = pat.search(line)
        if m:
            x = int(m.group(1))
            if 10 <= x < 60:
                line = "192.168.1." + str(x) + "\t" + m.group(2) + "\n"
        out_f.write(line)

When you have successfully written the output file, and there was no error, you can use os.rename() to rename the new file to the original filename, thus overwriting the old file. If you figure out that you didn’t need to change any lines in the old file, you can just delete the new file instead of renaming it; if you always rename, you will be updating the modification time stamp on your file even though you didn’t actually change anything.

EDIT: here is an example of running the last code sample. Say you put the code into a file called move_subnet.py, then on Linux or other *NIX you can run it like this:

python move_subnet.py /etc/hosts ./hosts_copy.txt

If you are using Windows, it will be something like this:

python move_subnet.py C:/Windows/system32/drivers/etc/hosts ./hosts_copy.txt

Note that Windows can use backslashes or forward slashes in filenames. I used forward slashes in this example.

Your output file will be hosts_copy.txt in the current directory.