Question

Really been struggling with this one for some time now, i have many text files with a specific format from which i need to extract all the data and file into different fields of a database. The struggle is tweaking the parameters for parsing, ensuring i get all the info correctly.

the format is shown below:

WHITESPACE HERE of unknown length.
K       PA   DETAILS 
2 4565434   i need this sentace as one DB record
2 4456788   and this one 
5 4879870   as well as this one, content will vary! 

X Max - there sometimes is a line beginning with 'Max' here which i don't need
There is a Line here that i do not need!
WHITESPACE HERE of unknown length.

The tough parts were 1) Getting rid of whitespace, and 2)defining the fields from each other, see my best attempt, below:

dict = {}
    XX = (open("XX.txt", "r")).readlines()

    for line in XX:
            if line.isspace():
            pass
        elif line.startswith('There is'):
            pass
        elif line.startswith('Max', 2):
            pass
        elif line.startswith('K'):
            pass
        else:
            for word in line.split():
                if word.startswith('4'):                    
                    tmp_PA = word
                elif word == "1" or word == "2" or word == "3" or word == "4" or word == "5":
                    tmp_K = word
                else:
                    tmp_DETAILS = word
            cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',(tmp_PA,tmp_K,tmp_DETAILS))

At the minute, i can pull the K & PA fields no problem using this, however my DETAILS is only pulling one word, i need the entire sentance, or at least 25 chars of it.

Thanks very much for reading and I hope you can help! 🙂

K

Answer 1

1. You are splitting the whole line into words. You need to split into first word, second word and the rest. Like line.split(None, 2).

2. It would probably use regular expressions. And use the oposite logic, that is if it starts with number 1 through 5, use it, otherwise pass. Like:

   pattern = re.compile(r'([12345])\s+\(d+)\s+\(.*\S)')
   f = open('XX.txt', 'r') # No calling readlines; lazy iteration is better
   for line in f:
       m = pattern.match(line)
       if m:
           cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',
               (m.group(2), m.group(1), m.group(3)))
   

Oh, and of course, you should be using prepared statement. Parsing SQL is orders of magnitude slower than executing it.