I am reading the news.ycombinator RSS feed using python and storing them in a database using sqlite3. For example, a sample entry that is being fed to the database is (id,title,ur)
('3814508', 'Github is making me feel stupid(er)', 'http://www.serpentine.com/blog/2012/04/08/github-is-making-me-feel-stupider/')
Where the id is the comment id used by the website in their comment thread urls. The above list is made by individually extracting ids, title and url then zipping them. Now I wish to populate a database with such entries sans duplication,
import sqlite3 as lite
con = lite.connect('/path/to/rss.db')
con.text_factory = str
cur=con.cursor()
# --- Extract ids, links, urls ----
zipped = zip( ids, titles, targets)
cur.execute("SELECT Id FROM Posts")
existing_ids = cur.fetchall()
for i in range(0,len(zipped)):
if ids[i] not in existing_ids:
cur.executemany("INSERT INTO Posts VALUES(?, ?, ?)", zipped)
The problem is that there are thirty feeds at a time. Printing the lists show normal behaviour, with 30 entires. But when I try to write to the database there is a large number of entries, the same thirty items are repeated over and over 31 times in the table “Posts”
sqlite> SELECT Count(*) FROM Posts;
930
The db has the schema CREATE TABLE Posts(Id TEXT, Title TEXT, Target TEXT);
It looks like you’re inserting the entire list of all the entries once you find one that is missing? Perhaps you mean to iterate over each tuple of zipped, and check each one if it’s already existing?