Question

I am new to Python and WampServer. I want to retrieve values from csv files (more than 10 GB) and, after some processing, store them in a MySQL database using Wamp efficiently.

I have installed Python and Django on Wamp server, and have checked the previous posts, but since I am a complete novice in this field, I am not getting much from them.

Can somebody please suggest appropriate resources for a beginner like me? I have already looked into Python Power! The Comprehensive Guide, but I did not get much from it. Any help would be appreciated.

Answer 1

You can read data from CSV files using the built-in csv module. For example:

>>> import csv
>>> open('test.csv','w').write('1,one\n2,two\n3,three\n')
>>> list(csv.reader(open('test.csv')))
[['1', 'one'], ['2', 'two'], ['3', 'three']]

You can write data to MySQL using the MySQL-python package, which conforms to the Python database API and is documented here. For example, after you’ve installed this package, you can do this kind of thing:

>>> import MySQLdb
>>> conn = MySQLdb.connect(host, user, passwd, db)
>>> cursor = conn.cursor()
>>> cursor.executemany('INSERT INTO numbers (value, name) VALUES (%, %)',
...                    [['1', 'one'], ['2', 'two'], ['3', 'three']])
... 
>>> cursor.commit()

However, if you’re using Django to manage the database, then you may be better off using Django’s own database interface, for example Model.save to save an instance of a model to the database.