Home/ Questions/Q 3272960
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-17T18:52:54+00:00

I will be writing a little Python script tomorrow, to retrieve all the data

  • 0

I will be writing a little Python script tomorrow, to retrieve all the data from an old MS Access database into a CSV file first, and then after some data cleansing, munging etc, I will import the data into a mySQL database on Linux.

I intend to use pyodbc to make a connection to the MS Access db. I will be running the initial script in a Windows environment.

The db has IIRC well over half a million rows of data. My questions are:

  1. Is the number of records a cause for concern? (i.e. Will I hit some limits)?
  2. Is there a better file format for the transitory data (instead of CSV)?

I chose CSv because it is quite simple and straightforward (and I am a Python newbie) – but
I would like to hear from someone who may have done something similar before.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Memory usage for csvfile.reader and csvfile.writer isn’t proportional to the number of records, as long as you iterate correctly and don’t try to load the whole file into memory. That’s one reason the iterator protocol exists. Similarly, csvfile.writer writes directly to disk; it’s not limited by available memory. You can process any number of records with these without memory limitations.

    For simple data structures, CSV is fine. It’s much easier to get fast, incremental access to CSV than more complicated formats like XML (tip: pulldom is painfully slow).

    • 0