Question

I have one table with almost 2 million records on my SQLite database, some records have null birthday and I would like to fix with data from antoher database since the ids are the same on both I can reference it for this update.

However I am having an issue for reading and I believe I will have antoher for updating it.

Reading issue, is there a way to get all the ids from the result without having to read 1 by 1 ?

Query being executed is (birthday is indexed):

SELECT id FROM personal WHERE birthday IS NULL OR birthday = ''

Currently I do this to retrieve all ids (which take something like 3 minutes for the first 300k records and then increase to something like 9 minutes for 250k records and keep growing the delay):

while (reader.Read())
{
    ids.Add(reader.GetInt32(0));
}

After I will reuse the ids list to execute an update to each record with the correct birthday which I assume I will run into the same issue, it will take a long time to get it done by inserting 1 by 1, which bring us to what I believe that will be the second issue.

Is there a fast way to update all the records with the needed data ?

NOTE: the birthdays I will read for updating the SQLITE will come from a remote MySQL server and as such I cannot link the 2 in 1 query like update select etc.

Table format is:

id
first_name
last_name
birthday
email
status

When I run the same query on SQLite Admin it takes 2371ms to spit all the data, so I assume I must be reading it wrong or reading 1 by 1 must be hurting it really bad.

Answer 1

Execute this SQL on your database. No need to iterate over all the records. It does all you need in one action:

UPDATE
    Persons
SET
    Persons.birthday = OtherTable.birthday
FROM
    Persons
INNER JOIN
    OtherTable
ON 
    Persons.ID = OtherTable.ID
WHERE 
    Persons.birthday IS NULL OR Persons.birthday = ''