Question

I’ve got 2 tables, named: csv (a csv dump), and items (primary data table) with 7M (csv dump) and 15M rows respectively. I need to update a column in items that exists in table csv.

Both tables have a commonly indexed join ID (a VARCHAR(255)).

An UPDATE query with a join on the mutual ID column (indexed) still takes multiple days to run. After researching it I believe the inefficiency is in MySQL scanning the csv table and making per-row random-access queries against the items table.

Even though there are indexes, those indexes don’t fit in memory, so the required 7M random access queries are nose diving performance.

Are there “typical” ways of addressing this kind of issue?

---

Update:

We’re basically taking multiple catalogs of “items” and storing them
in our items table (this is bit of a simplification for discussion).
Each of say 10 catalogs will have 7M items (some duplicates across catalogs that we
normalize to 1 row in our item table). We need to compare and
verify changes to those 10 catalogs daily (UPDATES w/ joins between two big
tables, or other such mechanism).

In reality we have an items table and an items_map table, but no
need to discuss that additional level of abstraction here. I’d be
happy to find a way to perform an update between the csv dump table
and an items table (given that they both have a common ID that’s
indexed in both tables). But given that the items table might have
20M rows, and the csv table might have 7M rows.

In this case indexes don’t fit in memory and we’re hammering the drive with random seeks I believe

Answer 1

Well, I finally put this query to an 8 core box with 12 GB of ram dedicated to InnoDB and it did completed but after 7 hours!

Our solution: We’re moving this process off of MySQL. We’ll use MapReduce (Hadoop) to maintain the entire large table in flat file format, do the major update processes in parallel and then finally use LOAD DATA INFILE to update the table once quickly (~daily).