Question

I have 2 big CSV file with millions of rows.
Because those 2 CSVs are from MySQL, I want to merge those 2 tables into one Document in couch DB.

What is the most efficient way to do this?

My current method is:

1. import 1st CSV
2. import 2nd CSV

To prevent duplication, the program will search the Document with the key for each row. after the row is found, then the Document is updated with the columns from the 2nd CSV

The problem is, it really takes a long time to search for each row.
while importing the 2nd CSV, it updates 30 documents/second and I have about 7 million rows. rough calculation, it will take about 64 hours to complete the whole importing.

Thank You

Answer 1

Another option is simply to do what you are doing already, but use the bulk document API to get a performance boost.

For each batch of documents:

1. POST to /db/_all_docs?include_docs=true with a body like this:

   { "keys": [ "some_id_1"
             , "some_id_2"
             , "some_id_3"
             ]
   }
   

2. Build your _bulk_docs update depending on the results you get.

   • Doc already exists, you must update it: {"key":"some_id_1", "doc": {"existing":"data"}}
   • Doc does not exist, you must create it: {"key":"some_id_2", "error":"not_found"}

3. POST to /db/_bulk_docs with a body like this:

   { "docs": [ { "_id": "some_id_1"
               , "_rev": "the _rev from the previous query"
               , "existing": "data"
               , "perhaps": "some more data I merged in"
               }
             , { "_id": "some_id_2"
               , "brand": "new data, since this is the first doc creation"
               }
             ]
   }