Question

So my map reduce operation sums up a list of micro payments into a lump sum that I owe a particular user. The user_id ends up being the _id. I also store an array ids of the micro payments that need to be paid. The output goes into a permeant collection called payments.

The output looks like this for one document

{ "_id" : ObjectId("4f48855606164f4765000004"), "value" : { "payment" : "5.0", "conversions" : [ ObjectId("4f5bd23baa113e964700000e") ] } }

I’d kind of like to track these payments so I was thinking about just building a mongoid document around the payments collection. I kind of know it can be done but I haven’t really seen anyone doing it so it makes me think there must be a better way.

Also one problem with this approach is I’m making the payments every month so the _id being the user_id is going to conflict. Additionally I think there is a possible transaction problem because I need to update the micro payments to a different state so I know not to pay them ever again and what happens if one of the payments fails? These state change via state_machine if that makes any difference.

Answer 1

Should I be accessing Map Reduce output as a Mongoid Document?

Sure you can definitely do this. That’s kind of the reason the M/R is output to a collection rather than just “some file”.

Also one problem with this approach is I’m making the payments every month so the _id being the user_id is going to conflict.

So clearly, the output of your M/R is important data. Do not leave this data in a collection that could be “hammered” by a future M/R. Instead, rename the collection you have created, or run a for loop that manually appends the data to a “keeper” collection.

In the “keeper” collection change the _id to something like _id: { uid: ObjectId('...'), month: "201203" }. You may also want to “fan out” the values field into several fields. And you will need to add a field for transaction ID.

Also remember that MongoDB uses “fire & forget” writes by default. These are low safety. You have financial data, so ensure that you are following all of the best practices for availability and data safety:

• Journaling On
• Replica Sets (with secondary data center)
• Ensure that all writes to this collection/db are done with w: majority and journal: true. This will slow down DB throughput on this operation as these writes can take a few hundred milliseconds.
• Database passwords
• Non-standard MongoDB port, IP white-listing (usual DB security)

what happens if one of the payments fails?

This a non-trivial problem and way too complicated to explain here. Instead, see this document on the two-phase commit with MongoDB.

Note that two-phase commit requires MongoDB’s findAndModify command. You will have to learn how to handle this with Mongoid.