Question

I understand the concepts around multithreading and using Thread Pools. One concept I am trying to figure out is how to keep track of what emails have been sent to on each thread. So imagine, each thread is responsible for pulling x number of records, iterating through those emails, applying an email template, then saving the email to a pick up directory. Obviously, I need a way to tell each thread not to pull the same data as another thread.

One solution I was thinking was to page the data, have a global variable or array to keep track of the pages already sent to, have each thread examine that variable and start from the next available page. The only issue I can think of is if the data changes, then the pages available might get out of sync.

another solution is to set a boolean value in the database to determine if an account has been emailed to or not. So, EF would pull X amount of records and update those records as being ready to email on. This way each query would only look for emails that are not ready to be emailed to.

I wanted to get some other suggestions, if possible, or expand on the solutions I provided.

Answer 1

Given that you may one day want to scale to more than one app server, memory synchronization implementations might also not be sufficient to guarantee that emails are not duplicated.

One of the simplest ways to solve is to implement a batch processing mechanism would be at the database level.

Under a Unit of Work

• Read N x records, with Pessimistic Locking (i.e. preventing concurrent reads by other threads pulling the same emails)
• Stamp these records with a batch id (or a IsProcessed Indicator)
• Return the records to your app

e.g. a Batching PROC in SQL server might look something like (Assuming table = dbo.Emails, which has a PK EmailId and a processed indicator BIT field IsProcessed):

CREATE PROC dbo.GetNextBatchOfEmails
AS
    BEGIN
        -- Identify the next N emails to be batched. UPDLOCK is to prevent another thread batching same emails
        SELECT top 100 EmailId 
        INTO #tmpBatch
            FROM dbo.Emails WITH (UPDLOCK)
            WHERE IsProcessed = 0

        -- Stamp emails as sent. Assumed that PROC is called under a UOW. The batch IS the UOW
        UPDATE e
            SET e.IsProcessed = 1
            FROM dbo.Emails e
            INNER JOIN #tmpBatch t
                on e.EmailId = t.EmailId

        -- Return the batch of emails to caller
        SELECT e.*
            FROM dbo.Emails e
            INNER JOIN #tmpBatch t
                on e.EmailId = t.EmailId
    END

Then expose the PROC as an EF Function Import mapped to your Email Entity. Under a TransactionScope ts, you can then call the EF Function Import, and send emails, and call ts.Complete() on success.