Question

In our application we need to import transaction data from paypal through an API for the users of my application and store in the database. I’ve thousands (approx 5k now) of users and it is increasing day by day.

This application is a .net windows service.

This imports data on hourly basis for all the users. At present we are importing data for users one user after the other, but sometimes what happens one user data may be so large that it takes around 5hrs to get his entire data, so we are blocking other users till this user data import is finished. This hourly import for all the other users completely gone for a toss.

To avoid this we thought of creating threads for each user import and run them every hour using windows service. Here we have a situation where we need to think about bandwidth at any point of time as all the threads will start at the same time. Is this an issue at all?

Now, I want to know whether our new implementation is right way or not? Also I want to know how it is done usually? If anyone has come across this kind of functionality then please let us know how it is done.

If my question is not clear enough please let me know I’ll provide more info.

Edit: If I send so many requests to Paypal from a single IP, how does it handle it? Any idea whether it is limiting requests per IP?

Update: Thanks for all the suggestions and feedback.

I thought of using jgauffin’s solution as it was perfect mimic of ThreadPool. But here I need some more features like changing thread limit dynamically and recursive calling of call back method.

After lot of research and analysing thread pool, I’ve decided to use SmartThreadPool which is made based on threadpool logic but with more features. It is quite good and serving my purpose perfectly.

Answer 1

I would use a queue and let’s say five threads for this. Each time a thread is is completed it will get a new user from the queue.

Example code:

public class Example
{

    public static void Main(string[] argv)
    {
        //setup
        DownloadQueue personQueue = new DownloadQueue();
        personQueue.JobTriggered += OnHandlePerson;
        personQueue.ThreadLimit = 10; //can be changed at any time and will be adjusted when a job completed (or a new one is enqueued)

        // enqueue as many persons as you like
        personQueue.Enqueue(new Person());

        Console.ReadLine();
    }

    public static void OnHandlePerson(object source, PersonEventArgs e)
    {
        //download persno here.
    }
}

public class DownloadQueue
{
    Queue<Person> _queue = new Queue<Person>();
    int _runningThreads = 0;

    public int ThreadLimit { get; set; }

    /// <summary>
    /// Enqueue a new user.
    /// </summary>
    /// <param name="person"></param>
    public void Enqueue(Person person)
    {
        lock (_queue)
        {
            _queue.Enqueue(person);
            if (_runningThreads < ThreadLimit)
                ThreadPool.QueueUserWorkItem(DownloadUser);
        }
    }

    /// <summary>
    /// Running using a ThreadPool thread.
    /// </summary>
    /// <param name="state"></param>
    private void DownloadUser(object state)
    {
        lock (_queue)
            ++_runningThreads;

        while (true)
        {
            Person person;
            lock (_queue)
            {
                if (_queue.Count == 0)
                {
                    --_runningThreads;
                    return; // nothing more in the queue. Lets exit
                }
                person = _queue.Dequeue();
            }

            JobTriggered(this, new PersonEventArgs(person));
        }
    }

    public event EventHandler<PersonEventArgs> JobTriggered = delegate { };
}


public class PersonEventArgs : EventArgs
{
    Person _person;

    public PersonEventArgs(Person person)
    {
        _person = person;
    }

    public Person Person { get { return _person; } }
}
public class Person
{
    public Person(string fName, string lName)
    {
        this.firstName = fName;
        this.lastName = lName;
    }

    public string firstName;
    public string lastName;
}