Question

Why i really need this: (simple description of problem in the bottom, under edit3)

Lets imagine, i try to create application for twitter. It has as much accounts, as i gonna add now or later (for example 10). So, for every account we have an options:

1. twite from rss;
2. follow people;
3. retwite any twites.

Every of this task should work in background and sync within other, when we talk about one account. There wouldn’t be twite and following in progress in one time (too much requests) for one of account.

Okey, this mean, i have to make just one task per account in a time. Does that mean, i have to organize one thread for each active in this moment account? But what is the thread, to tell every account what to do next (fill up queue)? One account want to twite, another just retwites and follows people:

1. (thread 1) Account A: want to twites from RSS/file all day long;
2. (thread 2) Account B: want to retwites every 30 seconds and follows people every 15 minuites;
3. (thread 3) : should fill up queue for “Account A” (make twite, make twite, make twite);
4. (thread 4) : should fill up queue for “Account B” (make retwite, make retwite (during 15 minuites, every 30 seconds) … follow people, make retwite …);

In this case, when one thread per account just make work (what’s in queue->so he did) and another just fill up queue, i could add in my functionality manual tasks (right now, all my accs, make retwite of this twite or follow this man),- in every queue i’ll add a new task, which every thread then execute as soon, as possible.

So, my goal (target), to make an application, where every account should make his work and i have an option to tell them, what to do next (change their priority for tasks).

I’d like to ask you, am i right about 2 thread per acc, about queue (maybe it should be a global queue, where threads take their tasks using account, as mark (delimiter)) and, maybe, you could advice me something about my tasks.

Edit:
And the biggest problem there, that if i make 20 accounts, but will have just 5 thread (limit)- so it shouldn’t be 1 acc->2 thread, that should be 1 thread to fill up queue (looks like global), and other for making work- but they could make works for one account (one thread make twite, other thread start to follow users), and that makes me in a corner,- can’t understand, how should i organize all this threads.

Edit2: One more problem: time of execution. Should i check in every thread time, when taken task should be executed (plan next one hour of my tasks) (now 23:55, this task gonna be executed in 23:59,- what’s next? skip? sleep?).

If i shouldn’t check time and in queue adding just tasks, that’s ready to be executed, i have again problem of one account-one task (set add twite 15 sec delay, add retwite 20 sec delay, follow people 60 sec delay)=> per minuit i’ll have 3 tasks in queue, but should execute just one by one (and i have 100 accounts, but only 5 working threads (they should sync between each other, whats account task now execute other threads?)).

Edit3 (lets make it easy):
Okey, let’s forget about twitter. I’m going to write windows desktop application, which should write in console something (let it be ‘hello from Bob’, ‘hi from Bob’, ‘bye from Bob’).

I could add much names to my program (Bob, Jack, John, Dave, Scott) and check, what phrases this names should write:

1. Bob write only ‘hello from Bob’ every 15 seconds;
2. Jack write ‘hello from Jack’ every 1 min and ‘bye from Jack’ every 5 seconds;
3. John write ‘hello’, ‘hi’, ‘bye’ + from ‘John’, 5 seconds, 10 seconds, 15
   seconds;
4. e.t.c. (let it be 20 names);

I hope, that’s okey?

So, now, when i start my application, every of this 20 names should write their phrases in console, but make it in their time and for all threads (let limit it with 10 workers thread)- should be task just one task per name (If one thread now writing ‘hello from Jack’, other shouldn’t write ‘bye from Jack’, even if this tasks should be executed in 01:42:00. Only when one task had been completed for Jack, other starts).

(+) Plus, i should have an option, to add for somebody from this names to write any phrase i want as soon, as their closest tasks been completed.

I hope, i’ll find your help.

Edit4: so:

Thx for help, now i know what i need and how to organize my application.

Finally realisation:

I’ve done my task using my own code, and solution for me looks like the best.

So, when i generate my queue for every account and put them in one global queue of application tasks, i pass it to TimerCallBack function, which check- if there are in queues any ready to be executed tasks (but check only event_elements[0]),- if it’s so, set isactive = true (for personal queue, prevent execute next event, before previous finished), and put this queue, that should be executed each first event after iterate.

Then, pass it as collection to Parallel.ForEach:

 Parallel.ForEach(readyToExec, new ParallelOptions { MaxDegreeOfParallelism = 5 }, eq =>
            {
                lock (eq)
                {
                    QueueElement exec_elem = eq.elements[0];
                    Console.WriteLine(exec_elem.timeToExecute + ": " + eq.account.Name + " " + exec_elem.EventType);
                    exec_elem.timeToExecute = DateTime.Now + exec_elem.timeDelay;
                    eq.timeLastExecute = DateTime.Now;
                    eq.elements.Sort();
                    eq.isactive = false;
                }
            });

By the way, added checking before adding events to be execute:

        foreach (var equeue in queue)
        {
            if (!equeue.isactive && equeue.elements[0].readyToExecute)
            {
                if (DateTime.Now > equeue.timeLastExecute + TimeSpan.FromSeconds(5))
                {
                    equeue.isactive = true;
                    readyToExec.Add(equeue);
                }
                else
                {
                    equeue.elements[0].timeToExecute = DateTime.Now + TimeSpan.FromSeconds(5);
                }
            }
        }

to make execution of events for one account with manual delay (for example, if in 00:00:00 write ‘Hello’, next event couldn’t be started to execute until 00:00:05, even if time of this execute 00:00:02).

So, it works for me great 😉

Answer 1

My first inclination would be to use a priority queue and a Timer. Here’s how I’d do it.

First, create a global Stopwatch to maintain the application time. You don’t want Daylight Saving Time or other time changes to mess up your timings, because that can cause you to wait too long between events or to make events happen too quickly. So, somewhere you have:

static Stopwatch ApplicationTime = Stopwatch.StartNew();

You also have an Account class that contains information about the Twitter account and, most importantly, the time that the last action for that account was taken:

class Account
{
    // information about the account goes here

    public TimeSpan LastEventTime { get; set; }
}

And an event class that defines an action that will be taken.

class AccountEvent
{
    public Account acct { get; private set; }
    public TimeSpan dueTime { get; set; }
    public ActionType action { get; private set; } // follow, re-tweet, etc.

    // constructor, etc.
}

And, finally, a priority queue that’s keyed by event time:

PriorityQueue<TimeSpan, AccountEvent> EventQueue = new Priority<TimeSpan, AccountEvent>();

Unfortunately, there isn’t a priority queue class built into the .NET runtime library. If you do a Google search you can find several. One that I wrote many years ago is at http://www.devsource.com/c/a/Languages/A-Priority-Queue-Implementation-in-C/. You can get the latest source at http://mischel.com/pubs/priqueue.zip. Note that my priority queue is not concurrent; you’ll have to wrap a concurrency layer around it. For your purposes, a simple lock before accessing the queue in any way should be sufficient.

Now, how to use all that stuff:

Let’s say you want to check an account every 30 minutes. You would create a new AccountEvent like this:

var ev = new AccountEvent(userAccount, ApplicationTime.Elapsed.AddMinutes(30), ActionType.Follow);
EventQueue.Enqueue(ev.dueTime, ev);

Now, you need something to remove items from the queue and process them. That’s where the timer comes in:

Timer eventTimer = new System.Threading.Timer(QueueProc, null, TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(-1));

That creates a timer that will fire in five seconds. It’s a one-shot timer rather than a periodic timer, which prevents us from getting overlapping timer ticks.

Now, your timer handler checks the queue for events that need to be processed:

void QueueProc(object state)
{
    while (eventQueue.Count > 0 && eventQueue.Peek().dueTime < ApplicationTime.Elapsed)
    {
        AccountEvent ev = eventQueue.Dequeue();
        // process item. See comments below.
    }
    // reset the timer so that it will fire in five seconds
    eventTimer.Change(TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(-1));
}

Because you can have multiple events for a single account, event processing is a little bit complicated, but not overly so. You have to check the current time against the account’s last event time. If the last event was too recent, then add some time to the event time and put it back into the queue with the new time.

To process an item, you can either fire an asynchronous Web request (what I would suggest), use a BackgroundWorker, or fire off a TPL Task. The key is that when the asynchronous request completes, the last thing the completion callback does is update the time for the next event and put that event back into the queue.

I’ve glossed over some detail here, but I think you can glean the basic idea: at least enough to get started.