Question

I am developing a simple multi thread web crawler. I am using a sqlite database to store urls that will be scanned.I have only one handle to the database the problem is that the main thread queryes the database in order to spawn new threads.The threads are accessing the same handle but the main thread also.

I have defined critical sections for each of the threads including the main thread. But the main thread keeps executing code eaven if a thread is in a critical section too.

Here is some code :

  CreateDb;

  InitializeCriticalSection(critical);

  index := 0;

  repeat
   if threads < THREADS_MAX then
   begin
     EnterCriticalSection(critical);
     try
       sqldb.query('SELECT * FROM urls WHERE vizitat=0 AND id>' + IntToStr(index));
       urlcount:= sqldb.rowcount;

       for i:= 1 to urlcount do
       begin
         WriteLn(sqldb.Fs('adresa'));
         sqldb.next;
         index := sqldb.Fi('id');
         with TPageCrawl.Create(@threads,sqldb.Fs('adresa'),index,sqldb) do;
         if threads = THREADS_MAX then break;
       end;
       LeaveCriticalSection(critical);
     except
       LeaveCriticalSection(critical);
       Continue;
     end;
   end;

   Write(logo);
   Sleep(1000);
  until (threads = 0) and (urlcount  < 1);

Answer 1

I have defined critical sections for each of the threads

They must all use the same critical section in order to lock properly. If they all have their own critical sections then the locks are only applicable to themselves.

I assume your thread is in TPageCrawl, you can pass in the critical section in the call:

TPageCrawl.Create(critical,@threads,sqldb.Fs('adresa'),index,sqldb)

Then in your thread proc you can EnterCriticalSection() and LeaveCriticalSection() as needed.