Question

I have for project to code a virus scanner by using clamAV signature database.
For increasing speed I use threads. (combo & and wait)
How my code works:

It read all files in a folder and sub-folders

function recursive_files()
{
files=$(find $folder_path -type f)
for f in $files
do
    raw_and_scan "$f" &
done
wait
}

As you can see, for each file there is a thread.

function raw_and_scan()
{
raw_test_file $1
read_signature_db_by_line $1
}

Read_signature.. read each line of signature database

    function  read_signature_db_by_line()
    {
    cat $signature_path | (while read LINE ; do
        stringtokenizer_line_db $LINE $1 $raw_file &
    done
    wait
    )  }

As you can see, for each line of DB there is a thread.

I did the double thread implementation because I saw a huge performance (using time benchmark)

When I scan 50 files with 50 lines into the DB. It works fine.

But when I scan my home folder (800 files) it doesn’t work and worse I have got a warning(cant fork() anymore) and my computer freeze, It needs to reboot.

I watch the process (htop) until 5000 tasks it works.

You can file my poject https://github.com/peondusud/Bash.antivir

At the end, I would to scan folder with a database 65000 lines.

If you have any idea to limit threads or something like that.

Thanks.

Answer 1

The fact you see a huge improvement going from a single process (not thread) into two does not mean you will go super fast using 5000 processes! Actually it is the opposite- if you plan to have processes doing intensive work you should limit it to 2* number of cpu cores in your system (this is a generic rule of thumb)