Question

I need help with multithreading in Perl.

The basic logic is to initiate 20 threads. I have one array @dataarray and I want 20 chunks of data to be passed to each thread. Say, @dataarray has 200 rows of data in it, so first 10 rows will be going to thread 1, next 10 should be sent to thread 2, so they don’t overwrite each others data and ultimately after processing thread should update return result to @outputarray at the same index position as of source @datarray.

For example: row 19(index position 18) from @dataarray was sent to thread number 2 so after processing it thread 2 should update $outputarray[18] = $processed_string.

Just need to figure out how to send from and to positions of array to a particular thread.

#!/usr/bin/perl

use strict;
use threads;
my $num_of_threads = 20;
my @threads = initThreads();
my @dataarray;

foreach(@threads)
    {
    $_ = threads->create(\&doOperation);
    }

foreach(@threads)
     {
     $_->join();
     }

sub initThreads
     {
       my @initThreads;
         for(my $i = 1;$i<=$num_of_threads;$i++)
         {
         push(@initThreads,$i);
     }
     return @initThreads;
     }

sub doOperation
    {
    # Get the thread id. Allows each thread to be identified.
    my $id = threads->tid();
    # Process something--- on array chunk
    print "Thread $id done!\n";
    # Exit the thread
    threads->exit();
    }

Answer 1

I don’t think what I said earlier about having to use threads::shared is correct. I had to go check the documentation, and I am not sure. I am never sure when it comes to threads with Perl.

Update: As it turned out, my incomplete understanding was revealed again. At the very least, you need threads::shared to be able to put the results in @output from within each thread.

#!/usr/bin/env perl

use strict; use warnings;
use threads;
use threads::shared;

use List::Util qw( sum );
use YAML;

use constant NUM_THREADS => 20;

my @output :shared;
my @data = ( ([1 .. 10]) x 200);

# note that you'll need different logic to handle left over
# chunks if @data is not evenly divisible by NUM_THREADS
my $chunk_size = @data / NUM_THREADS;

my @threads;

for my $chunk ( 1 .. NUM_THREADS ) {
    my $start = ($chunk - 1) * $chunk_size;
    push @threads, threads->create(
        \&doOperation,
        \@data,
        $start,
        ($start + $chunk_size - 1),
        \@output,
    );
}

$_->join for @threads;

print Dump \@output;

sub doOperation{
    my ($data, $start, $end, $output) = @_;

    my $id = threads->tid;

    print "Thread [$id] starting\n";

    for my $i ($start .. $end) {
        print "Thread [$id] processing row $i\n";
        $output->[$i] = sum @{ $data->[$i] };
        sleep 1 if 0.2 > rand;
    }

    print "Thread $id done!\n";

    return;
}

Output:

- 55
- 55
- 55
…
- 55
- 55
- 55
- 55