Question

I have a simple program that checks webpages for strings, example:

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
    Dim urls() As String = TextBox1.Lines()
    Dim links() As String = TextBox2.Lines()
    For Each url As String In urls
        CheckForLinks(url, links)
    Next

End Sub
Private Sub CheckForLinks(ByVal url As String, ByVal links() As String)
    Dim wc As New WebClient()
    Dim source As String = wc.DownloadString(url)
    'MessageBox.Show(source)
    For Each link As String In links
        If (source.IndexOf(link) <> -1) Then
            TextBox3.AppendText("url: " + url + " link: " + link + vbCrLf)
            Exit For
        Else
            TextBox3.AppendText("url: " + url + " link: " + "NOT FOUND" + vbCrLf)
        End If
    Next

End Sub

It works fine, but is slow, as it checks one webpage at a time.

I realize i can use a parallel.for each in the button1_click sub, but im worried that it might generate a ton of threads and overload the web connection.

I would prefer to be able to set the exact amount of threads it uses, but im not sure where to start. Eg how would i assign each url to a thread etc.

Answer 1

Well you can certainly use Parallel.ForEach and specify the maximum degree of parallelization using this overload where you can specify a ParallelOptions.

Note that you shouldn’t do this in your click handler though – your UI will block until it’s all finished. You either want to perform the Parallel.ForEach in a new thread, or start new tasks (using Task.Factory.StartNew) and rely on the parallelism limits within that. (Or create a custom task factory for the purpose. There are docs around that within the TPL documentation, I believe.)

Likewise your CheckForLinks method mustn’t try to update the UI within a non-UI thread. For WinForms, you can use Control.Invoke to get back to the UI thread – or if you’re using tasks, you could add a continuation (Task.ContinueWith) and specify a task scheduler tied to the UI synchronization context.

Frankly, doing this in a console app would be significantly simpler – just use Parallel.ForEach and you’ll be fine 🙂