Question

I’m building a console application that have to process a bunch of document.

To stay simple, the process is :

1. for each year between X and Y, query the DB to get a list of document reference to process
2. for each of this reference, process a local file

The process method is, I think, independent and should be parallelized as soon as input args are different :

private static bool ProcessDocument(
        DocumentsDataset.DocumentsRow d,
        string langCode
)
{         
        try
        {                           
            var htmFileName = d.UniqueDocRef.Trim() + langCode + ".htm";
            var htmFullPath = Path.Combine("x:\path", htmFileName;

            missingHtmlFile = !File.Exists(htmFullPath);

            if (!missingHtmlFile)
            {
                var html = File.ReadAllText(htmFullPath);

                // ProcessHtml is quite long : it use a regex search for a list of reference
                // which are other documents, then sends the result to a custom WS

                ProcessHtml(ref html);

                File.WriteAllText(htmFullPath, html);

            }

            return true;             
        }
        catch (Exception exc)
        {
            Trace.TraceError("{0,8}Fail processing {1} : {2}","[FATAL]", d.UniqueDocRef, exc.ToString());
            return false;
        }
    }

In order to enumerate my document, I have this method :

    private static IEnumerable<DocumentsDataset.DocumentsRow> EnumerateDocuments()
    {           
        return Enumerable.Range(1990, 2020 - 1990).AsParallel().SelectMany(year => {
            return Document.FindAll((short)year).Documents;
        });
    }

Document is a business class that wrap the retrieval of documents. The output of this method is a typed dataset (I’m returning the Documents table). The method is waiting for a year and I’m sure a document can’t be returned by more than one year (year is part of the key actually).

Note the use of AsParallel() here, but I never got issue with this one.

Now, my main method is :

        var documents = EnumerateDocuments();

        var result = documents.Select(d => {
            bool success = true;
            foreach (var langCode in new string[] { "-e","-f" })
            {
                success &= ProcessDocument(d, langCode);
            }
            return new { 
                d.UniqueDocRef,
                success
            };
        });
        using (var sw = File.CreateText("summary.csv"))
        {
            sw.WriteLine("Level;UniqueDocRef");
            foreach (var item in result)
            {
                string level;
                if (!item.success) level = "[ERROR]";
                else level = "[OK]";

                sw.WriteLine(
                    "{0};{1}",
                    level,
                    item.UniqueDocRef
                    );
                //sw.WriteLine(item);
            }
        }

This method works as expected under this form. However, if I replace

        var documents = EnumerateDocuments();

by

        var documents = EnumerateDocuments().AsParrallel();

It stops to work, and I don’t understand why.

The error appears exactly here (in my process method):

File.WriteAllText(htmFullPath, html);

It tells me that the file is already opened by another program.

I don’t understand what can cause my program not to works as expected. As my documents variable is an IEnumerable returning unique values, why my process method is breaking ?

thx for advises

[Edit] Code for retrieving document :

    /// <summary>
    /// Get all documents in data store
    /// </summary>
    public static DocumentsDS FindAll(short? year)
    {
        Database db = DatabaseFactory.CreateDatabase(connStringName); // MS Entlib

        DbCommand cm = db.GetStoredProcCommand("Document_Select");
        if (year.HasValue) db.AddInParameter(cm, "Year", DbType.Int16, year.Value);

        string[] tableNames = { "Documents", "Years" };
        DocumentsDS ds = new DocumentsDS();
        db.LoadDataSet(cm, ds, tableNames);

        return ds;
    }

[Edit2] Possible source of my issue, thanks to mquander. If I wrote :

        var test = EnumerateDocuments().AsParallel().Select(d => d.UniqueDocRef);

        var testGr = test.GroupBy(d => d).Select(d => new { d.Key, Count = d.Count() }).Where(c=>c.Count>1);

        var testLst = testGr.ToList();

        Console.WriteLine(testLst.Where(x => x.Count == 1).Count());
        Console.WriteLine(testLst.Where(x => x.Count > 1).Count());

I get this result :

0
1758

Removing the AsParallel returns the same output.

Conclusion : my EnumerateDocuments have something wrong and returns twice each documents.

Have to dive here I think

This is probably my source enumeration in cause

Answer 1

Is Document.FindAll((short)year).Documents threadsafe? Because the difference between the first and the second version is that in the second (broken) version, this call is running multiple times concurrently. That could plausibly be the cause of the issue.