Question

Sorry for this long post….But i have a headache from this task.

I have a mile long xml document where I need to extract a list, use distinct values, and pass for transformation to web.

I have completed the task using xslt and keys, but the effort is forcing the server to its knees.

Description:
hundreds of products in xml, all with a number of named and Id’ed cattegories, all categories with at least one subcategory with name and id.

The categories are unique with ID, all subcategories are unique WITHIN that category:

Simplified example form the huge file (left our tons of info irrelevant to the task):

<?xml version="1.0" encoding="utf-8"?>
<root>
<productlist>
<product id="1">
<name>Some Product</name>
<categorylist>
<category id="1">
<name>cat1</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
<subcat id="2">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
<category id="2">
<name>cat1</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
<category id="3">
<name>cat1</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
</categorylist>
</product>
<product id="2">
<name>Some Product</name>
<categorylist>
<category id="1">
<name>cat1</name>
<subcategories>
<subcat id="2">
<name>subcat2</name>
</subcat>
<subcat id="4">
<name>subcat4</name>
</subcat>
</subcategories>
</category>
<category id="2">
<name>cat2</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
<category id="3">
<name>cat3</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
</categorylist>
</product>
</productlist>
</root>

DESIRED RESULT:

<?xml version="1.0" encoding="utf-8"?>
<root>
<maincat id="1">
<name>cat1</name>
<subcat id="1"><name>subcat1</name></subcat>
<subcat id="2"><name>subcat2</name></subcat>
<subcat id="3"><name>subcat3</name></subcat>
</maincat>
<maincat id="2">
<name>cat2</name>
<subcat id="1"><name>differentsubcat1</name></subcat>
<subcat id="2"><name>differentsubcat2</name></subcat>
<subcat id="3"><name>differentsubcat3</name></subcat>
</maincat>
<maincat id="2">
<name>cat2</name>
<subcat id="1"><name>differentsubcat1</name></subcat>
<subcat id="2"><name>differentsubcat2</name></subcat>
<subcat id="3"><name>differentsubcat3</name></subcat>
</maincat>
</root>

(original will from 2000 products produce 10 categories with from 5 to 15 subcategories)

Things tried:

1. Xslt with keys – works fine, but pooooor performance

2. Played around with linq:

          IEnumerable<XElement> mainCats =
                   from Category1 in doc.Descendants("product").Descendants("category") select Category1;
   
               var cDoc = new XDocument(new XDeclaration("1.0", "utf-8", null), new XElement("root"));
               cDoc.Root.Add(mainCats);
               cachedCategoryDoc = cDoc.ToString();
   

   Result was a “categories only” (not distinct values of categories or subcategories)

Applied the same xlst to that, and got fairly better performance….. but still far from usable…

Can i apply some sort of magic with the linq statement to have the desired output??

A truckload of good karma goes out to the ones that can point me in det right direction..

//Steen

NOTE:

• I am not stuck on using linq/XDocument if anyone has better options
• Currently on .net 3.5, can switch to 4 if needed

Answer 1

If I understood your question corectly, here’s a LINQ atempt.

The query below parses your XML data and creates a custom type which represents a category and contains the subcategories of that element.

After parsing, the data is grouped by category Id to get distinct subcategories for each category.

var doc = XElement.Load("path to the file");
var results = doc.Descendants("category")
    .Select(cat => new
    {
        Id = cat.Attribute("id").Value,
        Name = cat.Descendants("name").First().Value,
        Subcategories = cat.Descendants("subcat")
            .Select(subcat => new
            {
                Id = subcat.Attribute("id").Value,
                Name = subcat.Descendants("name").First().Value
            })
     })
     .GroupBy(x=>x.Id)
     .Select(g=>new
     {
         Id = g.Key,
         Name = g.First().Name,
         Subcategories = g.SelectMany(x=>x.Subcategories).Distinct()
     });

From the results above you can create your document using the code below:

var cdoc = new XDocument(new XDeclaration("1.0", "utf-8", null), new XElement("root")); 
cdoc.Root.Add(
    results.Select(x=>
    {
        var element = new XElement("maincat", new XAttribute("id", x.Id));
        element.Add(new XElement("name", x.Name));
        element.Add(x.Subcategories.Select(c=>
        {
            var subcat = new XElement("subcat", new XAttribute("id", c.Id));
            subcat.Add(new XElement("name", c.Name));
            return subcat;
        }).ToArray());
        return element;
    }));