Question

I have the following join statement:

  var query = from holding in dHoldList
              join client in clientList on
                new { holding.ClientNo }
                equals new { client.ClientNo } into clj
              from cl in clj.DefaultIfEmpty()
              join matter in matterList on 
                new { holding.ClientNo } 
                equals new { matter.ClientNo } into mlj
              from ml in mlj.DefaultIfEmpty()
              join stk in stockList on
                new { holding.Sedol }
                equals new { stk.Sedol } into slj
              from sl in slj.DefaultIfEmpty()
              select
                new GeneralHoldingsReport()
                {
                  ClientNo = holding.ClientNo,
                  Depot = holding.Depot,
                  HoldingSedol = holding.Sedol,
                  Value = holding.ValueOfStock,
                  NoOfUnits = holding.QuantityHeld,
                  ClientName = (cl == null ? null : cl.ClientName),
                  CountryOfResidence = (cl == null ? null : cl.CountryOfResidence),
                  BG = (cl == null ? null : cl.BusinessGetter),
                  ClientStockValue = (ml == null ? 0 : ml.FullValueOfPortfolio),
                  StockName = (sl == null ? null : sl.R1.Trim() + " " + sl.R2.Trim())
                };


  var reportList = query.ToList();

However when it runs I am getting an out of memory exception error.

I require the dHoldList to be the main table, with all other tables being left joins into it (i.e. if data is matched in other tables relating to each record in dHoldList then return data, if not just blank.) I believe I am doing this correctly, but obviously not.

Each of these lists contains approx 300k lines, apart from the client which is only 30k, so that may be causing some of the issues here.

Answer 1

1. You are instantiating a LOT of useless anonymous types in your joins. You can write them as follows:

   join client in clientList on holding.ClientNo equals client.ClientNo into clj 
   

   Maybe this already solves your memory problem?
   This does only apply to LINQ to Objects. If your query is translated into SQL as is the case with EF or LINQ to SQL these anonymous types will not be created.

2. Think about restructuring the query to make it more readable:

   var query = 
       from holding in dHoldList 
       join client in clientList on holding.ClientNo equals client.ClientNo into clj 
       join matter in matterList on holding.ClientNo equals matter.ClientNo into mlj 
       join stk in stockList on holding.Sedol equals stk.Sedol into slj 
   
       from cl in clj.DefaultIfEmpty() 
       from ml in mlj.DefaultIfEmpty() 
       from sl in slj.DefaultIfEmpty() 
       select new GeneralHoldingsReport() 
       { 
           ClientNo = holding.ClientNo, 
           Depot = holding.Depot, 
           HoldingSedol = holding.Sedol, 
           Value = holding.ValueOfStock, 
           NoOfUnits = holding.QuantityHeld, 
           ClientName = (cl == null ? null : cl.ClientName), 
           CountryOfResidence = (cl == null ? null : cl.CountryOfResidence), 
           BG = (cl == null ? null : cl.BusinessGetter), 
           ClientStockValue = (ml == null ? 0 : ml.FullValueOfPortfolio), 
           StockName = (sl == null ? null : sl.R1.Trim() + " " + sl.R2.Trim()) 
       }; 
   

   This will not have any impact on memory footprint but it does have an impact on readability and maintainability.

---

If I would write code to achieve your goal, especially with that many objects involved, I wouldn’t use a join. I would use hash tables. This would be dramatically faster.

Something like this (untested):

var clients = clientList.ToDictionary(x => x.ClientNo);
var matters = matterList.ToDictionary(x => x.ClientNo);
var stocks = stockList.ToDictionary(x => x.Sedol);

var reportList = new List<GeneralHoldingsReport>(dHoldList.Count);
Client client;
Matter matter;
Stock stock;    

foreach(var holding in dHoldList)
{
    if(!clients.TryGetValue(holding.ClientNo, out client))
        client = null;
    if(!matters.TryGetValue(holding.ClientNo, out matter))
        matter = null;
    if(!stocks.TryGetValue(holding.Sedol, out stock))
        stock = null;
    reportList.Add(new GeneralHoldingsReport()     
                   {     
                       ClientNo = holding.ClientNo,     
                       Depot = holding.Depot,     
                       HoldingSedol = holding.Sedol,     
                       Value = holding.ValueOfStock,     
                       NoOfUnits = holding.QuantityHeld,     
                       ClientName = (client == null ? null :
                                     client.ClientName),     
                       CountryOfResidence = (client == null ? null : 
                                             client.CountryOfResidence),     
                       BG = (client == null ? null : 
                             client.BusinessGetter),     
                       ClientStockValue = (matter == null ? 0 : 
                                           matter.FullValueOfPortfolio),     
                       StockName = (stock == null ? null : 
                                    stock.R1.Trim() + " " 
                                    + stock.R2.Trim())     
                   });
} 

The usage of ToDictionary will result in a crash if there is more than one client or matter per client number and more than one stock per sedol.