Question

We are using an extractor application that will export data from the database to csv files. Based on some condition variable it extracts data from different tables, and for some conditions we have to use UNION ALL as the data has to be extracted from more than one table. So to satisfy the UNION ALL condition we are using nulls to match the number of columns.

Right now all the queries in the system are pre-built based on the condition variable. The problem is whenever there is change in the table projection (i.e new column added, existing column modified, column dropped) we have to manually change the code in the application.

Can you please give some suggestions how to extract the column names dynamically so that any changes in the table structure do not require change in the code?

---

My concern is the condition that decides which table to query. The variable condition is
like

• if the condition is A, then load from TableX
• if the condition is B then load from TableA and TableY.

We must know from which table we need to get data. Once we know the table it is straightforward to query the column names from the data dictionary. But there is one more condition, which is that some columns need to be excluded, and these columns are different for each table.

I am trying to solve the problem only for dynamically generating the list columns. But my manager told me to make solution on the conceptual level rather than just fixing. This is a very big system with providers and consumers constantly loading and consuming data. So he wanted solution that can be general.

So what is the best way for storing condition, tablename, excluded columns? One way is storing in database. Are there any other ways? If yes what is the best? As I have to give at least a couple of ideas before finalizing.

Thanks,

Answer 1

Ok, MNC, try this for size (paste it into a new console app):

using System;
using System.Collections.Generic;
using System.Linq;
using Test.Api;
using Test.Api.Classes;
using Test.Api.Interfaces;
using Test.Api.Models;

namespace Test.Api.Interfaces
{
    public interface ITable
    {
        int Id { get; set; }
        string Name { get; set; }
    }
}

namespace Test.Api.Models
{
    public class MemberTable : ITable
    {
        public int Id { get; set; }
        public string Name { get; set; }
    }
    public class TableWithRelations
    {
        public MemberTable Member { get; set; }
        // list to contain partnered tables
        public IList<ITable> Partner { get; set; }

        public TableWithRelations()
        {
            Member = new MemberTable();
            Partner = new List<ITable>();
        }
    }
}

namespace Test.Api.Classes
{
    public class MyClass
    {
        private readonly IList<TableWithRelations> _tables;

        public MyClass()
        {
            // tableA stuff
            var tableA = new TableWithRelations { Member = { Id = 1, Name = "A" } };
            var relatedclasses = new List<ITable>
             {
                new MemberTable
                {
                   Id = 2,
                   Name = "B"
                }
             };
            tableA.Partner = relatedclasses;


            // tableB stuff
            var tableB = new TableWithRelations { Member = { Id = 2, Name = "B" } };
            relatedclasses = new List<ITable>
             {
                new MemberTable
                {
                   Id = 3,
                   Name = "C"
                }
             };
            tableB.Partner = relatedclasses;


            // tableC stuff
            var tableC = new TableWithRelations { Member = { Id = 3, Name = "C" } };
            relatedclasses = new List<ITable>
             {
                new MemberTable
                {
                   Id = 2,
                   Name = "D"
                }
             };
            tableC.Partner = relatedclasses;


            // tableD stuff
            var tableD = new TableWithRelations { Member = { Id = 3, Name = "D" } };
            relatedclasses = new List<ITable>
             {
                new MemberTable
                {
                   Id = 1,
                   Name = "A"
                },
                new MemberTable
                {
                   Id = 2,
                   Name = "B"
                },
             };
            tableD.Partner = relatedclasses;

            // add tables to the base tables collection
            _tables = new List<TableWithRelations> { tableA, tableB, tableC, tableD };
        }

        public IList<ITable> Compare(int tableId, string tableName)
        {
            return _tables.Where(table => table.Member.Id == tableId
                            && table.Member.Name == tableName)
                        .SelectMany(table => table.Partner).ToList();
        }
    }
}

namespace Test.Api
{
    public class TestClass
    {
        private readonly MyClass _myclass;
        private readonly IList<ITable> _relatedMembers;

        public IList<ITable> RelatedMembers
        {
            get { return _relatedMembers; }
        }

        public TestClass(int id, string name)
        {
            this._myclass = new MyClass();
            // the Compare method would take your two paramters and return
            // a mathcing set of related tables that formed the related tables
            _relatedMembers = _myclass.Compare(id, name);
            // now do something wityh the resulting list
        }
    }
}

class Program
{
    static void Main(string[] args)
    {
        // change these values to suit, along with rules in MyClass
        var id = 3;
        var name = "D";
        var testClass = new TestClass(id, name);

        Console.Write(string.Format("For Table{0} on Id{1}\r\n", name, id));
        Console.Write("----------------------\r\n");
        foreach (var relatedTable in testClass.RelatedMembers)
        {
            Console.Write(string.Format("Related Table{0} on Id{1}\r\n",
                      relatedTable.Name, relatedTable.Id));
        }
        Console.Read();
    }
}

I’ll get back in a bit to see if it fits or not.