I am attempting to optimize a process for creating a SQL report based upon our company’s financial data. The end result I am looking to achieve is a new table that contains the compiled data for the past three years (current year plus last year and the year before). I will take this data and display it in a graph with each year getting its own line (thus showing the trends, etc.)
The problems are as follows:
- The data is in two different tables
(one for current year and one for
previous years). - The data needs to be a cumulative SUM per week. Thus week two is a
sum from day one through the end of
week two while week three will be a
sum from day one through the end of
week three. - The fiscal year starts June 1, which splits a calendar week number
(thus if you operate by converting
dates to week numbers and do a GROUP
BY on the week numbers, you are
going to throw off the data because
data from the end of the year (week
23) gets put with data from the
beginning of the year (week 23). - The archive data is stored by year number, but that year number is
the same for the entire year, which
means the data stored under 2008
goes from 6/1/2008 to 5/31/2009.
Thus, you can’t use the year number
in the lookup as the year number of
the values you are trying to extract
(in case you decide to build dates
in order to uniformly look for
data).
While I am designing this so that it is run on a schedule (as a standard ETL process), I still want it to be efficient and manageable. Right now, I have a working solution. However, I am not comfortable with it. It is a cludge. I just got something to work (since I had to show the graph it is supporting to the president of our organization today). Now I need to do it right. What I have now is six update statements that write to my new table. Each statement pulls data for half a year. Two statements per year (thus six total). I do this because in my final table I have a list of months and days that I want to do the lookup based upon (6/7, 6/14, 6/21, etc.) This way each dataset is uniform and I don’t have the issue of week number rollover. I build the date off of this month and day number and add in the year number. Since the year number changes half way through (see problem #4), I go up to the end of the year with one SQL statement then pick up at the beginning of the calendar year with the next since one statement increments the year number for the date lookup and one does not.
Here is an example of my code (for those of you who might know, this is operating against a Microsoft Dynamics Great Plains database):
/* Second half of last year */
UPDATE ProfitLoss
SET ProfitLoss.monCashOneYearBack = upd.Balance
FROM ProfitLoss
INNER JOIN
(
SELECT (
SELECT (SUM(cumulative.CRDTAMNT) - SUM(cumulative.DEBITAMT)) AS Result
FROM GL30000 AS cumulative
INNER JOIN GL00100 ON cumulative.ActIndx = GL00100.ActIndx
WHERE GL00100.ActNumbr_1=10
AND GL00100.ActNumbr_2=1020
AND GL00100.ActNumbr_3=000
AND GL00100.ActNumbr_4=00
AND cumulative.HSTYEAR=2009
AND cumulative.TRXDATE <= CAST(CAST(2010 AS varchar) + '-' +
CAST(ProfitLoss.intMonthNumber AS varchar) + '-' +
CAST(ProfitLoss.intDayNumber AS varchar) AS DATETIME)
) AS Balance, intOrder, intWeekNumber
FROM ProfitLoss
WHERE intOrder > 30
) AS upd ON ProfitLoss.intWeekNumber = upd.intWeekNumber
In case you are wondering, this is just scratch code. The year number is still hard-coded. If I were to make this production code it would be a variable. The basic overview of this code is that I am doing a SELECT on my new table (ProfitLoss), doing a SUM of my credits minus a SUM of my debits from my history table based upon the date built using my new table’s month and date, then I am using that Balance data as the value to insert into the ProfitLoss table in the appropriate column (since I only operate on one column at a time, and only half of that).
All of this works, but it is terribly inefficient. What I want to know is if I am missing something obvious. Is there a better way to do this? I would love to create one statement that handled it all. I do have access to (and skill with) SSIS in case I need to bring in the big guns for a better method of ETL but I really think there is a better way of doing this using T-SQL. Any thought on what that way might be would be greatly appreciated.
Edit: The table schema that I am pulling from has only a few columns that matter. They are:
CRDTAMT, DEBITAMT, TRXDATE, and HSTYEAR (OPENYEAR for the current table)
The rest are just columns that I use for data filtering/etc. which is outside the scope of this question. Basically, the above fields hold the data I want. For example, when something is purchased on June 9th, we will get an entry here that has a zero in the CRDTAMT field, $100 in the DEBITAMT field, a TRXDATE of 6/9/2009, and a HSTYEAR of 2009. We will have multiple transactions like this. I want to collapse them down into one cumulative number each week. Thus, if this were the only transaction since June 1, we would have a zero for the week ending 6/7 and we would have a negative $100 for the week ending 6/14 and that negative $100 would carry over in 6/21 and on. This data would be one of the three years worth I would be capturing.
If you make a couple of auxiliary tables, then the problem becomes straight forward. I’ve prototyped the solution and from what information you’ve provided so far, I don’t see why this doesn’t meet your needs.
SQL containing DML and queries
Excel file used to create your weekly financial reporting structure
Cut to the chase – you wanted a simple query to replace your existing methodology. Here it is (with new schema that is defined in the attached SQL):
The attached SQL file contains sample data and queries that populate the new table schema and return the results.
If you populate the ProfitLossDaily table via nightly job, then you can run the MERGE INTO right afterward (see the attached SQL file) to always have updated data ready.
SCHEMA (in case the attached SQL goes away sometime)
Nightly Job to populate ProfitLoss
Populate WEEKS (with your financial reporting structure)
Populate #ProfitLossDaily with sample data