Question

So a site I work on is migrating to a new data feed. The current data feed is fed over from an FTP site and presented to me in a two denomlized files. This is then pushed into two tables into a MS SQL 2005 server which the site then uses to run all the searches and such for (real estate site).

The problem is, the new data feed is normilized, there is not just one file, but 12. All of these files though have been merged into a view to mimic the old legacy tables. Its slow though because each row has to have its data demormilzed from “a,d,e” to “Range,Refrigerator,Dishwasher”. There are at maximumn 17 of these changes per a row that have to be done. So my thought was to do the transformations once, and save them into a table as static values and not through the view. This works but it takes over 40 minutes to get the data into blank tables. It takes 1 hr 45 min to update just the needed rows, and insert the new rows. The primary key here is a char(8) for everything. I have an updated date for each row as well. What do I need to look for in terms of speeding this up. What changes do I need to make to the server,filegroups, indexes, tables, etc?

Also some other info, each row is right around 7K, the server is running SQL 2005 Workgroup Edition, I have both a dev and production box so I can easily test anything. Any help would be apperciated, my goal is to get this down to like 10 to 20 minutes, not 47 minutes.

Edit,
I was asked to post the view, here is one of the files merged together. Having issues posting the other, seems it mushes them all together after the next Union all.

---

        select LN,null as [PropertyType],ST,CTGF,TYP,UD,null as [LAG],null as [LO],null as [OFFICE_PHONE],null as [AGENT_FORM_NAME],
                    null as [OFFICE_FORM_NAME],LP,HSN,CP,STR,null as [UN],CIT,ZP,ADI,AR,null as [AREA_Name],SUB,CNY,RMS,BR,BTH,MBB,CARS,dbo.listLookupToString(gar,'a,Attached,b,Detached,c,1 Car Garage,d,1.5 Car Garage,e,2 Car Garage,f,2.5 Car Garage,g,3 Car Garage,h,3+ Car Garage,m,Garage Door Opener(s) (Auto),n,Transmitter(s),o,Carport,p,Heated,q,Leased,r,Owned,s,Underground,u,None,v,Deeded Sold Separately,w,On-Site,x,Off-Site,z,Tandem') as GAR,PKN,
                    PAR,FP,BLT,LSZ,ACR,WF,BMT,MOD,TAX,TXY,ASM,MBS,dbo.listLookupToString(MBF,'c,Carpet,h,Hardwood,p,Parquet,v,Vinyl,t,Ceramic Tile,o,Other') as MBF,B2S,dbo.listLookupToString(B2F,'c,Carpet,h,Hardwood,p,Parquet,v,Vinyl,t,Ceramic Tile,o,Other') as B2F,B3S,dbo.listLookupToString(B3F,'c,Carpet,h,Hardwood,p,Parquet,v,Vinyl,t,Ceramic Tile,o,Other') as B3F,B4S,dbo.listLookupToString(B4F,'c,Carpet,h,Hardwood,p,Parquet,v,Vinyl,t,Ceramic Tile,o,Other') as B4F,LRS,dbo.listLookupToString(LRF,'c,Carpet,h,Hardwood,p,Parquet,v,Vinyl,t,Ceramic Tile,o,Other') as LRF,DRS,dbo.listLookupToString(DRF,'c,Carpet,h,Hardwood,p,Parquet,v,Vinyl,t,Ceramic Tile,o,Other') as DRF,KTS,dbo.listLookupToString(KTF,'c,Carpet,h,Hardwood,p,Parquet,v,Vinyl,t,Ceramic Tile,o,Other') as KTF,FRS,FRF,REMARKS_INTERNET,REMARKS,
                    null as [RE1],null as [RE2],null as [RE3],null as [RE4],null as [RE5],null as [RE6],null as [RE7],null as [RE8],null as [RE9],
                    null as [RE10],A1N,A1S,A1F,A2N,A2S,A2F,A3N,A3S,A3F,A4N,A4S,A4F,A5N,A5S,A5F,GS,GSD,JH,JHD,HS,HSD,OT,OSD,FEA,dbo.listLookupToString(OTR,'a,1st Floor Bedroom,b,Darkroom,c,Den/Office/Study,d,Enclosed Balcony,e,Exercise Room,f,Family Room,g,Gallery/Foyer,h,Great Room,i,In-Law Arrangement,j,Kitchen Second,k,Library,l,Loft,m,Maids Room,n,Porch - Enclosed,o,Porch - Screened,p,Recreation,q,Sitting Room,r,Sunroom/Florida Room,t,Utility/Laundry - 1st Floor,u,Utility/Laundry - 2nd Floor,v,Workshop') as OTR,STY,TPE,
                    null as [TPC],null as [TMU],null as [TMF],DIR,null as [DR1],null as [DR2],null as [DR3],null as [UFL],null as [FLN],ASF,
                    null as [PTA],null as [CAA],EXP,null as [UFE],null as [TNU],null as [UF1],null as [RM1],null as [BR1],null as [BT1],
                    null as [UF2],null as [RM2],null as [BR2],null as [BT2],null as [UF3],null as [RM3],null as [BR3],null as [BT3],null as [UF4],
                    null as [RM4],null as [BR4],null as [BT4],null as [GRI],null as [NOI],null as [AP1],null as [AP2],null as [AP3],null as [AP4],
                    null as [AZN],null as [SLN],null as [ASQ],null as [TLA],null as [RU],null as [LT],null as [AML],null as [BIM],null as [CUU],
                    null as [FMT],null as [LND],null as [LOCAT],null as [PTU],null as [GSI],null as [TAE],null as [GSA],null as [TO],
                    null as [UNT],null as [SUBTPE],null as [RP],null as [MIN],null as [MAX],null as [BSQ],null as [BAG],null as [BUT],
                    null as [APT],null as [OFC],null as [STO],null as [DID],null as [NDK],null as [CEMXF],null as [CEMXI],null as [CEMIF],
                    null as [CEMII],null as [HT],null as [GD],null as [PARK],null as [DBL],null as [MK],PHOTOCOUNT as [NBR_PHOTOS],
                    null as [SUPP_PHOTOS],SP,CLOSED_DATE
from rets_de where st<>'CLSD'

Answer 1

Can you post your current view that pulls all of this data together?

It sounds like changing your tables to actually declare the primary keys as clustered and using some well written JOIN statements would help your queries run much faster without too much work. Also, whenever running selects in SQL Server, unless you have a specific need for locking a table your looking at, be sure to run the Select with NOLOCK as this can give quite the speed boost if you’re loading data and selecting from the same set of tables at the same time. Ex:

SELECT X, Y, Z
FROM Table AS t WITH(NOLOCK)

After looking at your query, it looks like you’re probably slowing down because you’re calling the ListLookupToString function all over the place, which has to be called for each row. The better, faster way to do this is with normalized tables and JOINing across the tables. I’ve taken one of the lookups and shown how to convert it below.

Current Lookup:

dbo.listLookupToString(gar,'a,Attached,b,Detached,c,1 Car Garage,d,1.5 Car Garage,e,2 Car Garage,f,2.5 Car Garage,g,3 Car Garage,h,3+ Car Garage,m,Garage Door Opener(s) (Auto),n,Transmitter(s),o,Carport,p,Heated,q,Leased,r,Owned,s,Underground,u,None,v,Deeded Sold Separately,w,On-Site,x,Off-Site,z,Tandem') as GAR

Create the Lookup Table

CREATE TABLE GarageLookup (GarageID VARCHAR(4), GarageTypeName VARCHAR(64))
INSERT INTO GarageLookup (GarageID, GarageTypeName) VALUES('a', 'Attached')
INSERT INTO GarageLookup (GarageID, GarageTypeName) VALUES('b', 'Detached')
/* Insert all rows into GarageLookup that are in the Lookup function currently */

Use the lookup table in the SELECT instead of the lookup function

SELECT /*Everything ommitted to be brief*/
        gl.GarageTypeName
FROM rets_de AS r WITH(NOLOCK)
JOIN GarageLookup AS gl WITH(NOLOCK) ON r.gar = gl.GarageID