How do you organize DB layer, business logic and cross-platform API of your information management system, if uploading and processing 500000 data records in one session is a normal operation (C# .NET 3.5 + MS SQL 2005)?
I’m specifically interested in production-proven paging patterns that behave well with the concurrency, scalability and reliability.
Does anybody have any ideas, in what direction to dig?
- Open Source Projects (don’t care about the language or platform, as long as it is not Ook)
- books
- articles
- Google keywords
- forums or newsgroups
Any help would greatly appreciated!
Update:
- simple paging (i.e.: rownumber in SQL 2005) does not work, since there are a lot of concurrent changes to the database. Item, that is deleted or inserted between the page requests, automatically makes current page index invalid.
Done the implementation. I have been informed recently that one of the uploads was about 2148849 records. Tiers did successfully deal with the a couple of broken connections and dozens of deadlocks at the DB level during this upload.
In case somebody else needs some info: