I’m brand new to OData, so please bear with me.
Assuming that I’m using IQueryable<> with OData, and the number of records generated by the server is large (say 10000), is it possible to start processing the entities on the client as they keep arriving (before the entire HTTP stream has been received)?
The concept I have in mind is similar to using LINQ with SqlDataReader (details), where data is fetched from SQL (by the underlying forward-only server-side cursor connected to the SqlDataReader) only as records are being enumerated by the LINQ2SQL provider.
HTTP does allow streaming. Can OData IQueryable<> take advantage of that support? If end-to-end streaming could be achieved, data would be enumerated on the server and flow (get serialised) into the HTTP stream one record at a time. As the stream is received by the client and deserialised, entities could be enumerated and keep arriving one at a time.
Is this wishful thinking?
Is it possible streaming content with WCF Data Services? Yes, it is. To do that you have to implement the streaming provider which is there to allow streaming of binary large object (BLOB) data, such as photos, videos, and documents (No Entities)
If well a big resultset is streamed from IIS, it is feed/xml/json and it is not valid until the response has finished. However, you could perform several parallel requests using $skip and $top parameters in order to split that resultset in several smaller requests and join them when they arrive.
Update:
Another option could by just create your own infraestructure for streaming over wcf ds as this guy http://blogs.msdn.com/b/tom_laird-mcconnell/archive/2010/01/18/using-ado-net-wcf-data-services-for-streaming-infinite-event-result-sets.aspx