The idea of doing remote rendering (typically for a video game) which is streamed to a client device is conceptually quite simple, barring obvious issues like lag for an interactive fast-paced game.
But – technically how could you do it? My understanding is that streaming video not only caches ahead of the current play-back position, but that video files are compressed by looking ahead many frames?
Are there libraries that would let you feed an arbitrary “display feed” into a serverside video-source, so that you could then play it on the client using regular Flash/HTML5 components? Avoiding the need for a custom app or bespoke browser-plugin would be a significant benefit… i.e. the client-side web-page doesn’t know it’s not a regular video.
It’s a bit like a web-cam I suppose… but I want the ‘camera’ to be ‘watching’ the output of a window rendered to on the server.
I’m targeting Windows-based servers and C++ rendering apps.
Interesting problem. There are a number of aspects to consider, in no particular order:
Encoding and streaming at the same time
The choice of container format for the rendered movie is very important. I think the main limitation is that the renderer is constrained to write the file sequentially. The reason is that the file needs to be streamed to clients, so while the renderer is writing the file there will be a web server process reading it at some potentially close distance from EOF. The renderer cannot use random access to write the movie file because any data that is already on disk might have already be sent to clients, so clearly everything that is written to disk has to be in final form.
It seems that the F4V format (successor of FLV from Adobe) fits the bill, as it can be written in streaming friendly fashion. This format is widely supported by clients, you just need to have the Flash player plugin installed. For iPhone/iPad you will need another alternative that does not involve Flash, so for iOS you can use MP4. Note that F4V derives from MP4, both are extremely alike.
Of course, the 3D engine running on the server will have to have the ability to render to F4V/MP4, and this may require a custom export plugin for your engine.
Performance
Your server must be able to render and encode frames at equal or faster speed than the intended playback frame rate. Hardware acceleration is your friend.
Latency
Efficient video encoding formats work by encoding only the differences between frames, so to decode any given frame you may need to decode a few others first. One of the most interesting aspects of modern encoding formats is that they not only encode differences from past frames, but also from future frames. This clearly increases latency, as the encoder needs to postpone encoding a frame until it receives a few frames more. It seems that to reduce latency you would need to limit the ‘future’ side of the encoding to a very short amount, and thus possibly reduce encoding efficiency and/or quality.
Client-side buffering
This is possibly a tough one if you want to avoid a custom playback plugin. Video players download streams to a buffer that is typically several seconds long, and only begin to play when the buffer is full. The idea here is that a full buffer helps ride out any network interruptions and slow downs. But unfortunately a large buffer means an increase in latency. You will need to find out how many seconds of material the client players want to have in their playback buffer, and that will determine how far ahead the server-side rendering/encoding process always needs to be. A custom playback plugin could reduce or eliminate the buffer to reduce latency, but then it will be more sensitive to network hiccups.
HTTP server support
I’m not sure how an HTTP server will like to stream a file as it is being generated by another process. I suspect this is not something that regular servers test or intend to support. There is this not very known extension to FTP called “tail-mode FTP” which basically uses the behavior you want for this. The tail-mode enabled FTP server knows the file is growing, so it makes no assumption about size and just transfers bytes as they appear in the file. The server even waits for the file to grow if it finds it is consuming the file too fast and reaches EOF. You may need a customized HTTP server that supports a similar feature.
A dedicated streaming server may be a good option here. Links of interest are the open source Darwin Streaming Server and the QuickTime Broadcaster streaming application. For the Adobe side of things there is the Adobe Streaming Server which is a commercial product. And yet another option from Microsoft, the Smooth Streaming server extension for IIS.
Interactivity
You didn’t say anything about this, but I would imagine a good application of this technology would allow the client to send input events back to the server, which will then use that to affect the contents of the movie. This effectively would be a game engine that is hosted entirely on the server with only the input and display components running on the client. Once again, this will be challenging to do with low enough latency for the application to feel responsive. Also you will now be having to encode per-client streams, as each client will be seeing a different version of the movie. Lots of problems here, a render/encoding farm might be necessary depending on the number of simultaneous clients that need to be supported. Having pre-rendered and pre-encoded chunks of animation that can be combined (in the style of the old Dragon’s Lair games) might be a good compromise solution for this type of application.