Question

I’m trying to wrap my head around Node.js streams, not that I’m pretty new to JavaScript and node, the last languages I really got were Perl and PHP 😀

I’ve read the Buffer/Streams documentation @ nodejs.org, watched James Halliday @ LXJS, read his stream-handbook and Thorsten Lorenz event-stream post. I start to understand the basics 🙂

I process data which is serialized in RDF (which is neither JSON nor XML). I manage to fetch the data (in real code via request) and parse it into a JS object using rdfstore module.

So far I do this:

s.createReadStream('myRDFdata.ttl').pipe(serialize()).pipe(process.stdout);

Where serialize()does the job of parsing an serializing the code at the same time right now. I use through module to interface to the stream.

Now I have some more methods (not the real function declaration but I hope you get the point):

• getRecipe(parsedRDF) -> takes the parsed RDF (as a JavaScript object) and tells me how to use it
• createMeal(parsedRDF, recipe) -> takes the parsed RDF and the recipe from above and creates a new RDF object out of it
• this new object needs to get serialized and sent to the browser
• (In the real world getRecipe will have to do a user interaction in the browser)

I like the idea of chaining this together via pipes for higher flexibility when I enhance the code later. But I don’t want to serialize it to a RDF serialization every time but just send around the JS object. From what I’ve read in the documentation I could use the stringify module to get a string out of each step for piping it to the next step. But:

• does this actually make sense? In terms of do I add unnecessary overhead or is this negligible?
• I don’t see how I could give the parsedRDF to both methods with the dependency that getRecipe would have to be called first and the output is input for createMeal as well. Are there modules which help me on that?
• It might be that I have to ask the user for the final recipe selection so I might need to send stuff to the browser there to get the final answer. Can I do something like this over sockets while the pipe is “waiting”?

I hope this shows what I’m trying to do, if not I will try to give more details/rephrase.

Update: After sleeping over it I figured out some more things:

• It probably doesn’t make sense to serialize a format like RDF into something non-standard if there are official serialization formats. So instead of using stringify I will simply pass an official RDF serialization between the steps
• This does imply that I parse/serialize the objects in each step and this surely does add overhead. Question is do I care? I could extend the RDF module I use to parse from stream and serialize into one
• I can solve the problem with the dependency between getRecipe and createMeal by simply adding some information from getRecipe to parseRDF, this can be done very easily with RDF without breaking the original data model. But I would still be interested to know if I could handle dependencies like this with pipes

Answer 1

yes, It’s okay to make a stream of js objects,
you just have to remember to pipe it through something that will serialize the stream again after before writing it to IO.

I’d recomend writing a module called rdfStream that parses and serializes rdf, you would use it like this

var rdf = require('rdf-stream')

fs.createReadStream(file) //get a text stream
  .pipe(rdf.parse())      //turn it into objects 
  .pipe(transform)        //optional, do something with the objects
  .pipe(rdf.stringify())  //turn back into text
  .pipe(process.stdout)   //write to IO.

and it could also be used by other people working with rdf in node, awesome!