Question

I tried using PARSE on a PORT! and it does not work:

>> parse open %test-data.r [to end]  
** Script error: parse does not allow port! for its input argument

Of course, it works if you read the data in:

>> parse read open %test-data.r [to end]  
== true

…but it seems it would be useful to be able to use PARSE on large files without first loading them into memory.

Is there a reason why PARSE couldn’t work on a PORT! … or is it merely not implemented yet?

Answer 1

the easy answer is no we can’t…

The way parse works, it may need to roll-back to a prior part of the input string, which might in fact be the head of the complete input, when it meets the last character of the stream.

ports copy their data to a string buffer as they get their input from a port, so in fact, there is never any “prior” string for parse to roll-back to. its like quantum physics… just looking at it, its not there anymore.

But as you know in rebol… no isn’t an answer. 😉

This being said, there is a way to parse data from a port as its being grabbed, but its a bit more work.

what you do is use a buffer, and

APPEND buffer COPY/part connection amount

Depending on your data, amount could be 1 byte or 1kb, use what makes sense.

Once the new input is added to your buffer, parse it and add logic to know if you matched part of that buffer.

If something positively matched, you remove/part what matched from the buffer, and continue parsing until nothing parses.

you then repeat above until you reach the end of input.

I’ve used this in a real-time EDI tcp server which has an “always on” tcp port in order to break up a (potentially) continuous stream of input data, which actually piggy-backs messages end to end.

details

The best way to setup this system is to use /no-wait and loop until the port closes (you receive none instead of “”).

Also make sure you have a way of checking for data integrity problems (like a skipped byte, or erroneous message) when you are parsing, otherwise, you will never reach the end.

In my system, when the buffer was beyond a specific size, I tried an alternate rule which skipped bytes until a pattern might be found further down the stream. If one was found, an error was logged, the partial message stored and a alert raised for sysadmin to sort out the message.

HTH !