Question

Not sure if this is a SO, SuperUser or ServerFault question, but I just installed the FlumeNode service on my windows machine and I wanted to send logging infromation via a named pipe to the FlumeNode.

The Flume Cookbook states that we can tail the named pipe to Flume, but it seems that this requires cygwin and it’s unlikely that we’re going to install cygwin on all the Flume nodes. Here are the instructions from the cookbook:

Console sources are useful because we can pipe data into Flume
directly. The next example pipes data from a program into Flume which
then delivers it.

$ <external process> | flume node_nowatch -1 -s -n foo -c
'foo:console|agentBESink("collector");'

Ideally, you could write data to a named pipe and just have Flume read
data from a named pipe using text or tail. Unfortunately, this version
of Flume’s text and tail are not currently compatible with named pipes
in a Linux environment. However, you could pipe data to a Flume node
listening on the stdin console:

$ tail -f namedpipe | flume node_nowatch -1 -s -n foo -c
'foo:console|agentBESink;'

Or you can use the exec source to get its output data:

$ flume node_nowatch -1 -s -n bar -c 'bar:exec("cat
pipe")|agentBESink;'

I also found a discussion on google groups regarding this idea and it seems that the Claudera people say it’s possible, but there is nothing specific explaining how Flume would connect to a pipe and if it acts as the server or a client in that case.

Ideally I should be able to create a NamedPipeClientSteram or a NamedPipeServerStream in my application. The client requires a named pipe server to be running and the clientlooks for clients which connect to it. I assume that the Flume will be acting as a server, but I can’t confirm this. E.g.

NamedPipeClientStream pipeClient = new NamedPipeClientStream(".", "FlumeNamedPipe", PipeDirection.Out, pIpeOptions.None);
pipeClient.Connect();

Is there something that can be added to the command line arguments, when starting the Flume service in Windows, in order to tail a pipe?

Answer 1

I got a response from Claudera and they state that since windows does not have a built-in tail program, it’s required that one uses Cygwin for such functionality. We don’t want to install cygwin, so I’m looking for an alternative solution.

According to Claudera, the alternative is to use C# Thrift bindings which would send the data a Flume agent with a Thrift source on the same Windows box. I’m not exactly sure how you specify a Thrift source for a Flume agent, but supposedly it can be done.