Question

I have two different log flows which I would like to create in flume. These flows contain different log messages and it would be great to handle them separately.

Topology

My configuration looks like this:

+----+    +-----+
| a0 | -> | dst |
+----+    +-----+
             ^
+----+       |
| a1 | ------+
+----+

a0 and a1 is a source, where the logs created.

dst is the destination, where the log entries should be logged (to HDFS).

Configuration

Here is the node configuration from flume shell:

[flume localhost:35873:45678] getmappings
Master has the following mappings

Physical Node                  --> Logical Node(s)

a0.domain.local                --> [a0, a0.domain.local]
a1.domain.local                --> [a1, a1.domain.local]
dst.domain.local               --> [dst.domain.local, fooSink, barSink]

Problem

When I try to create a new flume flow as it is described here, I get the following error:

[flume localhost:35873:45678] exec config a0 fooFlow rpcSource(12345) agentE2ESink("dst")
Failed to run command 'exec config a0 fooFlow rpcSource(12345) agentE2ESink("dst")' due to null
12/03/13 14:06:05 ERROR util.FlumeShell: Failed to run command 'exec config a0 fooFlow rpcSource(12345) agentE2ESink("dst")'

There is nothing in the logs, I cannot do anything with this due to null error.

Answer 1

Okay, I’ve got it, the sources and sinks need to be quoted:

[flume localhost:35873:45678] exec config a0 fooFlow 'rpcSource(12345)' 'agentE2ESink("dst")'
[id: 35] Execing command : config
Command succeeded