Home/ Questions/Q 8485379
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-10T20:37:38+00:00

For some reason my Flume 1.2 avro sink is being removed from the configuration

  • 0

For some reason my Flume 1.2 avro sink is being removed from the configuration as being empty. I’ve included the agent config and debug output below.

#Flume agent configuration
collector_agent.sources = tailSource1
collector_agent.sinks = avroSink1
collector_agent.channels = memChannel1

collector_agent.sources.tailSource1.type = exec
collector_agent.sources.tailSource1.command = tail -n0 -F /f/f2-traffic/logs/archive_epc.log
collector_agent.sources.tailSource1.channels = memChannel1
collector_agent.sources.tailSource1.batchSize = 5
collector_agent.sources.tailSource1.restart = true

collector_agent.sinks.avroSink1.type = avro
collector_agent.sinks.avroSink1.channels = memChannel1
collector_agent.sinks.avroSink1.hostname = 10.11.14.83
collector_agent.sinks.avroSink1.port = 4545

collector_agent.channels.memChannel1.type = memory
collector_agent.channels.memChannel1.transactionCapacity = 5



#Console log output:
bin/flume-ng agent --conf conf/ --conf-file ../test_epc_config.conf --name collector_agent -Dflume.root.logger=DEBUG,console
+ exec /usr/java/latest/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/root/flume_test/apache-flume-1.2.0/conf:/root/flume_test/apache-flume-1.2.0/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file ../test_epc_config.conf --name collector_agent
2012-09-06 11:09:26,590 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-06 11:09:26,594 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - collector_agent
2012-09-06 11:09:26,597 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-06 11:09:26,599 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-06 11:09:26,599 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 10
2012-09-06 11:09:26,600 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)] Node manager started
2012-09-06 11:09:26,601 (lifecycleSupervisor-1-2) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] Configuration provider started
2012-09-06 11:09:26,601 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:../test_epc_config.conf for changes
2012-09-06 11:09:26,601 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:../test_epc_config.conf
2012-09-06 11:09:26,606 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created context for avroSink1: type
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: avroSink1 Agent: collector_agent
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,608 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] Starting validation of configuration for agent: collector_agent, initial-configuration: AgentConfiguration[collector_agent]
SOURCES: {tailSource1={ parameters:{restart=true, command=tail -n0 -F /f/f2-traffic/logs/archive_epc.log, batchSize=5, channels=memChannel1, type=exec} }}
CHANNELS: {memChannel1={ parameters:{transactionCapacity=5, type=memory} }}
SINKS: {avroSink1={ parameters:{port=4545, hostname=10.11.14.83, channels=memChannel1, type=avro} }}

2012-09-06 11:09:26,613 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] Created channel memChannel1
2012-09-06 11:09:26,626 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] Creating sink: avroSink1 using AVRO
2012-09-06 11:09:26,628 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)] Configuration empty for: avroSink1.Removed.
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)] Post validation configuration for collector_agent
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[collector_agent]
SOURCES: {tailSource1={ parameters:{restart=true, command=tail -n0 -F /f/f2-traffic/logs/archive_epc.log, batchSize=5, channels=memChannel1, type=exec} }}
CHANNELS: {memChannel1={ parameters:{transactionCapacity=5, type=memory} }}

2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:117)] Channels:memChannel1

2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:118)] Sinks null

2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Sources tailSource1

2012-09-06 11:09:26,629 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration  for agents: [collector_agent]
2012-09-06 11:09:26,629 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:68)] Creating instance of channel memChannel1 type memory
2012-09-06 11:09:26,668 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: memChannel1, registered successfully.
2012-09-06 11:09:26,669 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel memChannel1
2012-09-06 11:09:26,669 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:74)] Creating instance of source tailSource1, type exec
2012-09-06 11:09:26,678 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:92)] Starting new configuration:{ sourceRunners:{tailSource1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource@100c62c8 }} sinkRunners:{} channels:{memChannel1=org.apache.flume.channel.MemoryChannel@1d2940b3} }
2012-09-06 11:09:26,679 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:99)] Starting Channel memChannel1
2012-09-06 11:09:26,679 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] Waiting for channel: memChannel1 to start. Sleeping for 500 ms
2012-09-06 11:09:26,680 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: CHANNEL, name: memChannel1 started
2012-09-06 11:09:27,181 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:138)] Starting Source tailSource1
2012-09-06 11:09:27,182 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:145)] Exec source starting with command:tail -n0 -F /f/f2-traffic/logs/archive_epc.log
2012-09-06 11:09:27,184 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.source.ExecSource.start(ExecSource.java:163)] Exec source started
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Typo based off of copy & paste from the Flume Users Guide on AvroSink. The 

     collector_agent.sinks.avroSink1.channels

    should actually have been

      collector_agent.sinks.avroSink1.channel

    since a Sink can only have 1 channel.

    • 0