Question

I would like to tail multiple files concurrently using Python and send the logs to Scribe to the respective bucket

I get the buckets using:

dict(Config.items('files')).keys()

I get the files using:

dict(Config.items('files')).values()

I want to tail all the files concurrently and then push the log to Scribe in the specific bucket:

log_entry = scribe.LogEntry(category, message)

socket = TSocket.TSocket(host='localhost', port=1463)
transport = TTransport.TFramedTransport(socket)
protocol = TBinaryProtocol.TBinaryProtocol(trans=transport, strictRead=False, strictWrite=False)
client = scribe.Client(iprot=protocol, oprot=protocol)

transport.open()
result = client.Log(messages=[log_entry])
transport.close()

Answer 1

This is a really nice talk about Python generators by David Beazley, he has some very relevant examples including sample implementations of tail: http://www.slideshare.net/dabeaz/python-generator-hacking/82