Question

I have a server application for handling XMLRPC requests. Each connection creates its own thread. I would like to setup logging so that some thread/connection specific information gets in the log. I could do something like:

import logging

@add_unique_request_id
def thread_top(request_id):
    logging.info('thread %d says: hello'%request_id)
    logging.error('thread %d says: darn!!'%request_id)

and setup the global root logger as I wish, but I do not like this solution. I would like the following instead:

import logging

@setup_logger
def thread_top():
    logging.info('hello')
    logging.error('darn!!')

But I have no idea how the setup_logger deco should look like. I came up with a workaround to use separate process for each request, then setting the root logger in each process would do exactly what I want. Is there some way to make this work without being forced to use multiprocessing?

Thank you!

Answer 1

I’m doing this with a custom formatter and threadlocal storage:

from collections import defaultdict
import logging
import threading

class ContextAwareFormatter(logging.Formatter):
    """
    Makes use of get_context() to populate the record attributes.
    """

    def format(self, record):
        # Using defaultdict to avoid KeyErrorS when a key is not in the context.
        def factory():
            return ""
        record.__dict__ = defaultdict(factory, record.__dict__)

        for k, v in get_context().iteritems():
            if not hasattr(record, k):
                setattr(record, k, v)
        return logging.Formatter.format(self, record)

THREADLOCAL_ATTR = "logging_context"
_threadlocal = threading.local()

def get_context():
    result = getattr(_threadlocal, THREADLOCAL_ATTR, None)
    if result is None:
        result = {}
        setattr(_threadlocal, THREADLOCAL_ATTR, result)
    return result

def set_context(**context):
    c = get_context()
    c.clear()
    c.update(**context)
    return c

def update_context(**context):
    c = get_context()
    c.update(**context)
    return c

Then in logger configuration:

"formatters": {
    "default": {
        "()": "log.ContextAwareFormatter",
        "format": "%(asctime)s %(levelname)s [%(request_id)s] %(message)s (%(module)s:%(lineno)d)",
    },
}

Before logging the context is populated with:

update_context(request_id=request_id)

You may want to use different formatters for different parts of application where you might not need request_id in the log records.