Question

Python’s logging module lets modules or classes define their own loggers. And different loggers can have different handlers. Some of them may choose to log to a file, while some choose to log to, say, stdout.

Now my application uses several of these modules, each with their own loggers that have various handlers. Can I unify the logging behavior so that all logs go to a log file that I specified? In other words, is there a way to .config() all the loggers’ handlers at once, from a single place?

Answer 1

You should probably look into the Python Logging HOWTO to understand how it works.

In short, all that modules usually do is getting a logger of the form G_LOG = logging.getLogger('package.name') and sending messages to the logger: G_LOG.info('some message'), G_LOG.exception('something bad happened'). Modules should not usually configure anything.

The application that uses the modules can turn the logging on and configure the handlers based on the logger names:

• listen all messages, or
• listen only messages above a certain threshold, or
• listen messages only from loggers whose name starts with package, or
• listen messages only from loggers whose name starts woth package.name, etc

The easiest way is to configure logging through logging.basicConfig somewhere in the beginning of your application:

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(levelname)-8s %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S',
                    filename=log_file, filemode='a')

That way you will write all logging messages from all modules to the log_file.

If you need a more detailed logging strategy (put logs from different loggers to different files, or send stacktraces to a separate file), it is better to define a logging config file and configure logging using logging.config.dictConfig or logging.config.fileConfig.

P.S. I usually create two loggers as module variables:

G_LOG = logging.getLogger(__name__)
ST_LOG = logging.getLogger('stacktrace.' + __name__)

to G_LOG I send only one-line messages. To ST_LOG I duplicate important messages using ST_LOG.exception which implicitly has exc_info=True and writes the stacktrace of the current exception.

At the start of the application I load a configuration that configures two loggers (and two file handlers for them): one that receives messages that start with stacktrace and has propagate=0 (that is stacktrace messages are not visible at the top) and the root logger that handles the rest of the messages. I will not put here my complete log config files, since it is a useful home work to understand how it all works.