Question

I am new to Python and just starting one project. I am used to use log4j in Java and I would like to log all modules and classes in Python as I do in Java.

In Java I have one log configuration file in src folder named log4j.properties like below:

log4j.rootLogger=DEBUG, Console, fileout

log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{dd/MM/yyyy HH:mm:ss} %5p [%t] (%F:%L) - %m%n

log4j.appender.fileout=org.apache.log4j.RollingFileAppender
log4j.appender.fileout.File=servidor.log
log4j.appender.fileout.layout=org.apache.log4j.PatternLayout
log4j.appender.fileout.layout.ConversionPattern=%d{dd/MM/yyyy HH:mm:ss} (%F:%L) %p %t %c - %m%n

It logs to console and a file.

In my classes, I only have to import log4j and add a static attribute to recovery the log4j logger with config loaded then all classes will be logging in console and file. The configuration file is loaded automaticaly by the name. For example:

import org.apache.log4j.Logger;

public class Main {
    public static Logger logger = Logger.getLogger(Main.class);
    public static void main(String[] args) {
        logger.info("Hello");
    }
}

Now I am having problem to setup logging in Python, I have read the docs but I couldn’t find a way to use it in many modules/classes. How could I setup Python logging in an easy way to log my modules and classes without code much in every module/class? Is it possible to reproduce same code that I have wrote in Python?

Answer 1

Actually in Python it looks pretty much similar. There are different ways to do it. I usually create a logger class which is very simple:

import os
import logging 
import settings   # alternativly from whereever import settings  

class Logger(object):

    def __init__(self, name):
        name = name.replace('.log','')
        logger = logging.getLogger('log_namespace.%s' % name)    # log_namespace can be replaced with your namespace 
        logger.setLevel(logging.DEBUG)
        if not logger.handlers:
            file_name = os.path.join(settings.LOGGING_DIR, '%s.log' % name)    # usually I keep the LOGGING_DIR defined in some global settings file
            handler = logging.FileHandler(file_name)
            formatter = logging.Formatter('%(asctime)s %(levelname)s:%(name)s %(message)s')
            handler.setFormatter(formatter)
            handler.setLevel(logging.DEBUG)
            logger.addHandler(handler)
        self._logger = logger

    def get(self):
        return self._logger

Then if I want to log something in a class or module I simply import the logger and create an instance. Passing the class name will create one file for each class. The logger can then log messages to its file via debug, info, error, etc.:

from module_where_logger_is_defined import Logger

class MyCustomClass(object):

    def __init__(self):
        self.logger = Logger(self.__class__.__name__).get()   # accessing the "private" variables for each class

    def do_something():
        ...
        self.logger.info('Hello')

    def raise_error():
        ...
        self.logger.error('some error message')

Updated answer

Over the years I changed how I am using Python logging quite a bit. Mostly based in good practices I configure the logging of the whole application once in whatever module is loaded first during startup of the application and then use individual loggers in each file. Example:

# app.py (runs when application starts)

import logging
import os.path

def main():
    logging_config = {
        'version': 1,
        'disable_existing_loggers': False,
        'formatters': {
            'standard': {
                'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
            },
        },
        'handlers': {
            'default_handler': {
                'class': 'logging.FileHandler',
                'level': 'DEBUG',
                'formatter': 'standard',
                'filename': os.path.join('logs', 'application.log'),
                'encoding': 'utf8'
            },
        },
        'loggers': {
            '': {
                'handlers': ['default_handler'],
                'level': 'DEBUG',
                'propagate': False
            }
        }
    }
    logging.config.dictConfig(logging_config)
    # start application ...

if __name__ == '__main__':
    main()

# submodule.py (any application module used later in the application)

import logging

# define top level module logger
logger = logging.getLogger(__name__)

def do_something():
    # application code ...
    logger.info('Something happended')
    # more code ...
    try:
        # something which might break
    except SomeError:
        logger.exception('Something broke')
        # handle exception
    # more code ...

The above is the recommended way of doing this. Each module defines its own logger and can easily identify based on the __name__ attribute which message was logged in which module when you inspect the logs. This removes the boilerplate from my original answer and instead uses the logging.config module from the Python standard library.