Question

I am trying to set up a Fluentd log handler from a Django project using a logger from code:

def get_fluentd_logger(name):
    import logging
    from fluent import handler

    logging.basicConfig(level=logging.DEBUG)
    logger = logging.getLogger(name)
    logger.addHandler(handler.FluentHandler(name, host='localhost', port=24224))
    return logger

handler.FluentHandler comes from package fluent-logger and I am running fluent locally.

fluent.conf:

<source>
  type forward
  port 24224
</source>

<match **>
  type copy
  <store>
        type stdout
  </store>

  <store>
          type forward
          send_timeout 60s
          recover_wait 10s
          heartbeat_interval 1s
          phi_threshold 8
          hard_timeout 60s

          <server>
            name monitoring
            host 1.2.3.4
            port 24224
            weight 100
          </server>
  </store>
</match>

When I run this from a non-django python project it works fine, but when called from django it just does not do anything.

The question is: is there a way to see the currently installed loggers and their handlers so I can debug this situation?

[EDIT]

When done from the django settings like this:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '%(levelname)s: %(message)s'
        },
    },
    'handlers': {
       'fluentdebug':{
            'level':'DEBUG',
            'class':'fluent.handler.FluentHandler',
            'formatter': 'simple',
            'tag':'foo.bar.baz',
            'host':'localhost',
            'port':24224,
        },
    },
    'loggers': {
        'foo.bar.baz': {
            'handlers': ['fluentdebug'],
            'level': 'DEBUG',
            'propagate': True,
        },
    }
}

It does work. I however would like to be able to do this from code because foo.bar.baz can take many values and I dont want to pollute this file with 20 loggers and handlers that do exactly the same.

Maybe the real question is: Why cant I add loggers from code to logging after Django has performed it’s setup?

Answer 1

I am going to answer myself because if somebody runs in the same problem it is easier to find.
The problem seems to be with the FluentHandler. Using Brandon’s Logging tree as mentioned by @Thomas we could see that the loggers were correctly added.

However further debugging showed that if you not explicitly set the loglevel it is set to NotSet. Somehow in “ordinary” python this is ok and the record is emitted. In Django however NotSet level logs are thrown away. I don’t know if this is standard Django behaviour, but it caused a lot of headaces. Below is the code that worked:

def get_fluentd_logger(name):
    import logging
    from fluent import handler

    logging.basicConfig(level=logging.DEBUG)
    logger = logging.getLogger(name)
    logger.level = logging.INFO
    handler = handler.FluentHandler(name, host='localhost', port=24224)
    handler.level = logging.INFO
    logger.addHandler(handler)
    return logger