Question

I am trying to use the combination Django custom logger and Celery task to capture certain application log messages and dump them in DynamoDB asynchronously. I have created a Django Celery task that takes a log message and transfer it to DynamoDB asynchronously. I tried to call this celery task from my custom logger to transfer it to DynamoDB asynchronously.

However, Django custom logger does not allow me to import:

from celery.task import task, Task, PeriodicTask, periodic_task

My server crashes with the below error:

ValueError: Unable to configure handler 'custom_handler': Cannot resolve 'myApp.analytics.tasks.LogHandler': cannot import name cache

I know that Django Logger docs warns against circular imports if the custom logger file
includes settings.py but I have made sure thats not the case. But it is still giving me the same error as that of circular imports.

Am I doing something wrong or is there any other way to achieve asynchronous data transfer to DynamoDB using Django custom logger and DjCelery?

Thanks for any help.

Answer 1

I found the solution.

The problem was “If your settings.py specifies a custom handler class and the file defining that class also imports settings.py a circular import will occur.”

To resolve this we need to do the import in the method body instead of the file defining the class.

Here’s my custom LogHandler:

import logging
#Do not import settings here, as this would lead to circular import.


#This custom log handler parses the message and inserts the entry to the DynamoDB tables.
 class LogHandler(logging.Handler):
   def __init__(self):
    logging.Handler.__init__(self)
    self.report_logger = logging.getLogger('reporting')
    self.report_logger.setLevel(logging.INFO)

   def emit(self, record):
    #Submit the task to "reporting" queue to be picked up and processed by the worker lazily.
    #myApp.analytics.tasks imports celery.task
    from myApp.analytics import tasks
    tasks.push_row_to_dynamodb.apply_async(args=[record])
    return 

Hope it helps someone.