Question

I have several different parsers for different websites, I also have a file called shared.py which has the lxml functions for the special parsing and a base.py file which takes care of the database (saving, and others).
If something has failed (no image found) or passed (we found an image) I need to log this to a log file, for this I am using the standard logging module.

I have wrote a logger.py file where I create a Log class so I can call this in my parsers or base.py/shared.py, the file looks like this:

import logging
import os.path

__metaclass__ = type

class Log:
    def __init__(self, filename, loggername="fetchers", path="logs/", formater="%(asctime)s %(levelname)s %(message)s"):
        self.logger = logging.getLogger(loggername)
        self.hdlr = logging.FileHandler(path + os.path.splitext(filename)[0] + ".log")
        self.formater = logging.Formatter(formater)
        self.hdlr.setFormatter(self.formater)
        self.logger.addHandler(self.hdlr)
        self.logger.setLevel(logging.INFO)
    def info(self, msg):
        self.logger.info(msg)
    def warning(self, msg):
        self.logger.warning(msg)
    def error(self, msg):
        self.logger.error(msg)
    def critical(self, msg):
        self.logger.critical(msg)

if __name__ == "__main__":
    pass

The folder hierarchy looks like this;

• /Parsers
• /Parsers/base.py
• /Parsers/shared.py
• /Parsers/parserone.py
• /Parsers/parsertwo.py
• /Parsers/parsersthree.py
• /Parsers/…
• /Parsers/logs/base.log
• /Parsers/logs/shared.log
• /Parsers.logs/parserOne.log

Every parser imports the logger file (base.py and shared.py also) and they initialize the logger like this:

logger = Log(os.path.basename(__file__))
logger.warning("Something has happened..")

This works fine, it does write to the logs, but the problem is, base.py writes logs about queries and etc, and the parsers about failures and etc (the same with shared.py)
The problem is it’s writing the logs but they all look the same..

➜  logs  tail parserOne.log 
2012-10-26 16:35:21,250 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/scherpe-omzetdaling-televisiereclame/
2012-10-26 16:35:21,286 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/nominaties-mercurs-bekend2/
2012-10-26 16:35:21,322 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/traditionele-media-nog-steeds-populair-bij-jongeren/
2012-10-26 16:35:21,371 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/persgroep-blijft-sponsor-san/
2012-10-26 16:35:21,407 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/pg-overtreft-verwachtingen/
2012-10-26 16:35:21,443 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/discovery-networks-introduceert-discovery-client-productions/
2012-10-26 16:35:21,479 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/stoelendans-bij-wehkamp.nl/
2012-10-26 16:35:21,563 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/amazon-duikt-in-rode-cijfers/
2012-10-26 16:35:21,599 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/linkedin-rabobank-meest-populaire-werkgever/
2012-10-26 16:35:21,683 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/veronica-uitgeverij-wil-naar-amsterdam/

➜  logs  tail base.log 
2012-10-26 16:35:21,250 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/scherpe-omzetdaling-televisiereclame/
2012-10-26 16:35:21,286 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/nominaties-mercurs-bekend2/
2012-10-26 16:35:21,322 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/traditionele-media-nog-steeds-populair-bij-jongeren/
2012-10-26 16:35:21,371 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/persgroep-blijft-sponsor-san/
2012-10-26 16:35:21,407 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/pg-overtreft-verwachtingen/
2012-10-26 16:35:21,443 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/discovery-networks-introduceert-discovery-client-productions/
2012-10-26 16:35:21,479 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/stoelendans-bij-wehkamp.nl/
2012-10-26 16:35:21,563 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/amazon-duikt-in-rode-cijfers/
2012-10-26 16:35:21,599 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/linkedin-rabobank-meest-populaire-werkgever/
2012-10-26 16:35:21,683 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/veronica-uitgeverij-wil-naar-amsterdam/

➜  logs  tail shared.log 
2012-10-26 16:35:21,250 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/scherpe-omzetdaling-televisiereclame/
2012-10-26 16:35:21,286 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/nominaties-mercurs-bekend2/
2012-10-26 16:35:21,322 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/traditionele-media-nog-steeds-populair-bij-jongeren/
2012-10-26 16:35:21,371 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/persgroep-blijft-sponsor-san/
2012-10-26 16:35:21,407 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/pg-overtreft-verwachtingen/
2012-10-26 16:35:21,443 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/discovery-networks-introduceert-discovery-client-productions/
2012-10-26 16:35:21,479 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/stoelendans-bij-wehkamp.nl/
2012-10-26 16:35:21,563 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/amazon-duikt-in-rode-cijfers/
2012-10-26 16:35:21,599 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/linkedin-rabobank-meest-populaire-werkgever/
2012-10-26 16:35:21,683 INFO Data was sucessfuly saved: http://www.adformatie.nl/nieuws/bericht/veronica-uitgeverij-wil-naar-amsterdam/

Why are all the log files the same? If they are different files?!

Cheers.

Answer 1

It looks like its getting the logger using loggerName, and that is always set to "fetchers". So you are using the same logger everywhere which explains why the output is the same.