Question

I wrote a program in Python yesterday that checks hundreds of URLs for validity (that they won’t 404 mainly) but currently when my program runs into a URL that is not valid it throws many exceptions and then crashes. I am using urllib.urlopen to visit the links. Is there anyway for my program to ignore these exceptions and continue running but log somewhere the lines of code that caused the problem?

Answer 1

You can use the Logger.exception() method. For example :

LOGGER = logging.getLogger("my_logger_name")
for my_url in urls_to_test:
   try:
       #test your url
   except Exception:   # or BaseException if you really want to catch all python exceptions
      LOGGER.exception("something weird happened") 

That will display the traceback of the url that raise exceptions, and still continue to test the other URL.