Question

As a hobby/learning project, I’m writing a parser generator in Python. One of my code files is named “token.py” – which contains a couple of classes for turning plain strings into Token objects. I’ve just discovered that using the “help()” function from the console in Python raises an error for any module defined in a directory that contains a “token.py”.

Here’s a way to reproduce the error. Create a new directory with the following files:

/New Folder
  main.py
  token.py

Leave ‘token.py’ blank. In main.py, write a simple function – for example:

def test():
    pass

Then, in your Python console, import ‘main’ and call ‘help(main.test)’ – here’s what you’ll get:

C:\New Folder>python
Python 3.1.1 (r311:74483, Aug 17 2009, 17:02:12) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import main
>>> help(main.test)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python31\lib\site.py", line 428, in __call__
    import pydoc
  File "C:\Python31\lib\pydoc.py", line 55, in <module>
    import sys, imp, os, re, inspect, builtins, pkgutil
  File "C:\Python31\lib\inspect.py", line 40, in <module>
    import tokenize
  File "C:\Python31\lib\tokenize.py", line 37, in <module>
    COMMENT = N_TOKENS
NameError: name 'N_TOKENS' is not defined
>>>

If you delete the ‘token.py’ file, help() will behave normally. Both Python 3.1 and Python 2.5 exhibit this behavior.

Is this a known issue? If not, how do I report it?

EDIT:

Several comments state that this behavior isn’t a bug. The module that defines “help” imports a module called “token” from Python’s standard library. However, Python looks in the application folder before it looks in its library to find modules. In the example above, “help” tries to use my “token.py” instead of Python’s, which causes the error.

Since Python is defined to exhibit this behavior, I suppose it isn’t a bug. But why do people think that this behavior is acceptable? It implies that adding new modules to Python’s library – even without changing existing modules – could break existing applications. It also implies that programmers are expected to have memorized the names of all modules in Python’s library – how is that any less ridiculous than expecting programmers to memorize every namespace in .NET or Java? Why don’t Python applications get their own namespaces? Why aren’t Python standard library modules in their own namespace?

Answer 1

The problem is that your local token.py is being imported by help() instead of Python’s actual token.py. This will occur for any number of .py files whose names collide with built-in modules. For example, try creating a pydoc.py file in the CWD and then try help() in Python. The help() function is just a built-in Python function, so it follows the same import path as any other Python code.