Question

Looking to improve quality of a fairly large Python project. I am happy with the types of warnings PyLint gives me. However, they are just too numerous and hard to enforce across a large organization. Also I believe that some code is more critical/sensitive than others with respect to where the next bug may come. For example I would like to spend more time validating a library method that is used by 100 modules rather than a script that was last touched 2 years ago and may not be used in production. Also it would be interesting to know modules that are frequently updated.

Is anyone familiar with tools for Python or otherwise that help with this type of analysis?

Answer 1

You problem is similar to the one I answered over at SQA https://sqa.stackexchange.com/a/3082. This problem was associated with Java which made the tooling a bit easier, but I have a number of suggestions below.

A number of other answers suggest that there is no good runtime tools for Python. I disagree on this in several ways:

1. Coverage tools work very well
2. Based on my experience in tooling in Java, static and dynamic analysis tools in Python are weaker than in a strongly typed less dynamic language but will work more than well enough to give good heuristics for you here. Unless you use an unusually large pathological number of dynamic features (including adding and removing methods, intercepting method and property invocations, playing with import, manually modifying the namespace) – in which case any problems you have may well be associated with this dynamism…
3. Pylint picks up simpler problems, and will not detect problems with dynamic class/instance modifications and decorators – so it doesn’t matter that the metric tools don’t measure these
4. In any case, where you can usefully focus can be determined by much more than a dependency graph.

Heuristics for selecting code

I find that there are a number of different considerations for selecting code for improvement which work both individually and together. Remember that, at first, all you need to do is find a productive seam of work – you don’t need to find the absolutely worst code before you start.

Use your judgement.

After a few cycles through the codebase, you will have a huge amount of information and be much better positioned to continue your work – if indeed more needs to be done.

That said, here are my suggestions:

High value to the business: For example any code that could cost your company a lot of money. Many of these may be obvious or widely known (because they are important), or they may be detected by running the important use cases on a system with the run-time profiler enabled. I use Coverage.

Static code metrics: There are a lot of metrics, but the ones that concern us are:

• High afferent couplings. This is code that a lot of other files depends on. While I don’t have a tool that directly outputs this, snakefood is a good way to dump the dependencies directly to file, one line per dependency, each being a tuple of afferent and efferent file. I hate to say it, but computing the afferent coupling value from this file is a simple exercise left to the reader.
• High McCabe (cyclomatic) complexity: This is more complex code. PyMetrics seems to produce this measure although I have not used the tool.
• Size: You can get a surprising amount of information by viewing the size of your project using a visualiser (eg https://superuser.com/questions/8248/how-can-i-visualize-the-file-system-usage-on-windows or https://superuser.com/questions/86194/good-program-to-visualize-file-system-usage-on-mac?lq=1. Linux has KDirStat at Filelight). Large files are a good place to start as fixing one file fixes many warnings.

Note that these tools are file-based. This is probably fine-enough resolution since you mention the project is itself has hundreds of modules (files).

Changes frequently: Code that changes frequently is highly suspect. The code may:

• Historically have had many defects, and empirically may continue to do so
• Be undergoing changes from feature development (high number of revisions in your VCS)

Find areas of change using a VCS visualisation tool such as those discussed later in this answer.

Uncovered code: Code not covered by tests.

If you run (or can run) your unit tests, your other automated tests and typical user tests with coverage, take a look at the packages and files with next to no coverage. There are two logical reasons why there is no coverage:

• The code is needed (and important) but not tested at all (at least automatically). These areas are extremely high risk
• The code may be unused and is a candidate for removal.

Ask other developers

You may be surprised at the ‘smell’ metrics you can gather by having a coffee with the longer-serving developers. I bet they will be very happy if someone cleans up a dirty area of the codebase where only the bravest souls will venture.

Visibility – detecting changes over time

I am assuming that your environment has a DVCS (such as Git or Mercurial) or at least a VCS (eg SVN). I hope that you are also using an issue or bug tracker of some kind. If so, there is a huge amount of information available. It’s even better if developers have reliably checked in with comments and issue numbers. But how do you visualise it and use it?

While you can tackle the problem on a single desktop, it is probably a good idea to set up a Continuous Integration (CI) environment, perhaps using a tool like Jenkins. To keep the answer short, I will assume Jenkins from now on. Jenkins comes with a large number of plugins that really help with code analysis. I use:

• py.test with JUnit test output picked up by the JUnit test report Jenkins plugin
• Coverage with the Cobertura plugin
• SLOCCount and SLOCCount plugin
• Pylint and Violations plugin
• Apparently there is a plugin for McCabe (cyclometric) complexity for Python, although I have not used it. It certainly looks interesting.

This gives me visibility of changes over time, and I can drill in from there. For example, suppose PyLint violations start increasing in a module – I have evidence of the increase, and I know the package or file in which this is occurring, so I can find out who’s involved and go speak with them.

If you need historic data and you have just installed Jenkins, see if you can run a few manual builds that start at the beginning of the project and take a series of jumps forward in time until the present. You can choose milestone release tags (or dates) from the VCS.

Another important area, as mentioned above, is detecting the loci of changes in the code base. I have really liked Atlassian Fisheye for this. Besides being really good at searching for commit messages (eg bug id) or file contents at any point in time, it allows me to easily see metrics:

• Linecount by directory and subdirectory
• Committers at any point in time or in specific directories and/or files
• Patterns of committal, both by time and also location in the source code