Question

We have some tests that fail periodically for no reason, mainly, JUnit times out. I want to know if I can get the number of times each test has failed. With that, I can see if it is certain tests that keep have issues, or it is not tied to tricky tests and more an issue with the stability of Jenkins on that server.

Answer 1

I encountered the same problems and we made a python that can grab the failings tests in the last N builds :

# -*- coding: utf-8 -*-
#! /usr/bin/python
import urllib
import re
import sys

project = "HERE_THE_PROJECT_NAME"
jenkin_host = "http://path.to.your.jenkins/jenkins/job/%s" % project

last_build = int(re.search("%s #(\d+)" % project, urllib.urlopen(jenkin_host + "/rssAll").read()).group(1))

start_build = last_build
nb_build = 200

REG_EXP = """All Failed Tests(.*)All Tests"""
FAILURE_REG_EXP = """javascript:hideStackTrace\(([^<]*)\)"""

all_failures = {}
last_seen = {}

print "Loading %s builds starting from build number %s" % (nb_build, start_build)

build_ok = 0

for build_id in range(start_build - nb_build, start_build + 1):
    test_page = jenkin_host + "/%s/testReport/" % build_id
    sys.stdout.write(".")
    sys.stdout.flush()
    failures = set()
    for line in urllib.urlopen(test_page).readlines():
        line_piece = re.search(REG_EXP, line)
        if line_piece:
            piece = line_piece.group(1)
            match = re.search(FAILURE_REG_EXP, piece)
            while (match):
                failures.add(match.group(1))
                match = re.search(FAILURE_REG_EXP, piece[match.start():match.end()])
    if not failures:
        build_ok += 1
    for failure in failures:
        all_failures[failure] = all_failures.get(failure, 0) + 1
        last_seen[failure] = build_id
print
print "Done (found %s build OK)" % build_ok

nbs = [ x for x in list(set(all_failures.values())) if x > 1]
nbs.sort(reverse=True)

for i in nbs:
    for test, nb in all_failures.iteritems():
        if nb == i :
            print "%d : %s (last seen : %s)" % (nb, test, last_seen[test])

And I obtain :

Loading 200 builds starting from build number 11032
.........................................................................................................................................................................................................
Done (found 148 build OK)
8 : 'one failing test' (last seen : 10906)
7 : 'another-failing-test' (last seen : 11019)