ThoughtSpark.org

Coming from the Java world, I'm used to easily-available code metrics as part of my build process. Ant can do it, Maven can do it and life was good. About two years ago, I realized I started using Python for a lot of things and so it was only natural to use Python for a simple web-based management tool I was prototyping. I decided to go with Django and again, life was good. As my prototype got larger, primarily due to me having to write my own model-validation framework, I began to worry about the integrity of my unit tests. I immediately Google for "Python Code Coverage" and find coverage.py. This excellent and simple script/module made code coverage instantly available but with no integration into Django's manage.py, I had to run coverage.py manually on top of the Django test cycle. Again, I turn to Google and I found a great article on exactly what I wanted: Code Coverage for Your Django Code. The only problem was that Siddhi's post assumed I wanted HTML reports, which I didn't, and his suggestion was to modify the actual Django test code, which can be very volatile. Not being 100% satisfied, I embarked on my own journey to allow for a simple and safe way to integrate coverage.py into Django.

With Django 0.97+, there is a setting available to settings.py called TEST_RUNNER. This setting allows you to write your own test runner and have Django use that test runner in place of its own. Since I can write my own test runner now, anything is possible and that is where Siddhi's post gave me an idea: Why not extend the settings.py to have the stuff I need for coverage.py and use the TEST_RUNNER setting to run my test runner instead? With that approach, I have the following samples of code:

settings.py

...
# Specify your custom test runner to use
TEST_RUNNER='sample.tests.test_runner_with_coverage'
 
# List of modules to enable for code coverage
COVERAGE_MODULES = ['sample.views', 'sample.urls',]
...

tests.py

import os, shutil, sys, unittest
 
# Look for coverage.py in __file__/lib as well as sys.path
sys.path = [os.path.join(os.path.dirname(__file__), "lib")] + sys.path
 
import coverage
from django.test.simple import run_tests as django_test_runner
 
from django.conf import settings
 
def test_runner_with_coverage(test_labels, verbosity=1, interactive=True, extra_tests=[]):
  """Custom test runner.  Follows the django.test.simple.run_tests() interface."""
  # Start code coverage before anything else if necessary
  if hasattr(settings, 'COVERAGE_MODULES') and not test_labels:
    coverage.use_cache(0) # Do not cache any of the coverage.py stuff
    coverage.start()
 
  test_results = django_test_runner(test_labels, verbosity, interactive, extra_tests)
 
  # Stop code coverage after tests have completed
  if hasattr(settings, 'COVERAGE_MODULES') and not test_labels:
    coverage.stop()
 
    # Print code metrics header
    print ''
    print '----------------------------------------------------------------------'
    print ' Unit Test Code Coverage Results'
    print '----------------------------------------------------------------------'
 
  # Report code coverage metrics
  if hasattr(settings, 'COVERAGE_MODULES') and not test_labels:
    coverage_modules = []
    for module in settings.COVERAGE_MODULES:
      coverage_modules.append(__import__(module, globals(), locals(), ['']))
 
    coverage.report(coverage_modules, show_missing=1)
 
    # Print code metrics footer
    print '----------------------------------------------------------------------'
 
  return test_results
 
# test_runner_with_coverage()

With the code above in place, now when I run 'python manage.py test', I now get get a code coverage report that looks like this:

.....
------------------------------------------------------------------
 Unit Test Code Coverage Results
------------------------------------------------------------------
Name           Stmts   Exec  Cover   Missing
--------------------------------------------
sample.urls        2      0     0%   1-3
sample.views       3      0     0%   1-5
--------------------------------------------
TOTAL              5      0     0%
------------------------------------------------------------------

As you can see, for ever module added to COVERAGE_MODULES in settings.py, you get a report telling the number of executable statements, the number executed, the percentage executed and a list of code lines not executed. (Obviously I've not written any tests.) Being able to see your code coverage is not a way to guarantee better tests but they sure let you know what code is being executed as part of running your unit tests.

The most important parts about my approach as opposed to other approaches is that my approach does the following:

• It does not require modifying Django core code, which can be volatile in the event that you're running Django from the non-released 0.97 trunk.
• It is very simple to maintain since you pretty much separate your specific needs from Django's internal needs.
• It works as part of Django meaning no extra work required on your part to get the metrics you need.
• You now have access to code coverage metrics by running the same Django test command you're use to.

In then end, you have easily integrated code coverage metrics via coverage.py into Django. I hope this information is as useful to you as it has been for me.