Style Guides for Testing in Python

In the last decade language communities (especially JavaScript) have adopted style guides to shape individual contributions and establish clear expectations for new work. I’ve often wondered why there isn’t the same sort of interest in establishing guides for testing. The Unit testing guidelines from the Pylons project are a good counterexample and there are individuals that have pursued this idea (like this guide from Thea Flowers), but they are exceptions.

Here’s my small contribution to the topic. This post was originally published on the Safari Flow Blog on . I’ve updated the links.

Python programmers are fortunate to have a clear, reasonable style guide in PEP8. While PEP8 is widely followed by professional and amateur Python programmers, there’s no widely adopted equivalent style guide for testing in Python.

To help our own team improve code reviews and train newcomers (both to the company and to Python), I’ve started a draft style guide outlining how we write tests for Python code at Safari. It is meant to be expanded and refined over time.

Here’s the outline of the draft style guide (the complete version is available at https://github.com/safarijv/python-testing-style-guide). What would you add or remove?

Prerequisites

  • PEP8 first
  • Lint in your editor
  • Prefer imperfect tests to no tests

The practice of testing

  • Use Tox
  • Master should always pass 100% of the time
  • Manually run tests often; automatically run tests always
  • Use TDD if it makes sense
  • Pay attention to coverage, but don’t obsess over metrics
  • Remember the QA team: some things are still best tested by humans
  • Delete dead or misleading tests aggressively
  • Focus on tests during code reviews
  • Reproduce before testing
  • Always “Click the thing”: it’s not fixed until you’ve tried it like a real user, regardless of how many unit tests you wrote
  • Write regression tests whenever possible
  • Fail tests first
  • Prefer fewer asserts per test

Structure

  • Prefer small tests
  • Separate tests into different files
  • Follow import conventions

Write to be read

  • Always write a docstring
  • Use “should” in every docstring
  • Prefer descriptive variables

Expectations

  • Always bind an expected value
  • Assert the expected value, then the returned value
  • Prefer explicit expected values

Things that people will yell about

  • Never leave behind print statements: use logging

Unusual testing conventions

  • Repeat yourself

Django

  • Avoid fixtures: they’re fast, but you’ll forget to update them when the data model changes
  • Use CSS selectors to find specific nodes rather than just looking for ‘substring in undifferentiated HTML blob’
  • Prefer extremely targeted functional tests
  • Target testable HTML with CSS classes prefaced with t-, rather than using presentational classes that your front-end team might change.
  • Consider templatetags

Testing tools and techniques

  • Use mocks with care
  • Enjoy assertRaises

How to avoid traps

  • Really think about boundary values
  • Date-related functionality should test oddball cases like leap years
  • Test the obvious positive and negative cases separately
  • Introduce some randomness: pick random numbers or strings within an expected range
  • Introduce some Unicode: we like upside-down input text or users named ☃.

Resources