Wednesday, April 13, 2016

Improving how I write Python tests

The main focus of this post is about what I've learning about writing Python tests, using mocks and patching functions properly. This is not an exhaustive post.

What I'm writing now is something I should have learned many years ago as a Python developer. It can be embarrassing to recognize it, however, I've thought of sharing this with you since I know it would have helped me earlier on my career and I hope it might help you as well.

Somebody has probably written about this topic and if you're aware of a good blog post covering this similar topic please let me know. I would like to see what else I've missed.

Also, if you want to start a Python project from scratch or to improve your current one, I suggest you read "Open Sourcing a Python Project the Right Way". Many of the things he mentions is what I follow for mozci.

This post might also be useful for new contributors trying to write tests for your project.

My takeaway

These are some of the things I've learned

  1. Make running tests easy
    • We use tox to help us create a Python virtual environment, install the dependencies for the project and to execute the tests
    • Here's the tox.ini I use for mozci
  2. If you use py.test learn how to not capture the output
    • Use the -s flag to not capture the output
    • If your project does not print but instead it uses logging, add the pytest-capturelog plugin to py.test and it will immediately log for you
  3. If you use py.test learn how to jump into the debugger upon failures
    • Use --pdb to using the Python debugger upon failure
  4. Learn how to use @patch and Mock properly

How I write tests

This is what I do:

  • If no tests exists for a module, create the file for it
    • If you're testing module.py create a test called test_module.py
  • If you already have tests but want to add coverage to a function, determine what is the minimal py.test call to only call the test or set of tests

@patch properly and use Mocks

What I'm doing now to patch modules is the following:

  • What function are you testing? (aka test subject)
    • Have a look at the function you're adding tests for and list which functions it calls (aka test resources)
  • Which of those test resources do you need to patch?
    • To patch the test resources I use @patch + I change the return_value. You can see an example in test_buildbot_bridge.py. I use two different style of patching if you're interested
    • I normally change test resources which hit the network (controlled environment) or that I can make the test execution faster
    • You can have pieces of code that are shared between tests to avoid duplicating mocking code
  • Determine if you need to Mock objects and function calls


The way that Mozilla CI tools is designed it begs for integration tests, however, I don't think it is worth doing beyond unit testing + mocking. The reason is that mozci might not stick around once we have fully migrated from Buildbot which was the hard part to solve.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.