What I'm writing now is something I should have learned many years ago as a Python developer. It can be embarrassing to recognize it, however, I've thought of sharing this with you since I know it would have helped me earlier on my career and I hope it might help you as well.
Somebody has probably written about this topic and if you're aware of a good blog post covering this similar topic please let me know. I would like to see what else I've missed.
This post might also be useful for new contributors trying to write tests for your project.
My takeawayThese are some of the things I've learned
- Make running tests easy
- We use tox to help us create a Python virtual environment, install the dependencies for the project and to execute the tests
- Here's the tox.ini I use for mozci
- Use the -s flag to not capture the output
- If your project does not print but instead it uses logging, add the pytest-capturelog plugin to py.test and it will immediately log for you
- Use --pdb to using the Python debugger upon failure
- The theory of how to mock is explained very well in "Using Mocks in Python"
- Learning where to @patch is golden. Read "Where to patch"
How I write testsThis is what I do:
- If no tests exists for a module, create the file for it
- If you're testing module.py create a test called test_module.py
- If you already have tests but want to add coverage to a function, determine what is the minimal py.test call to only call the test or set of tests
- You don't want to run all tests when you're developing
- You can read about it in "Specifying tests/selecting tests".
@patch properly and use MocksWhat I'm doing now to patch modules is the following:
- What function are you testing? (aka test subject)
- Have a look at the function you're adding tests for and list which functions it calls (aka test resources)
- Which of those test resources do you need to patch?
- To patch the test resources I use @patch + I change the return_value. You can see an example in test_buildbot_bridge.py. I use two different style of patching if you're interested
- I normally change test resources which hit the network (controlled environment) or that I can make the test execution faster
- You can have pieces of code that are shared between tests to avoid duplicating mocking code
- Determine if you need to Mock objects and function calls
The way that Mozilla CI tools is designed it begs for integration tests, however, I don't think it is worth doing beyond unit testing + mocking. The reason is that mozci might not stick around once we have fully migrated from Buildbot which was the hard part to solve.
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.