Learn the concepts of software testing and implement your own tests using the pytest library

Sam Thurman

Dec 17, 2020·9 min read

Overview

If you’re already familiar with the concepts of testing and terminology as it relates to software, absolutely skip ahead to a more relevant section

• Manual vs Automated Testing
• Unit vs Integration Testing
• Quick Intro to Python’s assert
• Test Runners
• Introducing Our Test Runner Selection: pytest
• Working With pytest

Manual vs Automated Testing

TLDR version:

• Manual testing involves testing features by hand, providing input, and observing output. It’s tedious
• Automated testing is the execution of your tests by a script or scripts. It’s much faster and more robust than manual testing.

— — — — — — — — — — — —

Manual

You’ve probably already performed tests without even realizing it. Opening up your application to see if things are functioning properly/in the right place, experimenting with your features to see if they perform as expected, these are both forms of exploratory testing.

Exploratory testing is a form of manual testing that is done “without a plan”. You’re simply exploring the application and its functionality. Are the components formatted correctly? What about the features, are they returning an expected output?

Manual testing involves testing each individual feature by hand, providing input to features, and observing the output. As you can imagine — or maybe you’ve experienced — manual testing can be an extremely tedious process. Entering test cases/edge cases by hand for each feature and visually inspecting the results is slow and unreliable. Not to mention that if you make a change to your code, you have to go back and redo every manual test to ensure everything is still functioning as designed

Automated

Automated testing is the execution of your test plan (code/features you want to test, the order to test them in, and expected outputs) by a script instead of manually. Automated tests are designed to simply test if a specific function or series of functions is producing the expected outputs, and it does this for every test in your test suite (your collection of test cases). This requires you to think of test and edge cases in advance but allows for a much quicker and cleaner execution of your test plan.

Python comes pre-built with a set of tools and libraries to help with automated testing. More on this in a bit.

Unit vs Integration Testing

TLDR version:

• Integration testing checks that components in an application operate with each other correctly
• Unit tests check that a single component of an application is providing the expected output

— — — — — — — — — — — —

Now that we’re familiar with the different types of tests and what those might look like, let’s briefly talk about scope.

Think about testing the audio on your stereo and speakers at home. Your test plan might look like:

1. Turn on stereo (test step)
2. Press play on device/drop needle on vinyl etc. (test step)
3. Is music coming out of speakers? (test assertion)

Testing multiple components is called integration testing. The problem with this kind of testing is that if an integration test fails, it can be very difficult to diagnose which component is failing.

Going off of our home audio scenario, if no music is playing, is it because the stereo itself is broken? Are there some cables or wires that are disconnected? Is our audio source not working properly?

To narrow down our possible malfunctions, we might try unplugging our phone and checking if audio is playing directly from the device. This is a form of unit testing.

A unit test is a smaller test that checks if a single component is operating in the right way.

Quick Intro to Python’s assert

You can write both integration and unit tests in Python. Tests are based on conditional logic — we are checking the output of some function or process against a known (expected) output.

For unit testing in Python, we use the assert command.

For example, if we wanted to test if a given input is of the correct datatype…

input = [‘x’, ‘y’, ‘z’]assert type(input) == ‘list’, ‘Should be a list’

This wouldn’t return any output because our test passes.

What’s the code doing?

• assert to state that we want to test a specific process.
• type(input) == ‘list’ is our conditional logic. If this line of code evaluates to true, our test passes.
• , ‘Should be a list’ is our custom AssertionError message. If our test fails, this message will be shown in the output.

If we ran the same test again, this time with an input we know would fail…

input = 'list'assert type(input) == 'list', 'Should be a list'

an AssertionError would be raised with the provided message…

Traceback (most recent call last):
  File "main.py", line 3, in <module>
    assert type(input) == 'list', 'Should be a list'
AssertionError: Should be a list

Test Runners

A test runner is the execution unit of the invocation flow. This is where the tests actually run. (src: source.andrioid.com)

This definition for test runners is accurate, but a little incomplete and difficult to make sense of if you aren’t familiar with testing or test runners already.

A test runner — sometimes called a testing framework — is the tool or library that hooks into your test’s assertions and analyzes failures so that it can provide more context when an AssertionError is raised. Test runners scan the assembly (source code directory) that contains your unit tests and settings, executes them, and finally writes the test results to either the console or log files.

There are a myriad of test runners available for Python. Some examples are…

• unittest
• nose/nose2
• pry
• pytest
• many more…

Most functional tests follow the Arrange-Act-Assert model:

1. Arrange, or set up, the conditions for the test
2. Act by calling some function or method
3. Assert that some end condition is true

unittest is a testing framework that comes with Python and provides you with a number of great out-of-the-box assertion utilities. Unfortunately, the downside to unittest and frameworks like it is that even a small number of tests requires a decent amount of boilerplate code (lots of code for act and arrange steps).

For example, if we simply wanted to makes sure our unittest framework is functioning properly, we could write two tests: one that always passes, and one that always fails.

We could achieve this using unittest with the following code…

# filename test_using_unitest.pyfrom unittest import TestCase
class TryTesting(TestCase):   def test_always_passes(self):      self.assertTrue(True)   def test_always_fails(self):      self.assertTrue(False)

We would then execute the tests in the command line using…

$ python -m unittest discover

Which would give us this output, showing our expected one-pass, one-fail result…

FAIL: test_always_fails (test_with_unittest.TryTesting)
------------------------------------------------------------
Traceback (most recent call last):File "/.../test_with_unittest.py", line 9, in test_always_failsself.assertTrue(False)AssertionError: False is not True
-----------------------------------------------------------
Ran 2 tests in 0.001sFAILED (failures=1)

In my opinion, that’s a lot of code to simply assert that True is truthy and False is not truthy. This is an opinion held by many other Python devs, and hence the creation of all our third-party test runners.

Introducing Our Test Runner Selection: pytest

pytest has been one of the most popular Python testing frameworks for many years running, and it’s easy to see why. pytest is a plugin-based ecosystem for testing Python code, and it was designed mainly with productivity and ease-of-use in mind.

It’s main claim-to-fame is that it requires quite a bit less boilerplate code than other Python test runners. It’s super robust, can run existing assertion tests right out of the box, and even includes functionality for reading tests written in unittest.

Revisiting our unittest one-passing one-failing example, in pytest that process would be as simple as…

# filename test_using_pytest.py
def test_always_pass():
   assert Truedef test_always_fail():
   assert False

Our arrange and act steps are all taken care of for us, no need to import any libraries or use any classes. We don’t even need to call our testing functions. All we to do to run our tests now is enter the pytest command in the command line…

$ pytest

We’ll look at output from pytest in just a second.

Working With pytest

Installation

To install pytest using pip type the following command in the command line…

$ pip install -U pytest

To verify installation and check what version you have installed…

$ pytest --version
pytest 6.2.1            # Output

How To Read Output

if you’re used to unittest (or even if you haven’t), pytest output might look a little foreign. Let’s try to make sense of pytest output by running our one-passing one-failing test.

============= test session starts =============platform darwin -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1rootdir: /Users/sam/self_study/blog_cells/testing_in_python/testsplugins: dash-1.17.0collected 2 itemstest_using_pytest.py .F                                                  [100%]================ FAILURES =========________________ test_always_false _____________def test_always_false():>      assert FalseE      assert Falsetest_using_pytest.py:19: AssertionError========== short test summary info ==========FAILED test_using_pytest.py::test_always_false - assert False========== 1 failed, 1 passed in 0.37s ===========

pytest reports show (in order)…

1. System state, including which version of Python/pytest/plugins you’re running
2. rootdir or the directory pytests is searching for configurations and tests
3. The number of tests the runner discovered
4. Status of each test

• A dot (.) means the test passed
• An F means the test failed
• An E means the test raised an unexpected exception (script contains errors other than AssertionErrors)

5. Breakdown of test failures

Writing Some Tests

Important:pytest gathers tests according to a naming convention. By default, any file that is to contain tests must be named starting with test_, classes that hold tests must be named starting with Test, and any function in a file that should be treated as a test must also start with test_ (src: cewing, StackOverflow).

Let’s set up some basic functions and use pytest to test them. Create an empty directory on your machine (mkdir dir_name), navigate inside it (cd dir_name), and create a new file adhering to the pytest naming convention (touch filename.py).

Inside our new file, let’s create a function and test it against some test cases.

Our function will simply take two lists as input, and return the single largest value of the two lists.

# filename test_using_pytest.py
def get_largest_number(array1, array2):
   return max([max(array1), max(array2)])

Now let’s write our tests. In pytest, this can be as simple as creating a function following the required naming guidelines and adding a single assert statement.

def test_largest1():
   assert get_largest_number([1,3,6,9], [8,7,8])==9, 'Should be 9'def test_largest2():
   assert get_largest_number([29,3,87,26], [16,97,4,12])==97, 'Should be 97'def test_largest3():
   assert get_largest_number([1,1,1,1,1,1,1], [1,1,1,1,1])==1, 'Should be 1'

Here we wrote 3 separate tests, let’s see what happens when we run our tests…

# in command line, inside current working directory$ pytest

We should see the following output indicating 3 successful tests…

========= test session starts ==============platform darwin -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1rootdir: /Users/sam/self_study/blog_cells/testing_in_python/testsplugins: dash-1.17.0collected 1 itemtest_using_pytest.py .                                                   [100%]============ 1 passed in 0.01s ==============(learn-env) Sams-MacBook-Pro:tests sam$ pytest=========== test session starts ==============platform darwin -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1rootdir: /Users/sam/self_study/blog_cells/testing_in_python/testsplugins: dash-1.17.0collected 1 itemtest_using_pytest.py .                                                   [100%]============= 1 passed in 0.01s ===============(learn-env) Sams-MacBook-Pro:tests sam$ pytest=========== test session starts =============platform darwin -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1rootdir: /Users/sam/self_study/blog_cells/testing_in_python/testsplugins: dash-1.17.0collected 1 itemtest_using_pytest.py .                                                   [100%]============== 1 passed in 0.01s ==============

And that’s it! We’ve just written our first test suite!

We can also combine assertions into a single function. However, this will be treated as a single unit test. If one of our assertions fails, our test will end early and we won’t receive a full debrief of our test.

def test_largest():   assert get_largest_number([1,3,6,9], [8,7,8])==9, 'Should be 9'   assert get_largest_number([1],[2])==1, 'Should raise AssertionError'   assert get_largest_number([1,1,1,1,1,1,1], [1,1,1,1,1])==1, 'Should be 1'

Output in command line…

============ test session starts =============platform darwin -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1rootdir: /Users/sam/self_study/blog_cells/testing_in_python/testsplugins: dash-1.17.0collected 1 itemtest_using_pytest.py F                                                   [100%]=============== FAILURES =================_______________ test_largest _______________def test_largest():assert get_largest_number([1,3,6,9], [8,7,8])==9, 'Should be 9'>      assert get_largest_number([1],[2])==1, 'Should raise AssertionError'E      AssertionError: Should raise AssertionErrorE      assert 2 == 1E       +  where 2 = get_largest_number([1], [2])test_using_pytest.py:17: AssertionError============ short test summary info ============FAILED test_using_pytest.py::test_largest - AssertionError: Should raise Asse...========== 1 failed in 0.40s ====================