Godkjenn tutorial

This will take you through the process of installing, setting up, and using godkjenn for approval testing in a Python project.

Note

This tutorial will use godkjenn’s pytest integration. Godkjenn does not mandate the use of pytest, though currently it’s the only testing framework for which godkjenn provides an integration. Integrating with other frameworks is straightforward and encouraged!

Installing godkjenn

First you need to install godkjenn. The simplest and most common way to do this is with pip:

pip install godkjenn

For pytest integration you’ll also want to install the necessary plugin:

pip install "godkjenn[pytest-plugin]"

A first test

Now let’s create our first test that uses godkjenn. Create a directory to contain the code for the rest of this tutorial. We’ll refer to it as TEST_DIR or $TEST_DIR.

Once you have that directory, create the file TEST_DIR/pytest.ini. This can be empty; it just exists to tell pytest where the “top” of your tests is.

Next create the file TEST_DIR/test_tutorial.py with these contents:

def test_demo(godkjenn):
    test_data = b'12345'
    godkjenn.verify(test_data, mime_type='application/octet-stream')

This will be mostly familiar if you’ve used pytest: it’s just a single test function with a fixture.

On line 1 we define the test function. The godkjenn parameter tells pytest that we want to use the godkjenn fixture. This fixture gives us an object that we use for verifying our test data.

On line 2 we simply invent some test data. Notice that it’s a bytes object. Godkjenn ultimately requires all of its data to be bytes, so for this tutorial we’ve just created some simple data. In practice, this data would be the output from some function that you want to test.

Finally on line 3 we call godkjenn.verify(), passing in our test_data. This call will take the data we pass in and compare it to the currently-accepted “golden” output. If the data we pass in does not match the accepted output, the test is flagged as a failure. Similarly - as will happen for us - we’ll get a failure if there is no existing accepted output.

Running the test

Now we can run the tests with pytest. For now just run the test from TEST_DIR:

cd $TEST_DIR
pytest .

You should see some output like this:

$ pytest .
========================================================================================================= test session starts =========================================================================================================
platform darwin -- Python 3.8.0, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/abingham/repos/sixty-north/godkjenn/docs/tutorial, configfile: pytest.ini
plugins: hypothesis-6.4.0, godkjenn-2.0.1
collected 1 item

test_tutorial.py F                                                                                                                                                                                                           [100%]

============================================================================================================== FAILURES ===============================================================================================================
______________________________________________________________________________________________________________ test_demo ______________________________________________________________________________________________________________
There is no accepted data

If you wish to accept the received result, run:

    godkjenn -C . accept "test_tutorial.py::test_demo"

======================================================================================================= short test summary info =======================================================================================================
FAILED test_tutorial.py::test_demo
========================================================================================================== 1 failed in 0.07s ==========================================================================================================

We see that - as expected - our test failed. The report tells us that “There is no accepted data”. This means that this is the first time we’ve run the test and haven’t accepted any output for the test. We’re also given instructions on how accept the output if we believe it to be correct.

This idea of of “accepting” output is central to the notion of approval testing. At some point we have to decide that our code is correct and that the output it produces is indicative of that proper functioning. For now let’s assume that our data is correct.

Status

Before accepting the data, let’s use the godkjenn status command to see the state of our approval tests:

$ godkjenn status
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Test ID                     ┃ Status      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ test_tutorial.py::test_demo │ initialized │
└─────────────────────────────┴─────────────┘

This tells us that we have one approval test in our system and that its state is “initialized”. This means that it has some received data (i.e. the data from the test we just ran) but no accepted data.

Accepting the data

Since we believe that the data we passed to godkjenn.verify() represents the correct output from our program, we want to accept it. We can use the command provided to us in the test output:

$ godkjenn accept "test_tutorial.py::test_demo"

Now if we run status we see don’t get any output:

$ godkjenn status

This is because all of our tests are “up-to-date”, i.e. all of that have accepted data and not received data. In order to see the status of all test-ids, including those that are up-to-date, you can use the “-a/–show-all” options of the ‘status’ command:

$ godkjenn status --show-all
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Test ID                     ┃ Status     ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ test_tutorial.py::test_demo │ up-to-date │
└─────────────────────────────┴────────────┘

And that’s it! You’ve created your first godkjenn approval test and accepted its output.

Accepting new data

Over time, of course, your code may change such that its correct output no longer matches your accepted output. When this happens your test will fail and you’ll have to accept the new data. To see this, let’s change our test_tutorial.py to look like this:

def test_demo(godkjenn):
    test_data = b'1234567890'
    godkjenn.verify(test_data, mime_type='application/octet-stream')

You can see on line 2 that test_data now has more digits. When we run our test we get a failure because of this change:

$ pytest test_tutorial.py
=================================================================================================================================== test session starts ===================================================================================================================================
platform darwin -- Python 3.8.0, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/abingham/repos/sixty-north/godkjenn/docs/tutorial, configfile: pytest.ini
plugins: hypothesis-6.4.0, godkjenn-2.0.1
collected 1 item

test_tutorial.py F                                                                                                                                                                                                                                                                  [100%]

======================================================================================================================================== FAILURES =========================================================================================================================================
________________________________________________________________________________________________________________________________________ test_demo ________________________________________________________________________________________________________________________________________
Received data does not match accepted

If you wish to accept the received result, run:

    godkjenn -C . accept "test_tutorial.py::test_demo"

================================================================================================================================= short test summary info =================================================================================================================================
FAILED test_tutorial.py::test_demo
==================================================================================================================================== 1 failed in 0.05s ====================================================================================================================================

You can see the failure was because “Received data does not match accepted”. That is, the data we’re passing to godkjenn.verify() doesn’t match the accepted data.

If we run godkjenn status again, we see a new status for our test:

$ godkjenn status
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Test ID                     ┃ Status   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ test_tutorial.py::test_demo │ mismatch │
└─────────────────────────────┴──────────┘

The status “mismatch” means that the received and accepted data are different.

Seeing the difference

Our job now is to decide if the received data is correct and should become the accepted. To make this decision it can be very helpful to see the accepted data, the received data, and the differenced between them.

To see the accepted data we can use the godkjenn accepted command:

$ godkjenn accepted "test_tutorial.py::test_demo" -
12345%

Similarly, we can see the received data using the godkjenn received command:

$ godkjenn received "test_tutorial.py::test_demo" -
1234567890%

In this case it’s pretty easy to see the difference. In other cases it might be more difficult. To help with this godkjenn also lets you view the difference between the files with the godkjenn diff command. By default godkjenn diff uses a very basic diff display:

$ godkjenn diff "test_tutorial.py::test_demo"
WARNING:godkjenn.cli:No review tools configured. Fallback differs will be used.
---
+++
@@ -1 +1 @@
-12345
+1234567890

Again, in a simple case like this, this default diff output is enough to make it clear what the difference is. In more complex cases you might need more powerful tools, though, and we’ll look at how to use those soon.

Configuring an external diff tool

The built-in diff tool in godkjenn is sufficient for simple cases, but many people have other, more sophisticated diff tools that they would prefer to use with approval testing. Godkjenn allows you to specify these tools in your configuration.

For this tutorial we’re going to configure godkjenn to use Beyond Compare as its default diff tool. To do this you need to create the file “.godkjenn/config.toml”. Put these contents in that file:

[godkjenn.differs]
default_command = "bcomp {accepted.path} {received.path}"

This is telling godkjenn to run the command “bcomp” (the Beyond Compare executble) to display diffs. The first argument to “bcomp” will be the path to the current accepted data, and the second is the path to the received data. With this configuration, whenever godkjenn needs to display a diff (e.g in the “diff” and “review” commands), it will use “bcomp”.

If you don’t have Beyond Compare installed, you can replace “bcomp” with many other commands like “diff”, “vimdiff”, and “p4diff”.

Once you’ve made this change, you can run your godkjenn diff command again and see your configure diff tool being used.

Note

Godkjenn supports fairly sophisticated configuration of diff tools, allowing you to use different diff tools for different MIME types. See the configuration documentation for details.

Accepting the new data

We’ll assume that our new data is actually correct and accept it:

godkjenn accept "test_tutorial.py::test_demo"

Once we do that we see that our status is back to “up-to-date”:

$ godkjenn status --show-all
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Test ID                     ┃ Status     ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ test_tutorial.py::test_demo │ up-to-date │
└─────────────────────────────┴────────────┘

Reviewing multiple tests

The godkjenn diff command lets you view the difference between the accepted and received data for a single test. In many cases, though, you have several - and in some cases a great many - tests for which you need to see the diff. The godkjenn review command lets you view all of the diffs for ‘mismatch’ tests in sequence.

In effect, the review command calls diff for each test that’s in the mismatch state, one after the other, using the diffing tools that you’ve configured.

To see review in action, let’s first add a new test. Here’s the new contents of “test_tutorial.py”:

def test_demo(godkjenn):
    test_data = b"1234567890"
    godkjenn.verify(test_data, mime_type="application/octet-stream")


def test_second_demo(godkjenn):
    test_data = b"8675309"
    godkjenn.verify(test_data, mime_type="application/octet-stream")

We’ll run the tests and accept the received data in order to lay a foundation for running review:

$ pytest test_tutorial.py
==================================================== test session starts =====================================================
platform darwin -- Python 3.8.0, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/abingham/repos/sixty-north/godkjenn/docs/tutorial/sandbox, configfile: pytest.ini
plugins: hypothesis-6.4.0, godkjenn-4.0.0
collected 2 items

test_tutorial.py FF                                                                                                    [100%]

========================================================== FAILURES ==========================================================
_________________________________________________________ test_demo __________________________________________________________
Received data does not match accepted

If you wish to accept the received result, run:

    godkjenn -C . accept "test_tutorial.py::test_demo"

______________________________________________________ test_second_demo ______________________________________________________
There is no accepted data

If you wish to accept the received result, run:

    godkjenn -C . accept "test_tutorial.py::test_second_demo"

================================================== short test summary info ===================================================
FAILED test_tutorial.py::test_demo
FAILED test_tutorial.py::test_second_demo
===================================================== 2 failed in 0.03s ======================================================
$ godkjenn accept-all

Now we’ll modify the tests so that each produces different output:

def test_demo(godkjenn):
    test_data = b"-- 1234567890 --"
    godkjenn.verify(test_data, mime_type="application/octet-stream")


def test_second_demo(godkjenn):
    test_data = b"-- 8675309 --"
    godkjenn.verify(test_data, mime_type="application/octet-stream")

If you run the tests again, godkjenn status shows that both tests are in the ‘mismatch’ state:

$ pytest test_tutorial.py
... elided ...
$ godkjenn status
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Test ID                            ┃ Status   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ test_tutorial.py::test_second_demo │ mismatch │
│ test_tutorial.py::test_demo        │ mismatch │
└────────────────────────────────────┴──────────┘

Now if you run godkjenn review, godkjenn will run your configured diff tool twice, once for each test, letting you view and potentially modify the accepted data.

Critically, if the received and accepted data are identical after godkjenn runs your diff tool (i.e. if you edit them through your diff tool), then godkjenn will accept the data and mark them as ‘up-to-date’. This gives you a conventient way to rapidly iterate through a set of received data, verifying each in turn and rapidly updating the accepted data for each affected test.

Note

The godkjenn review command can be very useful if you’ve a collection of received data for which you need to manually verify each one. However, if you somehow know that all of the received data should be accepted, then it’s even faster to use the godkjenn accept-all command.