Threads in pytest

22 July 2018

pytest is a wonderful unit-testing framework for Python. Unfortunately, I recently learned it has some problems in dealing with threads(or at least doesn’t behave the way I’d like or expect).

The Problem

The crux of the issue is that when a test function spawns another thread and there is an runtime error in that thread, pytest doesn’t seem to have any knowledge of it. For example:

def foo():
    bar()

def test_foo():
    t = Thread(target=foo)
    t.start()
    t.join()
    # presumably assert something here

foo should fail immediately after being called, as bar is not defined, and it likely does. As far as pytest is concerned though, the test passed.

The use-case that drove me to find this unexpected behavior involved a producer-consumer scheme with queues. Here’s a simplified example:

def queue_consumer(q):
    i = 0
    while not i < 0:
        i = q.get()
        secretly_cause_an_error()


def test_queue_consumer():
    q = Queue()
    q.put(1)
    q.put(-1)
    t = Thread(target=queue_consumer, args=(q,))
    t.start()
    while not q.empty():
        pass
    t.join()
    # assert something

This caused some very mysterious deadlock, with the test/producer waiting on the queue to be emptied by a thread that has thrown an exception and will never return. Before realizing this, I ventured down the dark road of deleting caches, running in a new virtualenv, on a different machine, etc. Finally, I ran the function in its normal context outside of pytest and saw the exception being raised, which led to a quick fix.

Conclusion

So far as I’ve read there’s not a lot of threading support built into pytest. It would be nice to at least see the exceptions being raised in non-main threads, or just fail the test entirely. This issue aside, I’d still highly recommend pytest.