python - pytest中的测试用例执行顺序

Question

我正在使用pytest。我在一个目录中有两个文件。在其中一个文件中，有一个长时间运行的测试用例会生成一些输出。在另一个文件中有一个读取该输出的测试用例。如何确保两个测试用例的正确执行顺序？除了以正确的顺序将测试用例放在同一个文件中之外，还有其他选择吗？

Answer 1

一般来说，您可以使用其明确指定的 hooks配置 pytest 基本上任何部分的行为。

在您的情况下，您需要“pytest_collection_modifyitems”钩子，它可以让您重新排序收集的测试。

话虽如此，您的测试排序似乎应该更容易——毕竟这是 Python！所以我写了一个用于排序测试的插件：“pytest-ordering”。查看文档或从​​pypi安装它。现在我推荐使用@pytest.mark.firstand@pytest.mark.second或其中一个@pytest.mark.order#标记，但我对更有用的 API 有一些想法。欢迎提出建议:)

编辑： pytest-ordering 目前似乎已被放弃，您还可以查看pytest-order（作者原始项目的一个分支）。

Edit2：在 pytest-order 中，仅order支持一个标记 ( )，并且提到的示例将显示为@pytest.mark.order("first"), @pytest.mark.order("second"), or @pytest.mark.order(#)(其中 # 是任意数字)。

Answer 2

也许您可以考虑使用依赖pytest 插件，您可以在其中轻松设置测试依赖项。

小心 - 评论表明这并不适合所有人。

@pytest.mark.dependency()
def test_long():
    pass

@pytest.mark.dependency(depends=['test_long'])
def test_short():
    pass

这种方式只有在成功test_short时才会执行，并且也会强制执行顺序。test_long

Answer 3

还有一个插件pytest-ordering似乎可以满足您的要求。

Answer 4

Pytest 的固定装置可用于以与它们订购固定装置的创建类似的方式对测试进行排序。虽然这是非常规的，但它利用了您可能已经拥有的夹具系统知识，它不需要单独的包，并且不太可能被 pytest 插件更改。

@pytest.fixture(scope='session')
def test_A():
    pass

@pytest.mark.usefixtures('test_A')
def test_B():
    pass

如果有多个依赖于它的测试，范围会阻止对 test_A 的多次调用。

Answer 5

It's important to keep in mind, while trying to fix pytest ordering "issue", that running tests in the same order as they are specified seems to be the default behavior of pytest.

It turns out that my tests were out of that order because of one of these packages - pytest-dependency, pytest-depends, pytest-order. Once I uninstalled them all with pip uninstall package_name, the problem was gone. Looks like they have side effects

Answer 6

Pytest by default, it's executes the steps randomly. So use this option for executing in order.

Install with:

pip install pytest-order

Write the test method as like below:

import pytest

@pytest.mark.order(1)
def test_foo():
    assert True

@pytest.mark.order(2)
def test_bar():
    assert True

Answer 7

As indicated by @Frank T in the accepted answer, the pytest_collection_modifyitems hook hook allows to modify the order of collected tests (items) in place. This approach has the advantage of not requiring any third party library.

A full example on how to enforce test case execution order by test class is already available in this answer.

In this case, though, it seems like you would like to enforce execution order by test module (i.e., the .py file where the test lives). The following adaptation would allow you to do so:

# conftest.py
def pytest_collection_modifyitems(items):
    """Modifies test items in place to ensure test modules run in a given order."""
    MODULE_ORDER = ["tests.test_b", "tests.test_c", "tests.test_a"]
    module_mapping = {item: item.module.__name__ for item in items}

    sorted_items = items.copy()
    # Iteratively move tests of each module to the end of the test queue
    for module in MODULE_ORDER:
        sorted_items = [it for it in sorted_items if module_mapping[it] != module] + [
            it for it in sorted_items if module_mapping[it] == module
        ]
    items[:] = sorted_items

Placing the snippet above in conftest.py replaces the default alphabetical test execution order test_a -> test_b -> test_c with test_b -> test_c -> test_a. Modules can live in different test sub-directories, and the order of tests inside a module is left unchanged.

Answer 8

主要.py：

import functools
import pytest
from demo import test_foo,test_hi

def check_depends(depends):
    try:
        for dep in depends:
            dep()
    except Exception as e:
        return dep
    else:
        return True

def pytest_depend(depends):
    def pytest_depend_decorator(func):
        stat = check_depends(depends)
        if stat is True:
            return func
        else:
            return pytest.mark.skip(True, reason="%s[skip] --> %s[Failed]" % (func.__name__, stat.__name__))(func)
    return pytest_depend_decorator


@pytest_depend([test_foo,test_hi])
def test_bar():
    pass

@pytest_depend([test_foo,test_hi])
def test_bar2():
    pass

演示.py：

def test_hi():
    pass
def test_foo():
    assert False

---

平台 linux -- Python 3.5.2、pytest-3.8.2、py-1.6.0、pluggy-0.7.1 -- /usr/bin/python3

pytest -vrsx ./plugin.py

Answer 9

使用pytest-randomly插件中可用的“--randomly-dont-reorganize”选项或“-p no:randomly”，这将按照您在模块中提到的顺序运行您的测试。

模块：

import pytest

def test_three():
    assert True

def test_four():
    assert True

def test_two():
    assert True

def test_one():
    assert True

执行：

(tmp.w95BqE188N) rkalaiselvan@dev-rkalaiselvan:~/$ py.test --randomly-dont-reorganize test_dumm.py
======================================================================== test session starts ========================================================================
platform linux2 -- Python 2.7.12, pytest-3.10.1, py-1.5.4, pluggy-0.7.1 -- /tmp/tmp.w95BqE188N/bin/python2
cachedir: .pytest_cache
Using --randomly-seed=1566829391
rootdir: /home/rkalaiselvan, inifile: pytest.ini
plugins: randomly-1.2.3, timeout-1.3.1, cov-2.6.0, mock-1.10.0, ordering-0.6
collected 4 items

test_dumm.py::test_three PASSED
test_dumm.py::test_four PASSED
test_dumm.py::test_two PASSED
test_dumm.py::test_one PASSED

(tmp.w95BqE188N) rkalaiselvan@dev-rkalaiselvan:~/$ py.test -p no:randomly test_dumm.py
======================================================================== test session starts ========================================================================
platform linux2 -- Python 2.7.12, pytest-3.10.1, py-1.5.4, pluggy-0.7.1 -- /tmp/tmp.w95BqE188N/bin/python2
cachedir: .pytest_cache
Using --randomly-seed=1566829391
rootdir: /home/rkalaiselvan, inifile: pytest.ini
plugins: randomly-1.2.3, timeout-1.3.1, cov-2.6.0, mock-1.10.0, ordering-0.6
collected 4 items

test_dumm.py::test_three PASSED
test_dumm.py::test_four PASSED
test_dumm.py::test_two PASSED
test_dumm.py::test_one PASSED

Answer 10

To me the simpliest way to fix tests execution order is to use them as fixtures, which are executed ordered by design.

@pytest.fixture()
def test_a():
    print("running test a first")
    pass

def test_b(test_a):
    print("running test b after test a")
    pass

Mark test dependency as fixture and pass it as an argument to dependent one.

Answer 11

确保你已经安装了 pytest-ordering 包。要确认，请转到 Pycharm 设置>>项目解释器>>并查找 pytest-ordering ：如果它不可用，请安装它。Pycharm设置>>项目解释器>>点击+按钮并搜索pytest-ordering安装它。瞧！！它肯定会奏效。