2

我想使用一个插件来显示并行调用的pytest-benchmark运行时统计信息，...，。 test_valid_submission(0)test_valid_submission(EXPERIMENT_SIZE)

为此，我有以下代码，它尝试使用pytest-benchmark（类似于Grouping Parametrized Benchmarks with pytest）来实现这一点：

@pytest.mark.parametrize("counter", range(EXPERIMENT_SIZE))
def test_performance_under_load(benchmark, counter):
    benchmark(test_valid_submission, counter)

当我打电话

pytest --workers auto --tests-per-worker auto -vv --benchmark-only --benchmark-verbose --group-by=func

我希望最后得到一个基准汇总表EXPERIMENT_SIZE，其中包含我的并行test_valid_submission()调用的每个运行时的最小值、最大值、平均值、标准偏差。不幸的是，没有打印基准摘要表（请参阅下面的详细信息）。

@hoefling 评论pytest-benchmark说不支持并行运行和收集基准数据。

是否有另一个 pytest 插件（或其他解决方案）可以

  • 收集EXPERIMENT_SIZE并行test_valid_submission(x)调用的数量并将它们组合在一起
  • 计算组中并行调用的运行时统计的最小值、最大值、平均值、标准差
  • 使用多个组，例如一个 fortest_valid_submission(x)和一个 fortest_invalid_submission(x)
  • 在我的测试结束时打印统计信息，类似于上面提到的 pytest-benchmark 汇总表？

关于 pytest-benchmark 的详细信息

使用pytest-benchmark、和EXPERIMENT_SIZE=3，我得到以下输出（但即使在 时，它也显示并且没有统计信息）。iterations=1rounds=1EXPERIMENT_SIZE>=5rounds=5

但我得到以下输出（EXPERIMENT_SIZE=3删除了不相关的行）：

 ============================== test session starts ==========================================
 platform linux -- Python 3.6.10, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /anaconda3/envs/reg/bin/python
 benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
 plugins: repeat-0.8.0, cov-2.8.1, bdd-3.2.1, clarity-0.3.0a0, benchmark-3.2.3, parallel-0.1.0
 collected 22 items
 pytest-parallel: 8 workers (processes), 3 tests per worker (threads)

 ...

 test/test_x.py::test_valid_submission SKIPPED

Computing precision for time.perf_counter ... 50.99ns.

Calibrating to target round 5.00us; will estimate when reaching 2.50us (using: time.perf_counter, precision: 50.99ns).

Computing precision for time.perf_counter ... 48.98ns.

Calibrating to target round 5.00us; will estimate when reaching 2.50us (using: time.perf_counter, precision: 48.98ns).

Computing precision for time.perf_counter ... 49.01ns.

Calibrating to target round 5.00us; will estimate when reaching
2.50us (using: time.perf_counter, precision: 49.01ns).

Measured 1 iterations: 105.72s.   Running 5 rounds x 1 iterations ...
Measured 1 iterations: 105.73s.   Running 5 rounds x 1 iterations ...
Measured 1 iterations: 117.20s.   Running 5 rounds x 1 iterations ...   
Ran for 339.53s.   Ran for 350.11s.   Ran for 461.53s.

test/test_x.py::test_performance_under_load[2] PASSED
test/test_x.py::test_performance_under_load[1] PASSED
test/test_x.py::test_performance_under_load[0] PASSED

========================== 3 passed, 19 skipped in 714.05s (0:11:54) ========================

交替使用benchmark.pedantic(test_valid_submission, args=[counter], iterations=1, rounds=1)也不会导致打印统计数据。

4

0 回答 0