4

假设您正在编写一些与 BenchmarkDotNet 一起使用的基准测试,这些基准测试是多目标的net48net6.0,并且其中一个基准测试只能针对net6.0目标编译。

显而易见的事情是使用这样的东西从net48构建中排除特定的基准:

#if NET6_0_OR_GREATER

[Benchmark]
public void UsingSpan()
{
    using var stream = new MemoryStream();
    writeUsingSpan(stream, _array);
}

static void writeUsingSpan(Stream output, double[] array)
{
    var span  = array.AsSpan();
    var bytes = MemoryMarshal.AsBytes(span);

    output.Write(bytes);
}

#endif // NET6_0_OR_GREATER

不幸的是,这不起作用,并且它不起作用的方式取决于TargetFrameworks项目文件中属性中指定的目标的顺序。

如果您订购框架,那么net6.0首先<TargetFrameworks>net6.0;net48</TargetFrameworks>(在上面的示例中)该UsingSpan()方法包含在两个目标中,导致net48目标和输出的 BenchmarkDotNet 构建错误,如下所示:

|            Method |                Job |            Runtime |       Mean |     Error |    StdDev |
|------------------ |------------------- |------------------- |-----------:|----------:|----------:|
| UsingBitConverter |           .NET 6.0 |           .NET 6.0 | 325.587 us | 2.0160 us | 1.8858 us |
|      UsingMarshal |           .NET 6.0 |           .NET 6.0 | 505.784 us | 4.3719 us | 4.0894 us |
|         UsingSpan |           .NET 6.0 |           .NET 6.0 |   4.942 us | 0.0543 us | 0.0482 us |
| UsingBitConverter | .NET Framework 4.8 | .NET Framework 4.8 |         NA |        NA |        NA |
|      UsingMarshal | .NET Framework 4.8 | .NET Framework 4.8 |         NA |        NA |        NA |
|         UsingSpan | .NET Framework 4.8 | .NET Framework 4.8 |         NA |        NA |        NA |

另一方面,如果您对框架进行排序,那么net48首先<TargetFrameworks>net48;net6.0</TargetFrameworks>(在上面的示例中)该UsingSpan()方法被排除在两个目标中,结果输出如下:

|            Method |                Job |            Runtime |     Mean |    Error |   StdDev |
|------------------ |------------------- |------------------- |---------:|---------:|---------:|
| UsingBitConverter |           .NET 6.0 |           .NET 6.0 | 343.1 us |  6.51 us | 11.57 us |
|      UsingMarshal |           .NET 6.0 |           .NET 6.0 | 539.5 us | 10.77 us | 22.94 us |
| UsingBitConverter | .NET Framework 4.8 | .NET Framework 4.8 | 331.2 us |  5.43 us |  5.08 us |
|      UsingMarshal | .NET Framework 4.8 | .NET Framework 4.8 | 588.9 us | 11.18 us | 10.98 us |    

我必须通过单一目标项目和编辑项目文件以分别针对框架来解决这个问题,然后为每个目标分别运行基准测试。

有没有办法让它与多目标项目一起工作?


为了完整起见,这里有一个完整的可编译测试应用程序来演示该问题。我正在使用 Visual Studio 2022。

项目文件:

<PropertyGroup>
  <OutputType>Exe</OutputType>
  <TargetFrameworks>net48;net6.0</TargetFrameworks>
  <ImplicitUsings>enable</ImplicitUsings>
  <LangVersion>latest</LangVersion>
  <Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
  <PackageReference Include="BenchmarkDotNet" Version="0.13.1" />
</ItemGroup>

“Program.cs”文件:

using System.Runtime.InteropServices;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

namespace Benchmark;

public static class Program
{
    public static void Main()
    {
        BenchmarkRunner.Run<UnderTest>();
    }
}

[SimpleJob(RuntimeMoniker.Net48)]
[SimpleJob(RuntimeMoniker.Net60)]
public class UnderTest
{
    [Benchmark]
    public void UsingBitConverter()
    {
        using var stream = new MemoryStream();
        writeUsingBitConverter(stream, _array);
    }

    static void writeUsingBitConverter(Stream output, double[] array)
    {
        foreach (var sample in array)
        {
            output.Write(BitConverter.GetBytes(sample), 0, sizeof(double));
        }
    }

    [Benchmark]
    public void UsingMarshal()
    {
        using var stream = new MemoryStream();
        writeUsingMarshal(stream, _array);
    }

    static void writeUsingMarshal(Stream output, double[] array)
    {
        const int SIZE_BYTES = sizeof(double);

        byte[] buffer = new byte[SIZE_BYTES];
        IntPtr ptr    = Marshal.AllocHGlobal(SIZE_BYTES);

        foreach (var sample in array)
        {
            Marshal.StructureToPtr(sample, ptr, true);
            Marshal.Copy(ptr, buffer, 0, SIZE_BYTES);
            output.Write(buffer, 0, SIZE_BYTES);
        }

        Marshal.FreeHGlobal(ptr);
    }

    #if NET6_0_OR_GREATER

    [Benchmark]
    public void UsingSpan()
    {
        using var stream = new MemoryStream();
        writeUsingSpan(stream, _array);
    }

    static void writeUsingSpan(Stream output, double[] array)
    {
        var span  = array.AsSpan();
        var bytes = MemoryMarshal.AsBytes(span);

        output.Write(bytes);
    }

    #endif // NET6_0_OR_GREATER

    readonly double[] _array = new double[10_000];
}
4

2 回答 2

2

从内存中,Benchmark.NET 将使用一些内部魔法为所有框架运行基准测试。因此,与其使用现有的预处理器符号,不如将测试拆分为具有不同RuntimeMoniker属性的两个类。例如:

[SimpleJob(RuntimeMoniker.Net48)]
public class UnderTestNet48
{
    // Benchmarks
}

[SimpleJob(RuntimeMoniker.Net60)]
public class UnderTestNet60
{
    // Benchmarks
}

现在您需要修改运行基准测试的代码,因为它们是跨类拆分的,这样的事情会起作用:

public static void Main()
{
    var config = DefaultConfig.Instance.
        .WithOptions(ConfigOptions.JoinSummary)
        .WithOptions(ConfigOptions.DisableLogFile);

    BenchmarkRunner.Run(typeof(Program).Assembly, config);
}

[来自 OP(马修·沃森)的编辑]

由于这个答案,我设法实现了这一点。

net48通过将通用测试方法放入受保护的基类中,然后提供两个派生类——一个用于基准测试,一个用于基准测试,我设法减少了代码重复net5.0

这是我最终得到的代码:

using System.Runtime.InteropServices;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

namespace Benchmark;

public static class Program
{
    public static void Main()
    {
        BenchmarkRunner.Run(
            typeof(Program).Assembly, 
            DefaultConfig.Instance
               .WithOptions(ConfigOptions.JoinSummary)
               .WithOptions(ConfigOptions.DisableLogFile));
    }
}

public abstract class UnderTestBase
{
    protected static Stream CreateStream()
    {
        return new MemoryStream(); // Or Stream.Null
    }

    protected void WriteUsingBitConverter(Stream output, double[] array)
    {
        foreach (var sample in array)
        {
            output.Write(BitConverter.GetBytes(sample), 0, sizeof(double));
        }
    }

    protected void WriteUsingMarshal(Stream output, double[] array)
    {
        const int SIZE_BYTES = sizeof(double);

        byte[] buffer = new byte[SIZE_BYTES];
        IntPtr ptr    = Marshal.AllocHGlobal(SIZE_BYTES);

        foreach (var sample in array)
        {
            Marshal.StructureToPtr(sample, ptr, true);
            Marshal.Copy(ptr, buffer, 0, SIZE_BYTES);
            output.Write(buffer, 0, SIZE_BYTES);
        }

        Marshal.FreeHGlobal(ptr);
    }

    #if NET6_0_OR_GREATER
    
    protected void WriteUsingSpan(Stream output, double[] array)
    {
        var span  = array.AsSpan();
        var bytes = MemoryMarshal.AsBytes(span);

        output.Write(bytes);
    }

    #endif // NET6_0_OR_GREATER

    protected readonly double[] Array = new double[100_000];
}

[SimpleJob(RuntimeMoniker.Net48)]
public class UnderTestNet48: UnderTestBase
{
    [Benchmark]
    public void UsingBitConverter()
    {
        using var stream = CreateStream();
        WriteUsingBitConverter(stream, Array);
    }

    [Benchmark]
    public void UsingMarshal()
    {
        using var stream = CreateStream();
        WriteUsingMarshal(stream, Array);
    }
}

[SimpleJob(RuntimeMoniker.Net60)]
public class UnderTestNet60: UnderTestBase
{
    [Benchmark]
    public void UsingBitConverter()
    {
        using var stream = CreateStream();
        WriteUsingBitConverter(stream, Array);
    }

    [Benchmark]
    public void UsingMarshal()
    {
        using var stream = CreateStream();
        WriteUsingMarshal(stream, Array);
    }

    #if NET6_0_OR_GREATER

    [Benchmark]
    public void UsingSpan()
    {
        using var stream = CreateStream();
        WriteUsingSpan(stream, Array);
    }

    #endif // NET6_0_OR_GREATER
}

这导致此输出:

|           Type |            Method |                Job |            Runtime |       Mean |     Error |    StdDev |
|--------------- |------------------ |------------------- |------------------- |-----------:|----------:|----------:|
| UnderTestNet60 | UsingBitConverter |           .NET 6.0 |           .NET 6.0 | 4,110.8 us |  81.53 us | 151.13 us |
| UnderTestNet60 |      UsingMarshal |           .NET 6.0 |           .NET 6.0 | 5,774.0 us | 114.78 us | 194.90 us |
| UnderTestNet60 |         UsingSpan |           .NET 6.0 |           .NET 6.0 |   521.6 us |   5.13 us |   4.80 us |
| UnderTestNet48 | UsingBitConverter | .NET Framework 4.8 | .NET Framework 4.8 | 2,987.2 us |  35.60 us |  29.73 us |
| UnderTestNet48 |      UsingMarshal | .NET Framework 4.8 | .NET Framework 4.8 | 5,616.9 us |  57.85 us |  48.30 us |

(顺便说一句,一个有趣的结果是,与相比,该UsingBitConverter()方法实际上运行得更快——尽管这与.net48net6.0Span<T>

[/来自 OP(马修·沃森)的编辑]

于 2021-12-08T16:41:54.333 回答
1

它已在https://github.com/dotnet/BenchmarkDotNet/issues/1226#issuecomment-532144829中进行了讨论:

当运行以 XYZ 框架为目标的宿主进程时,BDN 使用反射来获取可用方法(基准)的列表。如果您使用的是#if定义,那么每个主机进程目标框架的基准测试列表都会有所不同。

性能 repo 文档描述了如何在此处比较多个运行时性能: https ://github.com/dotnet/performance/blob/master/docs/benchmarkdotnet.md#multiple-runtimes

主机进程必须是您要比较的运行时的最低通用 API 分母!

您可以在https://benchmarkdotnet.org/articles/configs/toolchains.html#multiple-frameworks-support中阅读有关对多个 tfms 进行基准测试的更多信息

于 2022-02-21T12:25:20.297 回答