sql - 能否使用 CTE 或 OVER 将此递归解决方案写入 T-SQL 查询？

Question

假设您有一个名为 Table1 of Orders 的表格，按时间顺序从 In-line UDF 返回。请注意，OrderID 可能不同步，所以我故意在那里创建了一个异常（即我没有包含 Date 字段，但如果您更容易，我可以访问该列）。

   OrderID  BuySell  FilledSize  ExecutionPrice  RunningTotal AverageBookCost  RealisedPnL
   339      Buy      2           24.5            NULL         NULL             NULL
   375      Sell     3           23.5            NULL         NULL             NULL
   396      Sell     3           20.5            NULL         NULL             NULL
   416      Sell     1           16.4            NULL         NULL             NULL
   405      Buy      4           18.2            NULL         NULL             NULL
   421      Sell     1           16.7            NULL         NULL             NULL
   432      Buy      3           18.6            NULL         NULL             NULL

我有一个函数，我想从上到下递归地应用它来计算 3 个 NULL 列，但是该函数的输入将是上一次调用的输出。我创建的函数称为 mfCalc_RunningTotalBookCostPnL，我在下面附上了这个

CREATE FUNCTION [fMath].[mfCalc_RunningTotalBookCostPnL](
    @BuySell           VARCHAR(4),
    @FilledSize        DECIMAL(31,15),
    @ExecutionPrice    DECIMAL(31,15),
    @OldRunningTotal   DECIMAL(31,15),
    @OldBookCost       DECIMAL(31,15)
    )

RETURNS @ReturnTable TABLE(
    NewRunningTotal DECIMAL(31,15),
    NewBookCost DECIMAL(31,15),
    PreMultRealisedPnL  DECIMAL(31,15)
    )
AS
BEGIN
    DECLARE @SignedFilledSize   DECIMAL(31,15),
            @NewRunningTotal    DECIMAL(31,15),
            @NewBookCost        DECIMAL(31,15),
            @PreMultRealisedPnL DECIMAL(31,15)

    SET @SignedFilledSize = fMath.sfSignedSize(@BuySell, @FilledSize)
    SET @NewRunningTotal = @OldRunningTotal + @SignedFilledSize
    SET @PreMultRealisedPnL = 0
    IF SIGN(@SignedFilledSize) = SIGN(@OldRunningTotal)
        -- This Trade is adding to the existing position.
        SET @NewBookCost = (@SignedFilledSize * @ExecutionPrice +
            @OldRunningTotal * @OldBookCost) / (@NewRunningTotal)
    ELSE
    BEGIN
        -- This trade is reversing the existing position.
        -- This could be buying when short or selling when long.
        DECLARE @AbsClosedSize DECIMAL(31,15)
        SET @AbsClosedSize = fMath.sfMin(ABS(@SignedFilledSize), ABS(@OldRunningTotal));

        -- There must be Crystalising of PnL.
        SET @PreMultRealisedPnL = (@ExecutionPrice - @OldBookCost) * @AbsClosedSize * SIGN(-@SignedFilledSize)

        -- Work out the NewBookCost
        SET @NewBookCost = CASE
            WHEN ABS(@SignedFilledSize) < ABS(@OldRunningTotal) THEN @OldBookCost
            WHEN ABS(@SignedFilledSize) = ABS(@OldRunningTotal) THEN 0
            WHEN ABS(@SignedFilledSize) > ABS(@OldRunningTotal) THEN @ExecutionPrice
        END
    END

    -- Insert values into Return Table
    INSERT INTO @ReturnTable
        VALUES (@NewRunningTotal, @NewBookCost, @PreMultRealisedPnL)

    -- Return
    RETURN
END

因此，我正在寻找的 t-SQL 命令（我不介意是否有人也可以创建外部应用）将生成以下结果/解决方案集：

OrderID BuySell FilledSize ExecutionPrice RunningTotal AverageBookCost RealisedPnL
339     Buy     2          24.5           2            24.5            0
375     Sell    3          23.5           -1           23.5            -2
396     Sell    3          20.5           -4           21.25           0
416     Sell    1          16.4           -5           20.28           0
405     Buy     4          18.2           -1           20.28           8.32
421     Sell    1          16.7           -2           18.49           0
432     Buy     3          18.6           1            18.6            -0.29

需要注意的是，上面的存储过程调用了一个简单的函数 fMath.sfSignedSize，它只使 ('Sell',3) = -3。此外，为免生疑问，假设我的计算正确，我会看到按此顺序进行这些调用的解决方案！（请注意，我开始假设 OldRunningTotal 和 OldBookCost 都为零）：

SELECT * FROM fMath.mfCalc_RunningTotalBookCostPnL('Buy',2,24.5,0,0)
SELECT * FROM fMath.mfCalc_RunningTotalBookCostPnL('Sell',3,23.5,2,24.5)
SELECT * FROM fMath.mfCalc_RunningTotalBookCostPnL('Sell',3,20.5,-1,23.5)
SELECT * FROM fMath.mfCalc_RunningTotalBookCostPnL('Sell',1,16.4,-4,21.25)
SELECT * FROM fMath.mfCalc_RunningTotalBookCostPnL('Buy',4,18.2,-5,20.28)
SELECT * FROM fMath.mfCalc_RunningTotalBookCostPnL('Sell',1,16.7,-1,20.28)
SELECT * FROM fMath.mfCalc_RunningTotalBookCostPnL('Buy',3,18.6,-2,18.49)

显然，[fMath].[mfCalc_RunningTotalBookCostPnL] 可能需要调整，以便它可以从作为 OldRunningTotal 和 OldBookCost 的 NULL 条目开始，但这很简单。应用递归性质的 SQL 集理论有点困难。

非常感谢，伯蒂。

score 4 · Accepted Answer

运行总。更新临时表与 CTE

create table Test(
    OrderID int primary key,
    Qty int not null
);



declare @i int = 1;

while @i <= 5000 begin
    insert into Test(OrderID, Qty) values (@i * 2,rand() * 10); 
    set @i = @i + 1;
end;

递归求解需要 9 秒：

with T AS
(
    select ROW_NUMBER() over(order by OrderID) as rn, * from test
)
,R(Rn, OrderId, Qty, RunningTotal) as
(
    select Rn, OrderID, Qty, Qty
    from t 
    where rn = 1

    union all

    select t.Rn, t.OrderId, t.Qty, p.RunningTotal + t.Qty
    from t t
    join r p on t.rn = p.rn + 1

)
select R.OrderId, R.Qty, R.RunningTotal from r
option(maxrecursion 0);

UPDATE 表需要 0 秒：

create function TestRunningTotal()
returns @ReturnTable table(
    OrderId int, Qty int, RunningTotal int
)
as begin

    insert into @ReturnTable(OrderID, Qty, RunningTotal)
    select OrderID, Qty, 0 from Test
    order by OrderID;

    declare @RunningTotal int = 0;

    update @ReturnTable set 
           RunningTotal = @RunningTotal, 
           @RunningTotal = @RunningTotal + Qty;

    return;
end;

这两种方法至少可以为您提供一个构建查询的框架。

顺便说一句，在 SQL Server 中，与 MySQL 不同，变量赋值的顺序无关紧要。这：

update @ReturnTable set 
    RunningTotal = @RunningTotal, 
    @RunningTotal = @RunningTotal + Qty;

以及以下内容：

update @ReturnTable set 
    @RunningTotal = @RunningTotal + Qty,
    RunningTotal = @RunningTotal;

它们都以相同的方式执行，即变量赋值首先发生，而不管变量赋值在语句中的位置。两个查询都有相同的输出：

OrderId     Qty         RunningTotal
----------- ----------- ------------
2           4           4
4           8           12
6           4           16
8           5           21
10          3           24
12          8           32
14          2           34
16          9           43
18          1           44
20          2           46
22          0           46
24          2           48
26          6           54

在您的确切表格上，只需检测买入/卖出，您可以分别将其乘以 1 和 -1，或者您只需在字段上签名，例如：

update @ReturnTable set 
       @RunningTotal = @RunningTotal + 
                       CASE WHEN BuySell = 'Buy' THEN Qty ELSE -Qty END,
       RunningTotal = @RunningTotal;

如果您碰巧升级到 SQL Server 2012，以下是运行总计的简单实现：

select OrderID, Qty, sum(Qty) over(order by OrderID) as RunningTotal
from Test

关于您的确切问题：

select OrderID, Qty, 

   sum(CASE WHEN BuySell = 'Buy' THEN Qty ELSE -Qty END) 
   over(order by OrderID) as RunningTotal

from Test;

更新

如果您对古怪的 update感到不安，您可以添加一个保护子句来检查要更新的行的顺序是否与原始顺序匹配（由 identity(1,1) 辅助）：

create function TestRunningTotalGuarded()
returns @ReturnTable table(
    OrderId int, Qty int, 
    RunningTotal int not null, 
    RN int identity(1,1) not null
)
as begin

    insert into @ReturnTable(OrderID, Qty, RunningTotal)
    select OrderID, Qty, 0 from Test
    order by OrderID;

    declare @RunningTotal int = 0;

    declare @RN_check INT = 0;

    update @ReturnTable set 
            @RN_check = @RN_check + 1,
            @RunningTotal = 
                (case when RN = @RN_check then @RunningTotal + Qty else 1/0 end),
            RunningTotal = @RunningTotal;

    return;

end;

如果 UPDATE 真的以不可预知的顺序更新行（或任何机会），@RN_Check 将不再等于 RN（身份顺序），那么代码将引发被零除的错误。使用保护子句，不可预测的更新顺序将快速失败；如果发生这种情况，是时候向微软提交错误请愿书，让这个古怪的更新不那么古怪了 :-)

内在命令式操作（变量赋值）上的保护子句对冲实际上是顺序的。

score 2 · Accepted Answer

如果没有完全运行的 [fMath].[mfCalc_RunningTotalBookCostPnL] 进行测试，这有点暗中刺伤。我在测试前第一次获得递归 CTE 的记录只有大约 50%，但即使不完美，如果我正确理解您的要求，它也应该足以让您开始：

-- First, cache Table1 into #temp to improve recursive CTE performance
select
RowNum=ROW_NUMBER()OVER(ORDER BY OrderID)
, *
INTO #temp
FROM Table1;
GO

; WITH CTE (RowNum,OrderID, BuySell, FilledSize, ExecutionPrice, RunningTotal, AverageBookCost, RealisedPnL) AS (
    SELECT RowNum,OrderID, BuySell, FilledSize, ExecutionPrice, RunningTotal=0, AverageBookCost=0, RealisedPnL=0
    FROM #temp
    WHERE RowNum=1

    UNION ALL

    SELECT t.RowNum, t.OrderID, t.BuySell, t.FilledSize, t.ExecutionPrice
    , RunningTotal=c.NewRunningTotal, AverageBookCost=c.NewBookCost, RealisedPnL=c.PreMultRealisedPnL
    FROM #temp t
    INNER JOIN CTE ON CTE.RowNum+1 = t.RowNum
    CROSS APPLY [fMath].[mfCalc_RunningTotalBookCostPnL](t.BuySell, t.FilledSize, t.ExecutionPrice, CTE.RunningTotal, CTE.AverageBookCost) AS c
)
SELECT OrderID, BuySell, FilledSize, ExecutionPrice, RunningTotal, AverageBookCost, RealisedPnL
FROM CTE
/* Replace the above SELECT with the following after testing ok
UPDATE tab
SET RunningTotal=CTE.RunningTotal
, AverageBookCost=CTE.AverageBookCost
, RealisedPnL=CTE.RealisedPnL
FROM Table1 tab
INNER JOIN CTE on CTE.OrderID=tab.OrderID
*/
OPTION (MAXRECURSION 32767);
GO

-- clean up
DROP TABLE #temp
GO

还有一个免责声明 - 递归 CTE 适用于 32767 的最大深度。如果这限制性太大，您将需要探索一种不同的方法，或者对数据集进行某种窗口化。

score 0 · Accepted Answer

我重新制作了正在运行的总查询以包括一个分区（在客户上）

CTE方法：

with T AS
(
    select 
       ROW_NUMBER() over(partition by CustomerCode order by OrderID) as rn, * 
    from test
)
,R(CustomerCode, Rn, OrderId, Qty, RunningTotal) as
(
    select CustomerCode, Rn, OrderID, Qty, Qty
    from t 
    where rn = 1

    union all

    select t.CustomerCode, t.Rn, t.OrderId, t.Qty, p.RunningTotal + t.Qty
    from t t
    join r p on p.CustomerCode = t.CustomerCode and t.rn = p.rn + 1

)
select R.CustomerCode, R.OrderId, R.Qty, R.RunningTotal from r
order by R.CustomerCode, R.OrderId 
option(maxrecursion 0);

古怪的更新方法：

create function TestRunningTotalGuarded()
returns @ReturnTable table(
    CustomerCode varchar(50), OrderId int, Qty int, 
    RunningTotal int not null, RN int identity(1,1) not null
)
as begin

    insert into @ReturnTable(CustomerCode, OrderID, Qty, RunningTotal)
    select CustomerCode, OrderID, Qty, 0 from Test
    order by CustomerCode, OrderID;

    declare @RunningTotal int;

    declare @RN_check INT = 0;
    declare @PrevCustomerCode varchar(50) = NULL;

    update @ReturnTable set
            @RN_check = @RN_check + 1,
            @RunningTotal = 
                (case when RN = @RN_check then
                    case when @PrevCustomerCode = CustomerCode then
                        @RunningTotal + Qty 
                    else
                        Qty
                    end
                else
                    1/0 
                end),
            @PrevCustomerCode = CustomerCode,
            RunningTotal = @RunningTotal;

    return;
end;

光标方法（压缩代码以删除滚动条）

create function TestRunningTotalCursor()
returns @ReturnTable table(CustomerCode varchar(50), OrderId int, 
                           Qty int, RunningTotal int not null) as
begin     
    declare @c_CustomerCode varchar(50);
    declare @c_OrderID int;
    declare @c_qty int;

    declare @PrevCustomerCode varchar(50) = null;
    declare @RunningTotal int = 0;

    declare o_cur cursor for
    select CustomerCode, OrderID, Qty from Test order by CustomerCode, OrderID;     
    open o_cur;
    fetch next from o_cur into @c_CustomerCode, @c_OrderID, @c_Qty;

    while @@FETCH_STATUS = 0 begin

        if @c_CustomerCode = @PrevCustomerCode begin
            set @RunningTotal = @RunningTotal + @c_qty;
        end else begin
            set @RunningTotal = @c_Qty;
        end;

        set @PrevCustomerCode = @c_CustomerCode;

        insert into @ReturnTable(CustomerCode, OrderId, Qty, RunningTotal)
        values(@c_CustomerCode, @c_OrderID, @c_Qty, @RunningTotal);

        fetch next from o_cur into @c_CustomerCode, @c_OrderID, @c_Qty;
    end;

    close o_cur; deallocate o_cur; return;
end;

5,000 行的指标：

* Recursive CTE : 49 seconds
* Quirky Update : 0 second
* Cursor        : 0 second

那些 0 秒没有意义。在我将行数增加到 50,000 行后，以下是指标：

* Quirky Update : 1 second
* Cursor        : 3 second
* Recursive CTE : An hour

警告，我发现古怪的更新真的很古怪，有时它有效，有时它不（由五分之一的查询运行中存在被零除错误表示）。

这是数据的 DDL：

create table Test(
    OrderID int primary key,
    CustomerCode varchar(50),
    Qty int not null
);


declare @i int = 1;

while @i <= 20 begin
    insert into Test(OrderID, CustomerCode, Qty) values (
        @i * 2
        ,case @i % 4 
        when 0 then 'JOHN'
        when 1 then 'PAUL'
        when 2 then 'GEORGE'
        when 3 then 'RINGO'
        end
        ,rand() * 10);    
    set @i = @i + 1;
end;

更新

显然，纯 CTE 方法并不好。必须使用混合方法。将行编号具体化为实际表时，速度会上升

select ROW_NUMBER() over(partition by CustomerCode order by OrderID) as rn, * into #xxx 
from test;

with T AS
(
    select * from #xxx
)
,R(CustomerCode, Rn, OrderId, Qty, RunningTotal) as
(
    select CustomerCode, Rn, OrderID, Qty, Qty
    from t 
    where rn = 1

    union all

    select t.CustomerCode, t.Rn, t.OrderId, t.Qty, p.RunningTotal + t.Qty
    from t t
    join r p on p.CustomerCode = t.CustomerCode and t.rn = p.rn + 1

)
select R.CustomerCode, R.OrderId, R.Qty, R.RunningTotal from r
order by R.CustomerCode, R.OrderId 
option(maxrecursion 0);

drop table #xxx;

回顾一下，这里是在将纯 CTE 转换为使用物化行编号之前的指标（行编号结果在实际表中，即在临时表中）

* Quirky Update       : 1 second
* Cursor              : 3 second
* Recursive CTE(Pure) : An hour

将行编号具体化到临时表后：

* Quirky Update         : 1 second
* Cursor                : 3 second
* Recursive CTE(Hybrid) : 2 second (inclusive of row numbering table materialization)

混合递归 CTE 方法实际上比游标方法更快。

另一个更新

只需将聚集的主键放在顺序列上，UPDATE 就可以更新其物理顺序上的行。不再发生除以零（检测非顺序更新的保护子句）。例如

alter function TestRunningTotalGuarded()
returns @ReturnTable table(
    CustomerCode varchar(50), OrderId int, Qty int, 
    RunningTotal int not null, 
    RN int identity(1,1) not null primary key clustered
)

如果可能存在极端情况，我尝试运行古怪的更新（使用集群主键）100 次，但到目前为止我没有发现。我没有遇到任何被零除的错误。阅读这篇博文底部的结论：http ://www.ienablemuch.com/2012/05/recursive-cte-is-evil-and-cursor-is.html

即使有集群主键，它仍然很快。

这是 100,000 行的指标：

Quirky Update        : 3 seconds
Hybrid Recursive CTE : 5 seconds
Cursor               : 6 seconds

古怪的更新（毕竟不是那么古怪）仍然很快。它比混合递归 CTE 更快。

sql - 能否使用 CTE 或 OVER 将此递归解决方案写入 T-SQL 查询？

3 回答 3

运行总。更新临时表与 CTE

Related

Reference