2

I am creating an import data tool from several vendors. Unfortunately the data is not generated by me, so i have to work with it. I have come across the following situation.

I have a table like the following:

ID    |SartDate    |Availability
========================================
H1    |20130728    |YYYYYYNNNNQQQQQ
H2    |20130728    |NNNNYYYYYYY
A3    |20130728    |NNQQQQNNNNNNNNYYYYYY
A2    |20130728    |NNNNNYYYYYYNNNNNN

To explain what this data means is: Every letter in the Availability column is the availability flag for a specific date, starting from the date noted in the StartDate column.

  • Y : Available
  • N : Not Available
  • Q : On Request

For instance for ID H1 20130728 - 20130802 is available, then from 20130803 - 20130806 is not available and from 20130807 - 20130811 is available on request.

What i need to do is transform this table to the following setup:

ID    |Available   |SartDate    |EndDate     
========================================
H1    |Y           |20130728    |20130802    
H1    |N           |20130803    |20130806    
H1    |Q           |20130806    |20130811    
H2    |N           |20130728    |20130731
H2    |Y           |20130801    |20130807
A3    |N           |20130728    |20130729
A3    |Q           |20130730    |20130802
A3    |N           |20130803    |20130810
A3    |Y           |20130811    |20130816
A2    |Y           |20130728    |20130801
A2    |Y           |20130802    |20130807
A2    |Y           |20130808    |20130813

The initial table has approximately 40,000 rows. The Availability column may have several days (I've seen up to 800).

What i have tried is turn the Availability into rows and then group consecutive days together and then get min and max date for each group. For this i have used three or four CTEs

This works fine for a few IDs, but when i try to apply it to the whole table it take ages (I stopped the initial test run after a fool time sleep and it hadn't finish, and yes i mean i was sleeping while it was running!!!!)

I have estimated that if i turn each character in a single row then i end up with something like 14.5 million rows.

So, i am asking, is there a more efficient way of doing this? (I know there is, but i need you to tell me)

Thanks in advance.

4

4 回答 4

2

这可以在 SQL Server 中使用递归 CTE 完成。这是一个例子：

with t as (
    select 'H1' as id, cast('20130728' as date) as StartDate,
           'YYYYYYNNNNQQQQQ' as Availability union all
    select 'H2' as id, cast('20130728' as date) as StartDate,
           'NNNNYYYYYYY' as Availability union all
    select 'H3' as id, cast('20130728' as date) as StartDate,
           'NQ' as Availability 
   ),
   cte as (
     select id, left(Availability, 1) as Available,
            StartDate as thedate,
            substring(Availability, 2, 1000) as RestAvailability,
            1 as i,
            1 as periodcnt
     from t
     union all
     select t.id, left(RestAvailability, 1),
            dateadd(dd, 1, thedate),
            substring(RestAvailability, 2, 1000) as RestAvailability,
            1 + cte.i,
            (case when substring(t.Availability, i, 1) = substring(t.Availability, i+1, 1)
                  then periodcnt
                  else periodcnt + 1
             end)
     from t join
          cte
          on t.id = cte.id
     where len(RestAvailability) > 0

   )
select id, min(thedate), max(thedate), Available
from cte
group by id, periodcnt, Available;

它的工作方式是它首先分散日期。这将是 CTE 的“典型”用途。在此过程中，它还跟踪是否Available与之前的值（在变量中）发生了变化periodcnt。它为此使用了字符串操作。

有了这些信息，最终结果就是来自这个 CTE 的聚合。

于 2013-07-30T11:05:22.403 回答
0

As SQL Server is not the best tool, If I had to do this, I would probably set up an Integration Services package where I would use a script component to code the generate the several records from one in C#.

于 2013-07-30T09:38:42.600 回答
0

我尝试了不同的方法。我认为我可以使用 LINQ 进行转换，然后将输出批量加载到数据库，而不是使用带有初始 xml 文件的 SQLXMLBulkLoad 库。

所以我最初的 xml 是这样的：

<vacancies>
  <vacancy>
    <code>AT1010.200.1</code>
    <startday>2010-07-01</startday>
    <availability>YYYYYYNNNNQQQQQ</availability>
    <changeover>CCIIOOX</changeover>
    <minstay>GGGGGGGG</minstay>
    <flexbooking>YYYYY</flexbooking>
  </vacancy>
  <vacancy>
    <code>AT1010.200.2</code>
    <startday>2010-07-01</startday>
    <availability>NNNNYYYYYYY</availability>
    <changeover>CCIIOOX</changeover>
    <minstay>GGGGGGGG</minstay>
    <flexbooking>YYYYY</flexbooking>
  </vacancy>
  <vacancy>
    <code>AT1010.200.3</code>
    <startday>2010-07-01</startday>
    <availability>NNQQQQNNNNNNNNYYYYYY</availability>
    <changeover>CCIIOOX</changeover>
    <minstay>GGGGGGGG</minstay>
    <flexbooking>YYYYY</flexbooking>
  </vacancy>
  <vacancy>
    <code>AT1010.200.4</code>
    <startday>2010-07-01</startday>
    <availability>NNNNNYYYYYYNNNNNN</availability>
    <changeover>CCIIOOX</changeover>
    <minstay>GGGGGGGG</minstay>
    <flexbooking>YYYYY</flexbooking>
  </vacancy>
</vacancies>

这里的任务是创建一个新的 xml，其中包含每组可用性标志的开始日期和结束日期。

XElement xe = XElement.Load(file);

int i = 0;
char previousFlag = ' ';
int GroupIndex = 0;

XElement vacancies =
new XElement
(
    "vacancies",
    xe.Elements("vacancy")
    .Select
    (
        x =>
        {
            i = 0;
            GroupIndex = 0;
            return new
            {
                availabilities = x.Element("availability")
                .Value
                .Select
                (
                    v =>
                    {
                        if (previousFlag != v)
                        {
                            GroupIndex++;
                        }
                        previousFlag = v;
                        return new
                        {
                            Code = x.Element("code").Value,
                            startday = x.Element("startday").Value,
                            Date = DateTime.Parse(x.Element("startday").Value).AddDays(i++),
                            GIndex = GroupIndex
                        };
                    }
                )
            };
        }
    )
    .SelectMany
    (
        x =>
        x.availabilities
    )
    .GroupBy
    (
        g =>
        new
        {
            Code = g.Code,
            startday = g.startday,
            GroupIndex = g.GIndex
        }
    )
    .Select
    (
        x =>
        new XElement
        (
            "vacancy",
            new XElement("code", x.Key.Code),
            new XElement("startday", x.Key.startday),
            new XElement("GroupIndex", x.Key.GroupIndex),
            new XElement("minDate", x.Min(z => z.Date)),
            new XElement("maxDate", x.Max(z => z.Date))
        )
    )
);
vacancies.Save(outputfile);

打开输出文件，我有以下 xml 格式：

<vacancies>
  <vacancy>
    <code>AT1010.200.1</code>
    <startday>2010-07-01</startday>
    <GroupIndex>1</GroupIndex>
    <minDate>2010-07-01T00:00:00</minDate>
    <maxDate>2010-07-06T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.1</code>
    <startday>2010-07-01</startday>
    <GroupIndex>2</GroupIndex>
    <minDate>2010-07-07T00:00:00</minDate>
    <maxDate>2010-07-10T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.1</code>
    <startday>2010-07-01</startday>
    <GroupIndex>3</GroupIndex>
    <minDate>2010-07-11T00:00:00</minDate>
    <maxDate>2010-07-15T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.2</code>
    <startday>2010-07-01</startday>
    <GroupIndex>1</GroupIndex>
    <minDate>2010-07-01T00:00:00</minDate>
    <maxDate>2010-07-04T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.2</code>
    <startday>2010-07-01</startday>
    <GroupIndex>2</GroupIndex>
    <minDate>2010-07-05T00:00:00</minDate>
    <maxDate>2010-07-11T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.3</code>
    <startday>2010-07-01</startday>
    <GroupIndex>1</GroupIndex>
    <minDate>2010-07-01T00:00:00</minDate>
    <maxDate>2010-07-02T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.3</code>
    <startday>2010-07-01</startday>
    <GroupIndex>2</GroupIndex>
    <minDate>2010-07-03T00:00:00</minDate>
    <maxDate>2010-07-06T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.3</code>
    <startday>2010-07-01</startday>
    <GroupIndex>3</GroupIndex>
    <minDate>2010-07-07T00:00:00</minDate>
    <maxDate>2010-07-14T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.3</code>
    <startday>2010-07-01</startday>
    <GroupIndex>4</GroupIndex>
    <minDate>2010-07-15T00:00:00</minDate>
    <maxDate>2010-07-20T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.4</code>
    <startday>2010-07-01</startday>
    <GroupIndex>1</GroupIndex>
    <minDate>2010-07-01T00:00:00</minDate>
    <maxDate>2010-07-05T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.4</code>
    <startday>2010-07-01</startday>
    <GroupIndex>2</GroupIndex>
    <minDate>2010-07-06T00:00:00</minDate>
    <maxDate>2010-07-11T00:00:00</maxDate>
  </vacancy>
  <vacancy>
    <code>AT1010.200.4</code>
    <startday>2010-07-01</startday>
    <GroupIndex>3</GroupIndex>
    <minDate>2010-07-12T00:00:00</minDate>
    <maxDate>2010-07-17T00:00:00</maxDate>
  </vacancy>
</vacancies>

它是平坦的，可以由 SQLXMLBulkLoad 工具处理，无需进一步处理。

我的初始 xml 为 60MB，不到一分钟就转换为 45MB 文件，虽然我没有测试新文件中的 SQLXMLBulkLoad，但它会很快，因为我知道它在初始文件中的性能。

我仍然会尝试你所有的解决方案，因为你当然值得一试，我会接受其中最好的。

谢谢大家的努力。

于 2013-07-31T08:34:36.297 回答
0

您是否尝试使用CROSS APPLY它可以提供更好的性能？这不是一个完整的回应。只是另一种方式解析？

编辑：我现在使用table variable索引表。

DECLARE @MaxLen INT
SELECT @MaxLen = MAX(LEN(Availability))
FROM InputTable 
DECLARE @a TABLE (i int)

;WITH x AS
(
    SELECT 1 AS i
    UNION ALL SELECT i + 1 FROM x WHERE i <= @MaxLen
)
INSERT INTO @a 
SELECT i FROM x
OPTION (MAXRECURSION 0);


;WITH cte AS (
SELECT *, DATEADD(DAY, i-1, StartDate) StatusAtDay
FROM InputTable t
cross apply (
    select SUBSTRING(t.Availability, i, 1)  as c, i
    from @a
    WHERE LEN(Availability) >= i
    ) ca
)
SELECT *
FROM cte
order by 1

我尝试了大约 5000 行，长度Availability大于 1250 的地方花了 19 秒（将输出扔到临时表）。

于 2013-07-30T11:57:52.570 回答