performance - Oracle 10g 中的字符串匹配，其中任何一方都可以有通配符

Question

测试用例模式和数据如下：

create table tmp
(
 vals varchar(8),
 mask varchar(8)
);


insert into tmp values ('12345678','        ');

insert into tmp values ('12_45678','  _     ');

insert into tmp values ('12345678','   _    ');

insert into tmp values ('92345678','        ');

insert into tmp values ('92345678','     _  ');

现在忽略掩码列并假设存在 specialmatch 函数：

select VALS from tmp where specialmatch(vals,'12345678');

应该产生：

下一个，

select VALS from tmp where specialmatch(vals,'92345678');

应该产生：

VALS     
92345678 
92345678

下一个，

select VALS from tmp where specialmatch(vals,'_2345678');

应该产生：

下一个，

select VALS from tmp where specialmatch(vals,'12945678');

应该产生：

VALS     
12_45678

关于如何制作特殊匹配功能的任何想法？

我天真的方法是写一个特殊的字符串比较 udf（伪代码）：

bool function specialmatch(str1,str2) DETERMINISITC
{
 return false if either are null;
 for each char1,char2 of str1,str2
 {
  if (char1<>char2 && char1<>'_' && char2<>'_') return false;
 }
 return true;
}

同样在进行匹配之前，需要将掩码覆盖在 val 上。

例如：val='1_345678', mask=' _ _' => 1_34567_ 并且匹配 12345678 和 19345679 但不匹配 92345678。

但是如何做到这一点来利用索引、优化器等......

score 0 · Accepted Answer

Oracle 10g 确实有一个正则表达式函数，在这种情况下可能会对您有所帮助。 http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14251/adfns_regexp.htm

此外，如果您必须在数据库中执行此操作，您可以查看 java 存储过程。

我不知道在这种情况下会有什么索引可以帮助你，尽管 _ 可以出现在包括第一个字符在内的任何地方。

score 0 · Accepted Answer

面具只有一个字符吗？如果是这样，您可以通过以下方式限制可能性

select VALS from tmp 
where specialmatch(vals,'12945678')
and (substr(vals,1,4) = substr('12945678',1,4) 
      or substr(vals,5) = substr('12945678',5));

然后在 substr(vals,1,4) 和 substr(vals,5) 上有基于函数的索引。我似乎记得读过 FBI 可能没有为这些制定最佳计划的问题，因此替代 SQL 将是

    select VALS from tmp 
    where specialmatch(vals,'12945678')
    and substr(vals,1,4) = substr('12945678',1,4)
    union 
    select VALS from tmp 
    where specialmatch(vals,'12945678')
    substr(vals,5) = substr('12945678',5));

score 0 · Accepted Answer

接下来的建议。简单选项： _ 是 LIKE 的单个字符匹配，所以简单的解决方案是

SELECT * FROM tmp WHERE vals LIKE v_param OR v_param LIKE vals;

每次都是全表扫描，但是省去了SQL和PL/SQL层的切换

复杂选项位图索引 substr 上的每个单独的字符。那种多索引凝灰岩是位图擅长的。位图对于具有大量更新的列或具有大量小插入的表来说是个麻烦。

我已经建立了一个测试测试。首先，我将 10,000 个值加载到 TMP 中，几乎是随机生成的。不确定您的数据集有多大，或者没有通配符、一个通配符或多个通配符的条目的比例。这将对结果产生很大影响。

create table tmp ( vals varchar(8),  mask varchar(8));

insert into tmp
select new_val, translate(new_val,'0123456789','__________')
from
  (select case 
         when rn_3  is not null then translate(val,'34','__')
         when rn_5  is not null then translate(val,'2','_')
         when rn_7  is not null then translate(val,'78','__')
         when rn_11 is not null then translate(val,'12345','_____')
         else val end new_val
  from
    (select lpad(trunc(dbms_random.value(1,99999999)),8,'0') val, 
            decode(mod(rownum,3),0,1) rn_3, decode(mod(rownum,5),0,1) rn_5, 
            decode(mod(rownum,7),0,1) rn_7, decode(mod(rownum,11),0,1) rn_11
     from dual connect by level < 10000)
  )

declare
  cursor c_1 is
    select case 
         when rn_3  is not null then translate(val,'34','__')
         when rn_5  is not null then translate(val,'2','_')
         when rn_7  is not null then translate(val,'78','__')
         when rn_11 is not null then translate(val,'12345','_____')
         else val end try_val
    from
      (select lpad(trunc(dbms_random.value(1,99999999)),8,'0') val, 
              decode(mod(rownum,3),0,1) rn_3, decode(mod(rownum,5),0,1) rn_5, 
              decode(mod(rownum,7),0,1) rn_7, decode(mod(rownum,11),0,1) rn_11
       from dual connect by level < 1000);
  v_cnt number;
  v_start number;
  v_end number;
begin
 v_start := dbms_utility.get_time;
 for c_rec in c_1 loop
    select count(*) into v_cnt 
    from tmp 
    where (c_rec.try_val like vals or vals like c_rec.try_val);
 end loop;
 v_end := dbms_utility.get_time;
 dbms_output.put_line('Meth 1 :'||(v_end - v_start));
 v_start := dbms_utility.get_time;
 for c_rec in c_1 loop
    select count(*) into v_cnt from
      (select * from (select * from tmp where mask = '        ') v1 
       where vals like c_rec.try_val
       union all
       select * from (select * from tmp where mask > '        ') v2 
       where vals like maskmerge(mask,c_rec.try_val));
 end loop;
 v_end := dbms_utility.get_time;
 dbms_output.put_line('Meth 2 :'||(v_end - v_start));
end;
/

我已经将“双头 LIKE”与掩码合并进行了比较。在测试中，LIKE 通常得到大约 200-250（百分之一秒），而 maskmerge 大约需要十倍的时间。正如我所说，这在很大程度上取决于数据分布。

score 0 · Accepted Answer

我将表格“划分”为两个不同的集合，没有面具的人（v1）和有面具的人（v2）

select * from (select * from tmp where mask = '        ') v1 where vals like :srch
 union all
select * from (select * from tmp where mask > '        ') v2 where vals like maskmerge(mask,:srch);

现在，优化器说：

Operation                                Object Name       Rows Bytes Cost
SELECT STATEMENT Optimizer Mode=ALL_ROWS                      2          5    
  UNION-ALL
    TABLE ACCESS BY INDEX ROWID          SCHEMA.TMP           1    90    2  
      INDEX RANGE SCAN                   SCHEMA.I_TMP_MASK    1          1
    TABLE ACCESS BY INDEX ROWID          SCHEMA.TMP           1    90    3
      INDEX RANGE SCAN                   SCHEMA.I_TMP_MASK    2          1

这非常好，即使我的 :srch 中有通配符，Oracle 也可以优化掉。

最后，即使没有提示，关于 vals 和 mask cols 的标准索引也足以解决问题。在 10g 上测试。注意：我们使用 union all，因为 v1 和 v2 总是互斥的。

供参考：

CREATE OR REPLACE FUNCTION maskmerge (A IN VARCHAR, B IN VARCHAR)
 RETURN VARCHAR deterministic parallel_enable
 IS  
 alen int;
 blen int;
 mlen int;   
 res varchar(4000);
 ca char;
 cb char;
BEGIN
 if (a is null) then
  return b;
 end if;
 if (b is null) then
  return a;
 end if;
 alen:=length(a);
 blen:=length(b);
 if (alen<blen) then
  mlen:=alen;
 else
  mlen:=blen;
 end if;
 for i in 1 .. mlen loop
  ca:=substr(a,i,1);
  cb:=substr(b,i,1);
  if (ca='_' or cb='_') then
   res:=res||'_';
  elsif (ca=' ') then
   res:=res||cb;
  elsif (cb=' ') then
   res:=res||ca;
  else
   res:=res||cb;
  end if;
 end loop;
 return res;
END;

完整测试用例（具有典型数据分布）：

-----------------------------------------------------------------

CREATE OR REPLACE FUNCTION maskmerge (A IN VARCHAR, B IN VARCHAR)
 RETURN VARCHAR deterministic parallel_enable
 IS  
 alen int;
 blen int;
 mlen int;   
 res varchar(4000);
 ca char;
 cb char;
BEGIN
 if (a is null) then
  return b;
 end if;
 if (b is null) then
  return a;
 end if;
 alen:=length(a);
 blen:=length(b);
 if (alen<blen) then
  mlen:=alen;
 else
  mlen:=blen;
 end if;
 for i in 1 .. mlen loop
  ca:=substr(a,i,1);
  cb:=substr(b,i,1);
  if (ca='_' or cb='_') then
   res:=res||'_';
  elsif (ca=' ') then
   res:=res||cb;
  elsif (cb=' ') then
   res:=res||ca;
  else
   res:=res||cb;
  end if;
 end loop;
 return res;
END;
/

create table tmp
(
id int not null primary key,
ipv6address varchar(32) not null,
ipv6addressmask varchar(32) default ('                                ') not null
);

create sequence s_tmp;

create index i_tmp_addr on tmp(ipv6address);

create index i_tmp_mask on tmp(ipv6addressmask);

create or replace trigger t_i_tmp before insert on tmp referencing new as new old as old FOR EACH ROW
DECLARE
    tmpVar tmp.id%TYPE;
begin
SELECT s_tmp.NEXTVAL INTO tmpVar FROM dual;
:new.id:=tmpVar;
end;

exec dbms_random.initialize(17809465);

insert into tmp (ipv6address) 
select decode(trunc(dbms_random.value(0,2)),0,'20010db80000000000000000',1,'00000000000000000000ffff','00000000000000000000ffff') 
||trim(to_char(dbms_random.value(0, 4294967296),'0000000x')) 
as val from dual
connect by level <= 10000;

insert into tmp
SELECT * FROM
( SELECT * FROM tmp
ORDER BY dbms_random.value )
WHERE rownum <= 200;

insert into tmp values (null,'00000000000000000000ffff12345678','                                ');

insert into tmp values (null,'00000000000000000000ffff12345678','                              _ ');

insert into tmp values (null,'00000000000000000000ffff1234567_','                              __');

--select * from tmp order by ipv6address

-- network redaction of ipv4 
update tmp set ipv6addressmask=maskmerge('                        ______  ',ipv6addressmask),ipv6address=maskmerge('                        ______  ',ipv6address) where length(ipv6address)/32*dbms_random.value<0.005;

-- host redaction of ipv4 
update tmp set ipv6addressmask=maskmerge('                              __',ipv6addressmask),ipv6address=maskmerge('                              __',ipv6address) where length(ipv6address)/32*dbms_random.value<0.005;

-- full redaction of ipv4 
update tmp set ipv6addressmask=maskmerge('                              __',ipv6addressmask),ipv6address=maskmerge('                              __',ipv6address) where ipv6addressmask='                        ______  ' and length(ipv6address)/32*dbms_random.value<0.04;

-- network report redaction of ipv4 
update tmp set ipv6addressmask=maskmerge('                        ______  ',ipv6addressmask) where length(ipv6address)/32*dbms_random.value<0.005;

-- host report redaction of ipv4 
update tmp set ipv6addressmask=maskmerge('                              __',ipv6addressmask) where length(ipv6address)/32*dbms_random.value<0.005;

-- full report redaction of ipv4 
update tmp set ipv6addressmask=maskmerge('                              __',ipv6addressmask) where ipv6addressmask='                        ______  ' and length(ipv6address)/32*dbms_random.value<0.04;

select count(*) from tmp where instr(ipv6address,'_')>0;

select count(*) from tmp where ipv6addressmask > '                                '; 

-- srch := '00000000000000000000ffff12345678';  

select * from (select * from tmp where ipv6addressmask = '                                ') v1 where ipv6address like :srch
 union all
select * from (select * from tmp where ipv6addressmask > '                                ') v2 where ipv6address like maskmerge(ipv6addressmask,:srch);

/*
Operation                                Object Name Rows Bytes Cost
---------------------------------------- ----------- ---- ----- ----
SELECT STATEMENT Optimizer Mode=ALL_ROWS              510         29                                 
  UNION-ALL                                              
    TABLE ACCESS BY INDEX ROWID          TMP          500   23K   10                                 
      INDEX RANGE SCAN                   I_TMP_ADDR    92          2                                 
    TABLE ACCESS BY INDEX ROWID          TMP           10   490   19                                 
      INDEX RANGE SCAN                   I_TMP_MASK   207          2                                 

*/

SELECT * FROM tmp WHERE ipv6address LIKE :srch OR :srch LIKE ipv6address

/*

Operation                                Object Name Rows Bytes Cost
---------------------------------------- ----------- ---- ----- ----
SELECT STATEMENT Optimizer Mode=ALL_ROWS              995         22                                 
  TABLE ACCESS FULL                      TMP          995   47K   22                                 

*/

-----------------------------------------------------------------

drop table tmp;

drop sequence s_tmp;

drop function maskmerge;

-----------------------------------------------------------------

score 0 · Accepted Answer

仅供参考，我发现当您需要对 %var% 进行匹配并让它快速处理大量数据时，最好的方法是使用 Oracle 的 Oracle 文本索引。

performance - Oracle 10g 中的字符串匹配，其中任何一方都可以有通配符

5 回答 5

Related

Reference