3

我目前正在尝试将大约 2000 万行数据从文本文件导入表。它的速度太慢了（加上2000万还不够）。

有什么方法可以让这个过程变得更快吗？顺便说一句，我从 phpmyadmin 运行它...

<?php 

//connection to the database
$dbhandle = mysql_connect($hostname, $username, $password) 
  or die("Unable to connect to MySQL");
echo "Connected to MySQL<br>";

$selected = mysql_select_db("data",$dbhandle) 
  or die("Could not select data");

$handle = fopen ('text.txt', 'rt');  
while (!feof ($handle))  
{  
    ini_set('max_execution_time',10800);
    $buffer = fgets($handle, 4096);

    list($a,$b,$c)=explode(" ",$buffer);
    $lol = explode(".",$c);
    $rest = substr($c,-5,3);
    $date = date('Y-m-d H:i:s');


    if($rest == 'COM'){
    echo $a." | ".$b." | ".$c."<br>";
    $sqlquery = "INSERT INTO zonenet (date, domainname, dnstype, nameserver) VALUES('".$date."','".$a."','".$b."','".$c."')";   
    mysql_query($sqlquery,$dbhandle) or die(mysql_error());
    } 
    else {
    //$dnstype = array("a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", ".",
//          "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z");
    $str = str_replace($lol[1],".NET",$lol[1]);
    //$str = str_replace(end($lol),".NET",end($lol));
    echo $a." | ".$b." | ".$lol[0]."".$str."<br>";
    $sqlquery = "INSERT INTO zonenet (date, domainname, dnstype, nameserver) VALUES('".$date."','".$a."','".$b."','".$lol[0]."".$str."')";  
    mysql_query($sqlquery,$dbhandle) or die(mysql_error());
    }
    //dnstype reference @http://en.wikipeda.org/wiki/List_of_DNS_record_types
    $sqlquery1 = "DELETE FROM zonenet WHERE dnstype NOT IN ('A', 'AAAA', 'AFSDB', 'APL', 'CAA', 'CERT', 'CNAME', 'DHCID'`enter code here`, 'DLV', 'DNAME', 'DNSKEY', 'DS', 'HIP', 'IPSECKEY', 'KEY', 'KX', 'LOC', 'MX', 'NAPTR', 'NS', 'NSEC', 'NSEC3', 'NSEC3PARAM', 'PTR', 'RRSIG', 'RP', 'SIG', 'SOA', 'SPF', 'SRV', 'SSHFP', 'TA', 'TKEY', 'TLSA', 'TSIG', 'TXT')"; 
        mysql_query($sqlquery1,$dbhandle) or die(mysql_error()); 

}   
fclose ($handle); 

//close the connection
mysql_close($dbhandle);

    //$net=explode(".",$c);
    //echo $a."-".$b."-".$c."<br>";
    //

    //
    //mysql_query($sqlquery,$dbhandle) or die(mysql_error()); 
?>
4

2 回答 2

1

20M 记录并不多，所以这应该很容易优化：

  1. 循环内不需要 ini_set，一次就足够了
  2. 使用批量插入。现在你为每条记录做一个插入
  3. 在插入而不是“删除”错误记录之前处理检查。

代码更改：（未经测试）

<?php

//connection to the database
$dbhandle = mysql_connect($hostname, $username, $password) or die("Unable to connect to MySQL");

echo "Connected to MySQL<br>";

$selected = mysql_select_db("data",$dbhandle) or die("Could not select data");

ini_set('max_execution_time',10800);

$allowedDnsType = array('A', 'AAAA', 'AFSDB', 'APL', 'CAA', 'CERT', 'CNAME', 'DHCID', 'DLV', 'DNAME', 'DNSKEY', 'DS', 'HIP', 'IPSECKEY', 'KEY', 'KX', 'LOC', 'MX', 'NAPTR', 'NS', 'NSEC', 'NSEC3', 'NSEC3PARAM', 'PTR', 'RRSIG', 'RP', 'SIG', 'SOA', 'SPF', 'SRV', 'SSHFP', 'TA', 'TKEY', 'TLSA', 'TSIG', 'TXT');
$lookup = array_flip($allowedDnsType);

$values = array();

$handle = fopen ('text.txt', 'rt');
while (!feof ($handle)) {
    $buffer = fgets($handle, 4096);

    list($domain,$dnstype,$nameserver)=explode(" ",$buffer);

    $lol = explode(".",$nameserver);
    $rest = substr($nameserver,-5,3);
    $date = date('Y-m-d H:i:s');


    if (isset($lookup[$dnstype])) {

        if($rest == 'COM'){
            echo $domain." | ".$dnstype." | ".$c."<br>";
            $values[] = "'".$date."','".$domain."','".$dnstype."','".$nameserver."'";
        }
        else {
            $str = str_replace($lol[1],".NET",$lol[1]);
            //$str = str_replace(end($lol),".NET",end($lol));
            echo $domain." | ".$dnstype." | ".$lol[0]."".$str."<br>";
            $values[] = "'".$date."','".$domain."','".$dnstype."','".$lol[0].$str."'";
        }
    }

    // insert per 200
    if (count($values) > 200) {
        $sqlquery = "INSERT INTO zonenet (date, domainname, dnstype, nameserver) VALUES(".implode('),(', $values).")";
        mysql_query($sqlquery,$dbhandle) or die(mysql_error());
        $values = array();
    }

    //dnstype reference @http://en.wikipeda.org/wiki/List_of_DNS_record_types
}
fclose ($handle);

if (count($values) > 0) {
    $sqlquery = "INSERT INTO zonenet (date, domainname, dnstype, nameserver) VALUES(".implode('),(', $values).")";
    mysql_query($sqlquery,$dbhandle) or die(mysql_error());
}

//close the connection
mysql_close($dbhandle);
于 2013-10-25T08:52:09.620 回答
0

2000 万行需要时间，因为磁盘限制了在 mysql 中实际可以执行的插入次数。

我对此的首要建议是使用准备好的语句。根据您的 mysql 驱动程序，它将类似于：

$driver->prepareStatement("insert into table (`field`) values (?)");
foreach ($rows as $row) {
   $driver->bindParam('s', $row->value);
   $driver->execute();
}

根据服务器上的内存，上述可能会崩溃，因为您一次将所有行都放在内存中。这可以通过逐行流式传输文件来解决，但是使用http://php.net/manual/en/function.stream-get-line.php之类的函数应该是相当直接的

于 2013-10-22T06:23:02.057 回答