2

I've tried many ways in an attempt to insert a large amount of data parsed from a text file (200,000 lines of text gets parsed into the array in about 2 seconds on my server so I know that part isn't the issue). The mysqli_multi_query I'm using sends the queries in blocks of 5,000 (for some reason it refuses to do anything larger if I want to use multi_query more then once) OR if I split the text file down to 50,000ish rows I can insert them all at once in 2 seconds. If I do that then it refuses to run another query (I did read I need to do while (mysqli_more_results($db) && mysqli_next_result($db)); to clear the results out and I do have that for my 5,000 split one)

The 50,000 insert one runs in 2 seconds, the 5,000 insert one (of 50,000 rows) takes 40 seconds, I'm kind of tempted to just create a new mysqli connection in the foreach (for 50,000 inserts), but it doesn't always work...

I'd like some tips on how to make my function run faster since 40 seconds to insert all 50,000 takes forever when the file I'm reading from can contain sever hundred thousand possibly millions of lines

function log2db(){
$db = mysqli_connect(SQLHOST, SQLUSER, SQLPASS, SQLDB);
if($db){
    $qar = logparse("world2.chat", loglen());
    $query = "";
    $i = 0;
    foreach($qar['data'] as $q){
        $i++;
        $uid = $q['uid'];
        $type = $q['type'];
        $chldst = $q['chldst'];
        $time = $q['date'];
        $msg = $q['msg'];
        $query .= "insert into `messages` (`uid`, `type`, `chldst`, `time`, `message`) values ('$uid', '$type', '$chldst', '$time', '$msg');\n";
        if(!($i % 5000)){
            $qquery[] = $query;
            $query = "";
        }
    }
    $qquery[] = $query;
    foreach($qquery as $qq){
        $q = mysqli_multi_query($db, $qq);
        if($q){
            while (mysqli_more_results($db) && mysqli_next_result($db));
        } else {
            mysqli_error($db);
            return false;
        }
    }
    if($qar !== null){
        loglen($qar['filelen']);
    }
    return true;
}
return false;

}

Is what I've come up with (always works but like I said is slow), any tips on how I can improve it?

4

1 回答 1

2

您是否意识到 INSERT INTO 查询可以在单个查询中插入多行？

$parts  = array();

foreach($qar['data'] as $q){

    $uid    = $q['uid'];
    $type   = $q['type'];
    $chldst = $q['chldst'];
    $time   = $q['date'];
    $msg    = $q['msg'];

    $parts[] = "('$uid', '$type', '$chldst', '$time', '$msg')";
 }

$query  = "INSERT INTO `messages` (`uid`, `type`, `chldst`, `time`, `message`)";
$query .= "VALUES ".implode(', ', $parts);

mysqli_query($db, $query)
于 2011-02-21T21:42:27.480 回答