optimization - Symfony2，Doctrine，添加\插入\更新大量查询的最佳解决方案

Question

假设我们有这样的代码：

while (true)
{
    foreach($array as $row)
    {
       $item = $em->getRepository('reponame')->findOneBy(array('filter'));

       if (!$item)
       {
           $needPersist = true;
           $item = new Item();
       }

       $item->setItemName()
       // and so on ...

       if ($needPersist)
       {
           $em->persist();
       }
    }
    $em->flush();
}

所以，关键是代码将被执行很多次（而服务器不会死:)）。我们想要优化它。每次我们：

1. 从存储库中选择已经条目。
2. 如果条目不存在，则创建它。
3. 为其设置新的（更新）变量。
4. 应用操作（刷新）。

所以问题是 - 如何避免不必要的查询并优化“检查条目是否存在”？因为当有 100-500 个查询时它并不那么可怕......但是当它在一个 while 循环中达到 1000-10000 时 - 它太多了。

PS：DB 中的每个条目在多列中都是唯一的（不仅仅是 ID）。

Answer 1

1. Instead of fetching results one-by-one, load all results with one query.

Eg. let's say your filter wants to load ids 1, 2, 10. So QB would be something like:

$allResults = ...
    ->where("o.id IN (:ids)")->setParameter("ids", $ids) 
    ->getQuery()
    ->getResults() ;

1. "foreach" of these results, do your job of updating them and flushing

2. While doing that loop, save ids of those fetched objects in new array

3. Compare that array with original one using array_diff. Now you have ids that were not fetched the first time

4. Rinse and repeat :)

5. And don't forget $em->clear() to free memory

While this can still be slow when working with 10.000 records (dunno, never tested), it will be much faster to have 2 big queries than 10.000 small ones.

Answer 2

Regardless if you need them to persist or not after the update, retrieving 10k+ and up entries from the database and hydrating them to php objects is going to need too much memory. In such cases you should better fallback to the Doctrine DBAL Layer and fire pure SQL queries.