0

由于未知原因，以下批量导入是错误的：

Warehouse.collection.insert(chunk)

其中 chunk 是一个哈希数组。我做了两次这个操作。一次有 1,000 个仓库，第二次迭代有 500 个仓库。只有 1,000 个仓库会持久保存在数据库中。我也能够通过 验证这一点mongostat。

起作用的是将插入基于 Mongoid：

warehouses = warehouses.map { |attributes| Warehouse.new(attributes) }
warehouses.each { |warehouse| listing.warehouses << warehouse }

这很好，但我想知道批量导入的问题是什么。

我该如何诊断这个问题？mongodb.log仅显示打开和关闭的断开连接。有没有类似错误日志的东西？

Warehouse.collection.insert(chunk, safe: true)

这也没有我的任何信息。

4

1 回答 1

0

在下面的测试中，请找到我希望能够阐明 Mongoid 和 Moped 中的批量插入的示例。测试 1 展示了如何使用 Mongoid 成功地进行批量插入，我推荐它而不是下拉到 Moped。如您所知，批处理应该是一个哈希数组。您可以使用 #serializable_hash 将 Mongoid 模型转换为哈希。

Moped 测试展示了如何使用 Moped 进行批量插入。包括对重复键插入错误的测试，无论是静默还是引发异常。我怀疑你为这个 SO 问题做了类似于测试#4 的事情。

希望这会有所帮助。如果您发现问题并可以继续前进，请告诉我们。

测试/单元/warehouse_test.rb

require 'test_helper'
require 'pp'

class WarehouseTest < ActiveSupport::TestCase
  def setup
    Warehouse.delete_all
    puts
  end
  def generate_warehouses(n = 1_000)
    n.times.collect { Warehouse.new.serializable_hash }
  end
  test "0. mongoid version" do
    puts "Mongoid::VERSION:#{Mongoid::VERSION}\nMoped::VERSION:#{Moped::VERSION}"
  end
  test "1. mongoid batch insert" do
    Warehouse.create(generate_warehouses(1_000))
    Warehouse.create(generate_warehouses(500))
    assert_equal 1_500, Warehouse.count
  end
  test "2. mongoid batch re-insert" do
    warehouses = generate_warehouses(1_000)
    Warehouse.create(warehouses) # object ids will be generated at a lower level and not be added to warehouses
    Warehouse.create(warehouses) # no duplicate key errors here
    assert_equal 2_000, Warehouse.count
  end
  test "3. moped batch insert" do
    Warehouse.collection.insert(generate_warehouses(1_000))
    Warehouse.collection.insert(generate_warehouses(500))
    assert_equal 1_500, Warehouse.count
  end
  test "4. moped batch re-insert duplicate keys silent failure" do
    warehouses = generate_warehouses(1_000)
    Warehouse.collection.insert(warehouses) # object ids will be generated and added to warehouses as a side effect
    Warehouse.collection.insert(warehouses) # re-insertion of previous object ids will fail with duplicate key errors
    assert_equal 1_000, Warehouse.count
  end
  test "5. moped batch re-insert duplicate keys exception" do
    warehouses = generate_warehouses(1_000)
    Warehouse.mongo_session.with(safe: true) do |session|
      session.use(Warehouse.database_name)
      collection = session[:warehouses]
      collection.insert(warehouses) # object ids will be generated and added to warehouses as a side effect
      begin
        collection.insert(warehouses) # re-insertion of previous object ids will fail with duplicate key errors
      rescue => exception
        puts "exception.message: #{exception.message}"
      end
    end
    assert_equal 1_000, Warehouse.count
  end
end

耙式试验

Run options: 

# Running tests:

[1/6] WarehouseTest#test_0._mongoid_version
Mongoid::VERSION:3.1.5
Moped::VERSION:1.5.1
[2/6] WarehouseTest#test_1._mongoid_batch_insert            
[3/6] WarehouseTest#test_2._mongoid_batch_re-insert              
[4/6] WarehouseTest#test_3._moped_batch_insert                      
[5/6] WarehouseTest#test_4._moped_batch_re-insert_duplicate_keys_silent_failure
[6/6] WarehouseTest#test_5._moped_batch_re-insert_duplicate_keys_exception                      
exception.message: The operation: #<Moped::Protocol::Command
  @length=85
  @request_id=3525
  @response_to=0
  @op_code=2004
  @flags=[]
  @full_collection_name="sandbox_mongoid3_test.$cmd"
  @skip=0
  @limit=-1
  @selector={:getlasterror=>1, :safe=>true}
  @fields=nil>
failed with error 11000: "E11000 duplicate key error index: sandbox_mongoid3_test.warehouses.$_id_  dup key: { : ObjectId('5283a3947f11ba868d001771') }"

See https://github.com/mongodb/mongo/blob/master/docs/errors.md
for details about this error.
Finished tests in 3.069318s, 1.9548 tests/s, 1.6290 assertions/s.                          
6 tests, 5 assertions, 0 failures, 0 errors, 0 skips
于 2013-11-13T16:13:48.710 回答