java - 避免嵌套对象中的 N+1 问题？

Question

我遇到了效率低下的 Hibernate 查询，我相信原因是我在其他地方读到的所谓的 N+1 问题。但是，我不确定如何解决这个问题，因为我的对象是嵌套的，而且我在网上看到的大多数示例解决方案都是微不足道的。

这是我的应用程序的简要说明。

我有一个ExecutionTrace类，它与类是一对多的关系TraceLine。TraceLine反过来与类具有一对多的关系TraceReference。这些类中的每一个都有一个由 Hibernate 生成的 id 属性。（所有的类都包含一些成员变量，但我现在将省略这些。）

在我的应用程序中，我正在查询一个特定ExecutionTrace的 ，然后我的应用程序将遍历它的所有TraceLine对象及其TraceReference对象。因此，我为所有这些关系设置了LazyCollectionOptionto ：false

// ExecutionTrace.java:
...
@Id @GeneratedValue
@Column(name = "file_id")
public long id;
...
@OneToMany(mappedBy = "container_trace_", cascade = CascadeType.ALL)
@LazyCollection(LazyCollectionOption.FALSE)
@OrderBy(clause = "id")
public List<TraceLine> lines_ = new ArrayList<TraceLine>();
...


// TraceLine.java:
...
@Id @GeneratedValue
@Column(name = "trace_line_id")
@Index(name = "traceIDIndex")
public long id;
...
@ManyToOne(cascade = CascadeType.ALL)
@JoinColumn(name = "file_id")
@LazyCollection(LazyCollectionOption.TRUE)
public ExecutionTrace container_trace_;
...
@OneToMany(mappedBy = "trace_line_", cascade = CascadeType.ALL)
@LazyCollection(LazyCollectionOption.FALSE)
@OrderBy(clause = "id")
public List<TraceReference> trace_references_ = new ArrayList<TraceReference>();
...


// TraceReference.java:
...
@Id @GeneratedValue
@Column(name = "trace_reference_id")
@Index(name="TraceReferenceIDIndex")
public long id;
...
@ManyToOne(cascade = CascadeType.ALL)
@JoinColumn(name = "trace_line_id")
@LazyCollection(LazyCollectionOption.TRUE)
public TraceLine trace_line_;
...

我选择一个休眠查询ExecutionTrace是：

String f_hash = ...;

Transaction tx3 = session.beginTransaction();
    String queryString = " from ExecutionTrace where hash_ = '"+ f_hash +"'";

List<ExecutionTrace> res2 = session.createQuery(queryString).list();
tx3.commit();

在我的一个简单测试用例中，有 1 个ExecutionTrace对象，其中包含 645TraceLine个对象，每个对象包含 1 到 5 个TraceReference对象。查询，以及遍历所有这些的代码，在普通硬件上需要 50 多秒。数据库 PostgreSQL 与应用程序在同一台机器上运行。

查看 Hibernate 抛出的 SQL 查询，我注意到：

• 有 1 个查询可以选择所有TraceLine对象。这很好。
• 有 645 个查询可以选择TraceReference对象。这是不好的。另外，它们都有很多外部连接，使它们效率低下。

例如，下面是对 TraceReference 的查询：

    select
        trace_refe0_.trace_line_id as trace5_6_18_,
        trace_refe0_.trace_reference_id as trace1_18_,
        trace_refe0_.trace_reference_id as trace1_7_17_,
        trace_refe0_.new_value as new2_7_17_,
        trace_refe0_.old_value as old3_7_17_,
        trace_refe0_.operand_id as operand4_7_17_,
        trace_refe0_.trace_line_id as trace5_7_17_,
        operand1_.operand_id as operand1_11_0_,
        operand1_.operand_index as operand2_11_0_,
        operand1_.is_dest as is3_11_0_,
        operand1_.is_source as is4_11_0_,
        operand1_.operand_size as operand5_11_0_,
        operand1_.operand_u_prop_id as operand6_11_0_,
        operand1_.maps_to_variable as maps7_11_0_,
        operandtyp2_.operand_prop_id as operand2_12_1_,
        operandtyp2_1_.constant_value as constant1_13_1_,
        operandtyp2_2_.register_name as register1_14_1_,
        operandtyp2_4_.base_operand_id as base2_16_1_,
        operandtyp2_4_.displacement_operand_id as displace3_16_1_,
        operandtyp2_4_.index_operand_id as index4_16_1_,
        operandtyp2_4_.ml_name as ml1_16_1_,
        operandtyp2_4_.scale_operand_id as scale5_16_1_,
        operandtyp2_.DTYPE as DTYPE12_1_,
        register3_.operand_prop_id as operand2_12_2_,
        register3_1_.register_name as register1_14_2_,
        constant4_.operand_prop_id as operand2_12_3_,
        constant4_1_.constant_value as constant1_13_3_,
        register5_.operand_prop_id as operand2_12_4_,
        register5_1_.register_name as register1_14_4_,
        constant6_.operand_prop_id as operand2_12_5_,
        constant6_1_.constant_value as constant1_13_5_,
        variable7_.variable_id as variable1_18_6_,
        variable7_.file_id as file6_18_6_,
        variable7_.function_id as function7_18_6_,
        variable7_.variable_is_argument as variable2_18_6_,
        variable7_.variable_is_global as variable3_18_6_,
        variable7_.variable_is_static as variable4_18_6_,
        variable7_.variable_name as variable5_18_6_,
        variable7_.data_type_id as data8_18_6_,
        sourcefile8_.file_id as file2_0_7_,
        sourcefile8_.file_path as file3_0_7_,
        sourcefile8_.file_name as file4_0_7_,
        sourcefile8_.source_lang as source5_0_7_,
        function9_.function_id as function1_8_8_,
        function9_.file_id as file6_8_8_,
        function9_.function_end_address as function2_8_8_,
        function9_.function_address as function3_8_8_,
        function9_.function_name as function4_8_8_,
        function9_.function_is_static as function5_8_8_,
        sourcefile10_.file_id as file2_0_9_,
        sourcefile10_.file_path as file3_0_9_,
        sourcefile10_.file_name as file4_0_9_,
        sourcefile10_.source_lang as source5_0_9_,
        datatype11_.data_type_id as data2_19_10_,
        datatype11_.defined_by as defined5_19_10_,
        datatype11_.data_type_length as data3_19_10_,
        datatype11_.data_type_name as data4_19_10_,
        datatype11_2_.pointer_to as pointer1_20_10_,
        datatype11_3_.array_size as array1_21_10_,
        datatype11_3_.contains_type as contains2_21_10_,
        datatype11_4_.enumerator_count as enumerator1_22_10_,
        datatype11_5_.type_ as type1_24_10_,
        datatype11_6_.constant_of_type as constant1_25_10_,
        datatype11_7_.return_type as return1_27_10_,
        datatype11_.DTYPE as DTYPE19_10_,
        sourcefile12_.file_id as file2_0_11_,
        sourcefile12_.file_path as file3_0_11_,
        sourcefile12_.file_name as file4_0_11_,
        sourcefile12_.source_lang as source5_0_11_,
        datatype13_.data_type_id as data2_19_12_,
        datatype13_.defined_by as defined5_19_12_,
        datatype13_.data_type_length as data3_19_12_,
        datatype13_.data_type_name as data4_19_12_,
        datatype13_2_.pointer_to as pointer1_20_12_,
        datatype13_3_.array_size as array1_21_12_,
        datatype13_3_.contains_type as contains2_21_12_,
        datatype13_4_.enumerator_count as enumerator1_22_12_,
        datatype13_5_.type_ as type1_24_12_,
        datatype13_6_.constant_of_type as constant1_25_12_,
        datatype13_7_.return_type as return1_27_12_,
        datatype13_.DTYPE as DTYPE19_12_,
        datatype14_.data_type_id as data2_19_13_,
        datatype14_.defined_by as defined5_19_13_,
        datatype14_.data_type_length as data3_19_13_,
        datatype14_.data_type_name as data4_19_13_,
        datatype14_2_.pointer_to as pointer1_20_13_,
        datatype14_3_.array_size as array1_21_13_,
        datatype14_3_.contains_type as contains2_21_13_,
        datatype14_4_.enumerator_count as enumerator1_22_13_,
        datatype14_5_.type_ as type1_24_13_,
        datatype14_6_.constant_of_type as constant1_25_13_,
        datatype14_7_.return_type as return1_27_13_,
        datatype14_.DTYPE as DTYPE19_13_,
        datatype15_.data_type_id as data2_19_14_,
        datatype15_.defined_by as defined5_19_14_,
        datatype15_.data_type_length as data3_19_14_,
        datatype15_.data_type_name as data4_19_14_,
        datatype15_2_.pointer_to as pointer1_20_14_,
        datatype15_3_.array_size as array1_21_14_,
        datatype15_3_.contains_type as contains2_21_14_,
        datatype15_4_.enumerator_count as enumerator1_22_14_,
        datatype15_5_.type_ as type1_24_14_,
        datatype15_6_.constant_of_type as constant1_25_14_,
        datatype15_7_.return_type as return1_27_14_,
        datatype15_.DTYPE as DTYPE19_14_,
        datatype16_.data_type_id as data2_19_15_,
        datatype16_.defined_by as defined5_19_15_,
        datatype16_.data_type_length as data3_19_15_,
        datatype16_.data_type_name as data4_19_15_,
        datatype16_2_.pointer_to as pointer1_20_15_,
        datatype16_3_.array_size as array1_21_15_,
        datatype16_3_.contains_type as contains2_21_15_,
        datatype16_4_.enumerator_count as enumerator1_22_15_,
        datatype16_5_.type_ as type1_24_15_,
        datatype16_6_.constant_of_type as constant1_25_15_,
        datatype16_7_.return_type as return1_27_15_,
        datatype16_.DTYPE as DTYPE19_15_,
        datatype17_.data_type_id as data2_19_16_,
        datatype17_.defined_by as defined5_19_16_,
        datatype17_.data_type_length as data3_19_16_,
        datatype17_.data_type_name as data4_19_16_,
        datatype17_2_.pointer_to as pointer1_20_16_,
        datatype17_3_.array_size as array1_21_16_,
        datatype17_3_.contains_type as contains2_21_16_,
        datatype17_4_.enumerator_count as enumerator1_22_16_,
        datatype17_5_.type_ as type1_24_16_,
        datatype17_6_.constant_of_type as constant1_25_16_,
        datatype17_7_.return_type as return1_27_16_,
        datatype17_.DTYPE as DTYPE19_16_ 
    from
        TraceReference trace_refe0_ 
    left outer join
        Operand operand1_ 
            on trace_refe0_.operand_id=operand1_.operand_id 
    left outer join
        OperandProperties operandtyp2_ 
            on operand1_.operand_u_prop_id=operandtyp2_.operand_prop_id 
    left outer join
        Constant operandtyp2_1_ 
            on operandtyp2_.operand_prop_id=operandtyp2_1_.operand_prop_id 
    left outer join
        Register operandtyp2_2_ 
            on operandtyp2_.operand_prop_id=operandtyp2_2_.operand_prop_id 
    left outer join
        UnknownOperandProperties operandtyp2_3_ 
            on operandtyp2_.operand_prop_id=operandtyp2_3_.operand_prop_id 
    left outer join
        MemoryLocation operandtyp2_4_ 
            on operandtyp2_.operand_prop_id=operandtyp2_4_.operand_prop_id 
    left outer join
        OperandProperties register3_ 
            on operandtyp2_4_.base_operand_id=register3_.operand_prop_id 
    left outer join
        Register register3_1_ 
            on register3_.operand_prop_id=register3_1_.operand_prop_id 
    left outer join
        OperandProperties constant4_ 
            on operandtyp2_4_.displacement_operand_id=constant4_.operand_prop_id 
    left outer join
        Constant constant4_1_ 
            on constant4_.operand_prop_id=constant4_1_.operand_prop_id 
    left outer join
        OperandProperties register5_ 
            on operandtyp2_4_.index_operand_id=register5_.operand_prop_id 
    left outer join
        Register register5_1_ 
            on register5_.operand_prop_id=register5_1_.operand_prop_id 
    left outer join
        OperandProperties constant6_ 
            on operandtyp2_4_.scale_operand_id=constant6_.operand_prop_id 
    left outer join
        Constant constant6_1_ 
            on constant6_.operand_prop_id=constant6_1_.operand_prop_id 
    left outer join
        Variable variable7_ 
            on operand1_.maps_to_variable=variable7_.variable_id 
    left outer join
        FileOnDisk sourcefile8_ 
            on variable7_.file_id=sourcefile8_.file_id 
    left outer join
        ObjectFile sourcefile8_1_ 
            on sourcefile8_.file_id=sourcefile8_1_.file_id 
    left outer join
        SourceFile sourcefile8_2_ 
            on sourcefile8_.file_id=sourcefile8_2_.file_id 
    left outer join
        SourceFunction function9_ 
            on variable7_.function_id=function9_.function_id 
    left outer join
        FileOnDisk sourcefile10_ 
            on function9_.file_id=sourcefile10_.file_id 
    left outer join
        ObjectFile sourcefile10_1_ 
            on sourcefile10_.file_id=sourcefile10_1_.file_id 
    left outer join
        SourceFile sourcefile10_2_ 
            on sourcefile10_.file_id=sourcefile10_2_.file_id 
    left outer join
        DataType datatype11_ 
            on variable7_.data_type_id=datatype11_.data_type_id 
    left outer join
        Pointer datatype11_1_ 
            on datatype11_.data_type_id=datatype11_1_.data_type_id 
    left outer join
        Pointer datatype11_2_ 
            on datatype11_.data_type_id=datatype11_2_.data_type_id 
    left outer join
        SourceArray datatype11_3_ 
            on datatype11_.data_type_id=datatype11_3_.data_type_id 
    left outer join
        Enum datatype11_4_ 
            on datatype11_.data_type_id=datatype11_4_.data_type_id 
    left outer join
        Typedef datatype11_5_ 
            on datatype11_.data_type_id=datatype11_5_.data_type_id 
    left outer join
        ConstantType datatype11_6_ 
            on datatype11_.data_type_id=datatype11_6_.data_type_id 
    left outer join
        FunctionType datatype11_7_ 
            on datatype11_.data_type_id=datatype11_7_.data_type_id 
    left outer join
        FileOnDisk sourcefile12_ 
            on datatype11_.defined_by=sourcefile12_.file_id 
    left outer join
        ObjectFile sourcefile12_1_ 
            on sourcefile12_.file_id=sourcefile12_1_.file_id 
    left outer join
        SourceFile sourcefile12_2_ 
            on sourcefile12_.file_id=sourcefile12_2_.file_id 
    left outer join
        DataType datatype13_ 
            on datatype11_2_.pointer_to=datatype13_.data_type_id 
    left outer join
        Pointer datatype13_1_ 
            on datatype13_.data_type_id=datatype13_1_.data_type_id 
    left outer join
        Pointer datatype13_2_ 
            on datatype13_.data_type_id=datatype13_2_.data_type_id 
    left outer join
        SourceArray datatype13_3_ 
            on datatype13_.data_type_id=datatype13_3_.data_type_id 
    left outer join
        Enum datatype13_4_ 
            on datatype13_.data_type_id=datatype13_4_.data_type_id 
    left outer join
        Typedef datatype13_5_ 
            on datatype13_.data_type_id=datatype13_5_.data_type_id 
    left outer join
        ConstantType datatype13_6_ 
            on datatype13_.data_type_id=datatype13_6_.data_type_id 
    left outer join
        FunctionType datatype13_7_ 
            on datatype13_.data_type_id=datatype13_7_.data_type_id 
    left outer join
        DataType datatype14_ 
            on datatype13_3_.contains_type=datatype14_.data_type_id 
    left outer join
        Pointer datatype14_1_ 
            on datatype14_.data_type_id=datatype14_1_.data_type_id 
    left outer join
        Pointer datatype14_2_ 
            on datatype14_.data_type_id=datatype14_2_.data_type_id 
    left outer join
        SourceArray datatype14_3_ 
            on datatype14_.data_type_id=datatype14_3_.data_type_id 
    left outer join
        Enum datatype14_4_ 
            on datatype14_.data_type_id=datatype14_4_.data_type_id 
    left outer join
        Typedef datatype14_5_ 
            on datatype14_.data_type_id=datatype14_5_.data_type_id 
    left outer join
        ConstantType datatype14_6_ 
            on datatype14_.data_type_id=datatype14_6_.data_type_id 
    left outer join
        FunctionType datatype14_7_ 
            on datatype14_.data_type_id=datatype14_7_.data_type_id 
    left outer join
        DataType datatype15_ 
            on datatype14_5_.type_=datatype15_.data_type_id 
    left outer join
        Pointer datatype15_1_ 
            on datatype15_.data_type_id=datatype15_1_.data_type_id 
    left outer join
        Pointer datatype15_2_ 
            on datatype15_.data_type_id=datatype15_2_.data_type_id 
    left outer join
        SourceArray datatype15_3_ 
            on datatype15_.data_type_id=datatype15_3_.data_type_id 
    left outer join
        Enum datatype15_4_ 
            on datatype15_.data_type_id=datatype15_4_.data_type_id 
    left outer join
        Typedef datatype15_5_ 
            on datatype15_.data_type_id=datatype15_5_.data_type_id 
    left outer join
        ConstantType datatype15_6_ 
            on datatype15_.data_type_id=datatype15_6_.data_type_id 
    left outer join
        FunctionType datatype15_7_ 
            on datatype15_.data_type_id=datatype15_7_.data_type_id 
    left outer join
        DataType datatype16_ 
            on datatype15_6_.constant_of_type=datatype16_.data_type_id 
    left outer join
        Pointer datatype16_1_ 
            on datatype16_.data_type_id=datatype16_1_.data_type_id 
    left outer join
        Pointer datatype16_2_ 
            on datatype16_.data_type_id=datatype16_2_.data_type_id 
    left outer join
        SourceArray datatype16_3_ 
            on datatype16_.data_type_id=datatype16_3_.data_type_id 
    left outer join
        Enum datatype16_4_ 
            on datatype16_.data_type_id=datatype16_4_.data_type_id 
    left outer join
        Typedef datatype16_5_ 
            on datatype16_.data_type_id=datatype16_5_.data_type_id 
    left outer join
        ConstantType datatype16_6_ 
            on datatype16_.data_type_id=datatype16_6_.data_type_id 
    left outer join
        FunctionType datatype16_7_ 
            on datatype16_.data_type_id=datatype16_7_.data_type_id 
    left outer join
        DataType datatype17_ 
            on datatype16_7_.return_type=datatype17_.data_type_id 
    left outer join
        Pointer datatype17_1_ 
            on datatype17_.data_type_id=datatype17_1_.data_type_id 
    left outer join
        Pointer datatype17_2_ 
            on datatype17_.data_type_id=datatype17_2_.data_type_id 
    left outer join
        SourceArray datatype17_3_ 
            on datatype17_.data_type_id=datatype17_3_.data_type_id 
    left outer join
        Enum datatype17_4_ 
            on datatype17_.data_type_id=datatype17_4_.data_type_id 
    left outer join
        Typedef datatype17_5_ 
            on datatype17_.data_type_id=datatype17_5_.data_type_id 
    left outer join
        ConstantType datatype17_6_ 
            on datatype17_.data_type_id=datatype17_6_.data_type_id 
    left outer join
        FunctionType datatype17_7_ 
            on datatype17_.data_type_id=datatype17_7_.data_type_id 
    where
        trace_refe0_.trace_line_id=? 
    order by
        trace_refe0_.trace_reference_id
TRACE [main] 06/26/13 10:00:54 preparing statement
TRACE [main] 06/26/13 10:00:54 binding parameter [1] as [BIGINT] - 82883

我尝试修改我的查询，以便它执行一些内部连接：

    String queryString =    " from ExecutionTrace as et " +
               "inner join fetch et.lines_ as lines " +
        "where et.hash_ = '"+ f_hash +"'";

这导致了 0 个对 selectTraceLine的查询和 645 个对 selectTraceReference的查询。没有帮助。然后我尝试了多个内部连接：

    String queryString = 
    " from ExecutionTrace as et " +
       "inner join fetch et.lines_ as lines " +
       "inner join fetch lines.trace_references_ " +
    "where et.hash_ = '"+ f_hash +"'";

这导致了一个异常：

Exception in thread "main" org.hibernate.loader.MultipleBagFetchException: cannot simultaneously fetch multiple bags.

我觉得我接近解决问题，但只是错过了一些简单的事情。

Answer 1

在做了一些研究和实验之后，我找到了解决这个问题的两种可能的解决方案，每种都有自己的优缺点。

1.使用SUBSELECT获取模式

使用@Fetch注解使 Hibernate 产生一个子选择查询：

@OneToMany(mappedBy = "container_trace_", cascade = CascadeType.ALL)
@LazyCollection(LazyCollectionOption.TRUE)
@OrderBy(clause = "id")
@Fetch(FetchMode.SUBSELECT)
public List<TraceLine> lines_ = new ArrayList<TraceLine>();

• 优点：上面的注解会导致所有的TraceLine对象都被同时获取（即在同一个 select 语句中）ExecutionTrace，从而导致 select 语句的数量要少得多。
• 缺点：如果ExecutionTrace有很多TraceLine对象，子选择会导致你的 JVM 内存不足。

2. 使用批次

如果您有太多TraceLine对象无法放入内存，因此您不能如上所述使用 SUBSELECT，那么您还有另一个选择。使用@BatchSize注解使 HibernateTraceLine一次获取 10 个对象，而不仅仅是一个。

@OneToMany(mappedBy = "container_trace_", cascade = CascadeType.ALL)
@LazyCollection(LazyCollectionOption.TRUE)
@OrderBy(clause = "id")
@BatchSize(size = 10)
public List<TraceLine> lines_ = new ArrayList<TraceLine>();

• 优点：这有效地将选择次数减少了 10 倍
• 优点：不会耗尽内存
• 缺点：如果对象的数量很大，仍然会有大量的选择TraceLine。

两种解决方案都不是完美的。在我的应用程序中，我使用了批次。