1

当我在与此类似的时间序列数据上添加索引时,我遇到了奇怪的分析查询行为:

{
    "clientId": "account-123",
    "orderCode": "49776599-cd7f-458a-b0f2-2d4f28aab85f",
    "timestamp": 1591869354000 // UNIX time milliseconds
}

使用包含所有三个字段的索引

index_clientId_timestamp_orderCode (clientId:string)(timestamp:int64)(orderCode:string)`

奇怪的行为

  1. 当我过滤clientIdorderCode
  2. 如果我添加一个过滤器,timestamp那么 couchbase 服务器会出错。

在 Couchbase Enterprise Edition 6.5.0 build 4960 上重现的步骤

向桶中插入一些数据

INSERT INTO `data` (KEY, VALUE)
 VALUES ("key1", {"clientId": "account-123", "orderCode": "49776599-cd7f-458a-b0f2-2d4f28aab85f", "timestamp": 1591869354000}),
 VALUES ("key2", {"clientId": "account-123", "orderCode": "8e2a8061-c69e-40ee-8ff4-e1fc6f03f76a", "timestamp": 1591869355000}),
 VALUES ("key3", {"clientId": "account-123", "orderCode": "757d33a6-85e7-4053-ad8b-bacc69ad0200", "timestamp": 1591869356000}),
 VALUES ("key4", {"clientId": "account-123", "orderCode": "c6022d17-80bf-497f-8f9d-f865896231dc", "timestamp": 1591869357000})

创建分析数据集:

CREATE DATASET orders ON `data` WHERE `clientId` = "account-123";

运行以下分析查询,它应该可以按预期工作:

SELECT `orders`.*
FROM `orders` orders
WHERE
  `clientId` = 'account-123'
  AND `orderCode` = '49776599-cd7f-458a-b0f2-2d4f28aab85f'
  AND `timestamp` > 1591860000000

创建一个索引,以便查询可以更好地扩展:

CREATE INDEX index_clientId_timestamp_orderCode ON `orders`(`clientId`:STRING, `timestamp`:INT64, `orderCode`:STRING)

再次运行查询:

SELECT `orders`.*
FROM `orders` orders
WHERE
  `clientId` = 'account-123'
  AND `orderCode` = '49776599-cd7f-458a-b0f2-2d4f28aab85f'
  AND `timestamp` > 1591860000000

你应该得到一个错误:

[
  {
    "code": 25000,
    "msg": "Internal error",
    "query_from_user": "SELECT `orders`.*\nFROM `orders` orders\nWHERE\n  `clientId` = 'account-123'\n  AND `orderCode` = '49776599-cd7f-458a-b0f2-2d4f28aab85f'\n  AND `timestamp` > 1591860000000"
  }
]

当我查看时,/opt/couchbase/var/lib/couchbase/logs/analytics_error.log我看到以下错误

2020-06-11T14:58:34.561+00:00 ERRO CBAS.apache.asterix [Executor-104:ClusterController] Unexpected exception
java.lang.NullPointerException: null
        at org.apache.asterix.optimizer.rules.am.BTreeAccessMethod.createKeyVarsAndExprs(BTreeAccessMethod.java:773) ~[asterix-algebra.jar:6.5.0-4960]
        at org.apache.asterix.optimizer.rules.am.BTreeAccessMethod.createIndexSearchPlan(BTreeAccessMethod.java:585) ~[asterix-algebra.jar:6.5.0-4960]
        at org.apache.asterix.optimizer.rules.am.BTreeAccessMethod.applySelectPlanTransformation(BTreeAccessMethod.java:183) ~[asterix-algebra.jar:6.5.0-4960]
        at org.apache.asterix.optimizer.rules.am.IntroduceSelectAccessMethodRule.checkAndApplyTheSelectTransformation(IntroduceSelectAccessMethodRule.java:431) ~[asterix-algebra.jar:6.5.0-4960]
        at org.apache.asterix.optimizer.rules.am.IntroduceSelectAccessMethodRule.checkAndApplyTheSelectTransformation(IntroduceSelectAccessMethodRule.java:343) ~[asterix-algebra.jar:6.5.0-4960]
        at org.apache.asterix.optimizer.rules.am.IntroduceSelectAccessMethodRule.checkAndApplyTheSelectTransformation(IntroduceSelectAccessMethodRule.java:343) ~[asterix-algebra.jar:6.5.0-4960]
        at org.apache.asterix.optimizer.rules.am.IntroduceSelectAccessMethodRule.checkAndApplyTheSelectTransformation(IntroduceSelectAccessMethodRule.java:343) ~[asterix-algebra.jar:6.5.0-4960]
        at org.apache.asterix.optimizer.rules.am.IntroduceSelectAccessMethodRule.rewritePre(IntroduceSelectAccessMethodRule.java:168) ~[asterix-algebra.jar:6.5.0-4960]
        at org.apache.hyracks.algebricks.core.rewriter.base.AbstractRuleController.rewriteOperatorRef(AbstractRuleController.java:86) ~[algebricks-core.jar:6.5.0-4960]
        at org.apache.hyracks.algebricks.compiler.rewriter.rulecontrollers.SequentialFixpointRuleController.rewriteWithRuleCollection(SequentialFixpointRuleController.java:52) ~[algebricks-compiler.jar:6.5.0-4960]
        at org.apache.hyracks.algebricks.core.rewriter.base.HeuristicOptimizer.runOptimizationSets(HeuristicOptimizer.java:89) ~[algebricks-core.jar:6.5.0-4960]
        at org.apache.hyracks.algebricks.core.rewriter.base.HeuristicOptimizer.runLogicalOptimizationSets(HeuristicOptimizer.java:80) ~[algebricks-core.jar:6.5.0-4960]
        at org.apache.hyracks.algebricks.core.rewriter.base.HeuristicOptimizer.optimize(HeuristicOptimizer.java:61) ~[algebricks-core.jar:6.5.0-4960]
        at org.apache.hyracks.algebricks.compiler.api.HeuristicCompilerFactoryBuilder$1$1.optimize(HeuristicCompilerFactoryBuilder.java:94) ~[algebricks-compiler.jar:6.5.0-4960]
        at org.apache.asterix.api.common.APIFramework.compileQuery(APIFramework.java:255) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.asterix.app.translator.QueryTranslator.rewriteCompileQuery(QueryTranslator.java:1982) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.asterix.app.translator.QueryTranslator.lambda$handleQuery$3(QueryTranslator.java:2491) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.asterix.app.translator.QueryTranslator.createAndRunJob(QueryTranslator.java:2623) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.asterix.app.translator.QueryTranslator.deliverResult(QueryTranslator.java:2536) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.asterix.app.translator.QueryTranslator.handleQuery(QueryTranslator.java:2506) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:399) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java:153) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java:64) ~[asterix-app.jar:6.5.0-4960]
        at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$0(ApplicationMessageWork.java:68) ~[hyracks-control-cc.jar:6.5.0-4960]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

从谓词中删除时间戳过滤器会导致搜索按预期返回。但是,当我为我的雇主在更大的数据集上有效地执行相同的操作时,我根本找不到匹配项。不过,我无法通过上面的示例重现该行为。

任何帮助表示赞赏:)

4

1 回答 1

1

这似乎是优化器中的一个错误。

尝试创建两个索引:

CREATE INDEX index_clientId_orderCode ON `orders`(`clientId`:STRING, `orderCode`:STRING)
CREATE INDEX index_timestamp ON `orders`(`timestamp`:BIGINT)

此外,orderCode 值看起来是唯一的(或低基数)。您能否仅在其上创建一个索引并在查询执行时计算剩余的过滤器?

CREATE INDEX index_orderCode ON `orders`(`orderCode`:STRING)

另一种解决方法是保留现有的索引定义,但在时间戳上添加一个上限过滤器:

... AND `timestamp` > 1591860000000 and `timestamp` < 9999999999999999
于 2020-06-11T18:13:15.843 回答