apache-spark - 如何在 Spark 中增加 Presto 的查询执行时间

Question

我目前正在使用 Spark 连接到 Presto。我们的查询在之后超时60m，为了增加查询执行时间，我在下面设置query.max-execution-time了参数getDBProperties()

private def constructPrestoDataFrame(sparkSession : SparkSession, jobConfig : Config, query : String) : DataFrame = {
    sparkSession
      .read
      .jdbc(getPrestoConnectionUrl(jobConfig), query, getDBProperties(jobConfig))
  }

  private def getDBProperties(jobConfig : Config) : Properties = {
    val dbProperties = new Properties
    dbProperties.put("user", jobConfig.getString("presto.user"))
    dbProperties.put("password", jobConfig.getString("presto.password"))
    dbProperties.put("Driver", jobConfig.getString("presto.driver.name"))
    dbProperties.put("query.max-execution-time", "2d")

    dbProperties
  }

  private def getPrestoConnectionUrl(jobConfig : Config) : String = {
    s"jdbc:presto://${jobConfig.getString("presto.host")}:8443/${jobConfig.getString("presto.catalogue.name")}?SSL=true&SSLTrustStorePath=${jobConfig.getString("sslTrustStorePath")}"+
      "&SSLTrustStorePassword="+URLEncoder.encode(jobConfig.getString("sslTrustStorePassword"))
  }

当我运行工作时，我收到异常说exception caught: Cause = null Message = Unrecognized connection property 'query.max-execution-time'

我们使用apache-spark-2.3.x, presto-jdbc-driver-300。

Answer 1

添加MAX_EXECUTION_TIME到sessionVariablesURL 为我完成了这项工作：

jdbc:mysql://{host}:{port}/{database}?sessionVariables=MAX_EXECUTION_TIME=123456666

查询验证：

SELECT @@max_execution_time

预期输出：

+--------------------+ 
|@@max_execution_time| 
+--------------------+ 
| 123456666          | 
+--------------------+