我使用 Spark 使用这个 jar 将数据从 Hive Table 写入 Kinetica kinetica-spark-7.0.6.1-jar-with-dependencies.jar
:. 但是,当我运行时spark-submit
,jar 中的记录器正在打印 JDBC 连接字符串及其凭据,如下所示:
...
22/03/03 03:00:58 INFO spark.LoaderParams: Using JDBC connection string: jdbc:kinetica://10.xx.xx.xx:9191;UID=xxx;PWD=xxx
22/03/03 03:00:58 INFO spark.LoaderParams: Installing truststore to bypass certificate check.
22/03/03 03:00:58 INFO spark.LoaderParams: Using URL(s) http://10.xx.xx.xx:9191to create a GPUdb connection
22/03/03 03:00:59 INFO spark.LoaderParams: Connecting to http://10.xx.xx.xx:9191 as user <xxx>
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 47
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 42
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 53
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 52
...
这来自这段代码:kinetica.spark.LoaderParams
我想将专门针对此类的日志级别设置为spark.LoaderParams
更高级别,以便连接字符串不会显示在日志中的任何位置。
有什么办法可以做到吗?
这是我的log4j.properties
配置:
log4j.rootLogger=${root.logger}
root.logger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
shell.log.level=WARN
log4j.logger.org.eclipse.jetty=INFO
log4j.logger.org.spark-project.jetty=INFO
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
log4j.logger.org.apache.spark.repl.Main=${shell.log.level}
log4j.logger.org.apache.spark.api.python.PythonGatewayServer=${shell.log.level}
log4j.logger.jobLogger=INFO, RollingAppenderU
log4j.appender.RollingAppenderU=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingAppenderU.DatePattern='.'yyyy-MM-dd
log4j.appender.RollingAppenderU.layout=org.apache.log4j.PatternLayout
log4j.appender.RollingAppenderU.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.fileAppender.MaxFileSize=1MB
log4j.appender.fileAppender.MaxBackupIndex=1