3

将 Spark3 安装到 redhat 7 后，一切似乎都运行了。

```
os.environ['SPARK_HOME'] = "/users/spark/spark-3.0.0-bin-hadoop3.2"
os.environ['JAVA_HOME'] ="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64/jre"
```

一个简单的连接 dat.join(ids_df, on=['symbol']).show()

遇到这个错误。任何人都可以帮忙吗？非常感谢。


    Exception in thread "map-output-dispatcher-0" java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path
    Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing l                                         ibzstd-jni in your system.
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at com.github.luben.zstd.util.Native.load(Native.java:108)
        at com.github.luben.zstd.util.Native.load(Native.java:60)
        at com.github.luben.zstd.ZstdOutputStream.<clinit>(ZstdOutputStream.java:15)
        at org.apache.spark.io.ZStdCompressionCodec.compressedOutputStream(CompressionCodec.scala:224)
        at org.apache.spark.MapOutputTracker$.serializeMapStatuses(MapOutputTracker.scala:913)
        at org.apache.spark.ShuffleStatus.$anonfun$serializedMapStatus$2(MapOutputTracker.scala:210)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:72)
        at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:207)
        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:457)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
4

1 回答 1

2

您需要在 RedHat 7 上安装 libzstd，使用 root 用户执行 yum install 命令：

  • 对于 64 位系统

     yum install libzstd.x86_64

  • 对于 32 位系统

     yum install libzstd.i686
于 2020-12-03T13:59:28.733 回答