以下在 Cloudera CDSW 集群网关上成功运行。
import pyspark
from pyspark.sql import SparkSession
spark = (SparkSession
.builder
.config("spark.jars.packages","JohnSnowLabs:spark-nlp:1.2.3")
.getOrCreate()
)
产生这个输出。
Ivy Default Cache set to: /home/cdsw/.ivy2/cache
The jars for the packages stored in: /home/cdsw/.ivy2/jars
:: loading settings :: url = jar:file:/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
JohnSnowLabs#spark-nlp added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found JohnSnowLabs#spark-nlp;1.2.3 in spark-packages
found com.typesafe#config;1.3.0 in central
found org.fusesource.leveldbjni#leveldbjni-all;1.8 in central
downloading http://dl.bintray.com/spark-packages/maven/JohnSnowLabs/spark-nlp/1.2.3/spark-nlp-1.2.3.jar ...
[SUCCESSFUL ] JohnSnowLabs#spark-nlp;1.2.3!spark-nlp.jar (3357ms)
downloading https://repo1.maven.org/maven2/com/typesafe/config/1.3.0/config-1.3.0.jar ...
[SUCCESSFUL ] com.typesafe#config;1.3.0!config.jar(bundle) (348ms)
downloading https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar ...
[SUCCESSFUL ] org.fusesource.leveldbjni#leveldbjni-all;1.8!leveldbjni-all.jar(bundle) (382ms)
:: resolution report :: resolve 3836ms :: artifacts dl 4095ms
:: modules in use:
JohnSnowLabs#spark-nlp;1.2.3 from spark-packages in [default]
com.typesafe#config;1.3.0 from central in [default]
org.fusesource.leveldbjni#leveldbjni-all;1.8 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 3 | 3 | 3 | 0 || 3 | 3 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
3 artifacts copied, 0 already retrieved (5740kB/37ms)
Setting default log level to "ERROR".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
但是当我尝试按照John Snow Labs中描述的 pyspark 导入 sparknlp 时...
import sparknlp
# or
from sparknlp.annotator import *
我明白了:
ImportError: No module named sparknlp
ImportError: No module named sparknlp.annotator
我需要做什么才能使用 sparknlp?当然,这可以推广到任何 Spark 包。