0

大家好，我正在尝试在 Streaming Spark 作业中执行 KCL，我创建了一个初稿项目，连同我的依赖项，我可能能够导入几个包。但是有一个包org.apache.spark.streaming.kinesis，即使它包含在我的 POM.xml 中也不会导入它。

我的 pom.xml：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.streaming.sparkdemo</groupId>
  <artifactId>spark-demo</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>spark-demo</name>
  <url>http://maven.apache.org</url>

  <dependencies>

  <!-- Import Spark -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>1.4.0</version>
    </dependency>

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>

    <dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.11.78</version>  
  </dependency>

  <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>2.3.0</version>
    <scope>provided</scope>
</dependency>


  <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>amazon-kinesis-client</artifactId>
      <version>1.9.3</version>
</dependency>


  </dependencies>



</project>

但是，在构建它之后，我从导入中收到了这条可怕的消息：

在此处输入图像描述

4

1 回答 1

0

我已经解决了这个问题，在我的 POM.XML 中包含了 ASL Kinesis 集成包

https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kinesis-asl_2.11/2.3.0

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kinesis-asl -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kinesis-asl_2.11</artifactId>
    <version>2.3.0</version>
</dependency>
于 2019-03-28T15:37:34.017 回答