4

我创建了一个示例 Livy (Spark) 应用程序，使用com.cloudera.livy.Job该类计算 Pi 的近似值（来源：https ://github.com/cloudera/livy#using-the-programmatic-api ），作为 jar 文件导出到例如C:/path/to/the/pijob.jar.

实际上我正在从另一个这样的Main类中运行这项工作（也从上面的链接复制并改编）：

import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Properties;
import java.util.concurrent.ExecutionException;

import com.cloudera.livy.LivyClient;
import com.cloudera.livy.LivyClientBuilder;


public class Main {

    public static void main(String[] args) throws IOException, URISyntaxException, InterruptedException, ExecutionException {
        String livyUrl = "http://myserverIp:8998";
        String piJar = "C:/path/to/the/pijob.jar";
        int samples = 10000;

        LivyClient client = new LivyClientBuilder().setURI(new URI(livyUrl)).build();

        try {
            System.err.printf("Uploading %s to the Spark context...\n", piJar);
            client.uploadJar(new File(piJar)).get();

            System.err.printf("Running PiJob with %d samples...\n", samples);
            double pi = client.submit(new PiJob(samples)).get();

            System.out.println("Pi is roughly: " + pi);
        } finally {
            client.stop(true);
        }
    }
}

此应用程序在外部（从我的客户端开始）的不安全 Hadoop 集群中完美运行。但是当我尝试在启用 Kerberos 的集群上运行它时，它会失败。

我尝试在LivyClientBuilder类中设置相应的 Kerberos 属性：

Properties props = new Properties();
props.put("livy.environment", "production");
props.put("livy.impersonation.enabled", "true");
props.put("livy.server.auth.kerberos.keytab", "/etc/security/keytabs/spnego.service.keytab");
props.put("livy.server.auth.kerberos.principal", "HTTP/_HOST@MYCLUSTER.DE");
props.put("livy.server.auth.type", "kerberos");
props.put("livy.server.csrf_protection.enabled", "true");
props.put("livy.server.kerberos.keytab", "/etc/security/keytabs/livy.service.keytab");
props.put("livy.server.kerberos.principal", "livy/_HOST@MYCLUSTER.DE");
props.put("livy.server.port", "8998");
props.put("livy.server.session.timeout", "3600000");
props.put("livy.superusers", "zeppelin-MyCluster");

LivyClient client = new LivyClientBuilder().setAll(props).setURI(new URI(livyUrl)).build();

但我仍然得到一个例外，说需要身份验证：

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Authentication required: <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 </title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing /sessions/. Reason:
<pre>    Authentication required</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

    at com.cloudera.livy.client.http.HttpClient.propagate(HttpClient.java:185)
    at com.cloudera.livy.client.http.HttpClient.<init>(HttpClient.java:85)
    at com.cloudera.livy.client.http.HttpClientFactory.createClient(HttpClientFactory.java:38)
    at com.cloudera.livy.LivyClientBuilder.build(LivyClientBuilder.java:124)
    at livy.Main.main(Main.java:34)
Caused by: java.io.IOException: Authentication required: <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 </title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing /sessions/. Reason:
<pre>    Authentication required</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

    at com.cloudera.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:230)
    at com.cloudera.livy.client.http.LivyConnection.sendJSONRequest(LivyConnection.java:204)
    at com.cloudera.livy.client.http.LivyConnection.post(LivyConnection.java:180)
    at com.cloudera.livy.client.http.HttpClient.<init>(HttpClient.java:82)
    ... 3 more

在这一点上的问题是给我的：

  • 这些都是我需要的 Kerberos 设置吗？
    • 还是我必须添加更多内容才能登录？
  • 我必须在我的客户端机器上提供配置文件/密钥表吗？
    • 还是我仍然可以使用服务器路径（就像我到目前为止所做的那样）？
  • 是否有一些关于 Livy 的 Kerberos 内容的有用文档？
4

0 回答 0