hadoop - Using kerberized webhdfs/hdfs with JAVA API

Question

I have a Hadoop cluster with Kerberos enabled I want to put files on HDFS using a windows/linux machine outside the cluster.

Hadoop admin team have provided me with username to access hadoop and keytab file, how should I use them in my java code?

I went through many resources on internet but none of them give any guide for accessing kerberized hadoop from outside the cluster.

Also, Is it necessary to run the code using hadoop jar? if yes how will I run it fromo outside the cluster

Reference

http://blog.rajeevsharma.in/2009/06/using-hdfs-in-java-0200.html
http://princetonits.com/technology/using-filesystem-api-to-read-and-write-data-to-hdfs/

I got kerberos working ,able to generate ticket now

But curl is not working(windows)

curl -i  --negotiate u:qjdht93 "http://server:50070/webhdfs/v1/user/qjdht93/?op=LISTSTATUS"

Gives error as

HTTP/1.1 401 Authentication required
Cache-Control: must-revalidate,no-cache,no-store
Date: Mon, 01 Jun 2015 15:26:37 GMT
Pragma: no-cache
Date: Mon, 01 Jun 2015 15:26:37 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Version=1; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 G
MT; HttpOnly

Content-Length: 1416
Server: Jetty(6.1.26)

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 401 Authentication required</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /webhdfs/v1/user/qjdht93. Reason:
<pre>    Authentication required</pre></p><hr /><i><small>Powered by Jetty://</s
mall></i><br/>
<br/>
<br/>

Please suggest

score -1 · Accepted Answer

这可以通过 hdfs 命令来实现。您所需要的只是出现在 namenode 上的 hadoop 分发和配置文件。

复制客户端节点上的 hadoop 分发。这意味着您必须将完整的 hadoop 包复制到客户端计算机。参考这个
使用kinit命令从 keytab 获取用户票， kinit命令是 java 的命令行工具。
一个。在您的客户端机器上安装 jdk。
湾。设置 JAVA_HOME，见这里
c. 在位置 C:\windows\krb5.ini 中创建一个 krb5.ini 文件。该文件应包含以下信息，

    [libdefaults]
        default_realm = 领域
    [领域]
        领域 = {
            kdc = kdc 值    
            admin_server = kdc 值
            default_domain = kdc 值
        }
    [domain_realm]
        .kdcvalue = 领域
        kdc 值 = 领域

REALM - 服务器领域名称
kdcvalue - 服务器主机名或 IP 地址

d。确保在 Windows 机器的 PATH 变量中设置了 java bin 路径。打开命令提示符，键入以下命令以获取用户票。

kinit -k -t keytabfile username

现在您可以使用“hadoop fs -put src dest”或使用 java 将文件放入 HDFS。

hadoop - Using kerberized webhdfs/hdfs with JAVA API

1 回答 1

Related

Reference