1

由于需要 CDH 支持,我编译了最新的 spring-xd。我可以启动服务器,但是当我通过 xd-shell 连接到服务器时,我尝试更改“配置”。这也是一个 kerberized 集群,我不确定 xd 将/可以如何处理它。

第一种情况:

  1. 管理员配置服务器 --uri http:// testdomain:10111
  2. hadoop config fs --namenode hdfs://nameservice1:8020
  3. hadoop config props 设置 hadoop.security.group.mapping=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
  4. hadoop 配置道具加载 hadoop.security.group.mapping
  5. hadoop fs ls

错误信息:

xd:>hadoop fs ls
-ls: Fatal internal error
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:55)
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
        at org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)
        at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:277)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:668)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:573)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2428)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2420)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2288)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
        at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:270)
        at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
        at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:254)
        at org.springframework.xd.shell.hadoop.FsShellCommands.run(FsShellCommands.java:412)
        at org.springframework.xd.shell.hadoop.FsShellCommands.runCommand(FsShellCommands.java:407)
        at org.springframework.xd.shell.hadoop.FsShellCommands.ls(FsShellCommands.java:110)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:191)
        at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:64)
        at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:48)
        at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:127)
        at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:483)
        at org.springframework.shell.core.JLineShell.run(JLineShell.java:157)
        at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:126)
        ... 35 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.JniBasedUnixGroupsMapping
        at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.<init>(JniBasedUnixGroupsMappingWithFallback.java:38)
        ... 40 more

第二种情况

或者我删除了一些 java opts

从上一个场景运行步骤 1、2

然后

  1. hadoop 配置道具设置 hadoop.security.authorization=true
  2. hadoop config props 设置 hadoop.security.authentication=kerberos

下面的错误


16:50:29,682 WARN Spring Shell util.NativeCodeLoader:62 - 无法为您的平台加载本机 hadoop 库...在适用的情况下使用内置 java 类 ls:授权(hadoop.security.authorization)已启用,但身份验证(hadoop .security.authentication) 配置为简单。请配置另一种方法,如 kerberos 或摘要。

感谢您的帮助 - 迫不及待地想让这个工作!

4

2 回答 2

0

感谢您提出这个问题 - 我们已经有一段时间没有在 shell 中测试授权/身份验证了 - 尽管它作为项目的一部分进行了测试https://github.com/vmware-serengeti/serengeti-ws

您是否能够使用标准的 hadoop 文件系统 shell 执行操作。例如

hdfs dfs -ls /user/hadoop/file1

于 2013-10-08T22:25:11.687 回答
0

XD 目前没有针对安全 Hadoop 集群运行的特定支持。

请随时在https://jira.springsource.org/browse/XD上打开 JIRA 票证——这是我们知道我们必须尽快解决的问题。

于 2013-10-08T22:23:21.067 回答