0

我正在使用 Apache Twill 将一个使用 Janino 动态编译类的开源项目移植到 YARN。这很好用，除了最后一个错误。当 Janino 与斜纹一起使用时，我得到一个找不到类的异常，尽管该类在 Classpath 中甚至使用过。

我得到的例外是：

2014-06-09T18:30:40,093Z 错误 oadepipProjectRecordBatch [zk1] [37daf04b-7d82-4d2f-987c-59851f2aeafe:frag:0:0] AbstractSingleRecordBatch:next(AbstractSingleRecordBatch.java:60) - 查询 org.apache 期间失败。 Drill.exec.exception.SchemaChangeException：尝试在 org.apache.drill.exec.record 的 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:243) 加载生成的类时失败.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next (AbstractSingleRecordBatch.java:45) 在 org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) 在 org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) 在 org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) 在 org .apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) at java.util .concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 原因： org.apache.drill.exec.exception.ClassTransformationException：失败svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) 在 org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) 在 org.apache.drill.exec.work.fragment。 FragmentExecutor.run(FragmentExecutor.java:104) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 在 java.lang .Thread.run(Thread.java:744) 引起：org.apache.drill.exec.exception.ClassTransformationException：失败svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) 在 org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) 在 org.apache.drill.exec.work.fragment。 FragmentExecutor.run(FragmentExecutor.java:104) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 在 java.lang .Thread.run(Thread.java:744) 引起：org.apache.drill.exec.exception.ClassTransformationException：失败runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 原因：org.apache.drill .exec.exception.ClassTransformationException：失败runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 原因：org.apache.drill .exec.exception.ClassTransformationException：失败

为值生成转换类：

package org.apache.drill.exec.test.generated;

import org.apache.drill.exec.exception.SchemaChangeException;
import org.apache.drill.exec.expr.holders.BitHolder;
import org.apache.drill.exec.expr.holders.VarCharHolder;
import org.apache.drill.exec.ops.FragmentContext;
import org.apache.drill.exec.record.RecordBatch;
import org.apache.drill.exec.vector.RepeatedVarCharVector;
import org.apache.drill.exec.vector.VarCharVector;
import org.apache.drill.exec.vector.complex.impl.RepeatedVarCharReaderImpl;

public class ProjectorGen0 {

    RepeatedVarCharVector vv0;
    RepeatedVarCharReaderImpl reader4;
    VarCharVector vv5;

    public boolean doEval(int inIndex, int outIndex)
        throws SchemaChangeException
    {
        {
            VarCharHolder out3 = new VarCharHolder();
            complex:
            vv0 .getAccessor().getReader().setPosition((inIndex));
            reader4 .read(0, out3);
            BitHolder out8 = new BitHolder();
            out8 .value = 1;
            if (!vv5 .getMutator().setSafe((outIndex), out3)) {
                out8 .value = 0;
            }
            if (out8 .value == 0) {
                return false;
            }
        }
        {
            return true;
        }
    }

    public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
        throws SchemaChangeException
    {
        {
            int[] fieldIds1 = new int[ 1 ] ;
            fieldIds1 [ 0 ] = 0;
            Object tmp2 = (incoming).getValueAccessorById(RepeatedVarCharVector.class, fieldIds1).getValueVector();
            if (tmp2 == null) {
                throw new SchemaChangeException("Failure while loading vector vv0 with id: org.apache.drill.exec.record.TypedFieldId@1cf4a5a0.");
            }
            vv0 = ((RepeatedVarCharVector) tmp2);
            reader4 = ((RepeatedVarCharReaderImpl) vv0 .getAccessor().getReader());
            int[] fieldIds6 = new int[ 1 ] ;
            fieldIds6 [ 0 ] = 0;
            Object tmp7 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds6).getValueVector();
            if (tmp7 == null) {
                throw new SchemaChangeException("Failure while loading vector vv5 with id: org.apache.drill.exec.record.TypedFieldId@1ce776c0.");
            }
            vv5 = ((VarCharVector) tmp7);
        }
    }

}

在 org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302) 在 org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) 在 org.apache.drill.exec .physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) 在 org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) 在 org.apache.drill.exec.physical.impl .project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) 在 org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) 在 org.apache.drill.exec.physical.impl.limit.LimitRecordBatch .next(LimitRecordBatch.java:99) 在 org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) 在 org.apache.drill.exec.physical。在 org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) 在 org.apache.drill.exec.work 的 impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94)。 fragment.FragmentExecutor.run(FragmentExecutor.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java .lang.Thread.run(Thread.java:744) 原因：org.codehaus.commons.compiler.CompileException：第 4 行第 8 列：无法导入的类“org.apache.drill.exec.exception.SchemaChangeException”在 org.codehaus.janino.UnitCompiler.import2(UnitCompiler.java:192) 的 org.codehaus.janino.UnitCompiler 的 org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) 加载。访问 $000(UnitCompiler.java:104) at org.codehaus.janino.UnitCompiler$1.visitSingleTypeImportDeclaration(UnitCompiler.java:166) at org.codehaus.janino.Java$CompilationUnit$SingleTypeImportDeclaration.accept(Java.java:171) at org .codehaus.janino.UnitCompiler.(UnitCompiler.java:164) 在 org.apache.drill.exec.compile.JaninoClassCompiler.getClassByteCode(JaninoClassCompiler.java:53) 在 org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode( QueryClassLoader.java:69) at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:256) at org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) at org .apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) 在 org.apache.drill.exec.record。AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next( AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java: 45) 在 org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) 在 org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java: 80) 在 org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 在 java.lang.Thread.run(Thread.java:744)

如您所见，异常的类型是，SchemaChangeException但内部异常是ClassNotFoundExceptionfor SchemaChangeException：

第 4 行第 8 列：无法加载导入的类“org.apache.drill.exec.exception.SchemaChangeException”

所以类加载器有问题，当应用程序使用 Apache Twill 运行时，它会发生变化。它独立工作，但在这两种情况下，底层 jar 都是相同的。

Apache Twill 还有一个添加额外资源的功能，但是在那里添加我的 jar 也不起作用，相反我得到了一个异常，即 jar 已经包含在内：

线程“ServiceDelegate STARTING”中的异常 java.lang.RuntimeException：java.util.zip.ZipException：重复条目：lib/drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar

在 com.google.common.base.Throwables.propagate(Throwables.java:160) 在 org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:133) 在 org.apache.twill.internal.AbstractZKServiceController.startUp (AbstractZKServiceController.java:82) 在 org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:109) 在 com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43 ) 在 java.lang.Thread.run(Thread.java:744) 引起：java.util.zip.ZipException：重复条目：lib/drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed。 jar 在 java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:215) 在 java.util.jar.JarOutputStream.putNextEntry(JarOutputStream.java:109) 在 org.apache.twill.internal。ApplicationBundler.copyResource(ApplicationBundler.java:347) 在 org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:140) 在 org.apache.twill.yarn.YarnTwillPreparer.createContainerJar(YarnTwillPreparer.java:388) 在 org .apache.twill.yarn.YarnTwillPreparer.access$300(YarnTwillPreparer.java:106) 在 org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:264) 在 org.apache.twill.yarn.YarnTwillPreparer$1。在 org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:98) 处调用(YarnTwillPreparer.java:253) ... 还有 4 个yarn.YarnTwillPreparer.access$300(YarnTwillPreparer.java:106) at org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:264) at org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java :253) 在 org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:98) ... 4 更多yarn.YarnTwillPreparer.access$300(YarnTwillPreparer.java:106) at org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:264) at org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java :253) 在 org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:98) ... 4 更多

使用的底层类加载器是URLClassLoader. 它是用一个空数组初始化的，但它适用于独立应用程序，问题只是当它与 Apache Twill 一起运行时，它从哪里获取它应该查找的 URL？我怎么能检查它？

类加载器定义：

public class QueryClassLoader extends URLClassLoader {

  static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(QueryClassLoader.class);

  private final ClassCompiler classCompiler;
  private AtomicLong index = new AtomicLong(0);
  private ConcurrentMap<String, byte[]> customClasses = new MapMaker().concurrencyLevel(4).makeMap();

  public QueryClassLoader(boolean useJanino) {
    super(new URL[0]);
    if (useJanino) {
      this.classCompiler = new JaninoClassCompiler(this);
    } else {
      throw new UnsupportedOperationException("Drill no longer supports using the JDK class compiler.");
    }
  }
  ...

我可以研究的任何想法，为什么会发生错误或如何解决它？

4

1 回答 1

-1

The same question was asked in the Apache Twill mailing list. Here is the discussion and proposed solution to it.

http://mail-archives.apache.org/mod_mbox/twill-dev/201406.mbox/%3CCAHqY-MOa8jBYs%3DEZENxxNZg-9YGMR5SASg76P_k6%2Bm6p2L9JuQ%40mail.gmail.com%3E

Repeat my answer in the mail content:

I am not familiar with how janino works, but it seems to me that it may not be using context ClassLoader to load classes or as least the thread that is compiling the generated class does not have the context ClassLoader set properly.

The way that Twill works is pretty straightforward. It creates a "launcher.jar", which has no dependency on any library and start the JVM in a YARN container like this:

java -cp launcher.jar ....

Hence the system classloader has no user/library classes, but only the Launcher class.

Then in the Launcher.main() method, it creates a URLClassLoader, using all the jars + .class files inside the "container.jar" file, to load the user TwillRunnable. It also sets it as the context ClassLoader of the thread that calls the "run()" method. So, if you want to load class manually (through ClassLoader or Class.forName) in a different thread than the "run()" thread, you'll have to use set the context ClassLoader of that thread or explicitly construct the ClassLoader with the correct parent ClassLoader.

于 2014-11-05T08:27:21.223 回答