3

className据我了解,KryonumberID在每个writeObject. 这张地图太窄了。因为在您的对象模型中,实例往往属于相同的类,接下来 writeObject 将再次构建和序列化类似的映射(一次又一次,一次又一次)。我知道可以通过手动注册类来共享地图,但这是繁琐的手动硬编码。我希望映射将由第一个对象写入启动,就像它通常那样,但是会话中的所有后续写入都将重用和扩展它。这样,注册将在运行时自动发生,没有额外的运行时开销,更频繁使用的对象自然会收到低 ID 号。该映射可以在之后单独存储在附件中,作为解密密钥。反序列化器将从加载此映射开始。你喜欢这个想法,如何实施?

我的问题类似于 使用 kryo 注册类的策略,其中用户可以使用列表将所有写入组合在单个 writeObject 下。正如我建议的那样,它甚至比单独存储地图更简单。但是,他似乎并不想这样做。就我而言,这种组合甚至是不可能的,因为大型 java 模型我避免通过分段序列化将其完全保存在内存中。在我的场景中,用户打开一个项目,进行更改并刷新它们。因此,该项目可以维护类映射并将其用于所有序列化。

更新!我意识到有类/对象注册器和自动重置。它们似乎是为这项任务而创建的。但是,我不明白这些事情如何解决它。Autoreset=false确实使第二次写入要小得多。但是,在这种情况下,我无法反序列化对象。正如您在示例中看到的,第二次反序列化失败:

public class A {
    String f1;
    A(String a) {
        f1 = a;
    }
    List list = new ArrayList();
    public String toString() {
        return "f1 = " + f1 + ":" + f1.getClass().getSimpleName();
    }

    public static void main(String[] args) {
        test(true);
        test(false);
    }


    static void write(String time, Kryo kryo, ByteArrayOutputStream baos, Object o) {
        Output output = new Output(baos); 
        kryo.writeClassAndObject(output, o); 
        output.close();
        System.err.println(baos.size() + " after " + time + " write");
    }

    private static void test(boolean autoReset) {
        Kryo kryo = new Kryo();
        kryo.setAutoReset(autoReset);
        kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());
        System.err.println("-------\ntesting autoreset = " + autoReset);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        A a = new A("a"), b = new A("b");
        write("first", kryo, baos, a);
        write("second", kryo, baos, b);
        A o1 = restore("first", baos, kryo);
        A o2 = restore("second", baos, kryo); // this fails
        System.err.println((o1.f1.equals(o2.f1)) ? "SUCCESS" : "FAILURE");

    }

    private static A restore(String time, ByteArrayOutputStream baos, Kryo k) {
        ByteArrayInputStream in = new ByteArrayInputStream(baos.toByteArray());
        Input input = new Input(in);
        A o = (A) k.readClassAndObject(input);
        System.err.println("reading object " + time + " time, got " + o);
        return o;
    }

输出是

-------
testing autoreset = true
41 after first write
82 after second write
reading object first time, got f1 = a:String
reading object second time, got f1 = a:String
SUCCESS
-------
testing autoreset = false
41 after first write
52 after second write
reading object first time, got f1 = a:String
reading object second time, got null
Exception in thread "main" java.lang.NullPointerException
    at kryo_test.AutoresetDemo.test(AutoresetDemo.java:40)
    at kryo_test.AutoresetDemo.main(AutoresetDemo.java:18)

Update2除了类名之外,autoReset=false 还可能记录对象引用。确实值得自动重置。

Update3我发现很难序列化类的映射(即类 -> 注册),因为注册包含引用 kryo 对象并保持某些状态的序列化程序。那时很难在许多 kryo 对象之间共享地图。

4

1 回答 1

3

好的,这是kryo-2.20的解决方案

public class GlobalClassKryo extends Kryo {

    public static class ExternalizableClassResolver implements ClassResolver {

        //local serializers
        final Map<Class, Registration> fromClass = new HashMap();
        final Map<Integer, Registration> fromId = new HashMap();

        public static class GlobalRegistration { int id; Class type; Class<? extends Serializer> serializer; }

        public final Map<Integer, GlobalRegistration> globalIds; 
        public final Map<Class, GlobalRegistration> globalClasses; 

        // I synchronize because I have one reader and one writer thread and 
        // writer may break the reader when adds something into the map. 
        public ExternalizableClassResolver() {this (
                Collections.synchronizedMap(new HashMap()), 
                Collections.synchronizedMap(new HashMap())
            ) ;}

        public ExternalizableClassResolver(Map<Integer, GlobalRegistration> ids, Map<Class, GlobalRegistration> classes) {
            globalIds = ids;
            globalClasses = classes;
        }

        public ExternalizableClassResolver (DataInput in) throws ClassNotFoundException, IOException {
            this();
            int id;
            while ((id = in.readInt()) != -1) {
                GlobalRegistration e = new GlobalRegistration();
                globalIds.put(e.id = id, e);
                e.type = Class.forName(in.readUTF());
                e.serializer = (Class<? extends Serializer>) Class.forName(in.readUTF());
                globalClasses.put(e.type, e);
            }
        }

        public void save(DataOutput out) throws IOException {
            for (GlobalRegistration entry : globalIds.values()) {
                    out.writeInt(entry.id);
                    out.writeUTF(entry.type.getName());
                    out.writeUTF(entry.serializer.getName());
            }
            out.writeInt(-1);
        }

        static final boolean TRACE = false;
        void log(String msg) {
            System.err.println(kryo != null ? Utils.fill(kryo.getDepth(), ' ')  + msg : msg);
        }
        @Override
        public Registration writeClass(Output output, Class type) {
            if (type == null) {output.writeInt(0, true); return null;}
            Registration registration = kryo.getRegistration(type);
            output.writeInt(registration.getId(), true);
            return registration;
        }
        @Override
        public Registration readClass(Input input) {
            int classID = input.readInt(true);
            if (classID == 0) return null;
            Registration registration = fromId.get(classID);
            if (registration == null) {
                registration = tryGetFromGlobal(globalIds.get(classID), classID + "");
            }
            if (registration == null) throw new KryoException("Encountered unregistered class ID: " + classID);
            return registration;
        }

        public Registration register(Registration registration) {
            throw new KryoException("register(registration) is not allowed. Use register(type, serializer)");
        }

        public Registration getRegistration(int classID) {
            throw new KryoException("getRegistration(id) is not implemented");
        }

        Registration tryGetFromGlobal(GlobalRegistration globalClass, String title) {
            if (globalClass != null) {
                Serializer serializer = kryo.newSerializer(globalClass.serializer, globalClass.type);
                Registration registration = register(globalClass.type, serializer, globalClass.id, "local");
                if (TRACE) log("getRegistration(" + title + ") taken from global => " + registration);
                return registration;
            } else
                if (TRACE) log("getRegistration(" + title + ") was not found");
            return null;
        }
        public Registration getRegistration(Class type) {
            Registration registration = fromClass.get(type);
            if (registration == null) {
                registration = tryGetFromGlobal(globalClasses.get(type), type.getSimpleName());
            } else
                if (TRACE) log("getRegistration(" + type.getSimpleName() + ") => " + registration);

            return registration;
        }

        Registration register(Class type, Serializer serializer, int id, String title) {
            Registration registration = new Registration(type, serializer, id);
            fromClass.put(type, registration);
            fromId.put(id, registration);

            if (TRACE) log("new " + title + " registration, " + registration);

            //why dont' we put into fromId?
            if (registration.getType().isPrimitive()) fromClass.put(getWrapperClass(registration.getType()), registration);
            return registration;
        }

        int primitiveCounter = 1; // 0 is reserved for NULL
        static final int PRIMITIVE_MAX = 20;

        //here we register anything that is missing in the global map. 
        // It must not be the case that something available is registered for the second time, particularly because we do not check this here
        // and use registered map size as identity counter. Normally, check is done prior to callig this method, in getRegistration
        public Registration register(Class type, Serializer serializer) {

            if (type.isPrimitive() || type.equals(String.class))
                return register(type, serializer, primitiveCounter++, "primitive");

            GlobalRegistration global = globalClasses.get(type);

            if (global != null )
                    throw new RuntimeException("register(type,serializer): we have " + type + " in the global map, this method must not be called");

            global = new GlobalRegistration();
            globalIds.put(global.id = globalClasses.size() + PRIMITIVE_MAX, global); 
            globalClasses.put(global.type = type, global);
            global.serializer= serializer.getClass();

            return register(global.type, serializer, global.id, "global");
        }

        public Registration registerImplicit(Class type) {
            throw new RuntimeException("registerImplicit is not needed since we register missing automanically in getRegistration");
        }

        @Override
        public void reset() {
            // super.reset(); //no need to reset the classes
        }

        Kryo kryo;
        public void setKryo(Kryo kryo) {
            this.kryo = kryo;
        }
    }




    public ExternalizableClassResolver ourClassResolver() {
        return (ExternalizableClassResolver) classResolver;
    }

    public GlobalClassKryo(ClassResolver resolver) {
        super(resolver, new MapReferenceResolver()); 
        setInstantiatorStrategy(new StdInstantiatorStrategy());
        this.setRegistrationRequired(true);
    }
    public GlobalClassKryo() {
        this(new ExternalizableClassResolver());
    }

    @Override
    public Registration getRegistration (Class type) {
        if (type == null) throw new IllegalArgumentException("type cannot be null.");

        if (type == memoizedClass) return memoizedClassValue;
        Registration registration = classResolver.getRegistration(type);
        if (registration == null) {
            if (Proxy.isProxyClass(type)) {
                // If a Proxy class, treat it like an InvocationHandler because the concrete class for a proxy is generated.
                registration = getRegistration(InvocationHandler.class);
            } else if (!type.isEnum() && Enum.class.isAssignableFrom(type)) {
                // This handles an enum value that is an inner class. Eg: enum A {b{}};
                registration = getRegistration(type.getEnclosingClass());
            } else if (EnumSet.class.isAssignableFrom(type)) {
                registration = classResolver.getRegistration(EnumSet.class);
            }
            if (registration == null) {
                //registration = classResolver.registerImplicit(type);
                return register(type, getDefaultSerializer(type));
            }
        }
        memoizedClass = type;
        memoizedClassValue = registration;
        return registration;
    }

    public Registration register(Class type, Serializer serializer) {
        return ourClassResolver().register(type, serializer);}

    public Registration register(Registration registration) {
        throw new RuntimeException("only register(Class, Serializer) is allowed");}

    public Registration register(Class type) {
        throw new RuntimeException("only register(Class, Serializer) is allowed");}

    public Registration register(Class type, int id) {
        throw new RuntimeException("only register(Class, Serializer) is allowed");}

    public Registration register(Class type, Serializer serializer, int id) {
        throw new RuntimeException("only register(Class, Serializer) is allowed");
    }

    static void write(String title, Kryo k, ByteArrayOutputStream baos, Object obj) {
        Output output = new Output(baos);
        k.writeClassAndObject(output, obj);
        output.close();
        System.err.println(baos.size() + " bytes after " + title + " write");
    }
    static class A {
        String field = "abcABC";
        A a = this;
        //int b = 1; // adds 1 byte to serialization
        @Override
        public String toString() {
            return field 
                    + " " + list.size() 
                    //+ ", " + b
                    ;
        }

        // list adds 3 bytes to serialization, two 3-byte string items add additionally 10 bytes in total
        ArrayList list = new ArrayList(100); // capacity is trimmed in serialization
        {
            list.add("LLL");
            list.add("TTT");

        }
    }

    private static void test() throws IOException, ClassNotFoundException {
        GlobalClassKryo k = new GlobalClassKryo();

        ByteArrayOutputStream baos = new ByteArrayOutputStream();

        write("first", k, baos, new A()); // write takes 24 byts

        //externalize the map
        ByteArrayOutputStream mapOut = new ByteArrayOutputStream();
        DataOutputStream dataOut = new DataOutputStream(mapOut);
        k.ourClassResolver().save(dataOut);
        dataOut.close();

        //deserizalize the map
        DataInputStream serialized = new DataInputStream(new ByteArrayInputStream(mapOut.toByteArray()));
        ExternalizableClassResolver resolver2 = new ExternalizableClassResolver(serialized);

        //use the map
        k = new GlobalClassKryo(resolver2);
        write("second", k, baos, new A()); // 24 bytes

        Input input = new Input(new ByteArrayInputStream(baos.toByteArray()));
        Object read = k.readClassAndObject(input);
        System.err.println("output " + read);
    }

    public static void main(String[] args) throws IOException, ClassNotFoundException {
        Kryo k = new Kryo();
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        write("first", k, baos, new A()); // write takes 78 bytes
        write("second", k, baos, new A()); // +78 bytes
        System.err.println("----------------");

        test();
    }
}

结果流没有类名。不幸的是,与默认的 java 序列化(2x 或更多)相比,Kryo 太慢了,尽管它产生的流更加密集。仅 Kryo 就使我的示例序列化几乎缩小了 10 倍。您会看到此答案中提出的解决方案增加了 3 倍的附加因子。但是,在我序列化兆字节的字段中,相对于 java 序列化,我得到的压缩只有 2 倍,而使用 Kryo 存储到磁盘时,速度会降低 2 倍。

于 2012-09-11T22:08:26.773 回答