1

在序列化具有大约 250 个属性和大约 20,000 行的动态对象时，我收到以下错误。当属性数量约为 20 时，相同的代码工作正常。错误发生在点Serializer.Serialize(stream, lst);

An unhandled exception of type 'System.OutOfMemoryException' occurred in System.ServiceModel.Internals.dll

at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at ProtoBuf.ProtoWriter.Flush(ProtoWriter writer) in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 534
at ProtoBuf.ProtoWriter.Dispose() in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 478
at ProtoBuf.ProtoWriter.System.IDisposable.Dispose() in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 472
at ProtoBuf.Meta.TypeModel.Serialize(Stream dest, Object value, SerializationContext context) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 218
at ProtoBuf.Meta.TypeModel.Serialize(Stream dest, Object value) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 201
at ProtoBuf.Serializer.Serialize[T](Stream destination, T instance) in c:\Dev\protobuf-net\protobuf-net\Serializer.cs:line 87
at WcfService1.DynamicWrapper.Serialize(DynamicWrapper lst) in c:\Users\rkohli\Documents\Visual Studio 2012\Projects\WindowsFormsApplication3\WcfService1\SerializeObject.cs:line 136
at WcfService1.Service1.GetData(String sVisibleColumnList) in c:\Users\rkohli\Documents\Visual Studio 2012\Projects\WindowsFormsApplication3\WcfService1\Service1.svc.cs:line 22
at SyncInvokeGetData(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
at System.ServiceModel.Dispatcher.ChannelHandler.OnContinueAsyncReceive(Object state)
at System.Runtime.ActionItem.DefaultActionItem.TraceAndInvoke()
at System.Runtime.ActionItem.DefaultActionItem.Invoke()
at System.Runtime.ActionItem.CallbackHelper.InvokeWithoutContext(Object state)
at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

下面是代码示例。

[Serializable]
[ProtoContract]
public class DynamicWrapper
{
    [ProtoMember(1, DataFormat = DataFormat.Group)]
    public List<DictWrapper> Items { get; set; }

    public DynamicWrapper()
    {
        Items = new List<DictWrapper>();
    }

    public static byte[] Serialize(DynamicWrapper lst)
    {
        byte[] msgOut;

        using (var stream = new MemoryStream())
        {
            Serializer.Serialize(stream, lst);
            msgOut = stream.ToArray();
        }

        return msgOut;
    }

    public static DynamicWrapper Deserialize(byte[] message)
    {
        DynamicWrapper msgOut;

        using (var stream = new MemoryStream(message))
        {
            msgOut = Serializer.Deserialize<DynamicWrapper>(stream);
        }

        return msgOut;
    }
}

[Serializable]
[ProtoContract]
public class DictWrapper
{
    [ProtoMember(1, DataFormat = DataFormat.Group)]
    public Dictionary<string, string > Dictionary { get; set; }

    public DictWrapper()
    {
        Dictionary = new Dictionary<string, string>();
    }
}
4

1 回答 1

1

这里没有什么魔法。基于代码和单独发送的工作项目 - 数据很简单：大。准确地说是 120446305 字节（基于样本数据）。这里的主要问题是您正在使用字符串属性名称，并且一遍又一遍地重复它们。现在，protobuf-net 可以支持字符串缓存和重用，但默认情况下它不支持 - 并且没有简单的方法将它应用到Dictionary<string,string>. 但坦率地说，在我想出让它在这种情况下工作的疯狂方法之前（这必然是一个重大变化），我必须首先指出这根本不适合 protobuf。Protobuf 不提供“总是更小”的保证：它提供了在典型场景下做得很好，即您的架构是预先知道的并且是可预测的。一切都不是这个特定的场景。

实际上，在给出的示例中，它正在从 a 加载数据DataSet- 特别是原始数据中只有 284MB。您在此处使用 protobuf-net，对于不针对的场景，导致大小增长了 4 倍。

坦率地说，您最好发送原始DataSet有效载荷。甚至更好：将数据集切换为二进制模式并发送（162 MB）。

using (var file = File.Create("binary-ds"))
{
    dataSet.WriteXml(file, XmlWriteMode.WriteSchema);
}

或者更好 - 将其切换到二进制模式并通过 gzip 运行，总共 15MB：

using (var file = File.Create("binary-ds"))
using (var gzip = new GZipStream(file, CompressionLevel.Optimal))
{
    dataSet.WriteXml(gzip, XmlWriteMode.WriteSchema);
}

如果我使用常规 POCO/DTO 类重新编写您的示例并通过 protobuf-net 运行它，我怀疑结果会相似（但没有DataTable这里的所有开销），但不会有一种简单的改变方法您的场景可以很好地与 protobuf-net 配合使用，而无需更改数据布局。如果您需要更改数据布局 - 上面的内容要容易得多。

于 2013-09-30T18:59:05.507 回答