delphi - TClientDataSet and limit by memory

Question

We have a system that creates reports out of our data. And we can deal with a lot of data. The idea of over 150,000 rows is not out of the question.

Unfortunately, our experience with TClientDataSet is its limitations, because it often results in an 'insufficient memory for this operation' error, when the data gets too big.

So the question is this: Does there exist a generally available implementation of TDataSet that can handle a large amount of data (such as streaming directly to a file and not keeping the entire dataset in memory)?

I am willing to implement such a class myself. But as far as I understand TClientDataSet, it needs to be able to contain the data itself before it can save it to a file/stream. In addition, loading the data again should also be possible as a stream rather than loading in an entire TClientDataSet object, because then we wouldn't have solved the issue.

Answer 1

您可以在嵌入式模式下使用 FireBird 或 Interbase 。

Answer 2

真的有必要在报告之前缓存客户端上的所有数据吗？如果没有，也许重新考虑如何查询和处理数据以生成这些报告，看看是否有一种方法可以减少客户端数据（这会带来通过网络传输的数据更少的好处）。

如果您之前已经走这条路，并且确实需要所有这些数据客户端，那么您可以查看自定义数据结构。A TList<T>of 记录（即使您需要建立自己的索引）比 a 占用的内存要少得多TClientDataSet。

Answer 3

KBMMemTable 是 TClientDataset 的一个不错的替代品

http://www.components4programmers.com/products/kbmmemtable/

我们使用它已有多年，它非常有用且快速。

Answer 4

想要强调 TClientDataset 的容量可能更大。

测试 TClientDataset 的限制 - 附加 xxx,xxx 记录，将单个记录放在整个（重复）以创建大小的想法。

// 开始将记录加载到 TCLientDataset 以进行反向（反向）处理

dxMemData1.Append;

dxMemData1['NT_Rec_No'] := 1000;

dxMemData1['NT_User'] := 'DEV\Administrator';

dxMemData1['NT_Type'] := '信息';

dxMemData1['Ora_Timestamp'] := '20170706033859.000';

dxMemData1['Ora_Host'] := 'DEV0001';

dxMemData1['Ora_SID'] := 'Oracle.orcl';

dxMemData1['Ora_Event_Id'] := '34';

dxMemData1['NT_Message'] := Memo1.Text;

dxMemData1.Post；

// 结束加载记录到 TCLientDataset 进行反向（on Reverse）处理

memo1 上的字符串是 100 个字符的字符串 - ansi

进行了几次测试，并设法继续处理 600,000 到 900,000 条记录而不会崩溃。

可以通过使备忘录上的文本更大来产生差异 - 这确实减少了崩溃前的最大数量 - 这意味着它不是确切的最大值问题。记录数-但消耗的大小-我的猜测。

用 TdxMemData (devexpress) 进行了同样的测试，这次我几乎可以达到两倍的记录