c# - C# 数据集、分页和大量数据

Question

我想在数据集中显示大量数据，100,000 条记录大约 10 列，这会消耗大量 ram 700MB。我也尝试过使用分页，这可以减少大约 15-20%，但我不太喜欢使用分页时涉及的上一个/下一个按钮。我目前没有将数据写入磁盘，应该是吗？如果是这样，最常用的方法是什么？数据不会在查看时永久存储，然后可能会运行新查询并查看另外 70,000 条记录。最好的方法是什么？

感谢您的建议。

Answer 1

The reality is that the end-user rarely needs to see the totality of their dataset, so I would use which method you like for presenting the data (listview) and build a custom pager so that the dataset is only fed with the results of the number of records desired. Otherwise, each page load would result in re-calling the dataset.

The XML method to a temp file or utilizing a temp table created through a stored proc are alternatives but you still must sift and present the data.

Answer 2

An important question is where this data comes from. That will help determine what options are available to you. Writing to disk would work, but it probably isn't the best choice, for three reasons:

1. As a user, I'd be pretty annoyed if your app suddenly chewed up 700Mb of disk space with no warning at all. But, then, I'd notice such things. I suppose a lot of users wouldn't. Still: it's a lot of space.
2. Depending on the source of the data, even the initial transfer could take longer than your really want to allow.
3. Again, as a user, there's NO WAY I'm manually digging through 700Mb worth of data. That means you almost certainly never need to show it. You want to only load the requested page, one (or a couple) pages at a time.

Answer 3

That is a lot of data to be working with and keeping aroudn in memory.

Is this a ASP.NET app? Or a Windows app?

I personally have found that going with a custom pager setup (to control, next previous links) and paging at the database level to be the only possible way to get the best performance, only get the data needed....

Answer 4

implement paging in SQL if you want to reduce the memory footprint

Answer 5

我建议使用内存映射文件...不确定 .NET 是否包含对它的支持。