问题标签 [memory-mapped-files]

在可滚动框中显示大型内存映射文件时，让视图代表整个文件还是仅代表当前正在显示的文件区域更合适？
更广泛地说，.Net 的内存映射文件在什么抽象级别进行分页？内存中保存的文件的页面大小/数量与 MemoryMappedViewAccessors 相关，还是独立于这些对象？

I am writing a routine to compare two files using memory-mapped file. In case files are too big to be mapped at one go. I split the files and map them part by part. For example, to map a 1049MB file, I split it into 512MB + 512MB + 25MB.

Every thing works fine except one thing: it always take much, much longer to compare the remainder (25MB in this example), though the code logic is exactly the same. 3 observations:

1. it does not matter which is compared first, whether the main part (512MB * N) or the remainder (25MB in this example) comes first, the result remains the same
2. the extra time in the remainder seems to be spent in the user mode
3. Profiling in VS2010 beta 1 shows, the time is spent inside t std::_Equal(), but this function is mostly (profiler says 100%) waiting for I/O and other threads.

I tried

• changing the VIEW_SIZE_FACTOR to another value
• replacing the lambda functor with a member function
• changing the file size under test
• changing the order of execution of the remainder to before/after the loop

The result was quite consistent: it takes a lot more time in the remainder part and in the User Mode.

I suspect it has something to do with the fact that the mapped size is not a multiple of mapping alignment (64K on my system), but not sure how.

Below is the complete code for the routine and a timing measured for a 3G file.

Can anyone please explain it, Thanks?

output:

segment size = 354410496 bytes for the remaining round

real 116.892s, cpu 56.201s (48.1%), user 54.548s, system 1.652s

segment size = 536870912 bytes, round #0

real 72.258s, cpu 2.273s (3.1%), user 0.320s, system 1.953s

segment size = 536870912 bytes, round #1

real 75.304s, cpu 1.943s (2.6%), user 0.240s, system 1.702s

segment size = 536870912 bytes, round #2

real 84.328s, cpu 1.783s (2.1%), user 0.320s, system 1.462s

segment size = 536870912 bytes, round #3

real 73.901s, cpu 1.702s (2.3%), user 0.330s, system 1.372s

---

More observations after the suggestions by responders

Further split the remainder into body and tail(remainder = body + tail), where

• body = N * alignment(), and tail < 1 * alignment()
• body = m * alignment(), and tail < 1 * alignment() + n * alignment(), where m is even.
• body = m * alignment(), and tail < 1 * alignment() + n * alignment(), where m is exponents of 2.
• body = N * alignment(), and tail = remainder - body. N is random.

the total time remains unchanged, but I can see that the time does not necessary relate to tail, but to size of body and tail. the bigger part takes more time. The time is USER TIME, which is most incomprehensible to me.

I also look at the pages faults through Procexp.exe. the remainder does NOT take more faults than the main loop.

---

Updates 2

I've performed some test on other workstations, and it seem the issue is related to the hardware configuration.

Test Code

Observation:

On another 2 PCs with the same h/w configurations with mine, the result is consistent as following:

------VS2010Beta1ENU_VSTS.iso [1319909376 bytes] ------

Remainder size = 44840960 bytes

Remainder_tail size = 14336 bytes

real 0.060s, cpu 0.040s (66.7%), user 0.000s, system 0.040s

Remainder_body size = 44826624 bytes

real 13.601s, cpu 7.731s (56.8%), user 7.481s, system 0.250s

segment size = 67108864 bytes, total round# = 19

real 172.476s, cpu 4.356s (2.5%), user 0.731s, system 3.625s

However, running the same code on a PC with a different h/w configuration yielded:

------VS2010Beta1ENU_VSTS.iso [1319909376 bytes] ------ Remainder size = 44840960 bytes

Remainder_tail size = 14336 bytes

real 0.013s, cpu 0.000s (0.0%), user 0.000s, system 0.000s

Remainder_body size = 44826624 bytes

real 2.468s, cpu 0.188s (7.6%), user 0.047s, system 0.141s

segment size = 67108864 bytes, total round# = 19

real 65.587s, cpu 4.578s (7.0%), user 0.844s, system 3.734s

System Info

My workstation yielding imcomprehensible timing:

OS Name: Microsoft Windows XP Professional

OS Version: 5.1.2600 Service Pack 3 Build 2600

OS Manufacturer: Microsoft Corporation

OS Configuration: Member Workstation

OS Build Type: Uniprocessor Free

Original Install Date: 2004-01-27, 23:08

System Up Time: 3 Days, 2 Hours, 15 Minutes, 46 Seconds

System Manufacturer: Dell Inc.

System Model: OptiPlex GX520

System type: X86-based PC

Processor(s): 1 Processor(s) Installed.

BIOS Version: DELL - 7

Windows Directory: C:\WINDOWS

System Directory: C:\WINDOWS\system32

Boot Device: \Device\HarddiskVolume2

System Locale: zh-cn;Chinese (China)

Input Locale: zh-cn;Chinese (China)

Time Zone: (GMT+08:00) Beijing, Chongqing, Hong Kong, Urumqi

Total Physical Memory: 3,574 MB

Available Physical Memory: 1,986 MB

Virtual Memory: Max Size: 2,048 MB

Virtual Memory: Available: 1,916 MB

Virtual Memory: In Use: 132 MB

Page File Location(s): C:\pagefile.sys

NetWork Card(s): 3 NIC(s) Installed.

Another workstation yielding "correct" timing: OS Name: Microsoft Windows XP Professional

OS Version: 5.1.2600 Service Pack 3 Build 2600

OS Manufacturer: Microsoft Corporation

OS Configuration: Member Workstation

OS Build Type: Multiprocessor Free

Original Install Date: 5/18/2009, 2:28:18 PM

System Up Time: 21 Days, 5 Hours, 0 Minutes, 49 Seconds

System Manufacturer: Dell Inc.

System Model: OptiPlex 755

System type: X86-based PC

Processor(s): 1 Processor(s) Installed.

BIOS Version: DELL - 15

Windows Directory: C:\WINDOWS

System Directory: C:\WINDOWS\system32

Boot Device: \Device\HarddiskVolume1

System Locale: zh-cn;Chinese (China)

Input Locale: en-us;English (United States)

Time Zone: (GMT+08:00) Beijing, Chongqing, Hong Kong, Urumqi

Total Physical Memory: 3,317 MB

Available Physical Memory: 1,682 MB

Virtual Memory: Max Size: 2,048 MB

Virtual Memory: Available: 2,007 MB

Virtual Memory: In Use: 41 MB

Page File Location(s): C:\pagefile.sys

NetWork Card(s): 3 NIC(s) Installed.

Any explanation theory? Thanks.

内容提供者消费者的客户端我可以做这样的事情，以获得正确的 InputStream 来读取图片：

这是一个很好的 API，并且在服务器端，实际的内容提供者会调用：

但是，如果映射到 URI 的图片不在文件系统上找到，而是作为内存资源，或者动态生成，该怎么办。

我可以创建一个内存映射的FileorInputStream或其他任何东西，这样我就不需要将临时文件保存到磁盘，只是为了能够将它返回给我的内容提供者消费者？

在 Windows 中，如何跟踪由 CreateFileMapping() 创建的文件映射对象？基本上我想知道有多少内存映射文件正在使用以及它们的名称、大小等信息。谢谢！

我意识到.NET 4.0 处于测试阶段，但我希望有人对此有解决方案。我正在尝试从 DLL 创建内存映射文件：

不幸的是，无论我做什么，我总是得到一个UnauthorizedAccessException，MSDN 文档指出：

操作系统拒绝了对文件的指定访问；例如，访问权限设置为 Write 或 ReadWrite，但文件或目录是只读的。

我已经使用 Sysinternals Process Monitor 监控了我的应用程序，这表明该文件确实已成功打开。我也尝试过内存映射其他非 DLL 文件，但结果相同。

我有一个使用大量内存映射文件的进程。
问题是这些文件保存在物理内存中，即使机器内存不足，其他进程也需要此内存。

我尝试使用SetProcessWorkingSetSize来限制进程工作集，但它没有帮助，进程的工作集不断增长超过最大值。

有没有更好的方法来限制进程的工作集？
我可以更改 Windows 的启发式方法来分页内存映射文件吗？

在 Windows 中，可以通过CreateFile(). 我以前成功地使用过它来ReadFile()从设备，但现在我想切换到内存映射。在下面的代码中，我收到INVALID_HANDLE_VALUE, 的值b并c设置为87, ERROR_INVALID_PARAMETER。

我怎样才能让它工作？

使用内存映射文件时，它似乎是只读的或只写的。我的意思是你不能：

• 有一个可以写，后来决定不保存
• 已打开阅读，后来决定保存

我们的应用程序使用一个可写的内存映射文件来保存数据文件，但是由于用户可能想退出而不保存更改，我们必须使用用户实际编辑的临时文件。当用户选择保存更改时，原始文件将被临时文件覆盖，因此它具有最新的更改。这很麻烦，因为文件可能非常大（> 1GB）并且复制它们需要很长时间。

我尝试了许多用于创建文件映射的标志组合，但似乎没有一个允许按需保存的灵活性。任何人都可以确认是这种情况吗？我们的应用程序是用 Delphi 编写的，但它使用标准的 Windows API 来创建映射，在我们的例子中

我对人们比较在传统的基于 SQL 的数据库中存储数据或使用内存映射文件（例如新的 .Net 4.0 运行时中的文件）的想法感兴趣。有问题的数据将是简单结构的数组。

明显的优点和缺点：

SQL 数据库专家

1. 即席查询支持
2. SQL 管理工具
3. 架构更改（添加更多列并设置默认值）

内存映射的优点

4. 头顶更轻？（这是我的一个假设）
5. 可在进程线程之间共享
6. 还有其他人吗？

为了性能提升值得吗？

When we map a file to memory, a system call is required. Do subsequent accesses to the file require system calls or is the virtual memory page of the process mapped to the actual page cache in memory?

update: what i also want to know is that if multiple processes are accessing the same file through mmap. they will be accessing the same physical memory portion write.