The difference is probably because your managed program has greater needs than the unmanaged program. And so there are fewer resources left for your DLL. Programs can run out of memory even when there appears to be memory left. This happens when the address space is fragmented and the virtual memory allocator cannot find a large enough contiguous block of memory to meet an allocation request.
There's not much you can do about that I suspect. I doubt you'll have much luck trying to make your managed process consume fewer resources. If you run on a 64 bit system and your process is large address aware then your process can access have 4GB of memory.
Marking your .net executable as LARGEADDRESSAWARE
might well help. For details please refer to another SO question: How to make a .NET application "large address aware"? In summary you need to run the EDITBIN
tool to mark your executable:
EDITBIN /LARGEADDRESSAWARE YourExeName.exe
Be aware though that on a 32 bit OS you are pretty much stuck with the 2GB limit. Yes you can boot your system with the /3GB switch but this is generally not to be advised.
The other obvious option would be to house the DLL in an out of process COM server. By putting it in a different process you could keep the .net runtime out of the process and leave as much space as possible for the hungry DLL.