The JVM increases the heap size when it needs to up to the maximum heap size you set. It doesn't take all the memory as it has to preallocate this on startup and you might want to use some memory for something else, like thread stacks, share libraries, off heap memory etc.
Why Java does not expand the heap size until it hits the OS-imposed process memory limit, in the same way .NET CLR does?
If you set the maximum heap size large enough, or use off heap memory, it will. It just won't do this by default. One reason is that heap memory has to be in main memory and cannot be swapped out without killing the performance of your machine (if not killing your machine) This is not true of C programs and expanding so much is worse than failing to expand.
If you have a JVM with a heap size of 10% more than main memory and you use that much, as soon as you perform a GC, which has to touch every page more than once, you are likely to find you need to power cycle the box.
Linux has a process killer when resources run out, and this doesn't trigger you might be luck enough to restart.
Is it just a policy made by JVM developers, or is an advantage of .NET CLR's architecture over JVM's one
A key feature of the JVM is that it is platform independent, so it has its own control. The JVM running at the limit of your process space is likely to prevent your machine from working (from heavy swapping) I don't know .NET avoids this from happening.
In other words, if Oracle engineers want to implement automatic heap expansion for the JVM, are they able to do that?
It does already as I have said, it's just not a good idea to allow it to use too much memory.