I have an implementation of a RNN for sequence classification, on my local machine (Win 10, 16Gb ram) when I run the training, it reaches sometimes 100% memory usage. When I try to run it on a Azure VM (Linux Ubuntu, 14gb ram) the process get killed as soon as it reaches high ram usages.
I am currently using a batch size of 5000 on the local machine, so I tried to reduce the size for the VM, but even if I put it at 2000, the process got killed anyway.
Also on the vm it gives me warnings: tensorflow/core/framework/allocator.cc:113] Allocation of 1152000000 exceeds 10% of system memory.
What I don't understand is how the local machine can handle the memory usage, and the vm cannot even if they differ by only 2gb of ram.
Does someone have any clue?
Let me know if you need more information, thanks in advance!