I am using liferay 6.0.6 with tomcat 6 and using terracotta 3.5.4. The terracotta installation in on another server and works fine.
The debian and redhat systems are virtual machines and use the exact same virtual hardware. 1 cpu , 4gb ram, both 64bit OS.
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.3) (rhel-1.48.1.11.3.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
If im using a debian system and boot the application then it everything works as expected.
If im using the redhat system then i get an error :
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at java.util.Timer.<init>(Timer.java:176)
at com.tc.object.locks.ClientLockManagerImpl.<init>(ClientLockManagerImpl.java:39)
at com.tc.object.StandardDSOClientBuilder.createLockManager(StandardDSOClientBuilder.java:190)
at com.tc.object.DistributedObjectClient.start(DistributedObjectClient.java:639)
at com.tc.object.bytecode.ManagerImpl$2.execute(ManagerImpl.java:263)
at com.tc.lang.StartupHelper.startUp(StartupHelper.java:39)
at com.tc.object.bytecode.ManagerImpl.startClient(ManagerImpl.java:281)
at com.tc.object.bytecode.ManagerImpl.init(ManagerImpl.java:202)
at com.tc.object.bytecode.ManagerImpl.init(ManagerImpl.java:190)
at com.tc.object.bytecode.hook.impl.DSOContextImpl.createStandaloneContext(DSOContextImpl.java:179)
at org.terracotta.express.StandaloneL1Boot.call(StandaloneL1Boot.java:190)
at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:309)
at org.terracotta.express.ClientFactoryImpl.newClient(ClientFactoryImpl.java:232)
at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:225)
at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:212)
at org.terracotta.express.ClientFactoryImpl.getOrCreateClient(ClientFactoryImpl.java:190)
at org.terracotta.express.ClientFactory.getOrCreateClient(ClientFactory.java:28)
at net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory.<init>(StandaloneTerracottaClusteredInstanceFactory.java:37)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
The weirdest part of this error is that memory is not used. I have set the Xmx at 3500m but the memory usage never climbs above 2.3gb.
I've been searching for a possible cause and found this : Low Java single process thread limit in Red Hat Linux
So i changed my ulimit -n and ulimit -u to 80000 but still nothing changes.
I then edited my /etc/security/limits.conf
* soft nproc 81920
* hard nproc 81920
* soft nofile 81920
* hard nofile 81920
I also edited /etc/sysctl.conf
fs.file-max = 100000
then i rebooted.
I also added -Xms=2g to my setenv.sh
I increased -Xss to -Xss5000k
I run echo 200000 > /proc/sys/kernel/threads-max
But the same error still comes up. Any ideas ?
free -m at begining of booting
total used free shared buffers cached
Mem: 3963 1027 2935 0 25 748
-/+ buffers/cache: 253 3709
Swap: 2047 18 2029
free -m just before crash
total used free shared buffers cached
Mem: 3963 2897 1065 0 31 823
-/+ buffers/cache: 2043 1920
Swap: 2047 18 2029
Update : i also tried with a machine with 8gb ram and the same result appears.