Tuning big java heap and linux huge pages

I recently had to run a jvm with large heap size, without getting into technical details we need the jvm to carry about 25G of cached memory that we load on stratup, so we want to start the jvm with 30G of minimum and maximum memory.  we have a 64G 16DC machine running centos 5.5. the machine is running both our application on tomcat and a postgresql database.

what ever we did we couldn't start the jvm with more then 25G with -Xms25g -Xmx25g, apparently the jvm needs contiguous memory blocks, and even with 64G physical memory the OS can't find 30G of contiguous memory,although free -m shows that there are more then 45G available. we even tried starting the machine without postgresql on startup and then try to allocate the jvm heap but it didn't help, 25G is the maximum we could do. and we tried it with sum jvm, ibm and jrockit, they all behave the same way.

one solution suggested in this post was to increase the swap area, it did the job and after adding a large swap file we could allocate 30G and even 40G heap size. but this will be swapable memory which could hurt performance really bad, we wanted to have a non swapable memory.

So we decided to drop the swap file and use a linux kernel feature called 'Huge Pages', available since 2.6.18. (find more about it here and here and here).

Huge pages is about allocating larger memory pages then the default 4k, a page size is hardware+arch dependent and in our machine its 2M. You usually see Huge Pages used by heavy weight servers like oracle, mysql and WebSphere.

But whats more relevant to us is that you can ask the linux kernel to reserve a certain amount of huge pages to be used by applications that explicitly ask the OS to use huge pages, this memory will not be available to other applications. and the best thing is that huge pages are not swapable and always reside in physical memory. this sounds good especially that our application needs to keep a large amount of memory for as long as it lives.

I'm not going to explain how to setup huge page support, you can find good howto here , here and here and this is a calculator.

so because we wanted to be able to allocate 30G for the jvm we configured the kernel to reserve 17920 huge pages , our page size is 2M so we have 35G of huge pages reserved.

then add a jvm argument -XX:+UseLargePages, and we can now start the jvm with 30G or more of heap size with no problem.

after starting the jvm this is our huge page info:

[root@localhost ~]# cat /proc/meminfo |grep Huge
HugePages_Total: 17920
HugePages_Free:   9156
HugePages_Rsvd:   7365
Hugepagesize:     2048 kB
 

The problem is that now there is no memory available to start any application, even 'java -version' will fail to allocate memory unless you start it like:  java -Xms12m -Xmx12m -XX:+UseLargePages -XX:LargePageSizeInBytes=2m -version

Now we are going on tuning this jvm for best performance, I will be happy to hear your ideas and I will continue editing this post as we improve our large heap jvm performance.