Tomcat Performance

Posted Thu, 06/08/2009 - 15:41 by zvika
, , ,

Greetings,

 

I'm currently consulting at a client site where Tomcat is used along with Apache via mod_jk, and from time to time simple requests (=>ping.jsp, which just prints out "OK") take as long as 10-15 secs.

 

Has anyone ever tried to analyze the thread/request ratio automatically (this is a production site and we need constant monitoring)?

 

Is mod_proxy considered a better alternative for mod_jk for long http connections?

 

cheers,

z

 

‹ How to compare JAVA Web Frameworks user management $ profiling ›

Comments

by lior.kanfi - 06/08/2009 - 16:03

Can you add the http listener configuration, how many clients are calling ping.jsp at time, is it correlate with the listener configuration

by zvika - 06/08/2009 - 16:57

here's the configuration, based on the defaults (150 or 200 threads + 15 extra):


<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />


clients are not only invoking ping.jsp, they are also calling real web services - I mentioned ping.jsp just to show that the problem is not applicative.

by lior.kanfi - 06/08/2009 - 17:03

Can you take a JVM thread dump correlate with the performance decrease?

by zvika - 06/08/2009 - 18:10

I am not there, at the scene of crime, long enough.
It is a production site and there is only so much I can do... I'll try writing a daemon to do the job.

by shlomo - 06/08/2009 - 19:56

use jstat , no need to re-start the JVM.

by shlomo - 06/08/2009 - 17:16

There is a tight relationship between the tomcat mod_jk and apache settings and if you are using mod prefork.

See:
http://www.tikalk.com/java/performance-tunning-case-study

Is it an SMP machine?
Can you attach the apache error and access logs?

by zvika - 06/08/2009 - 18:11

ok, reading... in the meantime, what's SMP machine?
I'll try get the logs, same as in previous reply - by writing a daemon

by shlomo - 06/08/2009 - 19:55

http://computingtech.blogspot.com/2008/08/enabling-multiple-cpus-smp-in-...

by zvika - 06/08/2009 - 23:05

all machines here are windows... including an 8CPU production server - I kid you not.

by udid - 07/08/2009 - 17:36

Zvi,

I would suspect this to be caused by java gc pauses.

To verify you can start tomcat with -verbose:gc or to use JConsole.

If you see pauses of more than, say, 200 ms - it's a problem.
In some cases, mostly when the application is leaking memory, pauses time can easily grow to 10 sec.

On machines with lots of CPUs and Memory, you need to configure the JVM to for a parallel short pause algorithm such as ConcMarkSweep.

For example:
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:+DisableExplicitGC

Udi

by zvika - 08/08/2009 - 00:23

Thanks!

I'll try it next week as soon as I get the correlation results from a little test (I'll post it in one of the next posts, in case someone ever needs something like it) I am running this weekend, trying to establish a connection between a large number of incoming requests and the unexplained delay.



The GC explanation sounds perfectly reasonable and defintely worth checking, it's just that there is a performance expert on board hired to fine tune the JVM, and he already had his say at the freaky GC flags. In case the problem isn't thread contention, I'll set off to testing the GC.