Threads stuck, gradually, slowly...Have you any suggestions?

Posted Wed, 22/06/2011 - 15:26 by peterk
, , ,

Hi!

 

Java 5 threading here.

 

I have an Executer with 10 threads,

 

threads are busy in e-mail sending tasks execution.

 

Gradually (1 thread in 5 hours) threads stuck in TIMED_WAITING state, from which they never revoked,

 

example of thread state is here, taken by jconsole:

 

Name: pool-4-thread-1
State: TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3125fe1
Total blocked: 0  Total waited: 7,609

Stack trace: 
 sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
com.idi.mailer.executor.impl.ResultsQueue.pull(ResultsQueue.java:66)
com.idi.mailer.executor.impl.SerialMailExecutor$FinalizerTask.run(SerialMailExecutor.java:466)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)

 

Gradually all 10 threads get into that state.

 

They are not using common resources. They are not waiting each other. No locks possible.

 

What I do now to fix the matter, I recognize the failure at the end and restart the entire executer.

 

It looks like a VM bug, because I looked into similar posts and all they end up in some VM problem...

 

Have you any idea?

 

Thanks.

 

Peter.

 

 

 

‹ Question When to use Tomcat CATALINA_OPTS instead of JAVA_OPTS ›

Comments

by shlomo - 22/06/2011 - 15:52

Hi,

1-Which JDK are you using?  sun OpenJDK? which version? 64 bit?

2-Are you sure you don't have stack overflow error somewhere? how much memory did you allocate to each thread?

3-How are you synchronizing between the threads?

4-Which executor are you using? do you have a custom rejected execution handler? what is the capacity of the queue? is it unlimited?

5-If there are only 10 threads in the pool, why not keep re-creating the pool as you did? there should not be significant overhead in doing that

6-Please attach the full thread dump

 

Also have you read this: http://www.cubeia.com/index.php/blog/archives/19 ?

 

by andrew - 22/06/2011 - 15:52

I see that in 3rd party class SerialMailExecutor tries to read from queue and gets blocked. Can you understand from the code, what is it expecting to read from queue.

 

And seeing finalizers is also rare (and sometimes means the code is not written in good way). Why are they used? What is this mail library?

by peterk - 22/06/2011 - 18:25

Now it all above is not important,

what I really found, that threads block in E-mail sending process:

 

That what I need to resolve.

Threads just stuck while sending e-mail to SMTP server with regular java mail API.

Name: pool-1-thread-10
State: RUNNABLE
Total blocked: 415  Total waited: 324

Stack trace: 
 java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:110)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
   - locked java.io.BufferedInputStream@19c9e919
com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:88)
com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:1589)
com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1369)
com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:412)
javax.mail.Service.connect(Service.java:310)
   - locked com.sun.mail.smtp.SMTPTransport@742e811d
javax.mail.Service.connect(Service.java:169)
javax.mail.Service.connect(Service.java:118)
javax.mail.Transport.send0(Transport.java:188)
javax.mail.Transport.send(Transport.java:118)
com.idi.mailer.client.JMailClient.sendMail(JMailClient.java:189)
com.idi.mailer.executor.impl.SerialMailTask.call(SerialMailTask.java:49)
com.idi.mailer.executor.impl.SerialMailTask.call(SerialMailTask.java:15)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)

 

by andrew - 22/06/2011 - 19:35

the second stack trace shows a normal blocking, SMTP tries to read the response, and waits for server to send it over socket. Do I miss your point?

by peterk - 22/06/2011 - 21:47

this normal blocking never ends.

 

gradually.

 

from 10 threads started at the morning, now 6 are in blocked state.

 

they are in normal WAITING mode , BUT smtp server somehow has forgotten to return a response.

 

I know , that our SMTP is problematic... But how to wrap, that request in therad, how to suspend it if timeout expired...

 

 

by andrew - 23/06/2011 - 01:02

well, you are stuck in javax.mail.Transport.send, and that API does not have a maximum timeout parameter. Probably, you can track the time before calling that API, and terminate the thread if it is stuck on that call for too long.

by peterk - 23/06/2011 - 09:35

create parallel thread, that tracks this call is one solution,

but I'm afraid to create additional therads.

 

I found, that people close Socket, when that happens. Closed socked tells everybody: "continue"...

I need find out how to reach the Socket, which have been used to create that smtp connection and then close it.

Anyway, any solution is quite complex.

so I leave it as is: restart upon all threads are stuck.

They do not pay me for more, it's quite ok.