javax.ejb.ConcurrentAccessTimeoutException: No instances available in Stateless Session Bean pool. Waited 30 SECONDS

[Kalyan]

Hello,
I am getting following error in my application.

avax.ejb.ConcurrentAccessTimeoutException: No instances available in
Stateless Session Bean pool.  Waited 30 SECONDS
        at
org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:226)
        at
org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)
        at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)
        at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)
        at
org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)
        at
org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:349)



My application is heavily used. Looks like not enough instances of beans.

I saw this post to increase the pool size

https://tomee.apache.org/admin/configuration/containers.html


and changed my configuration as
java.naming.factory.initial=org.apache.openejb.core.LocalInitialContextFactory
openejb.deployments.classpath.ear=false

myApp = new://Container?type=STATELESS
myApp.AccessTimeout = 30 seconds
*myApp.MaxSize = 40*
myApp.MinSize = 0
myApp.StrictPooling = true
myApp.MaxAge = 0 hours
myApp.ReplaceAged = true
myApp.ReplaceFlushed = false
myApp.MaxAgeOffset = -1
myApp.IdleTimeout = 0 minutes
myApp.GarbageCollection = false
myApp.SweepInterval = 5 minutes
myApp.CallbackThreads = 5
myApp.CloseTimeout = 5 minutes
myApp.UseOneSchedulerThreadByBean = false
myApp.EvictionThreads = 1


On the server start up I see

INFO - Configuring Service(id=myApp, type=Container, provider-id=Default
Stateless Container)

DEBUG - Containers        : 1
DEBUG - Type        Container ID
DEBUG -    STATELESS   myApp
DEBUG - Deployments       : 5
DEBUG - Type        Deployment ID

/Question i have is /
How would i know if the bean size is increased ?  I don't see any log
message for it.

/Secondly I also see /

DEBUG - Using default 'openejb.tempclassloader.skip=none'  Possible values
are: none, annotations, enums, all or NONE or ALL
DEBUG - Using default 'AsynchronousPool.Size=5'
DEBUG - Using default 'AsynchronousPool.CorePoolSize=5'
DEBUG - Using default 'AsynchronousPool.MaximumPoolSize=5'
DEBUG - Using default 'AsynchronousPool.QueueSize=5'
DEBUG - Using default 'AsynchronousPool.KeepAliveTime=60 SECONDS'
DEBUG - Using default 'AsynchronousPool.AllowCoreThreadTimeOut=true'
DEBUG - Using default 'AsynchronousPool.QueueType=linked'.  Possible values
are: array, linked, priority, synchronous
DEBUG - Using default 'AsynchronousPool.OfferTimeout=30 SECONDS'
DEBUG - Using default 'AsynchronousPool.ShutdownWaitDuration=1 MINUTES'


What's the AsynchronousPool ???


Please help me to sort this issue?
I'm running into this issues in production.



thanks
Kalyan







--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[jgallimore]

Hi

What version of TomEE are you using?

Jon

On Thu, Oct 3, 2019 at 9:16 PM Kalyan <[hidden email]> wrote:

> Hello,
> I am getting following error in my application.
>
> avax.ejb.ConcurrentAccessTimeoutException: No instances available in
> Stateless Session Bean pool.  Waited 30 SECONDS
>         at
>
> org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:226)
>         at
>
> org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)
>         at
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)
>         at
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)
>         at
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)
>         at
>
> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:349)
>
>
>
> My application is heavily used. Looks like not enough instances of beans.
>
> I saw this post to increase the pool size
>
> https://tomee.apache.org/admin/configuration/containers.html
>
>
> and changed my configuration as
>
> java.naming.factory.initial=org.apache.openejb.core.LocalInitialContextFactory
> openejb.deployments.classpath.ear=false
>
> myApp = new://Container?type=STATELESS
> myApp.AccessTimeout = 30 seconds
> *myApp.MaxSize = 40*
> myApp.MinSize = 0
> myApp.StrictPooling = true
> myApp.MaxAge = 0 hours
> myApp.ReplaceAged = true
> myApp.ReplaceFlushed = false
> myApp.MaxAgeOffset = -1
> myApp.IdleTimeout = 0 minutes
> myApp.GarbageCollection = false
> myApp.SweepInterval = 5 minutes
> myApp.CallbackThreads = 5
> myApp.CloseTimeout = 5 minutes
> myApp.UseOneSchedulerThreadByBean = false
> myApp.EvictionThreads = 1
>
>
> On the server start up I see
>
> INFO - Configuring Service(id=myApp, type=Container, provider-id=Default
> Stateless Container)
>
> DEBUG - Containers        : 1
> DEBUG - Type        Container ID
> DEBUG -    STATELESS   myApp
> DEBUG - Deployments       : 5
> DEBUG - Type        Deployment ID
>
> /Question i have is /
> How would i know if the bean size is increased ?  I don't see any log
> message for it.
>
> /Secondly I also see /
>
> DEBUG - Using default 'openejb.tempclassloader.skip=none'  Possible values
> are: none, annotations, enums, all or NONE or ALL
> DEBUG - Using default 'AsynchronousPool.Size=5'
> DEBUG - Using default 'AsynchronousPool.CorePoolSize=5'
> DEBUG - Using default 'AsynchronousPool.MaximumPoolSize=5'
> DEBUG - Using default 'AsynchronousPool.QueueSize=5'
> DEBUG - Using default 'AsynchronousPool.KeepAliveTime=60 SECONDS'
> DEBUG - Using default 'AsynchronousPool.AllowCoreThreadTimeOut=true'
> DEBUG - Using default 'AsynchronousPool.QueueType=linked'.  Possible values
> are: array, linked, priority, synchronous
> DEBUG - Using default 'AsynchronousPool.OfferTimeout=30 SECONDS'
> DEBUG - Using default 'AsynchronousPool.ShutdownWaitDuration=1 MINUTES'
>
>
> What's the AsynchronousPool ???
>
>
> Please help me to sort this issue?
> I'm running into this issues in production.
>
>
>
> thanks
> Kalyan
>
>
>
>
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>

[jgallimore]

I dropped your settings into TomEE 8 here, and checked the pool for my EJB
via JMX (openejb.management -> Pool -> openejb -> <empty> -> (file path) ->
(bean) -> Attributes. It showed with a max size of 40. 1 instance pooled,
but I'd expect that to grow to the max size.

I'd suggest getting a jstack and looking for threads
with org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)
in the trace, and see if you have something locked or running slowly. I'll
happily look at a jstack if you wish, but it may have some information you
don't want to post publicly (check the content of the jstack before
posting).

Jon

On Thu, Oct 3, 2019 at 9:50 PM Jonathan Gallimore <
[hidden email]> wrote:

> Hi
>
> What version of TomEE are you using?
>
> Jon
>
> On Thu, Oct 3, 2019 at 9:16 PM Kalyan <[hidden email]> wrote:
>
>> Hello,
>> I am getting following error in my application.
>>
>> avax.ejb.ConcurrentAccessTimeoutException: No instances available in
>> Stateless Session Bean pool.  Waited 30 SECONDS
>>         at
>>
>> org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:226)
>>         at
>>
>> org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)
>>         at
>>
>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)
>>         at
>>
>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)
>>         at
>>
>> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)
>>         at
>>
>> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:349)
>>
>>
>>
>> My application is heavily used. Looks like not enough instances of beans.
>>
>> I saw this post to increase the pool size
>>
>> https://tomee.apache.org/admin/configuration/containers.html
>>
>>
>> and changed my configuration as
>>
>> java.naming.factory.initial=org.apache.openejb.core.LocalInitialContextFactory
>> openejb.deployments.classpath.ear=false
>>
>> myApp = new://Container?type=STATELESS
>> myApp.AccessTimeout = 30 seconds
>> *myApp.MaxSize = 40*
>> myApp.MinSize = 0
>> myApp.StrictPooling = true
>> myApp.MaxAge = 0 hours
>> myApp.ReplaceAged = true
>> myApp.ReplaceFlushed = false
>> myApp.MaxAgeOffset = -1
>> myApp.IdleTimeout = 0 minutes
>> myApp.GarbageCollection = false
>> myApp.SweepInterval = 5 minutes
>> myApp.CallbackThreads = 5
>> myApp.CloseTimeout = 5 minutes
>> myApp.UseOneSchedulerThreadByBean = false
>> myApp.EvictionThreads = 1
>>
>>
>> On the server start up I see
>>
>> INFO - Configuring Service(id=myApp, type=Container, provider-id=Default
>> Stateless Container)
>>
>> DEBUG - Containers        : 1
>> DEBUG - Type        Container ID
>> DEBUG -    STATELESS   myApp
>> DEBUG - Deployments       : 5
>> DEBUG - Type        Deployment ID
>>
>> /Question i have is /
>> How would i know if the bean size is increased ?  I don't see any log
>> message for it.
>>
>> /Secondly I also see /
>>
>> DEBUG - Using default 'openejb.tempclassloader.skip=none'  Possible values
>> are: none, annotations, enums, all or NONE or ALL
>> DEBUG - Using default 'AsynchronousPool.Size=5'
>> DEBUG - Using default 'AsynchronousPool.CorePoolSize=5'
>> DEBUG - Using default 'AsynchronousPool.MaximumPoolSize=5'
>> DEBUG - Using default 'AsynchronousPool.QueueSize=5'
>> DEBUG - Using default 'AsynchronousPool.KeepAliveTime=60 SECONDS'
>> DEBUG - Using default 'AsynchronousPool.AllowCoreThreadTimeOut=true'
>> DEBUG - Using default 'AsynchronousPool.QueueType=linked'.  Possible
>> values
>> are: array, linked, priority, synchronous
>> DEBUG - Using default 'AsynchronousPool.OfferTimeout=30 SECONDS'
>> DEBUG - Using default 'AsynchronousPool.ShutdownWaitDuration=1 MINUTES'
>>
>>
>> What's the AsynchronousPool ???
>>
>>
>> Please help me to sort this issue?
>> I'm running into this issues in production.
>>
>>
>>
>> thanks
>> Kalyan
>>
>>
>>
>>
>>
>>
>>
>> --
>> Sent from:
>> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>>
>

[Kalyan]

Its 8.0.0-M3





--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[Kalyan]

our servers are basically using netty. OpenEjb is embedded in it. I can try
to get jstack and look into it.
Is there anything which is logged while server start up to see how much is
the max size? I don't have JMX

thanks
Kalyan



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[jgallimore]

I've had a quick run through of the code and I can't see a log statement
for that value. We can certainly add it to master, but that might not help
you on 8.0.0-M3. If you're feeling handy with a debugger, put a breakpoint
here:
https://github.com/apache/tomee/blob/tomee-8.0.0-M3/container/openejb-core/src/main/java/org/apache/openejb/core/stateless/StatelessContainerFactory.java#L138
and
inspect the pool field.

Do try and get a jstack. It should just be a matter of running jstack
<pid>. If you're not familiar with the output, you basically get a stack
trace for every thread in the JVM. Check how many threads are trying to
execute your EJB.

When you run into this issue, its always very tempting to just bump up the
pool size. I get asked this question a lot. My personal advice is that
increasing the pool size should be the absolute last thing you do. If you
have 10 threads all executing a business method on your EJB, and they are
all taking longer than 30 seconds, you have an issue and you need to find
the root cause, understand it, and fix it. Bumping it up to 40 will
probably just lead to 40 threads stuck executing your EJB business method
and getting stuck - the problem just ends up getting worse. If you know the
performance measurements, have everything tuned, and need more throughput,
that's the point to look at the pool size.

Also, take a look at the Singleton bean type as opposed to stateless. As
long as you really do not have state in your bean, a singleton
with @Lock(LockType.READ) should work well, and isn't pooled so there's no
limit on the number of threads that can call business methods all at once.
https://github.com/apache/tomee/tree/master/examples/simple-singleton

> our servers are basically using netty. OpenEjb is embedded in it.

This sounds like a very cool setup. If you wanted to contribute that to
community, I'd encourage you to do so. I would certainly be interested in
seeing it, and I'm sure others would be too.

Jon

On Thu, Oct 3, 2019 at 11:21 PM Kalyan <[hidden email]> wrote:

> our servers are basically using netty. OpenEjb is embedded in it. I can try
> to get jstack and look into it.
> Is there anything which is logged while server start up to see how much is
> the max size? I don't have JMX
>
> thanks
> Kalyan
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>

[Kalyan]

Thanks Jon,
Really appreciate for taking a look at code.
Certainly i will look into the jstack. We had similar issue when this
application was running in jboss too. So first thing we want to do is
increase the pool size. Absolutely i will try to get into the root cause of
this issue.
We don't see this in our lower environments , may be we don't have too much
traffic like production. I will try to do some load test and see if i can
reproduce this issue on dev environment.

Its very simple setup what i did on netty. Once i am out of this issue. I
will send you those details.

Thanks a lot

Regards
Kalyan



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[Kalyan]

Hello Jon,
Even after increasing the bean pool size. I am getting the exception.

Below is the jstack.
But I have enough connection configured on the database as well.

Could you please see what's going with this.


"JungoThreadPool-1-115-T348" #225 prio=5 os_prio=0 tid=0x00007fa5fc123000
nid=0x3204 waiting on condition [0x00007fa56d86d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006819f6760> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
        at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
        at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
        at
org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:127)
        at
org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:60)
        at
org.apache.openejb.resource.jdbc.dbcp.BasicManagedDataSource$1$1.<init>(BasicManagedDataSource.java:89)
        at
org.apache.openejb.resource.jdbc.dbcp.BasicManagedDataSource$1.getConnection(BasicManagedDataSource.java:89)
        at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
        at JDBCQueryElement.getConnection(JDBCQueryElement.java:3173)

thanks
Kalyan



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[jgallimore]

Are you able to share the whole thing? That's just one thread. If you have
a whole bunch of threads all hitting the database, that sounds like a
potential issue at the database end. If there are no other threads
accessing the database, then its possible there is a leak somewhere.

Jon

On Mon, Oct 7, 2019 at 11:23 PM Kalyan <[hidden email]> wrote:

> Hello Jon,
> Even after increasing the bean pool size. I am getting the exception.
>
> Below is the jstack.
> But I have enough connection configured on the database as well.
>
> Could you please see what's going with this.
>
>
> "JungoThreadPool-1-115-T348" #225 prio=5 os_prio=0 tid=0x00007fa5fc123000
> nid=0x3204 waiting on condition [0x00007fa56d86d000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000006819f6760> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
>
> org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
>         at
>
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
>         at
>
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>         at
>
> org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:127)
>         at
>
> org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:60)
>         at
>
> org.apache.openejb.resource.jdbc.dbcp.BasicManagedDataSource$1$1.<init>(BasicManagedDataSource.java:89)
>         at
>
> org.apache.openejb.resource.jdbc.dbcp.BasicManagedDataSource$1.getConnection(BasicManagedDataSource.java:89)
>         at
>
> org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
>         at JDBCQueryElement.getConnection(DLJDBCQueryElement.java:3173)
>
> thanks
> Kalyan
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>

[Kalyan]

Rest of the thread is pretty much as below

"JungoThreadPool-1-110" #220 prio=5 os_prio=0 tid=0x00007fa5fc119000
nid=0x31ff waiting on condition [0x00007fa56dd75000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000068339f010> (a
java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
        at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"JungoThreadPool-1-109-T888" #219 prio=5 os_prio=0 tid=0x00007fa5fc117000
nid=0x31fe waiting on condition [0x00007fa56de74000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000068022de00> (a
java.util.concurrent.Semaphore$NonfairSync)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:409)
        at org.apache.openejb.util.Pool.pop(Pool.java:247)
        at org.apache.openejb.util.Pool.pop(Pool.java:228)
        at
org.apache.openejb.core.stateless.StatelessInstanceManager$Data.poolPop(StatelessInstanceManager.java:512)
        at
org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:217)
        at
org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)
        at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)
        at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)
        at
org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)
        at
org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:349)
        at com.sun.proxy.$Proxy74.forwardToServiceBean(Unknown Source)



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[jgallimore]

The problem with getting single threads here and there is its difficult to
pin down the root cause. With the whole jstack file I can probably give you
some pointers pretty quickly (or I'll just say if not). A very long time
back (in the 1.7.3 era) there was an issue where you can lock a database
pool, but that was actually reproducible under small load with small pools.
Most of these types of issues (that I see, at least), particularly when you
see it in production but not other environments, are bottlenecks
communicating to the database or another external system. One thing to
remember is that your stateless pool will have a max of 40 instances *per
EJB* (not in total). If your database pool is also 40, you'll potentially
see issues there.

The first thread you sent is seeing contention issues trying to get a
database connection (the pool is exhausted, either leaking connections, or
the connections are in use in other threads). The more recent one is
showing contention on the stateless EJB pool (it just hasn't timedout
waiting yet). We can't learn to much from these, other than the fact that
they are waiting. What the other threads are doing will be of interest. I
need the full picture otherwise I'm guessing, which won't really help
either of us.

Jon

On Mon, Oct 7, 2019 at 11:39 PM Kalyan <[hidden email]> wrote:

> Rest of the thread is pretty much as below
>
> "JungoThreadPool-1-110" #220 prio=5 os_prio=0 tid=0x00007fa5fc119000
> nid=0x31ff waiting on condition [0x00007fa56dd75000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x000000068339f010> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
>         at
>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "JungoThreadPool-1-109-T888" #219 prio=5 os_prio=0 tid=0x00007fa5fc117000
> nid=0x31fe waiting on condition [0x00007fa56de74000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x000000068022de00> (a
> java.util.concurrent.Semaphore$NonfairSync)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
>         at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:409)
>         at org.apache.openejb.util.Pool.pop(Pool.java:247)
>         at org.apache.openejb.util.Pool.pop(Pool.java:228)
>         at
>
> org.apache.openejb.core.stateless.StatelessInstanceManager$Data.poolPop(StatelessInstanceManager.java:512)
>         at
>
> org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:217)
>         at
>
> org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)
>         at
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)
>         at
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)
>         at
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)
>         at
>
> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:349)
>         at com.sun.proxy.$Proxy74.forwardToServiceBean(Unknown Source)
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>

[Kalyan]

Hi Jon,
I sent you whole jstack to your email. Please take a look and let me know if
anything can be done.

thanks
Kalyan



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[jgallimore]

Got it. Going to be away from my computer for a couple of hours, but I'll
take a look later today.

Jon

On Tue, Oct 8, 2019 at 4:50 PM Kalyan <[hidden email]> wrote:

> Hi Jon,
> I sent you whole jstack to your email. Please take a look and let me know
> if
> anything can be done.
>
> thanks
> Kalyan
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>

[jgallimore]

At first glance, I'd say it looks like your database connection pool is
exhausted but there's nothing using the database, so either we have a bug
or connections are leaking. Next step will probably be some logging around
that.

Jon

On Tue, 8 Oct 2019, 17:10 Jonathan Gallimore, <[hidden email]>
wrote:

> Got it. Going to be away from my computer for a couple of hours, but I'll
> take a look later today.
>
> Jon
>
> On Tue, Oct 8, 2019 at 4:50 PM Kalyan <[hidden email]> wrote:
>
>> Hi Jon,
>> I sent you whole jstack to your email. Please take a look and let me know
>> if
>> anything can be done.
>>
>> thanks
>> Kalyan
>>
>>
>>
>> --
>> Sent from:
>> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>>
>

[Kalyan]

Thanks Jon for looking into it.
What do you suggest me to do ?



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[jgallimore]

We're probably going to make some small changes quite rapidly to 1) see if
we can reproduce your issue and 2) extract some more information in logs.
If you're willing and able to pick up some snapshots, or patch your
8.0.0-M3 with some PRs and report back, that helps.

Jon

On Tue, Oct 8, 2019 at 6:49 PM Kalyan <[hidden email]> wrote:

> Thanks Jon for looking into it.
> What do you suggest me to do ?
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>

[Kalyan]

Yes i am able to reproduce this issue.





--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[jgallimore]

I can reproduce your stack trace by deliberately not calling
connection.close().

"main" #1 prio=5 os_prio=31 cpu=1929.67ms elapsed=28.55s
tid=0x00007f8d1c800800 nid=0x1703 waiting on condition  [0x0000700000dd1000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
- parking to wait for  <0x000000070ea81590> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1
/LockSupport.java:194)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1
/AbstractQueuedSynchronizer.java:2081)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:127)
at
org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:60)
at
org.apache.openejb.resource.jdbc.dbcp.BasicManagedDataSource$1$1.<init>(BasicManagedDataSource.java:90)
at
org.apache.openejb.resource.jdbc.dbcp.BasicManagedDataSource$1.getConnection(BasicManagedDataSource.java:90)
at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
at
org.apache.openejb.resource.jdbc.ManagedDataSourceExhaustionTest$DBWorker.check(ManagedDataSourceExhaustionTest.java:74)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.1/Native
Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.1
/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.1
/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.1/Method.java:566)
at
org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205)
at
org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186)
at
org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191)
at
org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.1/Native
Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.1
/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.1
/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.1/Method.java:566)
at
org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205)
at
org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186)
at
org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85)
at
org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272)
at
org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)
at
org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:349)
at
org.apache.openejb.resource.jdbc.ManagedDataSourceExhaustionTest$DBWorker$$LocalBeanProxy.check(org/apache/openejb/resource/jdbc/ManagedDataSourceExhaustionTest$DBWorker.java)
at
org.apache.openejb.resource.jdbc.ManagedDataSourceExhaustionTest.test(ManagedDataSourceExhaustionTest.java:84)

Is the EJB making the call out to the database running in a transaction?

Jon

On Tue, Oct 8, 2019 at 7:36 PM Kalyan <[hidden email]> wrote:

> Yes i am able to reproduce this issue.
>
>
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>

[Kalyan]

yes that's correct



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html

[jgallimore]

Played around with this a little and opened this PR:
https://github.com/apache/tomee/pull/583. Take a look and see what you
think. I still need to do a full build and make sure I haven't
broken anything here. The theory is that is you turn the logging
on OpenEJB.resource.jdbc up to FINEST, you'll get a bunch of debug output
showing where connections are obtained from the datasource, and where they
are returned with connection.close(). If a transaction completes and the
connection has not been closed, it'll log that out, along with where the
connection was borrowed.

I wouldn't put this in production as it needs review and merging etc, as
well as being very verbose, but if you have a local copy of your
application that you can test with, it might be worth stepping through and
using it to ensure your connections are getting "closed", and returned to
the pool. From what you've indicated so far, it looks like you're leaking
connections (i.e., not calling close() on them).

Jon

On Tue, Oct 8, 2019 at 8:58 PM Kalyan <[hidden email]> wrote:

> yes that's correct
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Users-f979441.html
>