JTA JMS Spec question, connection leakage

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

JTA JMS Spec question, connection leakage

exabrial12
Hello Apache friends :) I have a question about the JTA and JMS/RA specs:

If you borrow something from a RA, like a JMS Connection, and you're in XA
Transaction, is it necessary to call connection.close()? It would seem JTA
should be smart enough to know the connection is enrolled for 2 phase
commit and should be smart enough to close it, but I'm not sure if that's
part of the spec.

In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
connectionFactory.createConnection(), and your code fails to call close()
before the transaction completion, the connection leaks. (And
unfortunately, calling close() after the transaction completes doesn't
mitigate the problem). It took awhile for us to track this down.

This becomes a huge problem if you're calling external services in your
transaction. Let's say you have a reasonable transaction timeout of 30s
set. You call three services, and they end up taking 15s a piece. Even if
you're doing the right thing and you have connection.close() in a finally
block, because your transaction isn't active when you call close, it leaks
and it just gets "stuck" as an active connection, which eventually you hit
the pool limit and your app freezes.

On a separate note, we noticed if you open a connection outside of the
scope of a transaction, then start a transaction, then create a session
with session_transacted option, the session does not participate in the JTA
(which seems out of spec). One most open the connection inside the
transaction, AND open the session in the transaction, and close the
connection in the transaction for everything to work.

I'll get a reproducing project created, but I was curious if anyone knew
offhand what the spec says.

cheers, and thanks,
-[the other] Jonathan

--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak

Here's a project that reproduces the bug. This project intentionally
exceeds the transaction timeout (of 1s). Each invocation, the connection is
not returned to the pool and eventually you run out, causing your
application to freeze.



On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <[hidden email]>
wrote:

> Hello Apache friends :) I have a question about the JTA and JMS/RA specs:
>
> If you borrow something from a RA, like a JMS Connection, and you're in XA
> Transaction, is it necessary to call connection.close()? It would seem JTA
> should be smart enough to know the connection is enrolled for 2 phase
> commit and should be smart enough to close it, but I'm not sure if that's
> part of the spec.
>
> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
> connectionFactory.createConnection(), and your code fails to call close()
> before the transaction completion, the connection leaks. (And
> unfortunately, calling close() after the transaction completes doesn't
> mitigate the problem). It took awhile for us to track this down.
>
> This becomes a huge problem if you're calling external services in your
> transaction. Let's say you have a reasonable transaction timeout of 30s
> set. You call three services, and they end up taking 15s a piece. Even if
> you're doing the right thing and you have connection.close() in a finally
> block, because your transaction isn't active when you call close, it leaks
> and it just gets "stuck" as an active connection, which eventually you hit
> the pool limit and your app freezes.
>
> On a separate note, we noticed if you open a connection outside of the
> scope of a transaction, then start a transaction, then create a session
> with session_transacted option, the session does not participate in the JTA
> (which seems out of spec). One most open the connection inside the
> transaction, AND open the session in the transaction, and close the
> connection in the transaction for everything to work.
>
> I'll get a reproducing project created, but I was curious if anyone knew
> offhand what the spec says.
>
> cheers, and thanks,
> -[the other] Jonathan
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
I've narrowed down the problem to AutoConnectionTracker. It's not
completing, which isn't allowing the connections to be returned to the pool.
https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174

getResource() is throwing an IllegalStateException. The JavaDoc (
https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-)
states it should throw an ISE if a current transaction is not Active. The
transaction is in the state ROLLED_BACK when AutoConnectionTracker tries to
call getResource().

I think the Geronimo implementation (
https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203)
maybe be a little too strict. The JTA Spec pdf doesn't offer exact hints of
which statuses (
https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html) should
be have getResource _not_ throw an ISE unfortunately. I was thinking of
changing Geronimo's implementation to check for anything
but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.

The other way is to cast Transaction to the Geronimo implementation and use
Geronimo specific APIs to get call getResource(). Do you guys have any
preference which route I should take to fix?


On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]>
wrote:

> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>
> Here's a project that reproduces the bug. This project intentionally
> exceeds the transaction timeout (of 1s). Each invocation, the connection is
> not returned to the pool and eventually you run out, causing your
> application to freeze.
>
>
>
> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>> Hello Apache friends :) I have a question about the JTA and JMS/RA specs:
>>
>> If you borrow something from a RA, like a JMS Connection, and you're in
>> XA Transaction, is it necessary to call connection.close()? It would seem
>> JTA should be smart enough to know the connection is enrolled for 2 phase
>> commit and should be smart enough to close it, but I'm not sure if that's
>> part of the spec.
>>
>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
>> connectionFactory.createConnection(), and your code fails to call close()
>> before the transaction completion, the connection leaks. (And
>> unfortunately, calling close() after the transaction completes doesn't
>> mitigate the problem). It took awhile for us to track this down.
>>
>> This becomes a huge problem if you're calling external services in your
>> transaction. Let's say you have a reasonable transaction timeout of 30s
>> set. You call three services, and they end up taking 15s a piece. Even if
>> you're doing the right thing and you have connection.close() in a finally
>> block, because your transaction isn't active when you call close, it leaks
>> and it just gets "stuck" as an active connection, which eventually you hit
>> the pool limit and your app freezes.
>>
>> On a separate note, we noticed if you open a connection outside of the
>> scope of a transaction, then start a transaction, then create a session
>> with session_transacted option, the session does not participate in the JTA
>> (which seems out of spec). One most open the connection inside the
>> transaction, AND open the session in the transaction, and close the
>> connection in the transaction for everything to work.
>>
>> I'll get a reproducing project created, but I was curious if anyone knew
>> offhand what the spec says.
>>
>> cheers, and thanks,
>> -[the other] Jonathan
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d

With these patches the JMS2.0 API works pretty well now :) If anyone wants
to review, please go ahead, otherwise I'll commit them tomorrow.

cheers,
- [The other] Jonathan

On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <[hidden email]>
wrote:

> I've narrowed down the problem to AutoConnectionTracker. It's not
> completing, which isn't allowing the connections to be returned to the pool.
>
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>
> getResource() is throwing an IllegalStateException. The JavaDoc (
> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-)
> states it should throw an ISE if a current transaction is not Active. The
> transaction is in the state ROLLED_BACK when AutoConnectionTracker tries to
> call getResource().
>
> I think the Geronimo implementation (
> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203)
> maybe be a little too strict. The JTA Spec pdf doesn't offer exact hints of
> which statuses (
> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
> should be have getResource _not_ throw an ISE unfortunately. I was thinking
> of changing Geronimo's implementation to check for anything
> but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>
> The other way is to cast Transaction to the Geronimo implementation and
> use Geronimo specific APIs to get call getResource(). Do you guys have any
> preference which route I should take to fix?
>
>
> On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>
>> Here's a project that reproduces the bug. This project intentionally
>> exceeds the transaction timeout (of 1s). Each invocation, the connection is
>> not returned to the pool and eventually you run out, causing your
>> application to freeze.
>>
>>
>>
>> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <[hidden email]>
>> wrote:
>>
>>> Hello Apache friends :) I have a question about the JTA and JMS/RA specs:
>>>
>>> If you borrow something from a RA, like a JMS Connection, and you're in
>>> XA Transaction, is it necessary to call connection.close()? It would seem
>>> JTA should be smart enough to know the connection is enrolled for 2 phase
>>> commit and should be smart enough to close it, but I'm not sure if that's
>>> part of the spec.
>>>
>>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
>>> connectionFactory.createConnection(), and your code fails to call close()
>>> before the transaction completion, the connection leaks. (And
>>> unfortunately, calling close() after the transaction completes doesn't
>>> mitigate the problem). It took awhile for us to track this down.
>>>
>>> This becomes a huge problem if you're calling external services in your
>>> transaction. Let's say you have a reasonable transaction timeout of 30s
>>> set. You call three services, and they end up taking 15s a piece. Even if
>>> you're doing the right thing and you have connection.close() in a finally
>>> block, because your transaction isn't active when you call close, it leaks
>>> and it just gets "stuck" as an active connection, which eventually you hit
>>> the pool limit and your app freezes.
>>>
>>> On a separate note, we noticed if you open a connection outside of the
>>> scope of a transaction, then start a transaction, then create a session
>>> with session_transacted option, the session does not participate in the JTA
>>> (which seems out of spec). One most open the connection inside the
>>> transaction, AND open the session in the transaction, and close the
>>> connection in the transaction for everything to work.
>>>
>>> I'll get a reproducing project created, but I was curious if anyone knew
>>> offhand what the spec says.
>>>
>>> cheers, and thanks,
>>> -[the other] Jonathan
>>>
>>> --
>>> Jonathan | [hidden email]
>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> half full.
>>> Engineers, of course, understand the glass is twice as big as it needs
>>> to be.
>>>
>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

David Jencks-3
In reply to this post by exabrial12
I think the java doc for getResource might have been written thoughtlessly, and more appropriate behavior would be an ISE only for STATUS_NO_TRANSACTION; literally the geronimo implementation is too lax, as “marked rollback” is not status “active”.  Is there anyone who’s opinion we might ask?

I rather thought the “ignore session type” logic was supposed to be put into the RA, but I don’t recall if or how I dealt with this in Geronimo.

So I’d prefer these issues be dealt with elsewhere but don’t see much practical alternative to your implementation.

Nice to see someone working on XA:-)

thanks!
David Jencks

> On Aug 26, 2019, at 1:45 PM, Jonathan S. Fisher <[hidden email]> wrote:
>
> I've narrowed down the problem to AutoConnectionTracker. It's not
> completing, which isn't allowing the connections to be returned to the pool.
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>
> getResource() is throwing an IllegalStateException. The JavaDoc (
> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-)
> states it should throw an ISE if a current transaction is not Active. The
> transaction is in the state ROLLED_BACK when AutoConnectionTracker tries to
> call getResource().
>
> I think the Geronimo implementation (
> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203)
> maybe be a little too strict. The JTA Spec pdf doesn't offer exact hints of
> which statuses (
> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html) should
> be have getResource _not_ throw an ISE unfortunately. I was thinking of
> changing Geronimo's implementation to check for anything
> but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>
> The other way is to cast Transaction to the Geronimo implementation and use
> Geronimo specific APIs to get call getResource(). Do you guys have any
> preference which route I should take to fix?
>
>
> On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>
>> Here's a project that reproduces the bug. This project intentionally
>> exceeds the transaction timeout (of 1s). Each invocation, the connection is
>> not returned to the pool and eventually you run out, causing your
>> application to freeze.
>>
>>
>>
>> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <[hidden email]>
>> wrote:
>>
>>> Hello Apache friends :) I have a question about the JTA and JMS/RA specs:
>>>
>>> If you borrow something from a RA, like a JMS Connection, and you're in
>>> XA Transaction, is it necessary to call connection.close()? It would seem
>>> JTA should be smart enough to know the connection is enrolled for 2 phase
>>> commit and should be smart enough to close it, but I'm not sure if that's
>>> part of the spec.
>>>
>>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
>>> connectionFactory.createConnection(), and your code fails to call close()
>>> before the transaction completion, the connection leaks. (And
>>> unfortunately, calling close() after the transaction completes doesn't
>>> mitigate the problem). It took awhile for us to track this down.
>>>
>>> This becomes a huge problem if you're calling external services in your
>>> transaction. Let's say you have a reasonable transaction timeout of 30s
>>> set. You call three services, and they end up taking 15s a piece. Even if
>>> you're doing the right thing and you have connection.close() in a finally
>>> block, because your transaction isn't active when you call close, it leaks
>>> and it just gets "stuck" as an active connection, which eventually you hit
>>> the pool limit and your app freezes.
>>>
>>> On a separate note, we noticed if you open a connection outside of the
>>> scope of a transaction, then start a transaction, then create a session
>>> with session_transacted option, the session does not participate in the JTA
>>> (which seems out of spec). One most open the connection inside the
>>> transaction, AND open the session in the transaction, and close the
>>> connection in the transaction for everything to work.
>>>
>>> I'll get a reproducing project created, but I was curious if anyone knew
>>> offhand what the spec says.
>>>
>>> cheers, and thanks,
>>> -[the other] Jonathan
>>>
>>> --
>>> Jonathan | [hidden email]
>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> half full.
>>> Engineers, of course, understand the glass is twice as big as it needs to
>>> be.
>>>
>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
> full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.

Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

jgallimore
In reply to this post by exabrial12
Hi Jonathan

Thanks for this. I think I had run into some of the things you fixed I
little while back. I'll take a look at your change later today. Happy for
you to commit and I can review after, too. :)

Cheers

The other Jonathan


On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <[hidden email]>
wrote:

>
> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
>
> With these patches the JMS2.0 API works pretty well now :) If anyone wants
> to review, please go ahead, otherwise I'll commit them tomorrow.
>
> cheers,
> - [The other] Jonathan
>
> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <[hidden email]>
> wrote:
>
> > I've narrowed down the problem to AutoConnectionTracker. It's not
> > completing, which isn't allowing the connections to be returned to the
> pool.
> >
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
> >
> > getResource() is throwing an IllegalStateException. The JavaDoc (
> >
> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
> )
> > states it should throw an ISE if a current transaction is not Active. The
> > transaction is in the state ROLLED_BACK when AutoConnectionTracker tries
> to
> > call getResource().
> >
> > I think the Geronimo implementation (
> >
> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
> )
> > maybe be a little too strict. The JTA Spec pdf doesn't offer exact hints
> of
> > which statuses (
> > https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
> > should be have getResource _not_ throw an ISE unfortunately. I was
> thinking
> > of changing Geronimo's implementation to check for anything
> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
> >
> > The other way is to cast Transaction to the Geronimo implementation and
> > use Geronimo specific APIs to get call getResource(). Do you guys have
> any
> > preference which route I should take to fix?
> >
> >
> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]>
> > wrote:
> >
> >> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
> >>
> >> Here's a project that reproduces the bug. This project intentionally
> >> exceeds the transaction timeout (of 1s). Each invocation, the
> connection is
> >> not returned to the pool and eventually you run out, causing your
> >> application to freeze.
> >>
> >>
> >>
> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <[hidden email]>
> >> wrote:
> >>
> >>> Hello Apache friends :) I have a question about the JTA and JMS/RA
> specs:
> >>>
> >>> If you borrow something from a RA, like a JMS Connection, and you're in
> >>> XA Transaction, is it necessary to call connection.close()? It would
> seem
> >>> JTA should be smart enough to know the connection is enrolled for 2
> phase
> >>> commit and should be smart enough to close it, but I'm not sure if
> that's
> >>> part of the spec.
> >>>
> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
> >>> connectionFactory.createConnection(), and your code fails to call
> close()
> >>> before the transaction completion, the connection leaks. (And
> >>> unfortunately, calling close() after the transaction completes doesn't
> >>> mitigate the problem). It took awhile for us to track this down.
> >>>
> >>> This becomes a huge problem if you're calling external services in your
> >>> transaction. Let's say you have a reasonable transaction timeout of 30s
> >>> set. You call three services, and they end up taking 15s a piece. Even
> if
> >>> you're doing the right thing and you have connection.close() in a
> finally
> >>> block, because your transaction isn't active when you call close, it
> leaks
> >>> and it just gets "stuck" as an active connection, which eventually you
> hit
> >>> the pool limit and your app freezes.
> >>>
> >>> On a separate note, we noticed if you open a connection outside of the
> >>> scope of a transaction, then start a transaction, then create a session
> >>> with session_transacted option, the session does not participate in
> the JTA
> >>> (which seems out of spec). One most open the connection inside the
> >>> transaction, AND open the session in the transaction, and close the
> >>> connection in the transaction for everything to work.
> >>>
> >>> I'll get a reproducing project created, but I was curious if anyone
> knew
> >>> offhand what the spec says.
> >>>
> >>> cheers, and thanks,
> >>> -[the other] Jonathan
> >>>
> >>> --
> >>> Jonathan | [hidden email]
> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> >>> half full.
> >>> Engineers, of course, understand the glass is twice as big as it needs
> >>> to be.
> >>>
> >>
> >>
> >> --
> >> Jonathan | [hidden email]
> >> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> >> half full.
> >> Engineers, of course, understand the glass is twice as big as it needs
> to
> >> be.
> >>
> >
> >
> > --
> > Jonathan | [hidden email]
> > Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> > half full.
> > Engineers, of course, understand the glass is twice as big as it needs to
> > be.
> >
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
> full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

David Jencks-3
In reply to this post by David Jencks-3
I checked the Open Liberty TransactionSynchronizationRegistry, and it interprets “active transaction” to mean “any transaction on the thread, no matter it’s state”.  So I think that it would be best to decide to do the same in the Geronimo TM, deciding that the java doc is ambiguous as to the meaning of “active” and the most useful meaning can be picked rather than the most literal.

Whether this is practical for the next TomEE, I don’t know.

David Jencks

> On Aug 27, 2019, at 8:25 AM, David Jencks <[hidden email]> wrote:
>
> I think the java doc for getResource might have been written thoughtlessly, and more appropriate behavior would be an ISE only for STATUS_NO_TRANSACTION; literally the geronimo implementation is too lax, as “marked rollback” is not status “active”.  Is there anyone who’s opinion we might ask?
>
> I rather thought the “ignore session type” logic was supposed to be put into the RA, but I don’t recall if or how I dealt with this in Geronimo.
>
> So I’d prefer these issues be dealt with elsewhere but don’t see much practical alternative to your implementation.
>
> Nice to see someone working on XA:-)
>
> thanks!
> David Jencks
>
>> On Aug 26, 2019, at 1:45 PM, Jonathan S. Fisher <[hidden email]> wrote:
>>
>> I've narrowed down the problem to AutoConnectionTracker. It's not
>> completing, which isn't allowing the connections to be returned to the pool.
>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>>
>> getResource() is throwing an IllegalStateException. The JavaDoc (
>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-)
>> states it should throw an ISE if a current transaction is not Active. The
>> transaction is in the state ROLLED_BACK when AutoConnectionTracker tries to
>> call getResource().
>>
>> I think the Geronimo implementation (
>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203)
>> maybe be a little too strict. The JTA Spec pdf doesn't offer exact hints of
>> which statuses (
>> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html) should
>> be have getResource _not_ throw an ISE unfortunately. I was thinking of
>> changing Geronimo's implementation to check for anything
>> but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>>
>> The other way is to cast Transaction to the Geronimo implementation and use
>> Geronimo specific APIs to get call getResource(). Do you guys have any
>> preference which route I should take to fix?
>>
>>
>> On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]>
>> wrote:
>>
>>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>>
>>> Here's a project that reproduces the bug. This project intentionally
>>> exceeds the transaction timeout (of 1s). Each invocation, the connection is
>>> not returned to the pool and eventually you run out, causing your
>>> application to freeze.
>>>
>>>
>>>
>>> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <[hidden email]>
>>> wrote:
>>>
>>>> Hello Apache friends :) I have a question about the JTA and JMS/RA specs:
>>>>
>>>> If you borrow something from a RA, like a JMS Connection, and you're in
>>>> XA Transaction, is it necessary to call connection.close()? It would seem
>>>> JTA should be smart enough to know the connection is enrolled for 2 phase
>>>> commit and should be smart enough to close it, but I'm not sure if that's
>>>> part of the spec.
>>>>
>>>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
>>>> connectionFactory.createConnection(), and your code fails to call close()
>>>> before the transaction completion, the connection leaks. (And
>>>> unfortunately, calling close() after the transaction completes doesn't
>>>> mitigate the problem). It took awhile for us to track this down.
>>>>
>>>> This becomes a huge problem if you're calling external services in your
>>>> transaction. Let's say you have a reasonable transaction timeout of 30s
>>>> set. You call three services, and they end up taking 15s a piece. Even if
>>>> you're doing the right thing and you have connection.close() in a finally
>>>> block, because your transaction isn't active when you call close, it leaks
>>>> and it just gets "stuck" as an active connection, which eventually you hit
>>>> the pool limit and your app freezes.
>>>>
>>>> On a separate note, we noticed if you open a connection outside of the
>>>> scope of a transaction, then start a transaction, then create a session
>>>> with session_transacted option, the session does not participate in the JTA
>>>> (which seems out of spec). One most open the connection inside the
>>>> transaction, AND open the session in the transaction, and close the
>>>> connection in the transaction for everything to work.
>>>>
>>>> I'll get a reproducing project created, but I was curious if anyone knew
>>>> offhand what the spec says.
>>>>
>>>> cheers, and thanks,
>>>> -[the other] Jonathan
>>>>
>>>> --
>>>> Jonathan | [hidden email]
>>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>>> half full.
>>>> Engineers, of course, understand the glass is twice as big as it needs to
>>>> be.
>>>>
>>>
>>>
>>> --
>>> Jonathan | [hidden email]
>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> half full.
>>> Engineers, of course, understand the glass is twice as big as it needs to
>>> be.
>>>
>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
>> full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>

Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
I just checked Wildfly, they do the same thing as Liberty. I agree with
your statement for the "completely correct" fix, ideally that's the place
to do it, but might take awhile to get a release out.

On another note: I know the spec says, "Ignore all arguments to
connection.create*(int mode)" methods. Yet I can think of a lot of
scenarios where having a non-JTA connection pool is very handy (for
instance, logging over JMS). We have the option to have non-JTA Database
connections, I feel though we should be able to declare whether or not a
jms connection pool participates in JTA.

I'm thinking maybe we should have an `xa=true/false` parameter in the
connection pool declaration. Would that be ok?


On Tue, Aug 27, 2019 at 3:43 PM David Jencks <[hidden email]>
wrote:

> I checked the Open Liberty TransactionSynchronizationRegistry, and it
> interprets “active transaction” to mean “any transaction on the thread, no
> matter it’s state”.  So I think that it would be best to decide to do the
> same in the Geronimo TM, deciding that the java doc is ambiguous as to the
> meaning of “active” and the most useful meaning can be picked rather than
> the most literal.
>
> Whether this is practical for the next TomEE, I don’t know.
>
> David Jencks
>
> > On Aug 27, 2019, at 8:25 AM, David Jencks <[hidden email]>
> wrote:
> >
> > I think the java doc for getResource might have been written
> thoughtlessly, and more appropriate behavior would be an ISE only for
> STATUS_NO_TRANSACTION; literally the geronimo implementation is too lax, as
> “marked rollback” is not status “active”.  Is there anyone who’s opinion we
> might ask?
> >
> > I rather thought the “ignore session type” logic was supposed to be put
> into the RA, but I don’t recall if or how I dealt with this in Geronimo.
> >
> > So I’d prefer these issues be dealt with elsewhere but don’t see much
> practical alternative to your implementation.
> >
> > Nice to see someone working on XA:-)
> >
> > thanks!
> > David Jencks
> >
> >> On Aug 26, 2019, at 1:45 PM, Jonathan S. Fisher <[hidden email]>
> wrote:
> >>
> >> I've narrowed down the problem to AutoConnectionTracker. It's not
> >> completing, which isn't allowing the connections to be returned to the
> pool.
> >>
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
> >>
> >> getResource() is throwing an IllegalStateException. The JavaDoc (
> >>
> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
> )
> >> states it should throw an ISE if a current transaction is not Active.
> The
> >> transaction is in the state ROLLED_BACK when AutoConnectionTracker
> tries to
> >> call getResource().
> >>
> >> I think the Geronimo implementation (
> >>
> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
> )
> >> maybe be a little too strict. The JTA Spec pdf doesn't offer exact
> hints of
> >> which statuses (
> >> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
> should
> >> be have getResource _not_ throw an ISE unfortunately. I was thinking of
> >> changing Geronimo's implementation to check for anything
> >> but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
> >>
> >> The other way is to cast Transaction to the Geronimo implementation and
> use
> >> Geronimo specific APIs to get call getResource(). Do you guys have any
> >> preference which route I should take to fix?
> >>
> >>
> >> On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]>
> >> wrote:
> >>
> >>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
> >>>
> >>> Here's a project that reproduces the bug. This project intentionally
> >>> exceeds the transaction timeout (of 1s). Each invocation, the
> connection is
> >>> not returned to the pool and eventually you run out, causing your
> >>> application to freeze.
> >>>
> >>>
> >>>
> >>> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <[hidden email]
> >
> >>> wrote:
> >>>
> >>>> Hello Apache friends :) I have a question about the JTA and JMS/RA
> specs:
> >>>>
> >>>> If you borrow something from a RA, like a JMS Connection, and you're
> in
> >>>> XA Transaction, is it necessary to call connection.close()? It would
> seem
> >>>> JTA should be smart enough to know the connection is enrolled for 2
> phase
> >>>> commit and should be smart enough to close it, but I'm not sure if
> that's
> >>>> part of the spec.
> >>>>
> >>>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
> >>>> connectionFactory.createConnection(), and your code fails to call
> close()
> >>>> before the transaction completion, the connection leaks. (And
> >>>> unfortunately, calling close() after the transaction completes doesn't
> >>>> mitigate the problem). It took awhile for us to track this down.
> >>>>
> >>>> This becomes a huge problem if you're calling external services in
> your
> >>>> transaction. Let's say you have a reasonable transaction timeout of
> 30s
> >>>> set. You call three services, and they end up taking 15s a piece.
> Even if
> >>>> you're doing the right thing and you have connection.close() in a
> finally
> >>>> block, because your transaction isn't active when you call close, it
> leaks
> >>>> and it just gets "stuck" as an active connection, which eventually
> you hit
> >>>> the pool limit and your app freezes.
> >>>>
> >>>> On a separate note, we noticed if you open a connection outside of the
> >>>> scope of a transaction, then start a transaction, then create a
> session
> >>>> with session_transacted option, the session does not participate in
> the JTA
> >>>> (which seems out of spec). One most open the connection inside the
> >>>> transaction, AND open the session in the transaction, and close the
> >>>> connection in the transaction for everything to work.
> >>>>
> >>>> I'll get a reproducing project created, but I was curious if anyone
> knew
> >>>> offhand what the spec says.
> >>>>
> >>>> cheers, and thanks,
> >>>> -[the other] Jonathan
> >>>>
> >>>> --
> >>>> Jonathan | [hidden email]
> >>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> >>>> half full.
> >>>> Engineers, of course, understand the glass is twice as big as it
> needs to
> >>>> be.
> >>>>
> >>>
> >>>
> >>> --
> >>> Jonathan | [hidden email]
> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> >>> half full.
> >>> Engineers, of course, understand the glass is twice as big as it needs
> to
> >>> be.
> >>>
> >>
> >>
> >> --
> >> Jonathan | [hidden email]
> >> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half
> >> full.
> >> Engineers, of course, understand the glass is twice as big as it needs
> to
> >> be.
> >
>
>

--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
Just noticed in JmsConnectionFactoryBuilder this is present already with
the attribute "transactionSupport". I need to tie that into my patch before
it's merged

On Tue, Aug 27, 2019 at 4:30 PM Jonathan S. Fisher <[hidden email]>
wrote:

> I just checked Wildfly, they do the same thing as Liberty. I agree with
> your statement for the "completely correct" fix, ideally that's the place
> to do it, but might take awhile to get a release out.
>
> On another note: I know the spec says, "Ignore all arguments to
> connection.create*(int mode)" methods. Yet I can think of a lot of
> scenarios where having a non-JTA connection pool is very handy (for
> instance, logging over JMS). We have the option to have non-JTA Database
> connections, I feel though we should be able to declare whether or not a
> jms connection pool participates in JTA.
>
> I'm thinking maybe we should have an `xa=true/false` parameter in the
> connection pool declaration. Would that be ok?
>
>
> On Tue, Aug 27, 2019 at 3:43 PM David Jencks <[hidden email]>
> wrote:
>
>> I checked the Open Liberty TransactionSynchronizationRegistry, and it
>> interprets “active transaction” to mean “any transaction on the thread, no
>> matter it’s state”.  So I think that it would be best to decide to do the
>> same in the Geronimo TM, deciding that the java doc is ambiguous as to the
>> meaning of “active” and the most useful meaning can be picked rather than
>> the most literal.
>>
>> Whether this is practical for the next TomEE, I don’t know.
>>
>> David Jencks
>>
>> > On Aug 27, 2019, at 8:25 AM, David Jencks <[hidden email]>
>> wrote:
>> >
>> > I think the java doc for getResource might have been written
>> thoughtlessly, and more appropriate behavior would be an ISE only for
>> STATUS_NO_TRANSACTION; literally the geronimo implementation is too lax, as
>> “marked rollback” is not status “active”.  Is there anyone who’s opinion we
>> might ask?
>> >
>> > I rather thought the “ignore session type” logic was supposed to be put
>> into the RA, but I don’t recall if or how I dealt with this in Geronimo.
>> >
>> > So I’d prefer these issues be dealt with elsewhere but don’t see much
>> practical alternative to your implementation.
>> >
>> > Nice to see someone working on XA:-)
>> >
>> > thanks!
>> > David Jencks
>> >
>> >> On Aug 26, 2019, at 1:45 PM, Jonathan S. Fisher <[hidden email]>
>> wrote:
>> >>
>> >> I've narrowed down the problem to AutoConnectionTracker. It's not
>> >> completing, which isn't allowing the connections to be returned to the
>> pool.
>> >>
>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>> >>
>> >> getResource() is throwing an IllegalStateException. The JavaDoc (
>> >>
>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>> )
>> >> states it should throw an ISE if a current transaction is not Active.
>> The
>> >> transaction is in the state ROLLED_BACK when AutoConnectionTracker
>> tries to
>> >> call getResource().
>> >>
>> >> I think the Geronimo implementation (
>> >>
>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>> )
>> >> maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>> hints of
>> >> which statuses (
>> >> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
>> should
>> >> be have getResource _not_ throw an ISE unfortunately. I was thinking of
>> >> changing Geronimo's implementation to check for anything
>> >> but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>> >>
>> >> The other way is to cast Transaction to the Geronimo implementation
>> and use
>> >> Geronimo specific APIs to get call getResource(). Do you guys have any
>> >> preference which route I should take to fix?
>> >>
>> >>
>> >> On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]
>> >
>> >> wrote:
>> >>
>> >>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>> >>>
>> >>> Here's a project that reproduces the bug. This project intentionally
>> >>> exceeds the transaction timeout (of 1s). Each invocation, the
>> connection is
>> >>> not returned to the pool and eventually you run out, causing your
>> >>> application to freeze.
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
>> [hidden email]>
>> >>> wrote:
>> >>>
>> >>>> Hello Apache friends :) I have a question about the JTA and JMS/RA
>> specs:
>> >>>>
>> >>>> If you borrow something from a RA, like a JMS Connection, and you're
>> in
>> >>>> XA Transaction, is it necessary to call connection.close()? It would
>> seem
>> >>>> JTA should be smart enough to know the connection is enrolled for 2
>> phase
>> >>>> commit and should be smart enough to close it, but I'm not sure if
>> that's
>> >>>> part of the spec.
>> >>>>
>> >>>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection
>> with
>> >>>> connectionFactory.createConnection(), and your code fails to call
>> close()
>> >>>> before the transaction completion, the connection leaks. (And
>> >>>> unfortunately, calling close() after the transaction completes
>> doesn't
>> >>>> mitigate the problem). It took awhile for us to track this down.
>> >>>>
>> >>>> This becomes a huge problem if you're calling external services in
>> your
>> >>>> transaction. Let's say you have a reasonable transaction timeout of
>> 30s
>> >>>> set. You call three services, and they end up taking 15s a piece.
>> Even if
>> >>>> you're doing the right thing and you have connection.close() in a
>> finally
>> >>>> block, because your transaction isn't active when you call close, it
>> leaks
>> >>>> and it just gets "stuck" as an active connection, which eventually
>> you hit
>> >>>> the pool limit and your app freezes.
>> >>>>
>> >>>> On a separate note, we noticed if you open a connection outside of
>> the
>> >>>> scope of a transaction, then start a transaction, then create a
>> session
>> >>>> with session_transacted option, the session does not participate in
>> the JTA
>> >>>> (which seems out of spec). One most open the connection inside the
>> >>>> transaction, AND open the session in the transaction, and close the
>> >>>> connection in the transaction for everything to work.
>> >>>>
>> >>>> I'll get a reproducing project created, but I was curious if anyone
>> knew
>> >>>> offhand what the spec says.
>> >>>>
>> >>>> cheers, and thanks,
>> >>>> -[the other] Jonathan
>> >>>>
>> >>>> --
>> >>>> Jonathan | [hidden email]
>> >>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>> as
>> >>>> half full.
>> >>>> Engineers, of course, understand the glass is twice as big as it
>> needs to
>> >>>> be.
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> Jonathan | [hidden email]
>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> >>> half full.
>> >>> Engineers, of course, understand the glass is twice as big as it
>> needs to
>> >>> be.
>> >>>
>> >>
>> >>
>> >> --
>> >> Jonathan | [hidden email]
>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half
>> >> full.
>> >> Engineers, of course, understand the glass is twice as big as it needs
>> to
>> >> be.
>> >
>>
>>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
Alrighty. I've got "transactionSupport" ready. Apparently that hasn't
worked for some time.

The nice thing too is that gives the user an "out" if they want to revert
to the non-spec behavior: Right now, connection factories are non-xa, by
the spec says they should be by default. If someone upgrades and desires
the old non-xa behavior, they can set transactionSupport=none on the
connection factory.

I'm fairly confident in this merge request, the only part I really don't
know about is the enlistResource() when I enlist XASession. It works
though, and from the examples I've seen it seems to be the correct way to
enlist a resource. I've just never messed with that API before.

Anyway, eyes appreciated, after I update my PRs on github I'll give it a
day for review and then merge it in.

Thanks!




On Tue, Aug 27, 2019 at 9:27 PM Jonathan S. Fisher <[hidden email]>
wrote:

> Just noticed in JmsConnectionFactoryBuilder this is present already with
> the attribute "transactionSupport". I need to tie that into my patch before
> it's merged
>
> On Tue, Aug 27, 2019 at 4:30 PM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>> I just checked Wildfly, they do the same thing as Liberty. I agree with
>> your statement for the "completely correct" fix, ideally that's the place
>> to do it, but might take awhile to get a release out.
>>
>> On another note: I know the spec says, "Ignore all arguments to
>> connection.create*(int mode)" methods. Yet I can think of a lot of
>> scenarios where having a non-JTA connection pool is very handy (for
>> instance, logging over JMS). We have the option to have non-JTA Database
>> connections, I feel though we should be able to declare whether or not a
>> jms connection pool participates in JTA.
>>
>> I'm thinking maybe we should have an `xa=true/false` parameter in the
>> connection pool declaration. Would that be ok?
>>
>>
>> On Tue, Aug 27, 2019 at 3:43 PM David Jencks <[hidden email]>
>> wrote:
>>
>>> I checked the Open Liberty TransactionSynchronizationRegistry, and it
>>> interprets “active transaction” to mean “any transaction on the thread, no
>>> matter it’s state”.  So I think that it would be best to decide to do the
>>> same in the Geronimo TM, deciding that the java doc is ambiguous as to the
>>> meaning of “active” and the most useful meaning can be picked rather than
>>> the most literal.
>>>
>>> Whether this is practical for the next TomEE, I don’t know.
>>>
>>> David Jencks
>>>
>>> > On Aug 27, 2019, at 8:25 AM, David Jencks <[hidden email]>
>>> wrote:
>>> >
>>> > I think the java doc for getResource might have been written
>>> thoughtlessly, and more appropriate behavior would be an ISE only for
>>> STATUS_NO_TRANSACTION; literally the geronimo implementation is too lax, as
>>> “marked rollback” is not status “active”.  Is there anyone who’s opinion we
>>> might ask?
>>> >
>>> > I rather thought the “ignore session type” logic was supposed to be
>>> put into the RA, but I don’t recall if or how I dealt with this in Geronimo.
>>> >
>>> > So I’d prefer these issues be dealt with elsewhere but don’t see much
>>> practical alternative to your implementation.
>>> >
>>> > Nice to see someone working on XA:-)
>>> >
>>> > thanks!
>>> > David Jencks
>>> >
>>> >> On Aug 26, 2019, at 1:45 PM, Jonathan S. Fisher <[hidden email]>
>>> wrote:
>>> >>
>>> >> I've narrowed down the problem to AutoConnectionTracker. It's not
>>> >> completing, which isn't allowing the connections to be returned to
>>> the pool.
>>> >>
>>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>>> >>
>>> >> getResource() is throwing an IllegalStateException. The JavaDoc (
>>> >>
>>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>>> )
>>> >> states it should throw an ISE if a current transaction is not Active.
>>> The
>>> >> transaction is in the state ROLLED_BACK when AutoConnectionTracker
>>> tries to
>>> >> call getResource().
>>> >>
>>> >> I think the Geronimo implementation (
>>> >>
>>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>>> )
>>> >> maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>>> hints of
>>> >> which statuses (
>>> >> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
>>> should
>>> >> be have getResource _not_ throw an ISE unfortunately. I was thinking
>>> of
>>> >> changing Geronimo's implementation to check for anything
>>> >> but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>>> >>
>>> >> The other way is to cast Transaction to the Geronimo implementation
>>> and use
>>> >> Geronimo specific APIs to get call getResource(). Do you guys have any
>>> >> preference which route I should take to fix?
>>> >>
>>> >>
>>> >> On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
>>> [hidden email]>
>>> >> wrote:
>>> >>
>>> >>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>> >>>
>>> >>> Here's a project that reproduces the bug. This project intentionally
>>> >>> exceeds the transaction timeout (of 1s). Each invocation, the
>>> connection is
>>> >>> not returned to the pool and eventually you run out, causing your
>>> >>> application to freeze.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
>>> [hidden email]>
>>> >>> wrote:
>>> >>>
>>> >>>> Hello Apache friends :) I have a question about the JTA and JMS/RA
>>> specs:
>>> >>>>
>>> >>>> If you borrow something from a RA, like a JMS Connection, and
>>> you're in
>>> >>>> XA Transaction, is it necessary to call connection.close()? It
>>> would seem
>>> >>>> JTA should be smart enough to know the connection is enrolled for 2
>>> phase
>>> >>>> commit and should be smart enough to close it, but I'm not sure if
>>> that's
>>> >>>> part of the spec.
>>> >>>>
>>> >>>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection
>>> with
>>> >>>> connectionFactory.createConnection(), and your code fails to call
>>> close()
>>> >>>> before the transaction completion, the connection leaks. (And
>>> >>>> unfortunately, calling close() after the transaction completes
>>> doesn't
>>> >>>> mitigate the problem). It took awhile for us to track this down.
>>> >>>>
>>> >>>> This becomes a huge problem if you're calling external services in
>>> your
>>> >>>> transaction. Let's say you have a reasonable transaction timeout of
>>> 30s
>>> >>>> set. You call three services, and they end up taking 15s a piece.
>>> Even if
>>> >>>> you're doing the right thing and you have connection.close() in a
>>> finally
>>> >>>> block, because your transaction isn't active when you call close,
>>> it leaks
>>> >>>> and it just gets "stuck" as an active connection, which eventually
>>> you hit
>>> >>>> the pool limit and your app freezes.
>>> >>>>
>>> >>>> On a separate note, we noticed if you open a connection outside of
>>> the
>>> >>>> scope of a transaction, then start a transaction, then create a
>>> session
>>> >>>> with session_transacted option, the session does not participate in
>>> the JTA
>>> >>>> (which seems out of spec). One most open the connection inside the
>>> >>>> transaction, AND open the session in the transaction, and close the
>>> >>>> connection in the transaction for everything to work.
>>> >>>>
>>> >>>> I'll get a reproducing project created, but I was curious if anyone
>>> knew
>>> >>>> offhand what the spec says.
>>> >>>>
>>> >>>> cheers, and thanks,
>>> >>>> -[the other] Jonathan
>>> >>>>
>>> >>>> --
>>> >>>> Jonathan | [hidden email]
>>> >>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>> as
>>> >>>> half full.
>>> >>>> Engineers, of course, understand the glass is twice as big as it
>>> needs to
>>> >>>> be.
>>> >>>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Jonathan | [hidden email]
>>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>> as
>>> >>> half full.
>>> >>> Engineers, of course, understand the glass is twice as big as it
>>> needs to
>>> >>> be.
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >> Jonathan | [hidden email]
>>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>> as half
>>> >> full.
>>> >> Engineers, of course, understand the glass is twice as big as it
>>> needs to
>>> >> be.
>>> >
>>>
>>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
Green build here:
https://ci.apache.org/builders/tomee-7.0.x-ubuntu/builds/52
Not sure why this got is all pissy:
https://ci.apache.org/builders/tomee-7.0.x-ubuntu-jvm8

Seems to be a different test failure each time. Builds fine on
OSX/AdoptOpenJdk8 locally. Any hints?


On Wed, Aug 28, 2019 at 10:08 AM Jonathan S. Fisher <[hidden email]>
wrote:

> Alrighty. I've got "transactionSupport" ready. Apparently that hasn't
> worked for some time.
>
> The nice thing too is that gives the user an "out" if they want to revert
> to the non-spec behavior: Right now, connection factories are non-xa, by
> the spec says they should be by default. If someone upgrades and desires
> the old non-xa behavior, they can set transactionSupport=none on the
> connection factory.
>
> I'm fairly confident in this merge request, the only part I really don't
> know about is the enlistResource() when I enlist XASession. It works
> though, and from the examples I've seen it seems to be the correct way to
> enlist a resource. I've just never messed with that API before.
>
> Anyway, eyes appreciated, after I update my PRs on github I'll give it a
> day for review and then merge it in.
>
> Thanks!
>
>
>
>
> On Tue, Aug 27, 2019 at 9:27 PM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>> Just noticed in JmsConnectionFactoryBuilder this is present already with
>> the attribute "transactionSupport". I need to tie that into my patch before
>> it's merged
>>
>> On Tue, Aug 27, 2019 at 4:30 PM Jonathan S. Fisher <[hidden email]>
>> wrote:
>>
>>> I just checked Wildfly, they do the same thing as Liberty. I agree with
>>> your statement for the "completely correct" fix, ideally that's the place
>>> to do it, but might take awhile to get a release out.
>>>
>>> On another note: I know the spec says, "Ignore all arguments to
>>> connection.create*(int mode)" methods. Yet I can think of a lot of
>>> scenarios where having a non-JTA connection pool is very handy (for
>>> instance, logging over JMS). We have the option to have non-JTA Database
>>> connections, I feel though we should be able to declare whether or not a
>>> jms connection pool participates in JTA.
>>>
>>> I'm thinking maybe we should have an `xa=true/false` parameter in the
>>> connection pool declaration. Would that be ok?
>>>
>>>
>>> On Tue, Aug 27, 2019 at 3:43 PM David Jencks <[hidden email]>
>>> wrote:
>>>
>>>> I checked the Open Liberty TransactionSynchronizationRegistry, and it
>>>> interprets “active transaction” to mean “any transaction on the thread, no
>>>> matter it’s state”.  So I think that it would be best to decide to do the
>>>> same in the Geronimo TM, deciding that the java doc is ambiguous as to the
>>>> meaning of “active” and the most useful meaning can be picked rather than
>>>> the most literal.
>>>>
>>>> Whether this is practical for the next TomEE, I don’t know.
>>>>
>>>> David Jencks
>>>>
>>>> > On Aug 27, 2019, at 8:25 AM, David Jencks <[hidden email]>
>>>> wrote:
>>>> >
>>>> > I think the java doc for getResource might have been written
>>>> thoughtlessly, and more appropriate behavior would be an ISE only for
>>>> STATUS_NO_TRANSACTION; literally the geronimo implementation is too lax, as
>>>> “marked rollback” is not status “active”.  Is there anyone who’s opinion we
>>>> might ask?
>>>> >
>>>> > I rather thought the “ignore session type” logic was supposed to be
>>>> put into the RA, but I don’t recall if or how I dealt with this in Geronimo.
>>>> >
>>>> > So I’d prefer these issues be dealt with elsewhere but don’t see much
>>>> practical alternative to your implementation.
>>>> >
>>>> > Nice to see someone working on XA:-)
>>>> >
>>>> > thanks!
>>>> > David Jencks
>>>> >
>>>> >> On Aug 26, 2019, at 1:45 PM, Jonathan S. Fisher <[hidden email]>
>>>> wrote:
>>>> >>
>>>> >> I've narrowed down the problem to AutoConnectionTracker. It's not
>>>> >> completing, which isn't allowing the connections to be returned to
>>>> the pool.
>>>> >>
>>>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>>>> >>
>>>> >> getResource() is throwing an IllegalStateException. The JavaDoc (
>>>> >>
>>>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>>>> )
>>>> >> states it should throw an ISE if a current transaction is not
>>>> Active. The
>>>> >> transaction is in the state ROLLED_BACK when AutoConnectionTracker
>>>> tries to
>>>> >> call getResource().
>>>> >>
>>>> >> I think the Geronimo implementation (
>>>> >>
>>>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>>>> )
>>>> >> maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>>>> hints of
>>>> >> which statuses (
>>>> >> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
>>>> should
>>>> >> be have getResource _not_ throw an ISE unfortunately. I was thinking
>>>> of
>>>> >> changing Geronimo's implementation to check for anything
>>>> >> but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>>>> >>
>>>> >> The other way is to cast Transaction to the Geronimo implementation
>>>> and use
>>>> >> Geronimo specific APIs to get call getResource(). Do you guys have
>>>> any
>>>> >> preference which route I should take to fix?
>>>> >>
>>>> >>
>>>> >> On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
>>>> [hidden email]>
>>>> >> wrote:
>>>> >>
>>>> >>>
>>>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>>> >>>
>>>> >>> Here's a project that reproduces the bug. This project intentionally
>>>> >>> exceeds the transaction timeout (of 1s). Each invocation, the
>>>> connection is
>>>> >>> not returned to the pool and eventually you run out, causing your
>>>> >>> application to freeze.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
>>>> [hidden email]>
>>>> >>> wrote:
>>>> >>>
>>>> >>>> Hello Apache friends :) I have a question about the JTA and JMS/RA
>>>> specs:
>>>> >>>>
>>>> >>>> If you borrow something from a RA, like a JMS Connection, and
>>>> you're in
>>>> >>>> XA Transaction, is it necessary to call connection.close()? It
>>>> would seem
>>>> >>>> JTA should be smart enough to know the connection is enrolled for
>>>> 2 phase
>>>> >>>> commit and should be smart enough to close it, but I'm not sure if
>>>> that's
>>>> >>>> part of the spec.
>>>> >>>>
>>>> >>>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection
>>>> with
>>>> >>>> connectionFactory.createConnection(), and your code fails to call
>>>> close()
>>>> >>>> before the transaction completion, the connection leaks. (And
>>>> >>>> unfortunately, calling close() after the transaction completes
>>>> doesn't
>>>> >>>> mitigate the problem). It took awhile for us to track this down.
>>>> >>>>
>>>> >>>> This becomes a huge problem if you're calling external services in
>>>> your
>>>> >>>> transaction. Let's say you have a reasonable transaction timeout
>>>> of 30s
>>>> >>>> set. You call three services, and they end up taking 15s a piece.
>>>> Even if
>>>> >>>> you're doing the right thing and you have connection.close() in a
>>>> finally
>>>> >>>> block, because your transaction isn't active when you call close,
>>>> it leaks
>>>> >>>> and it just gets "stuck" as an active connection, which eventually
>>>> you hit
>>>> >>>> the pool limit and your app freezes.
>>>> >>>>
>>>> >>>> On a separate note, we noticed if you open a connection outside of
>>>> the
>>>> >>>> scope of a transaction, then start a transaction, then create a
>>>> session
>>>> >>>> with session_transacted option, the session does not participate
>>>> in the JTA
>>>> >>>> (which seems out of spec). One most open the connection inside the
>>>> >>>> transaction, AND open the session in the transaction, and close the
>>>> >>>> connection in the transaction for everything to work.
>>>> >>>>
>>>> >>>> I'll get a reproducing project created, but I was curious if
>>>> anyone knew
>>>> >>>> offhand what the spec says.
>>>> >>>>
>>>> >>>> cheers, and thanks,
>>>> >>>> -[the other] Jonathan
>>>> >>>>
>>>> >>>> --
>>>> >>>> Jonathan | [hidden email]
>>>> >>>> Pessimists, see a jar as half empty. Optimists, in contrast, see
>>>> it as
>>>> >>>> half full.
>>>> >>>> Engineers, of course, understand the glass is twice as big as it
>>>> needs to
>>>> >>>> be.
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Jonathan | [hidden email]
>>>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>>> as
>>>> >>> half full.
>>>> >>> Engineers, of course, understand the glass is twice as big as it
>>>> needs to
>>>> >>> be.
>>>> >>>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Jonathan | [hidden email]
>>>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>>> as half
>>>> >> full.
>>>> >> Engineers, of course, understand the glass is twice as big as it
>>>> needs to
>>>> >> be.
>>>> >
>>>>
>>>>
>>>
>>> --
>>> Jonathan | [hidden email]
>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> half full.
>>> Engineers, of course, understand the glass is twice as big as it needs
>>> to be.
>>>
>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

jgallimore
In reply to this post by jgallimore
I had a play around with this, and got my previously failing
JMSContextInjectionTest to work -- some of the time. (
https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
).

I'm getting some log output that I'm not expecting so I've rolled back my
removal of @Ignore so I can dig into it some more.

Jon

On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
[hidden email]> wrote:

> Hi Jonathan
>
> Thanks for this. I think I had run into some of the things you fixed I
> little while back. I'll take a look at your change later today. Happy for
> you to commit and I can review after, too. :)
>
> Cheers
>
> The other Jonathan
>
>
> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>>
>> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
>>
>> With these patches the JMS2.0 API works pretty well now :) If anyone wants
>> to review, please go ahead, otherwise I'll commit them tomorrow.
>>
>> cheers,
>> - [The other] Jonathan
>>
>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <[hidden email]>
>> wrote:
>>
>> > I've narrowed down the problem to AutoConnectionTracker. It's not
>> > completing, which isn't allowing the connections to be returned to the
>> pool.
>> >
>> >
>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>> >
>> > getResource() is throwing an IllegalStateException. The JavaDoc (
>> >
>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>> )
>> > states it should throw an ISE if a current transaction is not Active.
>> The
>> > transaction is in the state ROLLED_BACK when AutoConnectionTracker
>> tries to
>> > call getResource().
>> >
>> > I think the Geronimo implementation (
>> >
>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>> )
>> > maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>> hints of
>> > which statuses (
>> > https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
>> > should be have getResource _not_ throw an ISE unfortunately. I was
>> thinking
>> > of changing Geronimo's implementation to check for anything
>> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>> >
>> > The other way is to cast Transaction to the Geronimo implementation and
>> > use Geronimo specific APIs to get call getResource(). Do you guys have
>> any
>> > preference which route I should take to fix?
>> >
>> >
>> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]>
>> > wrote:
>> >
>> >> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>> >>
>> >> Here's a project that reproduces the bug. This project intentionally
>> >> exceeds the transaction timeout (of 1s). Each invocation, the
>> connection is
>> >> not returned to the pool and eventually you run out, causing your
>> >> application to freeze.
>> >>
>> >>
>> >>
>> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <[hidden email]
>> >
>> >> wrote:
>> >>
>> >>> Hello Apache friends :) I have a question about the JTA and JMS/RA
>> specs:
>> >>>
>> >>> If you borrow something from a RA, like a JMS Connection, and you're
>> in
>> >>> XA Transaction, is it necessary to call connection.close()? It would
>> seem
>> >>> JTA should be smart enough to know the connection is enrolled for 2
>> phase
>> >>> commit and should be smart enough to close it, but I'm not sure if
>> that's
>> >>> part of the spec.
>> >>>
>> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection with
>> >>> connectionFactory.createConnection(), and your code fails to call
>> close()
>> >>> before the transaction completion, the connection leaks. (And
>> >>> unfortunately, calling close() after the transaction completes doesn't
>> >>> mitigate the problem). It took awhile for us to track this down.
>> >>>
>> >>> This becomes a huge problem if you're calling external services in
>> your
>> >>> transaction. Let's say you have a reasonable transaction timeout of
>> 30s
>> >>> set. You call three services, and they end up taking 15s a piece.
>> Even if
>> >>> you're doing the right thing and you have connection.close() in a
>> finally
>> >>> block, because your transaction isn't active when you call close, it
>> leaks
>> >>> and it just gets "stuck" as an active connection, which eventually
>> you hit
>> >>> the pool limit and your app freezes.
>> >>>
>> >>> On a separate note, we noticed if you open a connection outside of the
>> >>> scope of a transaction, then start a transaction, then create a
>> session
>> >>> with session_transacted option, the session does not participate in
>> the JTA
>> >>> (which seems out of spec). One most open the connection inside the
>> >>> transaction, AND open the session in the transaction, and close the
>> >>> connection in the transaction for everything to work.
>> >>>
>> >>> I'll get a reproducing project created, but I was curious if anyone
>> knew
>> >>> offhand what the spec says.
>> >>>
>> >>> cheers, and thanks,
>> >>> -[the other] Jonathan
>> >>>
>> >>> --
>> >>> Jonathan | [hidden email]
>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> >>> half full.
>> >>> Engineers, of course, understand the glass is twice as big as it needs
>> >>> to be.
>> >>>
>> >>
>> >>
>> >> --
>> >> Jonathan | [hidden email]
>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> >> half full.
>> >> Engineers, of course, understand the glass is twice as big as it needs
>> to
>> >> be.
>> >>
>> >
>> >
>> > --
>> > Jonathan | [hidden email]
>> > Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> > half full.
>> > Engineers, of course, understand the glass is twice as big as it needs
>> to
>> > be.
>> >
>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half
>> full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

jgallimore
In case it means anything to anyone, the unexpected output I'm getting is
an abandoned connection notification:

WARNING: Transaction complete, but connection still has handles associated:
ManagedConnectionInfo:
org.apache.geronimo.connector.outbound.ManagedConnectionInfo@5b5a4aed. mc:
[org.apache.openejb.resource.activemq.jms2.TomEEManagedConnection@37e69c43,ActiveMQConnection
{id=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-7:1,clientId=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-6:1,started=false}]]
Abandoned connection information:
  Connection handle opened at
org.apache.geronimo.connector.outbound.ConnectionInfo.setTrace(ConnectionInfo.java:119),
org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:57),
org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39),
org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66),
org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:81),
org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94),
org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67),
org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:83),
org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:99),
org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createQueue(JMSContextImpl.java:309),
org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createQueue(JMS2CDIExtension.java:371),
org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:37),
org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:33),
sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191),
org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102),
sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85),
org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272),
org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221),
org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265),
org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260),
org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89),
org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347),
org.apache.openejb.arquillian.tests.jms.JMSSenderBean$$LocalBeanProxy.sendToQueue(org/apache/openejb/arquillian/tests/jms/JMSSenderBean.java),
org.apache.openejb.arquillian.tests.jms.JMSContextInjectionTest.testShouldSendAndReceiveMessages(JMSContextInjectionTest.java:61),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50),
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12),
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47),
org.jboss.arquillian.junit.Arquillian$8$1.invoke(Arquillian.java:379),
org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50),
org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:109),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:136),
org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:372),
org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:246),
org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:260),
org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:324),
org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99),
org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
java.lang.reflect.Method.invoke(Method.java:498),
org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159),
org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:317),
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325),
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78),
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57),
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:205),
org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219),
org.junit.runners.ParentRunner.run(ParentRunner.java:363),
org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167),
org.junit.runners.Suite.runChild(Suite.java:128),
org.junit.runners.Suite.runChild(Suite.java:27),
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
org.junit.runners.ParentRunner.run(ParentRunner.java:363),
org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55),
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137),
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107),
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83),
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75),
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157),
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386),
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323),
org.apache.

Also, I see out by one errors - e.g. if I send 200 messages, sometimes I
receive 201 or 199. Not sure if its related to the above warning or not.

Jon

On Mon, Sep 2, 2019 at 8:25 PM Jonathan Gallimore <
[hidden email]> wrote:

> I had a play around with this, and got my previously failing
> JMSContextInjectionTest to work -- some of the time. (
> https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
> ).
>
> I'm getting some log output that I'm not expecting so I've rolled back my
> removal of @Ignore so I can dig into it some more.
>
> Jon
>
> On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
> [hidden email]> wrote:
>
>> Hi Jonathan
>>
>> Thanks for this. I think I had run into some of the things you fixed I
>> little while back. I'll take a look at your change later today. Happy for
>> you to commit and I can review after, too. :)
>>
>> Cheers
>>
>> The other Jonathan
>>
>>
>> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <[hidden email]>
>> wrote:
>>
>>>
>>> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
>>>
>>> With these patches the JMS2.0 API works pretty well now :) If anyone
>>> wants
>>> to review, please go ahead, otherwise I'll commit them tomorrow.
>>>
>>> cheers,
>>> - [The other] Jonathan
>>>
>>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <[hidden email]>
>>> wrote:
>>>
>>> > I've narrowed down the problem to AutoConnectionTracker. It's not
>>> > completing, which isn't allowing the connections to be returned to the
>>> pool.
>>> >
>>> >
>>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>>> >
>>> > getResource() is throwing an IllegalStateException. The JavaDoc (
>>> >
>>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>>> )
>>> > states it should throw an ISE if a current transaction is not Active.
>>> The
>>> > transaction is in the state ROLLED_BACK when AutoConnectionTracker
>>> tries to
>>> > call getResource().
>>> >
>>> > I think the Geronimo implementation (
>>> >
>>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>>> )
>>> > maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>>> hints of
>>> > which statuses (
>>> > https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
>>> > should be have getResource _not_ throw an ISE unfortunately. I was
>>> thinking
>>> > of changing Geronimo's implementation to check for anything
>>> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>>> >
>>> > The other way is to cast Transaction to the Geronimo implementation and
>>> > use Geronimo specific APIs to get call getResource(). Do you guys have
>>> any
>>> > preference which route I should take to fix?
>>> >
>>> >
>>> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <[hidden email]
>>> >
>>> > wrote:
>>> >
>>> >> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>> >>
>>> >> Here's a project that reproduces the bug. This project intentionally
>>> >> exceeds the transaction timeout (of 1s). Each invocation, the
>>> connection is
>>> >> not returned to the pool and eventually you run out, causing your
>>> >> application to freeze.
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
>>> [hidden email]>
>>> >> wrote:
>>> >>
>>> >>> Hello Apache friends :) I have a question about the JTA and JMS/RA
>>> specs:
>>> >>>
>>> >>> If you borrow something from a RA, like a JMS Connection, and you're
>>> in
>>> >>> XA Transaction, is it necessary to call connection.close()? It would
>>> seem
>>> >>> JTA should be smart enough to know the connection is enrolled for 2
>>> phase
>>> >>> commit and should be smart enough to close it, but I'm not sure if
>>> that's
>>> >>> part of the spec.
>>> >>>
>>> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection
>>> with
>>> >>> connectionFactory.createConnection(), and your code fails to call
>>> close()
>>> >>> before the transaction completion, the connection leaks. (And
>>> >>> unfortunately, calling close() after the transaction completes
>>> doesn't
>>> >>> mitigate the problem). It took awhile for us to track this down.
>>> >>>
>>> >>> This becomes a huge problem if you're calling external services in
>>> your
>>> >>> transaction. Let's say you have a reasonable transaction timeout of
>>> 30s
>>> >>> set. You call three services, and they end up taking 15s a piece.
>>> Even if
>>> >>> you're doing the right thing and you have connection.close() in a
>>> finally
>>> >>> block, because your transaction isn't active when you call close, it
>>> leaks
>>> >>> and it just gets "stuck" as an active connection, which eventually
>>> you hit
>>> >>> the pool limit and your app freezes.
>>> >>>
>>> >>> On a separate note, we noticed if you open a connection outside of
>>> the
>>> >>> scope of a transaction, then start a transaction, then create a
>>> session
>>> >>> with session_transacted option, the session does not participate in
>>> the JTA
>>> >>> (which seems out of spec). One most open the connection inside the
>>> >>> transaction, AND open the session in the transaction, and close the
>>> >>> connection in the transaction for everything to work.
>>> >>>
>>> >>> I'll get a reproducing project created, but I was curious if anyone
>>> knew
>>> >>> offhand what the spec says.
>>> >>>
>>> >>> cheers, and thanks,
>>> >>> -[the other] Jonathan
>>> >>>
>>> >>> --
>>> >>> Jonathan | [hidden email]
>>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>> as
>>> >>> half full.
>>> >>> Engineers, of course, understand the glass is twice as big as it
>>> needs
>>> >>> to be.
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >> Jonathan | [hidden email]
>>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> >> half full.
>>> >> Engineers, of course, understand the glass is twice as big as it
>>> needs to
>>> >> be.
>>> >>
>>> >
>>> >
>>> > --
>>> > Jonathan | [hidden email]
>>> > Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> > half full.
>>> > Engineers, of course, understand the glass is twice as big as it needs
>>> to
>>> > be.
>>> >
>>>
>>>
>>> --
>>> Jonathan | [hidden email]
>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> half
>>> full.
>>> Engineers, of course, understand the glass is twice as big as it needs to
>>> be.
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
There are actually a few log messages we regularly ignore all the time from
the transaction manager ::wince face:: I'm not sure if we should be
concerned with that one.

On your test, first, how is the broker xml declared? Often something that
trips our newbies up to TomEE is having a persistent broker that is storing
messages between TomEE runs. The tricky thing is that the broker store does
always appear in /target, so it might not get cleaned up with mvn clean
install. As such, for local development we use this
URL: broker:(vm://localhost)?persistent=false&amp;deleteAllMessagesOnStartup=true.
Next, on JMSReceiverBean, you're missing transactionAttribute. Technically
it should work without it, but I'd add it just in case. Finally, I'd add a
small thread.sleep, or check to the queue length == 0 before doing your
assert on messagecounter. It could be you're beating the receiver bean to
the finish line. The default messaging mode is auto-ack, so technically the
message just has to be on the broker, it doesn't have to be processed
before your sender bean will return.





On Mon, Sep 2, 2019 at 2:28 PM Jonathan Gallimore <
[hidden email]> wrote:

> In case it means anything to anyone, the unexpected output I'm getting is
> an abandoned connection notification:
>
> WARNING: Transaction complete, but connection still has handles associated:
> ManagedConnectionInfo:
> org.apache.geronimo.connector.outbound.ManagedConnectionInfo@5b5a4aed. mc:
> [org.apache.openejb.resource.activemq.jms2.TomEEManagedConnection@37e69c43
> ,ActiveMQConnection
>
> {id=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-7:1,clientId=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-6:1,started=false}]]
> Abandoned connection information:
>   Connection handle opened at
>
> org.apache.geronimo.connector.outbound.ConnectionInfo.setTrace(ConnectionInfo.java:119),
>
> org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:57),
>
> org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39),
>
> org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66),
>
> org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:81),
>
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94),
>
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67),
>
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:83),
>
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:99),
>
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createQueue(JMSContextImpl.java:309),
>
> org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createQueue(JMS2CDIExtension.java:371),
>
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:37),
>
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:33),
> sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
>
> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>
> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>
> org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191),
>
> org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102),
> sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
>
> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>
> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>
> org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85),
>
> org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272),
>
> org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221),
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265),
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260),
>
> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89),
>
> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347),
>
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean$$LocalBeanProxy.sendToQueue(org/apache/openejb/arquillian/tests/jms/JMSSenderBean.java),
>
> org.apache.openejb.arquillian.tests.jms.JMSContextInjectionTest.testShouldSendAndReceiveMessages(JMSContextInjectionTest.java:61),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50),
>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12),
>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47),
> org.jboss.arquillian.junit.Arquillian$8$1.invoke(Arquillian.java:379),
>
> org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>
> org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50),
>
> org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:109),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>
> org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>
> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>
> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>
> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>
> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>
> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>
> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:136),
> org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:372),
> org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:246),
> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
> org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:260),
> org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:324),
>
> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99),
>
> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>
> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>
> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>
> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>
> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>
> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> java.lang.reflect.Method.invoke(Method.java:498),
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>
> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159),
> org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:317),
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325),
>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78),
>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57),
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
> org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:205),
> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
> org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219),
> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
> org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167),
> org.junit.runners.Suite.runChild(Suite.java:128),
> org.junit.runners.Suite.runChild(Suite.java:27),
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55),
>
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137),
>
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107),
>
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83),
>
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75),
>
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157),
>
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386),
>
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323),
> org.apache.
>
> Also, I see out by one errors - e.g. if I send 200 messages, sometimes I
> receive 201 or 199. Not sure if its related to the above warning or not.
>
> Jon
>
> On Mon, Sep 2, 2019 at 8:25 PM Jonathan Gallimore <
> [hidden email]> wrote:
>
> > I had a play around with this, and got my previously failing
> > JMSContextInjectionTest to work -- some of the time. (
> >
> https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
> > ).
> >
> > I'm getting some log output that I'm not expecting so I've rolled back my
> > removal of @Ignore so I can dig into it some more.
> >
> > Jon
> >
> > On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
> > [hidden email]> wrote:
> >
> >> Hi Jonathan
> >>
> >> Thanks for this. I think I had run into some of the things you fixed I
> >> little while back. I'll take a look at your change later today. Happy
> for
> >> you to commit and I can review after, too. :)
> >>
> >> Cheers
> >>
> >> The other Jonathan
> >>
> >>
> >> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <[hidden email]>
> >> wrote:
> >>
> >>>
> >>>
> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
> >>>
> >>> With these patches the JMS2.0 API works pretty well now :) If anyone
> >>> wants
> >>> to review, please go ahead, otherwise I'll commit them tomorrow.
> >>>
> >>> cheers,
> >>> - [The other] Jonathan
> >>>
> >>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <[hidden email]
> >
> >>> wrote:
> >>>
> >>> > I've narrowed down the problem to AutoConnectionTracker. It's not
> >>> > completing, which isn't allowing the connections to be returned to
> the
> >>> pool.
> >>> >
> >>> >
> >>>
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
> >>> >
> >>> > getResource() is throwing an IllegalStateException. The JavaDoc (
> >>> >
> >>>
> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
> >>> )
> >>> > states it should throw an ISE if a current transaction is not Active.
> >>> The
> >>> > transaction is in the state ROLLED_BACK when AutoConnectionTracker
> >>> tries to
> >>> > call getResource().
> >>> >
> >>> > I think the Geronimo implementation (
> >>> >
> >>>
> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
> >>> )
> >>> > maybe be a little too strict. The JTA Spec pdf doesn't offer exact
> >>> hints of
> >>> > which statuses (
> >>> > https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
> >>> > should be have getResource _not_ throw an ISE unfortunately. I was
> >>> thinking
> >>> > of changing Geronimo's implementation to check for anything
> >>> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
> >>> >
> >>> > The other way is to cast Transaction to the Geronimo implementation
> and
> >>> > use Geronimo specific APIs to get call getResource(). Do you guys
> have
> >>> any
> >>> > preference which route I should take to fix?
> >>> >
> >>> >
> >>> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
> [hidden email]
> >>> >
> >>> > wrote:
> >>> >
> >>> >>
> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
> >>> >>
> >>> >> Here's a project that reproduces the bug. This project intentionally
> >>> >> exceeds the transaction timeout (of 1s). Each invocation, the
> >>> connection is
> >>> >> not returned to the pool and eventually you run out, causing your
> >>> >> application to freeze.
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
> >>> [hidden email]>
> >>> >> wrote:
> >>> >>
> >>> >>> Hello Apache friends :) I have a question about the JTA and JMS/RA
> >>> specs:
> >>> >>>
> >>> >>> If you borrow something from a RA, like a JMS Connection, and
> you're
> >>> in
> >>> >>> XA Transaction, is it necessary to call connection.close()? It
> would
> >>> seem
> >>> >>> JTA should be smart enough to know the connection is enrolled for 2
> >>> phase
> >>> >>> commit and should be smart enough to close it, but I'm not sure if
> >>> that's
> >>> >>> part of the spec.
> >>> >>>
> >>> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection
> >>> with
> >>> >>> connectionFactory.createConnection(), and your code fails to call
> >>> close()
> >>> >>> before the transaction completion, the connection leaks. (And
> >>> >>> unfortunately, calling close() after the transaction completes
> >>> doesn't
> >>> >>> mitigate the problem). It took awhile for us to track this down.
> >>> >>>
> >>> >>> This becomes a huge problem if you're calling external services in
> >>> your
> >>> >>> transaction. Let's say you have a reasonable transaction timeout of
> >>> 30s
> >>> >>> set. You call three services, and they end up taking 15s a piece.
> >>> Even if
> >>> >>> you're doing the right thing and you have connection.close() in a
> >>> finally
> >>> >>> block, because your transaction isn't active when you call close,
> it
> >>> leaks
> >>> >>> and it just gets "stuck" as an active connection, which eventually
> >>> you hit
> >>> >>> the pool limit and your app freezes.
> >>> >>>
> >>> >>> On a separate note, we noticed if you open a connection outside of
> >>> the
> >>> >>> scope of a transaction, then start a transaction, then create a
> >>> session
> >>> >>> with session_transacted option, the session does not participate in
> >>> the JTA
> >>> >>> (which seems out of spec). One most open the connection inside the
> >>> >>> transaction, AND open the session in the transaction, and close the
> >>> >>> connection in the transaction for everything to work.
> >>> >>>
> >>> >>> I'll get a reproducing project created, but I was curious if anyone
> >>> knew
> >>> >>> offhand what the spec says.
> >>> >>>
> >>> >>> cheers, and thanks,
> >>> >>> -[the other] Jonathan
> >>> >>>
> >>> >>> --
> >>> >>> Jonathan | [hidden email]
> >>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
> >>> as
> >>> >>> half full.
> >>> >>> Engineers, of course, understand the glass is twice as big as it
> >>> needs
> >>> >>> to be.
> >>> >>>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Jonathan | [hidden email]
> >>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see it
> as
> >>> >> half full.
> >>> >> Engineers, of course, understand the glass is twice as big as it
> >>> needs to
> >>> >> be.
> >>> >>
> >>> >
> >>> >
> >>> > --
> >>> > Jonathan | [hidden email]
> >>> > Pessimists, see a jar as half empty. Optimists, in contrast, see it
> as
> >>> > half full.
> >>> > Engineers, of course, understand the glass is twice as big as it
> needs
> >>> to
> >>> > be.
> >>> >
> >>>
> >>>
> >>> --
> >>> Jonathan | [hidden email]
> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> >>> half
> >>> full.
> >>> Engineers, of course, understand the glass is twice as big as it needs
> to
> >>> be.
> >>>
> >>
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
https://github.com/apache/tomee/pull/546/files

This passes consistently for me with no issues

On Tue, Sep 3, 2019 at 9:08 AM Jonathan S. Fisher <[hidden email]>
wrote:

> There are actually a few log messages we regularly ignore all the time
> from the transaction manager ::wince face:: I'm not sure if we should be
> concerned with that one.
>
> On your test, first, how is the broker xml declared? Often something that
> trips our newbies up to TomEE is having a persistent broker that is storing
> messages between TomEE runs. The tricky thing is that the broker store does
> always appear in /target, so it might not get cleaned up with mvn clean
> install. As such, for local development we use this
> URL: broker:(vm://localhost)?persistent=false&amp;deleteAllMessagesOnStartup=true.
> Next, on JMSReceiverBean, you're missing transactionAttribute. Technically
> it should work without it, but I'd add it just in case. Finally, I'd add a
> small thread.sleep, or check to the queue length == 0 before doing your
> assert on messagecounter. It could be you're beating the receiver bean to
> the finish line. The default messaging mode is auto-ack, so technically the
> message just has to be on the broker, it doesn't have to be processed
> before your sender bean will return.
>
>
>
>
>
> On Mon, Sep 2, 2019 at 2:28 PM Jonathan Gallimore <
> [hidden email]> wrote:
>
>> In case it means anything to anyone, the unexpected output I'm getting is
>> an abandoned connection notification:
>>
>> WARNING: Transaction complete, but connection still has handles
>> associated:
>> ManagedConnectionInfo:
>> org.apache.geronimo.connector.outbound.ManagedConnectionInfo@5b5a4aed.
>> mc:
>> [org.apache.openejb.resource.activemq.jms2.TomEEManagedConnection@37e69c43
>> ,ActiveMQConnection
>>
>> {id=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-7:1,clientId=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-6:1,started=false}]]
>> Abandoned connection information:
>>   Connection handle opened at
>>
>> org.apache.geronimo.connector.outbound.ConnectionInfo.setTrace(ConnectionInfo.java:119),
>>
>> org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:57),
>>
>> org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39),
>>
>> org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66),
>>
>> org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:81),
>>
>> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94),
>>
>> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67),
>>
>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:83),
>>
>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:99),
>>
>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createQueue(JMSContextImpl.java:309),
>>
>> org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createQueue(JMS2CDIExtension.java:371),
>>
>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:37),
>>
>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:33),
>> sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>>
>> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>>
>> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>>
>> org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191),
>>
>> org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102),
>> sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>>
>> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>>
>> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>>
>> org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85),
>>
>> org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272),
>>
>> org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221),
>>
>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265),
>>
>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260),
>>
>> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89),
>>
>> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347),
>>
>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean$$LocalBeanProxy.sendToQueue(org/apache/openejb/arquillian/tests/jms/JMSSenderBean.java),
>>
>> org.apache.openejb.arquillian.tests.jms.JMSContextInjectionTest.testShouldSendAndReceiveMessages(JMSContextInjectionTest.java:61),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>>
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50),
>>
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12),
>>
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47),
>> org.jboss.arquillian.junit.Arquillian$8$1.invoke(Arquillian.java:379),
>>
>> org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>>
>> org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50),
>>
>> org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:109),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>>
>> org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>
>> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>>
>> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>
>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>>
>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>
>> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>
>> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>
>> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>
>> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:136),
>> org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:372),
>> org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:246),
>> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
>> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
>> org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:260),
>> org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:324),
>>
>> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99),
>>
>> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>
>> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>>
>> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>
>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>>
>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>
>> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>
>> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>
>> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>> java.lang.reflect.Method.invoke(Method.java:498),
>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>
>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>
>> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159),
>> org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:317),
>> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325),
>>
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78),
>>
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57),
>> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
>> org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:205),
>> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
>> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
>> org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219),
>> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
>> org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167),
>> org.junit.runners.Suite.runChild(Suite.java:128),
>> org.junit.runners.Suite.runChild(Suite.java:27),
>> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
>> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
>> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55),
>>
>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137),
>>
>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107),
>>
>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83),
>>
>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75),
>>
>> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157),
>>
>> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386),
>>
>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323),
>> org.apache.
>>
>> Also, I see out by one errors - e.g. if I send 200 messages, sometimes I
>> receive 201 or 199. Not sure if its related to the above warning or not.
>>
>> Jon
>>
>> On Mon, Sep 2, 2019 at 8:25 PM Jonathan Gallimore <
>> [hidden email]> wrote:
>>
>> > I had a play around with this, and got my previously failing
>> > JMSContextInjectionTest to work -- some of the time. (
>> >
>> https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
>> > ).
>> >
>> > I'm getting some log output that I'm not expecting so I've rolled back
>> my
>> > removal of @Ignore so I can dig into it some more.
>> >
>> > Jon
>> >
>> > On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
>> > [hidden email]> wrote:
>> >
>> >> Hi Jonathan
>> >>
>> >> Thanks for this. I think I had run into some of the things you fixed I
>> >> little while back. I'll take a look at your change later today. Happy
>> for
>> >> you to commit and I can review after, too. :)
>> >>
>> >> Cheers
>> >>
>> >> The other Jonathan
>> >>
>> >>
>> >> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <[hidden email]
>> >
>> >> wrote:
>> >>
>> >>>
>> >>>
>> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
>> >>>
>> >>> With these patches the JMS2.0 API works pretty well now :) If anyone
>> >>> wants
>> >>> to review, please go ahead, otherwise I'll commit them tomorrow.
>> >>>
>> >>> cheers,
>> >>> - [The other] Jonathan
>> >>>
>> >>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <
>> [hidden email]>
>> >>> wrote:
>> >>>
>> >>> > I've narrowed down the problem to AutoConnectionTracker. It's not
>> >>> > completing, which isn't allowing the connections to be returned to
>> the
>> >>> pool.
>> >>> >
>> >>> >
>> >>>
>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>> >>> >
>> >>> > getResource() is throwing an IllegalStateException. The JavaDoc (
>> >>> >
>> >>>
>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>> >>> )
>> >>> > states it should throw an ISE if a current transaction is not
>> Active.
>> >>> The
>> >>> > transaction is in the state ROLLED_BACK when AutoConnectionTracker
>> >>> tries to
>> >>> > call getResource().
>> >>> >
>> >>> > I think the Geronimo implementation (
>> >>> >
>> >>>
>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>> >>> )
>> >>> > maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>> >>> hints of
>> >>> > which statuses (
>> >>> > https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
>> >>> > should be have getResource _not_ throw an ISE unfortunately. I was
>> >>> thinking
>> >>> > of changing Geronimo's implementation to check for anything
>> >>> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>> >>> >
>> >>> > The other way is to cast Transaction to the Geronimo implementation
>> and
>> >>> > use Geronimo specific APIs to get call getResource(). Do you guys
>> have
>> >>> any
>> >>> > preference which route I should take to fix?
>> >>> >
>> >>> >
>> >>> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
>> [hidden email]
>> >>> >
>> >>> > wrote:
>> >>> >
>> >>> >>
>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>> >>> >>
>> >>> >> Here's a project that reproduces the bug. This project
>> intentionally
>> >>> >> exceeds the transaction timeout (of 1s). Each invocation, the
>> >>> connection is
>> >>> >> not returned to the pool and eventually you run out, causing your
>> >>> >> application to freeze.
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
>> >>> [hidden email]>
>> >>> >> wrote:
>> >>> >>
>> >>> >>> Hello Apache friends :) I have a question about the JTA and JMS/RA
>> >>> specs:
>> >>> >>>
>> >>> >>> If you borrow something from a RA, like a JMS Connection, and
>> you're
>> >>> in
>> >>> >>> XA Transaction, is it necessary to call connection.close()? It
>> would
>> >>> seem
>> >>> >>> JTA should be smart enough to know the connection is enrolled for
>> 2
>> >>> phase
>> >>> >>> commit and should be smart enough to close it, but I'm not sure if
>> >>> that's
>> >>> >>> part of the spec.
>> >>> >>>
>> >>> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection
>> >>> with
>> >>> >>> connectionFactory.createConnection(), and your code fails to call
>> >>> close()
>> >>> >>> before the transaction completion, the connection leaks. (And
>> >>> >>> unfortunately, calling close() after the transaction completes
>> >>> doesn't
>> >>> >>> mitigate the problem). It took awhile for us to track this down.
>> >>> >>>
>> >>> >>> This becomes a huge problem if you're calling external services in
>> >>> your
>> >>> >>> transaction. Let's say you have a reasonable transaction timeout
>> of
>> >>> 30s
>> >>> >>> set. You call three services, and they end up taking 15s a piece.
>> >>> Even if
>> >>> >>> you're doing the right thing and you have connection.close() in a
>> >>> finally
>> >>> >>> block, because your transaction isn't active when you call close,
>> it
>> >>> leaks
>> >>> >>> and it just gets "stuck" as an active connection, which eventually
>> >>> you hit
>> >>> >>> the pool limit and your app freezes.
>> >>> >>>
>> >>> >>> On a separate note, we noticed if you open a connection outside of
>> >>> the
>> >>> >>> scope of a transaction, then start a transaction, then create a
>> >>> session
>> >>> >>> with session_transacted option, the session does not participate
>> in
>> >>> the JTA
>> >>> >>> (which seems out of spec). One most open the connection inside the
>> >>> >>> transaction, AND open the session in the transaction, and close
>> the
>> >>> >>> connection in the transaction for everything to work.
>> >>> >>>
>> >>> >>> I'll get a reproducing project created, but I was curious if
>> anyone
>> >>> knew
>> >>> >>> offhand what the spec says.
>> >>> >>>
>> >>> >>> cheers, and thanks,
>> >>> >>> -[the other] Jonathan
>> >>> >>>
>> >>> >>> --
>> >>> >>> Jonathan | [hidden email]
>> >>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see
>> it
>> >>> as
>> >>> >>> half full.
>> >>> >>> Engineers, of course, understand the glass is twice as big as it
>> >>> needs
>> >>> >>> to be.
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Jonathan | [hidden email]
>> >>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see
>> it as
>> >>> >> half full.
>> >>> >> Engineers, of course, understand the glass is twice as big as it
>> >>> needs to
>> >>> >> be.
>> >>> >>
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Jonathan | [hidden email]
>> >>> > Pessimists, see a jar as half empty. Optimists, in contrast, see it
>> as
>> >>> > half full.
>> >>> > Engineers, of course, understand the glass is twice as big as it
>> needs
>> >>> to
>> >>> > be.
>> >>> >
>> >>>
>> >>>
>> >>> --
>> >>> Jonathan | [hidden email]
>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> >>> half
>> >>> full.
>> >>> Engineers, of course, understand the glass is twice as big as it
>> needs to
>> >>> be.
>> >>>
>> >>
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
If I bump the number of messages up to 10k or so I get a VM Crash... I
cannot figure out how to get arquillian to give me a heap dump on exit
though.

On Tue, Sep 3, 2019 at 10:01 AM Jonathan S. Fisher <[hidden email]>
wrote:

> https://github.com/apache/tomee/pull/546/files
>
> This passes consistently for me with no issues
>
> On Tue, Sep 3, 2019 at 9:08 AM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>> There are actually a few log messages we regularly ignore all the time
>> from the transaction manager ::wince face:: I'm not sure if we should be
>> concerned with that one.
>>
>> On your test, first, how is the broker xml declared? Often something that
>> trips our newbies up to TomEE is having a persistent broker that is storing
>> messages between TomEE runs. The tricky thing is that the broker store does
>> always appear in /target, so it might not get cleaned up with mvn clean
>> install. As such, for local development we use this
>> URL: broker:(vm://localhost)?persistent=false&amp;deleteAllMessagesOnStartup=true.
>> Next, on JMSReceiverBean, you're missing transactionAttribute. Technically
>> it should work without it, but I'd add it just in case. Finally, I'd add a
>> small thread.sleep, or check to the queue length == 0 before doing your
>> assert on messagecounter. It could be you're beating the receiver bean to
>> the finish line. The default messaging mode is auto-ack, so technically the
>> message just has to be on the broker, it doesn't have to be processed
>> before your sender bean will return.
>>
>>
>>
>>
>>
>> On Mon, Sep 2, 2019 at 2:28 PM Jonathan Gallimore <
>> [hidden email]> wrote:
>>
>>> In case it means anything to anyone, the unexpected output I'm getting is
>>> an abandoned connection notification:
>>>
>>> WARNING: Transaction complete, but connection still has handles
>>> associated:
>>> ManagedConnectionInfo:
>>> org.apache.geronimo.connector.outbound.ManagedConnectionInfo@5b5a4aed.
>>> mc:
>>>
>>> [org.apache.openejb.resource.activemq.jms2.TomEEManagedConnection@37e69c43
>>> ,ActiveMQConnection
>>>
>>> {id=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-7:1,clientId=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-6:1,started=false}]]
>>> Abandoned connection information:
>>>   Connection handle opened at
>>>
>>> org.apache.geronimo.connector.outbound.ConnectionInfo.setTrace(ConnectionInfo.java:119),
>>>
>>> org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:57),
>>>
>>> org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39),
>>>
>>> org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66),
>>>
>>> org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:81),
>>>
>>> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94),
>>>
>>> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67),
>>>
>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:83),
>>>
>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:99),
>>>
>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createQueue(JMSContextImpl.java:309),
>>>
>>> org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createQueue(JMS2CDIExtension.java:371),
>>>
>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:37),
>>>
>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:33),
>>> sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>
>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>>>
>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>>>
>>> org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191),
>>>
>>> org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102),
>>> sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>
>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>>>
>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>>>
>>> org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85),
>>>
>>> org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272),
>>>
>>> org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221),
>>>
>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265),
>>>
>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260),
>>>
>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89),
>>>
>>> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347),
>>>
>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean$$LocalBeanProxy.sendToQueue(org/apache/openejb/arquillian/tests/jms/JMSSenderBean.java),
>>>
>>> org.apache.openejb.arquillian.tests.jms.JMSContextInjectionTest.testShouldSendAndReceiveMessages(JMSContextInjectionTest.java:61),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>
>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50),
>>>
>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12),
>>>
>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47),
>>> org.jboss.arquillian.junit.Arquillian$8$1.invoke(Arquillian.java:379),
>>>
>>> org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>>>
>>> org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50),
>>>
>>> org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:109),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>>>
>>> org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>
>>> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>>>
>>> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>
>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>>>
>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>
>>> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>
>>> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>
>>> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>
>>> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:136),
>>> org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:372),
>>> org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:246),
>>> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
>>> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
>>> org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:260),
>>> org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:324),
>>>
>>> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99),
>>>
>>> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>
>>> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>>>
>>> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>
>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>>>
>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>
>>> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>
>>> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>
>>> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>> java.lang.reflect.Method.invoke(Method.java:498),
>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>
>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>>
>>> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159),
>>> org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:317),
>>> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325),
>>>
>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78),
>>>
>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57),
>>> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
>>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
>>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
>>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
>>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
>>> org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:205),
>>> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
>>> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
>>> org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219),
>>> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
>>> org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167),
>>> org.junit.runners.Suite.runChild(Suite.java:128),
>>> org.junit.runners.Suite.runChild(Suite.java:27),
>>> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
>>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
>>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
>>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
>>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
>>> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
>>> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55),
>>>
>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137),
>>>
>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107),
>>>
>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83),
>>>
>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75),
>>>
>>> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157),
>>>
>>> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386),
>>>
>>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323),
>>> org.apache.
>>>
>>> Also, I see out by one errors - e.g. if I send 200 messages, sometimes I
>>> receive 201 or 199. Not sure if its related to the above warning or not.
>>>
>>> Jon
>>>
>>> On Mon, Sep 2, 2019 at 8:25 PM Jonathan Gallimore <
>>> [hidden email]> wrote:
>>>
>>> > I had a play around with this, and got my previously failing
>>> > JMSContextInjectionTest to work -- some of the time. (
>>> >
>>> https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
>>> > ).
>>> >
>>> > I'm getting some log output that I'm not expecting so I've rolled back
>>> my
>>> > removal of @Ignore so I can dig into it some more.
>>> >
>>> > Jon
>>> >
>>> > On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
>>> > [hidden email]> wrote:
>>> >
>>> >> Hi Jonathan
>>> >>
>>> >> Thanks for this. I think I had run into some of the things you fixed I
>>> >> little while back. I'll take a look at your change later today. Happy
>>> for
>>> >> you to commit and I can review after, too. :)
>>> >>
>>> >> Cheers
>>> >>
>>> >> The other Jonathan
>>> >>
>>> >>
>>> >> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <
>>> [hidden email]>
>>> >> wrote:
>>> >>
>>> >>>
>>> >>>
>>> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
>>> >>>
>>> >>> With these patches the JMS2.0 API works pretty well now :) If anyone
>>> >>> wants
>>> >>> to review, please go ahead, otherwise I'll commit them tomorrow.
>>> >>>
>>> >>> cheers,
>>> >>> - [The other] Jonathan
>>> >>>
>>> >>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <
>>> [hidden email]>
>>> >>> wrote:
>>> >>>
>>> >>> > I've narrowed down the problem to AutoConnectionTracker. It's not
>>> >>> > completing, which isn't allowing the connections to be returned to
>>> the
>>> >>> pool.
>>> >>> >
>>> >>> >
>>> >>>
>>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>>> >>> >
>>> >>> > getResource() is throwing an IllegalStateException. The JavaDoc (
>>> >>> >
>>> >>>
>>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>>> >>> )
>>> >>> > states it should throw an ISE if a current transaction is not
>>> Active.
>>> >>> The
>>> >>> > transaction is in the state ROLLED_BACK when AutoConnectionTracker
>>> >>> tries to
>>> >>> > call getResource().
>>> >>> >
>>> >>> > I think the Geronimo implementation (
>>> >>> >
>>> >>>
>>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>>> >>> )
>>> >>> > maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>>> >>> hints of
>>> >>> > which statuses (
>>> >>> > https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html
>>> )
>>> >>> > should be have getResource _not_ throw an ISE unfortunately. I was
>>> >>> thinking
>>> >>> > of changing Geronimo's implementation to check for anything
>>> >>> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>>> >>> >
>>> >>> > The other way is to cast Transaction to the Geronimo
>>> implementation and
>>> >>> > use Geronimo specific APIs to get call getResource(). Do you guys
>>> have
>>> >>> any
>>> >>> > preference which route I should take to fix?
>>> >>> >
>>> >>> >
>>> >>> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
>>> [hidden email]
>>> >>> >
>>> >>> > wrote:
>>> >>> >
>>> >>> >>
>>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>> >>> >>
>>> >>> >> Here's a project that reproduces the bug. This project
>>> intentionally
>>> >>> >> exceeds the transaction timeout (of 1s). Each invocation, the
>>> >>> connection is
>>> >>> >> not returned to the pool and eventually you run out, causing your
>>> >>> >> application to freeze.
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
>>> >>> [hidden email]>
>>> >>> >> wrote:
>>> >>> >>
>>> >>> >>> Hello Apache friends :) I have a question about the JTA and
>>> JMS/RA
>>> >>> specs:
>>> >>> >>>
>>> >>> >>> If you borrow something from a RA, like a JMS Connection, and
>>> you're
>>> >>> in
>>> >>> >>> XA Transaction, is it necessary to call connection.close()? It
>>> would
>>> >>> seem
>>> >>> >>> JTA should be smart enough to know the connection is enrolled
>>> for 2
>>> >>> phase
>>> >>> >>> commit and should be smart enough to close it, but I'm not sure
>>> if
>>> >>> that's
>>> >>> >>> part of the spec.
>>> >>> >>>
>>> >>> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection
>>> >>> with
>>> >>> >>> connectionFactory.createConnection(), and your code fails to call
>>> >>> close()
>>> >>> >>> before the transaction completion, the connection leaks. (And
>>> >>> >>> unfortunately, calling close() after the transaction completes
>>> >>> doesn't
>>> >>> >>> mitigate the problem). It took awhile for us to track this down.
>>> >>> >>>
>>> >>> >>> This becomes a huge problem if you're calling external services
>>> in
>>> >>> your
>>> >>> >>> transaction. Let's say you have a reasonable transaction timeout
>>> of
>>> >>> 30s
>>> >>> >>> set. You call three services, and they end up taking 15s a piece.
>>> >>> Even if
>>> >>> >>> you're doing the right thing and you have connection.close() in a
>>> >>> finally
>>> >>> >>> block, because your transaction isn't active when you call
>>> close, it
>>> >>> leaks
>>> >>> >>> and it just gets "stuck" as an active connection, which
>>> eventually
>>> >>> you hit
>>> >>> >>> the pool limit and your app freezes.
>>> >>> >>>
>>> >>> >>> On a separate note, we noticed if you open a connection outside
>>> of
>>> >>> the
>>> >>> >>> scope of a transaction, then start a transaction, then create a
>>> >>> session
>>> >>> >>> with session_transacted option, the session does not participate
>>> in
>>> >>> the JTA
>>> >>> >>> (which seems out of spec). One most open the connection inside
>>> the
>>> >>> >>> transaction, AND open the session in the transaction, and close
>>> the
>>> >>> >>> connection in the transaction for everything to work.
>>> >>> >>>
>>> >>> >>> I'll get a reproducing project created, but I was curious if
>>> anyone
>>> >>> knew
>>> >>> >>> offhand what the spec says.
>>> >>> >>>
>>> >>> >>> cheers, and thanks,
>>> >>> >>> -[the other] Jonathan
>>> >>> >>>
>>> >>> >>> --
>>> >>> >>> Jonathan | [hidden email]
>>> >>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see
>>> it
>>> >>> as
>>> >>> >>> half full.
>>> >>> >>> Engineers, of course, understand the glass is twice as big as it
>>> >>> needs
>>> >>> >>> to be.
>>> >>> >>>
>>> >>> >>
>>> >>> >>
>>> >>> >> --
>>> >>> >> Jonathan | [hidden email]
>>> >>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see
>>> it as
>>> >>> >> half full.
>>> >>> >> Engineers, of course, understand the glass is twice as big as it
>>> >>> needs to
>>> >>> >> be.
>>> >>> >>
>>> >>> >
>>> >>> >
>>> >>> > --
>>> >>> > Jonathan | [hidden email]
>>> >>> > Pessimists, see a jar as half empty. Optimists, in contrast, see
>>> it as
>>> >>> > half full.
>>> >>> > Engineers, of course, understand the glass is twice as big as it
>>> needs
>>> >>> to
>>> >>> > be.
>>> >>> >
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Jonathan | [hidden email]
>>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>> as
>>> >>> half
>>> >>> full.
>>> >>> Engineers, of course, understand the glass is twice as big as it
>>> needs to
>>> >>> be.
>>> >>>
>>> >>
>>>
>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
Two more updates:

For the log message, it looks like beforeCompletion() Is being called
before handleReleased(), leading to that warning. I ran a couple thousand
messages through and took a heap dump and didn't get any leak suspects, so
I think it's working correctly despite the warning. I'll add that to my
existing PR with your tests.

Second, on my 10K messages test, it looks like it might be Arquillian
crashing, not TomEE. I'm going to try the same test but without Arquillian,
just deploy those classes to a server and see what happens.

On Tue, Sep 3, 2019 at 10:54 AM Jonathan S. Fisher <[hidden email]>
wrote:

> If I bump the number of messages up to 10k or so I get a VM Crash... I
> cannot figure out how to get arquillian to give me a heap dump on exit
> though.
>
> On Tue, Sep 3, 2019 at 10:01 AM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>> https://github.com/apache/tomee/pull/546/files
>>
>> This passes consistently for me with no issues
>>
>> On Tue, Sep 3, 2019 at 9:08 AM Jonathan S. Fisher <[hidden email]>
>> wrote:
>>
>>> There are actually a few log messages we regularly ignore all the time
>>> from the transaction manager ::wince face:: I'm not sure if we should be
>>> concerned with that one.
>>>
>>> On your test, first, how is the broker xml declared? Often something
>>> that trips our newbies up to TomEE is having a persistent broker that is
>>> storing messages between TomEE runs. The tricky thing is that the broker
>>> store does always appear in /target, so it might not get cleaned up with
>>> mvn clean install. As such, for local development we use this
>>> URL: broker:(vm://localhost)?persistent=false&amp;deleteAllMessagesOnStartup=true.
>>> Next, on JMSReceiverBean, you're missing transactionAttribute. Technically
>>> it should work without it, but I'd add it just in case. Finally, I'd add a
>>> small thread.sleep, or check to the queue length == 0 before doing your
>>> assert on messagecounter. It could be you're beating the receiver bean to
>>> the finish line. The default messaging mode is auto-ack, so technically the
>>> message just has to be on the broker, it doesn't have to be processed
>>> before your sender bean will return.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Sep 2, 2019 at 2:28 PM Jonathan Gallimore <
>>> [hidden email]> wrote:
>>>
>>>> In case it means anything to anyone, the unexpected output I'm getting
>>>> is
>>>> an abandoned connection notification:
>>>>
>>>> WARNING: Transaction complete, but connection still has handles
>>>> associated:
>>>> ManagedConnectionInfo:
>>>> org.apache.geronimo.connector.outbound.ManagedConnectionInfo@5b5a4aed.
>>>> mc:
>>>>
>>>> [org.apache.openejb.resource.activemq.jms2.TomEEManagedConnection@37e69c43
>>>> ,ActiveMQConnection
>>>>
>>>> {id=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-7:1,clientId=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-6:1,started=false}]]
>>>> Abandoned connection information:
>>>>   Connection handle opened at
>>>>
>>>> org.apache.geronimo.connector.outbound.ConnectionInfo.setTrace(ConnectionInfo.java:119),
>>>>
>>>> org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:57),
>>>>
>>>> org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39),
>>>>
>>>> org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66),
>>>>
>>>> org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:81),
>>>>
>>>> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94),
>>>>
>>>> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67),
>>>>
>>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:83),
>>>>
>>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:99),
>>>>
>>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createQueue(JMSContextImpl.java:309),
>>>>
>>>> org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createQueue(JMS2CDIExtension.java:371),
>>>>
>>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:37),
>>>>
>>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:33),
>>>> sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>>>>
>>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>>>>
>>>> org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191),
>>>>
>>>> org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102),
>>>> sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>>>>
>>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>>>>
>>>> org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85),
>>>>
>>>> org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272),
>>>>
>>>> org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221),
>>>>
>>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265),
>>>>
>>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260),
>>>>
>>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89),
>>>>
>>>> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347),
>>>>
>>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean$$LocalBeanProxy.sendToQueue(org/apache/openejb/arquillian/tests/jms/JMSSenderBean.java),
>>>>
>>>> org.apache.openejb.arquillian.tests.jms.JMSContextInjectionTest.testShouldSendAndReceiveMessages(JMSContextInjectionTest.java:61),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50),
>>>>
>>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12),
>>>>
>>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47),
>>>> org.jboss.arquillian.junit.Arquillian$8$1.invoke(Arquillian.java:379),
>>>>
>>>> org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>>> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>>>>
>>>> org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50),
>>>>
>>>> org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:109),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>>> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>>>>
>>>> org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>>
>>>> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>>>>
>>>> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>
>>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>>>>
>>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>
>>>> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>
>>>> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>
>>>> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>>
>>>> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:136),
>>>> org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:372),
>>>> org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:246),
>>>> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
>>>> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
>>>> org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:260),
>>>> org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:324),
>>>>
>>>> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99),
>>>>
>>>> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>>
>>>> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>>>>
>>>> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>
>>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>>>>
>>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>
>>>> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>
>>>> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>
>>>> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>
>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>
>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>>>
>>>> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159),
>>>> org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:317),
>>>> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325),
>>>>
>>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78),
>>>>
>>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57),
>>>> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
>>>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
>>>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
>>>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
>>>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
>>>> org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:205),
>>>> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
>>>> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
>>>> org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219),
>>>> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
>>>> org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167),
>>>> org.junit.runners.Suite.runChild(Suite.java:128),
>>>> org.junit.runners.Suite.runChild(Suite.java:27),
>>>> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
>>>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
>>>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
>>>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
>>>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
>>>> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
>>>> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55),
>>>>
>>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137),
>>>>
>>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107),
>>>>
>>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83),
>>>>
>>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75),
>>>>
>>>> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157),
>>>>
>>>> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386),
>>>>
>>>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323),
>>>> org.apache.
>>>>
>>>> Also, I see out by one errors - e.g. if I send 200 messages, sometimes I
>>>> receive 201 or 199. Not sure if its related to the above warning or not.
>>>>
>>>> Jon
>>>>
>>>> On Mon, Sep 2, 2019 at 8:25 PM Jonathan Gallimore <
>>>> [hidden email]> wrote:
>>>>
>>>> > I had a play around with this, and got my previously failing
>>>> > JMSContextInjectionTest to work -- some of the time. (
>>>> >
>>>> https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
>>>> > ).
>>>> >
>>>> > I'm getting some log output that I'm not expecting so I've rolled
>>>> back my
>>>> > removal of @Ignore so I can dig into it some more.
>>>> >
>>>> > Jon
>>>> >
>>>> > On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
>>>> > [hidden email]> wrote:
>>>> >
>>>> >> Hi Jonathan
>>>> >>
>>>> >> Thanks for this. I think I had run into some of the things you fixed
>>>> I
>>>> >> little while back. I'll take a look at your change later today.
>>>> Happy for
>>>> >> you to commit and I can review after, too. :)
>>>> >>
>>>> >> Cheers
>>>> >>
>>>> >> The other Jonathan
>>>> >>
>>>> >>
>>>> >> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <
>>>> [hidden email]>
>>>> >> wrote:
>>>> >>
>>>> >>>
>>>> >>>
>>>> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
>>>> >>>
>>>> >>> With these patches the JMS2.0 API works pretty well now :) If anyone
>>>> >>> wants
>>>> >>> to review, please go ahead, otherwise I'll commit them tomorrow.
>>>> >>>
>>>> >>> cheers,
>>>> >>> - [The other] Jonathan
>>>> >>>
>>>> >>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <
>>>> [hidden email]>
>>>> >>> wrote:
>>>> >>>
>>>> >>> > I've narrowed down the problem to AutoConnectionTracker. It's not
>>>> >>> > completing, which isn't allowing the connections to be returned
>>>> to the
>>>> >>> pool.
>>>> >>> >
>>>> >>> >
>>>> >>>
>>>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>>>> >>> >
>>>> >>> > getResource() is throwing an IllegalStateException. The JavaDoc (
>>>> >>> >
>>>> >>>
>>>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>>>> >>> )
>>>> >>> > states it should throw an ISE if a current transaction is not
>>>> Active.
>>>> >>> The
>>>> >>> > transaction is in the state ROLLED_BACK when AutoConnectionTracker
>>>> >>> tries to
>>>> >>> > call getResource().
>>>> >>> >
>>>> >>> > I think the Geronimo implementation (
>>>> >>> >
>>>> >>>
>>>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>>>> >>> )
>>>> >>> > maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>>>> >>> hints of
>>>> >>> > which statuses (
>>>> >>> >
>>>> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
>>>> >>> > should be have getResource _not_ throw an ISE unfortunately. I was
>>>> >>> thinking
>>>> >>> > of changing Geronimo's implementation to check for anything
>>>> >>> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>>>> >>> >
>>>> >>> > The other way is to cast Transaction to the Geronimo
>>>> implementation and
>>>> >>> > use Geronimo specific APIs to get call getResource(). Do you guys
>>>> have
>>>> >>> any
>>>> >>> > preference which route I should take to fix?
>>>> >>> >
>>>> >>> >
>>>> >>> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
>>>> [hidden email]
>>>> >>> >
>>>> >>> > wrote:
>>>> >>> >
>>>> >>> >>
>>>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>>> >>> >>
>>>> >>> >> Here's a project that reproduces the bug. This project
>>>> intentionally
>>>> >>> >> exceeds the transaction timeout (of 1s). Each invocation, the
>>>> >>> connection is
>>>> >>> >> not returned to the pool and eventually you run out, causing your
>>>> >>> >> application to freeze.
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
>>>> >>> [hidden email]>
>>>> >>> >> wrote:
>>>> >>> >>
>>>> >>> >>> Hello Apache friends :) I have a question about the JTA and
>>>> JMS/RA
>>>> >>> specs:
>>>> >>> >>>
>>>> >>> >>> If you borrow something from a RA, like a JMS Connection, and
>>>> you're
>>>> >>> in
>>>> >>> >>> XA Transaction, is it necessary to call connection.close()? It
>>>> would
>>>> >>> seem
>>>> >>> >>> JTA should be smart enough to know the connection is enrolled
>>>> for 2
>>>> >>> phase
>>>> >>> >>> commit and should be smart enough to close it, but I'm not sure
>>>> if
>>>> >>> that's
>>>> >>> >>> part of the spec.
>>>> >>> >>>
>>>> >>> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS
>>>> Connection
>>>> >>> with
>>>> >>> >>> connectionFactory.createConnection(), and your code fails to
>>>> call
>>>> >>> close()
>>>> >>> >>> before the transaction completion, the connection leaks. (And
>>>> >>> >>> unfortunately, calling close() after the transaction completes
>>>> >>> doesn't
>>>> >>> >>> mitigate the problem). It took awhile for us to track this down.
>>>> >>> >>>
>>>> >>> >>> This becomes a huge problem if you're calling external services
>>>> in
>>>> >>> your
>>>> >>> >>> transaction. Let's say you have a reasonable transaction
>>>> timeout of
>>>> >>> 30s
>>>> >>> >>> set. You call three services, and they end up taking 15s a
>>>> piece.
>>>> >>> Even if
>>>> >>> >>> you're doing the right thing and you have connection.close() in
>>>> a
>>>> >>> finally
>>>> >>> >>> block, because your transaction isn't active when you call
>>>> close, it
>>>> >>> leaks
>>>> >>> >>> and it just gets "stuck" as an active connection, which
>>>> eventually
>>>> >>> you hit
>>>> >>> >>> the pool limit and your app freezes.
>>>> >>> >>>
>>>> >>> >>> On a separate note, we noticed if you open a connection outside
>>>> of
>>>> >>> the
>>>> >>> >>> scope of a transaction, then start a transaction, then create a
>>>> >>> session
>>>> >>> >>> with session_transacted option, the session does not
>>>> participate in
>>>> >>> the JTA
>>>> >>> >>> (which seems out of spec). One most open the connection inside
>>>> the
>>>> >>> >>> transaction, AND open the session in the transaction, and close
>>>> the
>>>> >>> >>> connection in the transaction for everything to work.
>>>> >>> >>>
>>>> >>> >>> I'll get a reproducing project created, but I was curious if
>>>> anyone
>>>> >>> knew
>>>> >>> >>> offhand what the spec says.
>>>> >>> >>>
>>>> >>> >>> cheers, and thanks,
>>>> >>> >>> -[the other] Jonathan
>>>> >>> >>>
>>>> >>> >>> --
>>>> >>> >>> Jonathan | [hidden email]
>>>> >>> >>> Pessimists, see a jar as half empty. Optimists, in contrast,
>>>> see it
>>>> >>> as
>>>> >>> >>> half full.
>>>> >>> >>> Engineers, of course, understand the glass is twice as big as it
>>>> >>> needs
>>>> >>> >>> to be.
>>>> >>> >>>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> --
>>>> >>> >> Jonathan | [hidden email]
>>>> >>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see
>>>> it as
>>>> >>> >> half full.
>>>> >>> >> Engineers, of course, understand the glass is twice as big as it
>>>> >>> needs to
>>>> >>> >> be.
>>>> >>> >>
>>>> >>> >
>>>> >>> >
>>>> >>> > --
>>>> >>> > Jonathan | [hidden email]
>>>> >>> > Pessimists, see a jar as half empty. Optimists, in contrast, see
>>>> it as
>>>> >>> > half full.
>>>> >>> > Engineers, of course, understand the glass is twice as big as it
>>>> needs
>>>> >>> to
>>>> >>> > be.
>>>> >>> >
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Jonathan | [hidden email]
>>>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>>> as
>>>> >>> half
>>>> >>> full.
>>>> >>> Engineers, of course, understand the glass is twice as big as it
>>>> needs to
>>>> >>> be.
>>>> >>>
>>>> >>
>>>>
>>>
>>>
>>> --
>>> Jonathan | [hidden email]
>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> half full.
>>> Engineers, of course, understand the glass is twice as big as it needs
>>> to be.
>>>
>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

jgallimore
In reply to this post by exabrial12
I knew you'd know some magic to help with this - thank you! Just looking at
your PR now. Will give it a test shortly, but it looks good to me.

Jon

On Tue, Sep 3, 2019 at 3:08 PM Jonathan S. Fisher <[hidden email]>
wrote:

> There are actually a few log messages we regularly ignore all the time from
> the transaction manager ::wince face:: I'm not sure if we should be
> concerned with that one.
>
> On your test, first, how is the broker xml declared? Often something that
> trips our newbies up to TomEE is having a persistent broker that is storing
> messages between TomEE runs. The tricky thing is that the broker store does
> always appear in /target, so it might not get cleaned up with mvn clean
> install. As such, for local development we use this
> URL:
> broker:(vm://localhost)?persistent=false&amp;deleteAllMessagesOnStartup=true.
> Next, on JMSReceiverBean, you're missing transactionAttribute. Technically
> it should work without it, but I'd add it just in case. Finally, I'd add a
> small thread.sleep, or check to the queue length == 0 before doing your
> assert on messagecounter. It could be you're beating the receiver bean to
> the finish line. The default messaging mode is auto-ack, so technically the
> message just has to be on the broker, it doesn't have to be processed
> before your sender bean will return.
>
>
>
>
>
> On Mon, Sep 2, 2019 at 2:28 PM Jonathan Gallimore <
> [hidden email]> wrote:
>
> > In case it means anything to anyone, the unexpected output I'm getting is
> > an abandoned connection notification:
> >
> > WARNING: Transaction complete, but connection still has handles
> associated:
> > ManagedConnectionInfo:
> > org.apache.geronimo.connector.outbound.ManagedConnectionInfo@5b5a4aed.
> mc:
> >
> [org.apache.openejb.resource.activemq.jms2.TomEEManagedConnection@37e69c43
> > ,ActiveMQConnection
> >
> >
> {id=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-7:1,clientId=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-6:1,started=false}]]
> > Abandoned connection information:
> >   Connection handle opened at
> >
> >
> org.apache.geronimo.connector.outbound.ConnectionInfo.setTrace(ConnectionInfo.java:119),
> >
> >
> org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:57),
> >
> >
> org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39),
> >
> >
> org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66),
> >
> >
> org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:81),
> >
> >
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94),
> >
> >
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67),
> >
> >
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:83),
> >
> >
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:99),
> >
> >
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createQueue(JMSContextImpl.java:309),
> >
> >
> org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createQueue(JMS2CDIExtension.java:371),
> >
> >
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:37),
> >
> >
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:33),
> > sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> >
> >
> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
> >
> >
> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
> >
> >
> org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191),
> >
> >
> org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102),
> > sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> >
> >
> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
> >
> >
> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
> >
> >
> org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85),
> >
> >
> org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272),
> >
> >
> org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221),
> >
> >
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265),
> >
> >
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260),
> >
> >
> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89),
> >
> >
> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347),
> >
> >
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean$$LocalBeanProxy.sendToQueue(org/apache/openejb/arquillian/tests/jms/JMSSenderBean.java),
> >
> >
> org.apache.openejb.arquillian.tests.jms.JMSContextInjectionTest.testShouldSendAndReceiveMessages(JMSContextInjectionTest.java:61),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> >
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50),
> >
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12),
> >
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47),
> > org.jboss.arquillian.junit.Arquillian$8$1.invoke(Arquillian.java:379),
> >
> >
> org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
> > org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
> >
> >
> org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50),
> >
> >
> org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:109),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
> > org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
> >
> >
> org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> >
> >
> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
> >
> >
> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> >
> >
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
> >
> >
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> >
> >
> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:136),
> > org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:372),
> > org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:246),
> > org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
> > org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
> > org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:260),
> > org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:324),
> >
> >
> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99),
> >
> >
> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> >
> >
> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
> >
> >
> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> >
> >
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
> >
> >
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > java.lang.reflect.Method.invoke(Method.java:498),
> > org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
> >
> >
> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159),
> > org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:317),
> > org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325),
> >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78),
> >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57),
> > org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
> > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
> > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
> > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
> > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
> > org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:205),
> > org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
> > org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
> > org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219),
> > org.junit.runners.ParentRunner.run(ParentRunner.java:363),
> > org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167),
> > org.junit.runners.Suite.runChild(Suite.java:128),
> > org.junit.runners.Suite.runChild(Suite.java:27),
> > org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
> > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
> > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
> > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
> > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
> > org.junit.runners.ParentRunner.run(ParentRunner.java:363),
> > org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55),
> >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137),
> >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107),
> >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83),
> >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75),
> >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157),
> >
> >
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386),
> >
> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323),
> > org.apache.
> >
> > Also, I see out by one errors - e.g. if I send 200 messages, sometimes I
> > receive 201 or 199. Not sure if its related to the above warning or not.
> >
> > Jon
> >
> > On Mon, Sep 2, 2019 at 8:25 PM Jonathan Gallimore <
> > [hidden email]> wrote:
> >
> > > I had a play around with this, and got my previously failing
> > > JMSContextInjectionTest to work -- some of the time. (
> > >
> >
> https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
> > > ).
> > >
> > > I'm getting some log output that I'm not expecting so I've rolled back
> my
> > > removal of @Ignore so I can dig into it some more.
> > >
> > > Jon
> > >
> > > On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
> > > [hidden email]> wrote:
> > >
> > >> Hi Jonathan
> > >>
> > >> Thanks for this. I think I had run into some of the things you fixed I
> > >> little while back. I'll take a look at your change later today. Happy
> > for
> > >> you to commit and I can review after, too. :)
> > >>
> > >> Cheers
> > >>
> > >> The other Jonathan
> > >>
> > >>
> > >> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <
> [hidden email]>
> > >> wrote:
> > >>
> > >>>
> > >>>
> >
> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
> > >>>
> > >>> With these patches the JMS2.0 API works pretty well now :) If anyone
> > >>> wants
> > >>> to review, please go ahead, otherwise I'll commit them tomorrow.
> > >>>
> > >>> cheers,
> > >>> - [The other] Jonathan
> > >>>
> > >>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <
> [hidden email]
> > >
> > >>> wrote:
> > >>>
> > >>> > I've narrowed down the problem to AutoConnectionTracker. It's not
> > >>> > completing, which isn't allowing the connections to be returned to
> > the
> > >>> pool.
> > >>> >
> > >>> >
> > >>>
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
> > >>> >
> > >>> > getResource() is throwing an IllegalStateException. The JavaDoc (
> > >>> >
> > >>>
> >
> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
> > >>> )
> > >>> > states it should throw an ISE if a current transaction is not
> Active.
> > >>> The
> > >>> > transaction is in the state ROLLED_BACK when AutoConnectionTracker
> > >>> tries to
> > >>> > call getResource().
> > >>> >
> > >>> > I think the Geronimo implementation (
> > >>> >
> > >>>
> >
> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
> > >>> )
> > >>> > maybe be a little too strict. The JTA Spec pdf doesn't offer exact
> > >>> hints of
> > >>> > which statuses (
> > >>> > https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html
> )
> > >>> > should be have getResource _not_ throw an ISE unfortunately. I was
> > >>> thinking
> > >>> > of changing Geronimo's implementation to check for anything
> > >>> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
> > >>> >
> > >>> > The other way is to cast Transaction to the Geronimo implementation
> > and
> > >>> > use Geronimo specific APIs to get call getResource(). Do you guys
> > have
> > >>> any
> > >>> > preference which route I should take to fix?
> > >>> >
> > >>> >
> > >>> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
> > [hidden email]
> > >>> >
> > >>> > wrote:
> > >>> >
> > >>> >>
> > https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
> > >>> >>
> > >>> >> Here's a project that reproduces the bug. This project
> intentionally
> > >>> >> exceeds the transaction timeout (of 1s). Each invocation, the
> > >>> connection is
> > >>> >> not returned to the pool and eventually you run out, causing your
> > >>> >> application to freeze.
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
> > >>> [hidden email]>
> > >>> >> wrote:
> > >>> >>
> > >>> >>> Hello Apache friends :) I have a question about the JTA and
> JMS/RA
> > >>> specs:
> > >>> >>>
> > >>> >>> If you borrow something from a RA, like a JMS Connection, and
> > you're
> > >>> in
> > >>> >>> XA Transaction, is it necessary to call connection.close()? It
> > would
> > >>> seem
> > >>> >>> JTA should be smart enough to know the connection is enrolled
> for 2
> > >>> phase
> > >>> >>> commit and should be smart enough to close it, but I'm not sure
> if
> > >>> that's
> > >>> >>> part of the spec.
> > >>> >>>
> > >>> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS Connection
> > >>> with
> > >>> >>> connectionFactory.createConnection(), and your code fails to call
> > >>> close()
> > >>> >>> before the transaction completion, the connection leaks. (And
> > >>> >>> unfortunately, calling close() after the transaction completes
> > >>> doesn't
> > >>> >>> mitigate the problem). It took awhile for us to track this down.
> > >>> >>>
> > >>> >>> This becomes a huge problem if you're calling external services
> in
> > >>> your
> > >>> >>> transaction. Let's say you have a reasonable transaction timeout
> of
> > >>> 30s
> > >>> >>> set. You call three services, and they end up taking 15s a piece.
> > >>> Even if
> > >>> >>> you're doing the right thing and you have connection.close() in a
> > >>> finally
> > >>> >>> block, because your transaction isn't active when you call close,
> > it
> > >>> leaks
> > >>> >>> and it just gets "stuck" as an active connection, which
> eventually
> > >>> you hit
> > >>> >>> the pool limit and your app freezes.
> > >>> >>>
> > >>> >>> On a separate note, we noticed if you open a connection outside
> of
> > >>> the
> > >>> >>> scope of a transaction, then start a transaction, then create a
> > >>> session
> > >>> >>> with session_transacted option, the session does not participate
> in
> > >>> the JTA
> > >>> >>> (which seems out of spec). One most open the connection inside
> the
> > >>> >>> transaction, AND open the session in the transaction, and close
> the
> > >>> >>> connection in the transaction for everything to work.
> > >>> >>>
> > >>> >>> I'll get a reproducing project created, but I was curious if
> anyone
> > >>> knew
> > >>> >>> offhand what the spec says.
> > >>> >>>
> > >>> >>> cheers, and thanks,
> > >>> >>> -[the other] Jonathan
> > >>> >>>
> > >>> >>> --
> > >>> >>> Jonathan | [hidden email]
> > >>> >>> Pessimists, see a jar as half empty. Optimists, in contrast, see
> it
> > >>> as
> > >>> >>> half full.
> > >>> >>> Engineers, of course, understand the glass is twice as big as it
> > >>> needs
> > >>> >>> to be.
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >> Jonathan | [hidden email]
> > >>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see
> it
> > as
> > >>> >> half full.
> > >>> >> Engineers, of course, understand the glass is twice as big as it
> > >>> needs to
> > >>> >> be.
> > >>> >>
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Jonathan | [hidden email]
> > >>> > Pessimists, see a jar as half empty. Optimists, in contrast, see it
> > as
> > >>> > half full.
> > >>> > Engineers, of course, understand the glass is twice as big as it
> > needs
> > >>> to
> > >>> > be.
> > >>> >
> > >>>
> > >>>
> > >>> --
> > >>> Jonathan | [hidden email]
> > >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
> as
> > >>> half
> > >>> full.
> > >>> Engineers, of course, understand the glass is twice as big as it
> needs
> > to
> > >>> be.
> > >>>
> > >>
> >
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
> full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.
>
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

exabrial12
Cool! And you're welcome! JMS, ActiveMQ, and XA transactions are pretty key
for us; they form the building blocks for us to scale horizontally (Kafka
like patterns).

I just ran a 100,000 messages through the code in the PR without problems
or memory leaks, so I'm not worried about that Arquillian test failing at
10k messages.

On Tue, Sep 3, 2019 at 2:37 PM Jonathan Gallimore <
[hidden email]> wrote:

> I knew you'd know some magic to help with this - thank you! Just looking at
> your PR now. Will give it a test shortly, but it looks good to me.
>
> Jon
>
> On Tue, Sep 3, 2019 at 3:08 PM Jonathan S. Fisher <[hidden email]>
> wrote:
>
> > There are actually a few log messages we regularly ignore all the time
> from
> > the transaction manager ::wince face:: I'm not sure if we should be
> > concerned with that one.
> >
> > On your test, first, how is the broker xml declared? Often something that
> > trips our newbies up to TomEE is having a persistent broker that is
> storing
> > messages between TomEE runs. The tricky thing is that the broker store
> does
> > always appear in /target, so it might not get cleaned up with mvn clean
> > install. As such, for local development we use this
> > URL:
> >
> broker:(vm://localhost)?persistent=false&amp;deleteAllMessagesOnStartup=true.
> > Next, on JMSReceiverBean, you're missing transactionAttribute.
> Technically
> > it should work without it, but I'd add it just in case. Finally, I'd add
> a
> > small thread.sleep, or check to the queue length == 0 before doing your
> > assert on messagecounter. It could be you're beating the receiver bean to
> > the finish line. The default messaging mode is auto-ack, so technically
> the
> > message just has to be on the broker, it doesn't have to be processed
> > before your sender bean will return.
> >
> >
> >
> >
> >
> > On Mon, Sep 2, 2019 at 2:28 PM Jonathan Gallimore <
> > [hidden email]> wrote:
> >
> > > In case it means anything to anyone, the unexpected output I'm getting
> is
> > > an abandoned connection notification:
> > >
> > > WARNING: Transaction complete, but connection still has handles
> > associated:
> > > ManagedConnectionInfo:
> > > org.apache.geronimo.connector.outbound.ManagedConnectionInfo@5b5a4aed.
> > mc:
> > >
> >
> [org.apache.openejb.resource.activemq.jms2.TomEEManagedConnection@37e69c43
> > > ,ActiveMQConnection
> > >
> > >
> >
> {id=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-7:1,clientId=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-6:1,started=false}]]
> > > Abandoned connection information:
> > >   Connection handle opened at
> > >
> > >
> >
> org.apache.geronimo.connector.outbound.ConnectionInfo.setTrace(ConnectionInfo.java:119),
> > >
> > >
> >
> org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:57),
> > >
> > >
> >
> org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39),
> > >
> > >
> >
> org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66),
> > >
> > >
> >
> org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:81),
> > >
> > >
> >
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94),
> > >
> > >
> >
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67),
> > >
> > >
> >
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:83),
> > >
> > >
> >
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:99),
> > >
> > >
> >
> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createQueue(JMSContextImpl.java:309),
> > >
> > >
> >
> org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createQueue(JMS2CDIExtension.java:371),
> > >
> > >
> >
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:37),
> > >
> > >
> >
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:33),
> > > sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> > >
> >
> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
> > >
> > >
> >
> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
> > >
> > >
> >
> org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191),
> > >
> > >
> >
> org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102),
> > > sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> > >
> >
> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
> > >
> > >
> >
> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
> > >
> > >
> >
> org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85),
> > >
> > >
> >
> org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272),
> > >
> > >
> >
> org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221),
> > >
> > >
> >
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265),
> > >
> > >
> >
> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260),
> > >
> > >
> >
> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89),
> > >
> > >
> >
> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347),
> > >
> > >
> >
> org.apache.openejb.arquillian.tests.jms.JMSSenderBean$$LocalBeanProxy.sendToQueue(org/apache/openejb/arquillian/tests/jms/JMSSenderBean.java),
> > >
> > >
> >
> org.apache.openejb.arquillian.tests.jms.JMSContextInjectionTest.testShouldSendAndReceiveMessages(JMSContextInjectionTest.java:61),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> > >
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50),
> > >
> > >
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12),
> > >
> > >
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47),
> > > org.jboss.arquillian.junit.Arquillian$8$1.invoke(Arquillian.java:379),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> > > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> > > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
> > > org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:109),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> > > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> > > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
> > > org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> > >
> > >
> >
> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
> > >
> > >
> >
> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > >
> > >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > >
> > >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > >
> > >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> > >
> > >
> >
> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:136),
> > > org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:372),
> > > org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:246),
> > >
> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
> > > org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
> > > org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:260),
> > > org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:324),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
> > >
> > >
> >
> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
> > >
> > >
> >
> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
> > >
> > >
> >
> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > >
> > >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > >
> > >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > >
> > >
> >
> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
> > > java.lang.reflect.Method.invoke(Method.java:498),
> > >
> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
> > >
> > >
> >
> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
> > > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
> > > org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
> > >
> > >
> >
> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159),
> > > org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:317),
> > > org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325),
> > >
> > >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78),
> > >
> > >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57),
> > > org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
> > > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
> > > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
> > > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
> > > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
> > > org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:205),
> > >
> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
> > > org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
> > > org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219),
> > > org.junit.runners.ParentRunner.run(ParentRunner.java:363),
> > > org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167),
> > > org.junit.runners.Suite.runChild(Suite.java:128),
> > > org.junit.runners.Suite.runChild(Suite.java:27),
> > > org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
> > > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
> > > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
> > > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
> > > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
> > > org.junit.runners.ParentRunner.run(ParentRunner.java:363),
> > > org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55),
> > >
> > >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137),
> > >
> > >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107),
> > >
> > >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83),
> > >
> > >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75),
> > >
> > >
> >
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157),
> > >
> > >
> >
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386),
> > >
> > >
> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323),
> > > org.apache.
> > >
> > > Also, I see out by one errors - e.g. if I send 200 messages, sometimes
> I
> > > receive 201 or 199. Not sure if its related to the above warning or
> not.
> > >
> > > Jon
> > >
> > > On Mon, Sep 2, 2019 at 8:25 PM Jonathan Gallimore <
> > > [hidden email]> wrote:
> > >
> > > > I had a play around with this, and got my previously failing
> > > > JMSContextInjectionTest to work -- some of the time. (
> > > >
> > >
> >
> https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
> > > > ).
> > > >
> > > > I'm getting some log output that I'm not expecting so I've rolled
> back
> > my
> > > > removal of @Ignore so I can dig into it some more.
> > > >
> > > > Jon
> > > >
> > > > On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
> > > > [hidden email]> wrote:
> > > >
> > > >> Hi Jonathan
> > > >>
> > > >> Thanks for this. I think I had run into some of the things you
> fixed I
> > > >> little while back. I'll take a look at your change later today.
> Happy
> > > for
> > > >> you to commit and I can review after, too. :)
> > > >>
> > > >> Cheers
> > > >>
> > > >> The other Jonathan
> > > >>
> > > >>
> > > >> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <
> > [hidden email]>
> > > >> wrote:
> > > >>
> > > >>>
> > > >>>
> > >
> >
> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
> > > >>>
> > > >>> With these patches the JMS2.0 API works pretty well now :) If
> anyone
> > > >>> wants
> > > >>> to review, please go ahead, otherwise I'll commit them tomorrow.
> > > >>>
> > > >>> cheers,
> > > >>> - [The other] Jonathan
> > > >>>
> > > >>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <
> > [hidden email]
> > > >
> > > >>> wrote:
> > > >>>
> > > >>> > I've narrowed down the problem to AutoConnectionTracker. It's not
> > > >>> > completing, which isn't allowing the connections to be returned
> to
> > > the
> > > >>> pool.
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
> > > >>> >
> > > >>> > getResource() is throwing an IllegalStateException. The JavaDoc (
> > > >>> >
> > > >>>
> > >
> >
> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
> > > >>> )
> > > >>> > states it should throw an ISE if a current transaction is not
> > Active.
> > > >>> The
> > > >>> > transaction is in the state ROLLED_BACK when
> AutoConnectionTracker
> > > >>> tries to
> > > >>> > call getResource().
> > > >>> >
> > > >>> > I think the Geronimo implementation (
> > > >>> >
> > > >>>
> > >
> >
> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
> > > >>> )
> > > >>> > maybe be a little too strict. The JTA Spec pdf doesn't offer
> exact
> > > >>> hints of
> > > >>> > which statuses (
> > > >>> >
> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html
> > )
> > > >>> > should be have getResource _not_ throw an ISE unfortunately. I
> was
> > > >>> thinking
> > > >>> > of changing Geronimo's implementation to check for anything
> > > >>> > but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
> > > >>> >
> > > >>> > The other way is to cast Transaction to the Geronimo
> implementation
> > > and
> > > >>> > use Geronimo specific APIs to get call getResource(). Do you guys
> > > have
> > > >>> any
> > > >>> > preference which route I should take to fix?
> > > >>> >
> > > >>> >
> > > >>> > On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
> > > [hidden email]
> > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> >>
> > > https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
> > > >>> >>
> > > >>> >> Here's a project that reproduces the bug. This project
> > intentionally
> > > >>> >> exceeds the transaction timeout (of 1s). Each invocation, the
> > > >>> connection is
> > > >>> >> not returned to the pool and eventually you run out, causing
> your
> > > >>> >> application to freeze.
> > > >>> >>
> > > >>> >>
> > > >>> >>
> > > >>> >> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
> > > >>> [hidden email]>
> > > >>> >> wrote:
> > > >>> >>
> > > >>> >>> Hello Apache friends :) I have a question about the JTA and
> > JMS/RA
> > > >>> specs:
> > > >>> >>>
> > > >>> >>> If you borrow something from a RA, like a JMS Connection, and
> > > you're
> > > >>> in
> > > >>> >>> XA Transaction, is it necessary to call connection.close()? It
> > > would
> > > >>> seem
> > > >>> >>> JTA should be smart enough to know the connection is enrolled
> > for 2
> > > >>> phase
> > > >>> >>> commit and should be smart enough to close it, but I'm not sure
> > if
> > > >>> that's
> > > >>> >>> part of the spec.
> > > >>> >>>
> > > >>> >>> In TomEE 7.0.6 we're noticing that if you borrow a JMS
> Connection
> > > >>> with
> > > >>> >>> connectionFactory.createConnection(), and your code fails to
> call
> > > >>> close()
> > > >>> >>> before the transaction completion, the connection leaks. (And
> > > >>> >>> unfortunately, calling close() after the transaction completes
> > > >>> doesn't
> > > >>> >>> mitigate the problem). It took awhile for us to track this
> down.
> > > >>> >>>
> > > >>> >>> This becomes a huge problem if you're calling external services
> > in
> > > >>> your
> > > >>> >>> transaction. Let's say you have a reasonable transaction
> timeout
> > of
> > > >>> 30s
> > > >>> >>> set. You call three services, and they end up taking 15s a
> piece.
> > > >>> Even if
> > > >>> >>> you're doing the right thing and you have connection.close()
> in a
> > > >>> finally
> > > >>> >>> block, because your transaction isn't active when you call
> close,
> > > it
> > > >>> leaks
> > > >>> >>> and it just gets "stuck" as an active connection, which
> > eventually
> > > >>> you hit
> > > >>> >>> the pool limit and your app freezes.
> > > >>> >>>
> > > >>> >>> On a separate note, we noticed if you open a connection outside
> > of
> > > >>> the
> > > >>> >>> scope of a transaction, then start a transaction, then create a
> > > >>> session
> > > >>> >>> with session_transacted option, the session does not
> participate
> > in
> > > >>> the JTA
> > > >>> >>> (which seems out of spec). One most open the connection inside
> > the
> > > >>> >>> transaction, AND open the session in the transaction, and close
> > the
> > > >>> >>> connection in the transaction for everything to work.
> > > >>> >>>
> > > >>> >>> I'll get a reproducing project created, but I was curious if
> > anyone
> > > >>> knew
> > > >>> >>> offhand what the spec says.
> > > >>> >>>
> > > >>> >>> cheers, and thanks,
> > > >>> >>> -[the other] Jonathan
> > > >>> >>>
> > > >>> >>> --
> > > >>> >>> Jonathan | [hidden email]
> > > >>> >>> Pessimists, see a jar as half empty. Optimists, in contrast,
> see
> > it
> > > >>> as
> > > >>> >>> half full.
> > > >>> >>> Engineers, of course, understand the glass is twice as big as
> it
> > > >>> needs
> > > >>> >>> to be.
> > > >>> >>>
> > > >>> >>
> > > >>> >>
> > > >>> >> --
> > > >>> >> Jonathan | [hidden email]
> > > >>> >> Pessimists, see a jar as half empty. Optimists, in contrast, see
> > it
> > > as
> > > >>> >> half full.
> > > >>> >> Engineers, of course, understand the glass is twice as big as it
> > > >>> needs to
> > > >>> >> be.
> > > >>> >>
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > Jonathan | [hidden email]
> > > >>> > Pessimists, see a jar as half empty. Optimists, in contrast, see
> it
> > > as
> > > >>> > half full.
> > > >>> > Engineers, of course, understand the glass is twice as big as it
> > > needs
> > > >>> to
> > > >>> > be.
> > > >>> >
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Jonathan | [hidden email]
> > > >>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
> > as
> > > >>> half
> > > >>> full.
> > > >>> Engineers, of course, understand the glass is twice as big as it
> > needs
> > > to
> > > >>> be.
> > > >>>
> > > >>
> > >
> >
> >
> > --
> > Jonathan | [hidden email]
> > Pessimists, see a jar as half empty. Optimists, in contrast, see it as
> half
> > full.
> > Engineers, of course, understand the glass is twice as big as it needs to
> > be.
> >
>


--
Jonathan | [hidden email]
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.
Reply | Threaded
Open this post in threaded view
|

Re: JTA JMS Spec question, connection leakage

David Jencks-3
In reply to this post by exabrial12
You might have already explained this,  but… why is beforeCompletion() called before handleReleased()?  If that’s happening, I’d expect something is wrong.  However, I haven’t looked at this code in years.

thanks!
David Jencks

> On Sep 3, 2019, at 12:27 PM, Jonathan S. Fisher <[hidden email]> wrote:
>
> Two more updates:
>
> For the log message, it looks like beforeCompletion() Is being called
> before handleReleased(), leading to that warning. I ran a couple thousand
> messages through and took a heap dump and didn't get any leak suspects, so
> I think it's working correctly despite the warning. I'll add that to my
> existing PR with your tests.
>
> Second, on my 10K messages test, it looks like it might be Arquillian
> crashing, not TomEE. I'm going to try the same test but without Arquillian,
> just deploy those classes to a server and see what happens.
>
> On Tue, Sep 3, 2019 at 10:54 AM Jonathan S. Fisher <[hidden email]>
> wrote:
>
>> If I bump the number of messages up to 10k or so I get a VM Crash... I
>> cannot figure out how to get arquillian to give me a heap dump on exit
>> though.
>>
>> On Tue, Sep 3, 2019 at 10:01 AM Jonathan S. Fisher <[hidden email]>
>> wrote:
>>
>>> https://github.com/apache/tomee/pull/546/files
>>>
>>> This passes consistently for me with no issues
>>>
>>> On Tue, Sep 3, 2019 at 9:08 AM Jonathan S. Fisher <[hidden email]>
>>> wrote:
>>>
>>>> There are actually a few log messages we regularly ignore all the time
>>>> from the transaction manager ::wince face:: I'm not sure if we should be
>>>> concerned with that one.
>>>>
>>>> On your test, first, how is the broker xml declared? Often something
>>>> that trips our newbies up to TomEE is having a persistent broker that is
>>>> storing messages between TomEE runs. The tricky thing is that the broker
>>>> store does always appear in /target, so it might not get cleaned up with
>>>> mvn clean install. As such, for local development we use this
>>>> URL: broker:(vm://localhost)?persistent=false&amp;deleteAllMessagesOnStartup=true.
>>>> Next, on JMSReceiverBean, you're missing transactionAttribute. Technically
>>>> it should work without it, but I'd add it just in case. Finally, I'd add a
>>>> small thread.sleep, or check to the queue length == 0 before doing your
>>>> assert on messagecounter. It could be you're beating the receiver bean to
>>>> the finish line. The default messaging mode is auto-ack, so technically the
>>>> message just has to be on the broker, it doesn't have to be processed
>>>> before your sender bean will return.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Sep 2, 2019 at 2:28 PM Jonathan Gallimore <
>>>> [hidden email]> wrote:
>>>>
>>>>> In case it means anything to anyone, the unexpected output I'm getting
>>>>> is
>>>>> an abandoned connection notification:
>>>>>
>>>>> WARNING: Transaction complete, but connection still has handles
>>>>> associated:
>>>>> ManagedConnectionInfo:
>>>>> org.apache.geronimo.connector.outbound.ManagedConnectionInfo@5b5a4aed.
>>>>> mc:
>>>>>
>>>>> [org.apache.openejb.resource.activemq.jms2.TomEEManagedConnection@37e69c43
>>>>> ,ActiveMQConnection
>>>>>
>>>>> {id=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-7:1,clientId=ID:MacBook-Pro.jlpsoftwareltd.net-50025-1567452049768-6:1,started=false}]]
>>>>> Abandoned connection information:
>>>>>  Connection handle opened at
>>>>>
>>>>> org.apache.geronimo.connector.outbound.ConnectionInfo.setTrace(ConnectionInfo.java:119),
>>>>>
>>>>> org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:57),
>>>>>
>>>>> org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39),
>>>>>
>>>>> org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66),
>>>>>
>>>>> org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:81),
>>>>>
>>>>> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94),
>>>>>
>>>>> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67),
>>>>>
>>>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:83),
>>>>>
>>>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:99),
>>>>>
>>>>> org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createQueue(JMSContextImpl.java:309),
>>>>>
>>>>> org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createQueue(JMS2CDIExtension.java:371),
>>>>>
>>>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:37),
>>>>>
>>>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean.sendToQueue(JMSSenderBean.java:33),
>>>>> sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>>>>>
>>>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>>>>>
>>>>> org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:191),
>>>>>
>>>>> org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:102),
>>>>> sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:205),
>>>>>
>>>>> org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:186),
>>>>>
>>>>> org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85),
>>>>>
>>>>> org.apache.openejb.core.singleton.SingletonContainer._invoke(SingletonContainer.java:272),
>>>>>
>>>>> org.apache.openejb.core.singleton.SingletonContainer.invoke(SingletonContainer.java:221),
>>>>>
>>>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265),
>>>>>
>>>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260),
>>>>>
>>>>> org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89),
>>>>>
>>>>> org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347),
>>>>>
>>>>> org.apache.openejb.arquillian.tests.jms.JMSSenderBean$$LocalBeanProxy.sendToQueue(org/apache/openejb/arquillian/tests/jms/JMSSenderBean.java),
>>>>>
>>>>> org.apache.openejb.arquillian.tests.jms.JMSContextInjectionTest.testShouldSendAndReceiveMessages(JMSContextInjectionTest.java:61),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50),
>>>>>
>>>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12),
>>>>>
>>>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47),
>>>>> org.jboss.arquillian.junit.Arquillian$8$1.invoke(Arquillian.java:379),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>>>> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:109),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>>>> org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>>>
>>>>> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>>>>>
>>>>> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>>
>>>>> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>>
>>>>> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>>
>>>>> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>>>
>>>>> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:136),
>>>>> org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:372),
>>>>> org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:246),
>>>>> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
>>>>> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
>>>>> org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:260),
>>>>> org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:324),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:85),
>>>>>
>>>>> org.apache.openejb.arquillian.common.TestObserver.switchLoader(TestObserver.java:98),
>>>>>
>>>>> org.apache.openejb.arquillian.common.TestObserver.observesTest(TestObserver.java:75),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142),
>>>>>
>>>>> org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>>
>>>>> org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>>
>>>>> org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>>
>>>>> org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73),
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
>>>>> java.lang.reflect.Method.invoke(Method.java:498),
>>>>>
>>>>> org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96),
>>>>>
>>>>> org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:92),
>>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:143),
>>>>> org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114),
>>>>>
>>>>> org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159),
>>>>> org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:317),
>>>>> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325),
>>>>>
>>>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78),
>>>>>
>>>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57),
>>>>> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
>>>>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
>>>>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
>>>>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
>>>>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
>>>>> org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:205),
>>>>> org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431),
>>>>> org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55),
>>>>> org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219),
>>>>> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
>>>>> org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167),
>>>>> org.junit.runners.Suite.runChild(Suite.java:128),
>>>>> org.junit.runners.Suite.runChild(Suite.java:27),
>>>>> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290),
>>>>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71),
>>>>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288),
>>>>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58),
>>>>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268),
>>>>> org.junit.runners.ParentRunner.run(ParentRunner.java:363),
>>>>> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55),
>>>>>
>>>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137),
>>>>>
>>>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107),
>>>>>
>>>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83),
>>>>>
>>>>> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75),
>>>>>
>>>>> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157),
>>>>>
>>>>> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386),
>>>>>
>>>>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323),
>>>>> org.apache.
>>>>>
>>>>> Also, I see out by one errors - e.g. if I send 200 messages, sometimes I
>>>>> receive 201 or 199. Not sure if its related to the above warning or not.
>>>>>
>>>>> Jon
>>>>>
>>>>> On Mon, Sep 2, 2019 at 8:25 PM Jonathan Gallimore <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> I had a play around with this, and got my previously failing
>>>>>> JMSContextInjectionTest to work -- some of the time. (
>>>>>>
>>>>> https://github.com/apache/tomee/blob/e58ff848a3a80938d4d99fc9bcfeaade5a72d644/arquillian/arquillian-tomee-tests/arquillian-tomee-jms-tests/src/test/java/org/apache/openejb/arquillian/tests/jms/JMSContextInjectionTest.java
>>>>>> ).
>>>>>>
>>>>>> I'm getting some log output that I'm not expecting so I've rolled
>>>>> back my
>>>>>> removal of @Ignore so I can dig into it some more.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Tue, Aug 27, 2019 at 5:36 PM Jonathan Gallimore <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>> Hi Jonathan
>>>>>>>
>>>>>>> Thanks for this. I think I had run into some of the things you fixed
>>>>> I
>>>>>>> little while back. I'll take a look at your change later today.
>>>>> Happy for
>>>>>>> you to commit and I can review after, too. :)
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> The other Jonathan
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 27, 2019 at 2:59 AM Jonathan S. Fisher <
>>>>> [hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> https://github.com/exabrial/tomee/commit/08dd4c744818702f3be5edfd8a1c4cf2b69d524d
>>>>>>>>
>>>>>>>> With these patches the JMS2.0 API works pretty well now :) If anyone
>>>>>>>> wants
>>>>>>>> to review, please go ahead, otherwise I'll commit them tomorrow.
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>> - [The other] Jonathan
>>>>>>>>
>>>>>>>> On Mon, Aug 26, 2019 at 3:45 PM Jonathan S. Fisher <
>>>>> [hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I've narrowed down the problem to AutoConnectionTracker. It's not
>>>>>>>>> completing, which isn't allowing the connections to be returned
>>>>> to the
>>>>>>>> pool.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>> https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/resource/AutoConnectionTracker.java#L174
>>>>>>>>>
>>>>>>>>> getResource() is throwing an IllegalStateException. The JavaDoc (
>>>>>>>>>
>>>>>>>>
>>>>> https://docs.oracle.com/javaee/7/api/javax/transaction/TransactionSynchronizationRegistry.html#getResource-java.lang.Object-
>>>>>>>> )
>>>>>>>>> states it should throw an ISE if a current transaction is not
>>>>> Active.
>>>>>>>> The
>>>>>>>>> transaction is in the state ROLLED_BACK when AutoConnectionTracker
>>>>>>>> tries to
>>>>>>>>> call getResource().
>>>>>>>>>
>>>>>>>>> I think the Geronimo implementation (
>>>>>>>>>
>>>>>>>>
>>>>> https://github.com/apache/geronimo-txmanager/blame/trunk/geronimo-transaction/src/main/java/org/apache/geronimo/transaction/manager/TransactionManagerImpl.java#L203
>>>>>>>> )
>>>>>>>>> maybe be a little too strict. The JTA Spec pdf doesn't offer exact
>>>>>>>> hints of
>>>>>>>>> which statuses (
>>>>>>>>>
>>>>> https://docs.oracle.com/javaee/7/api/javax/transaction/Status.html)
>>>>>>>>> should be have getResource _not_ throw an ISE unfortunately. I was
>>>>>>>> thinking
>>>>>>>>> of changing Geronimo's implementation to check for anything
>>>>>>>>> but STATUS_UNKNOWN, and STATUS_NO_TRANSACTION.
>>>>>>>>>
>>>>>>>>> The other way is to cast Transaction to the Geronimo
>>>>> implementation and
>>>>>>>>> use Geronimo specific APIs to get call getResource(). Do you guys
>>>>> have
>>>>>>>> any
>>>>>>>>> preference which route I should take to fix?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 26, 2019 at 9:15 AM Jonathan S. Fisher <
>>>>> [hidden email]
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>> https://github.com/exabrial/tomee-jms2-bug/tree/connection-pool-leak
>>>>>>>>>>
>>>>>>>>>> Here's a project that reproduces the bug. This project
>>>>> intentionally
>>>>>>>>>> exceeds the transaction timeout (of 1s). Each invocation, the
>>>>>>>> connection is
>>>>>>>>>> not returned to the pool and eventually you run out, causing your
>>>>>>>>>> application to freeze.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 23, 2019 at 2:37 PM Jonathan S. Fisher <
>>>>>>>> [hidden email]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Apache friends :) I have a question about the JTA and
>>>>> JMS/RA
>>>>>>>> specs:
>>>>>>>>>>>
>>>>>>>>>>> If you borrow something from a RA, like a JMS Connection, and
>>>>> you're
>>>>>>>> in
>>>>>>>>>>> XA Transaction, is it necessary to call connection.close()? It
>>>>> would
>>>>>>>> seem
>>>>>>>>>>> JTA should be smart enough to know the connection is enrolled
>>>>> for 2
>>>>>>>> phase
>>>>>>>>>>> commit and should be smart enough to close it, but I'm not sure
>>>>> if
>>>>>>>> that's
>>>>>>>>>>> part of the spec.
>>>>>>>>>>>
>>>>>>>>>>> In TomEE 7.0.6 we're noticing that if you borrow a JMS
>>>>> Connection
>>>>>>>> with
>>>>>>>>>>> connectionFactory.createConnection(), and your code fails to
>>>>> call
>>>>>>>> close()
>>>>>>>>>>> before the transaction completion, the connection leaks. (And
>>>>>>>>>>> unfortunately, calling close() after the transaction completes
>>>>>>>> doesn't
>>>>>>>>>>> mitigate the problem). It took awhile for us to track this down.
>>>>>>>>>>>
>>>>>>>>>>> This becomes a huge problem if you're calling external services
>>>>> in
>>>>>>>> your
>>>>>>>>>>> transaction. Let's say you have a reasonable transaction
>>>>> timeout of
>>>>>>>> 30s
>>>>>>>>>>> set. You call three services, and they end up taking 15s a
>>>>> piece.
>>>>>>>> Even if
>>>>>>>>>>> you're doing the right thing and you have connection.close() in
>>>>> a
>>>>>>>> finally
>>>>>>>>>>> block, because your transaction isn't active when you call
>>>>> close, it
>>>>>>>> leaks
>>>>>>>>>>> and it just gets "stuck" as an active connection, which
>>>>> eventually
>>>>>>>> you hit
>>>>>>>>>>> the pool limit and your app freezes.
>>>>>>>>>>>
>>>>>>>>>>> On a separate note, we noticed if you open a connection outside
>>>>> of
>>>>>>>> the
>>>>>>>>>>> scope of a transaction, then start a transaction, then create a
>>>>>>>> session
>>>>>>>>>>> with session_transacted option, the session does not
>>>>> participate in
>>>>>>>> the JTA
>>>>>>>>>>> (which seems out of spec). One most open the connection inside
>>>>> the
>>>>>>>>>>> transaction, AND open the session in the transaction, and close
>>>>> the
>>>>>>>>>>> connection in the transaction for everything to work.
>>>>>>>>>>>
>>>>>>>>>>> I'll get a reproducing project created, but I was curious if
>>>>> anyone
>>>>>>>> knew
>>>>>>>>>>> offhand what the spec says.
>>>>>>>>>>>
>>>>>>>>>>> cheers, and thanks,
>>>>>>>>>>> -[the other] Jonathan
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Jonathan | [hidden email]
>>>>>>>>>>> Pessimists, see a jar as half empty. Optimists, in contrast,
>>>>> see it
>>>>>>>> as
>>>>>>>>>>> half full.
>>>>>>>>>>> Engineers, of course, understand the glass is twice as big as it
>>>>>>>> needs
>>>>>>>>>>> to be.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jonathan | [hidden email]
>>>>>>>>>> Pessimists, see a jar as half empty. Optimists, in contrast, see
>>>>> it as
>>>>>>>>>> half full.
>>>>>>>>>> Engineers, of course, understand the glass is twice as big as it
>>>>>>>> needs to
>>>>>>>>>> be.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jonathan | [hidden email]
>>>>>>>>> Pessimists, see a jar as half empty. Optimists, in contrast, see
>>>>> it as
>>>>>>>>> half full.
>>>>>>>>> Engineers, of course, understand the glass is twice as big as it
>>>>> needs
>>>>>>>> to
>>>>>>>>> be.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jonathan | [hidden email]
>>>>>>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it
>>>>> as
>>>>>>>> half
>>>>>>>> full.
>>>>>>>> Engineers, of course, understand the glass is twice as big as it
>>>>> needs to
>>>>>>>> be.
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Jonathan | [hidden email]
>>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>>> half full.
>>>> Engineers, of course, understand the glass is twice as big as it needs
>>>> to be.
>>>>
>>>
>>>
>>> --
>>> Jonathan | [hidden email]
>>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>>> half full.
>>> Engineers, of course, understand the glass is twice as big as it needs to
>>> be.
>>>
>>
>>
>> --
>> Jonathan | [hidden email]
>> Pessimists, see a jar as half empty. Optimists, in contrast, see it as
>> half full.
>> Engineers, of course, understand the glass is twice as big as it needs to
>> be.
>>
>
>
> --
> Jonathan | [hidden email]
> Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
> full.
> Engineers, of course, understand the glass is twice as big as it needs to
> be.

12