Performance issue with JMS on Tomee 7.0.5

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance issue with JMS on Tomee 7.0.5

exabrial12
Hey guys,

We're noticing a pretty strange issue processing a large number of JMS
messages. After about 20k messages, messages consumed per second drops off
and there's heavy GC activity (smells like a memory leak). What interesting
though is the server continues to run and doesn't OutOfMemoryError. A simple
restart of TomEE (but not the broker) temporarily fixes the problem. We're
using an external broker, not the internal one

What's interesting, is that Jonathan Gallimore and I were talking about a
similar issue with Websockets. We noticed that eventually the servers will
exhibit the same behavior once enough websockets are opened and closed. This
issue occurs infrequently enough because we might handle 20,000 websockets
over the course of a few days, but we can process 20,000 JMS messages in a
few mins.


I have a heap dump from the issue and several jstacks. I'll be honest, I'm
not sure where to start. In the past I've solved memory leaks by careful
code audits. Our codebase happens to be mostly stateless, with everything
else being managed by CDI scopes (ApplicationScoped and TransactionScoped).

What's a good way to get started?



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

Wiesner, Martin
Hi Jonathan,

could you open a JIRA for the below topic, please? It makes it a lot easier to track the weird behaviour and it is so important to document your observations. Thereby, we’ll be able to reference this issue in upcoming discussions. Please put additional material into the JIRA, as long as you can provide it (legally speaking).

Optional: Can you a MWE that demonstrates the issue somehow?

Best
Martin

Am 07.01.2019 um 03:47 schrieb exabrial12 <[hidden email]>:

Hey guys,

We're noticing a pretty strange issue processing a large number of JMS
messages. After about 20k messages, messages consumed per second drops off
and there's heavy GC activity (smells like a memory leak). What interesting
though is the server continues to run and doesn't OutOfMemoryError. A simple
restart of TomEE (but not the broker) temporarily fixes the problem. We're
using an external broker, not the internal one

What's interesting, is that Jonathan Gallimore and I were talking about a
similar issue with Websockets. We noticed that eventually the servers will
exhibit the same behavior once enough websockets are opened and closed. This
issue occurs infrequently enough because we might handle 20,000 websockets
over the course of a few days, but we can process 20,000 JMS messages in a
few mins.


I have a heap dump from the issue and several jstacks. I'll be honest, I'm
not sure where to start. In the past I've solved memory leaks by careful
code audits. Our codebase happens to be mostly stateless, with everything
else being managed by CDI scopes (ApplicationScoped and TransactionScoped).

What's a good way to get started?



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html


smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

exabrial12
Thanks,

Issue opened here:
https://issues.apache.org/jira/projects/TOMEE/issues/TOMEE-2449

We'll try and create a project that demonstrates the issue.



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

Wiesner, Martin
Thanks Jonathan for such a quick reaction and the openness to provide a MWE for that issue to the community.

Best
Martin



smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

brunobat
In reply to this post by exabrial12
Hi Jon,

I'd like to have a look at the memory heap dump when available.

Cheers

Bruno Baptista
https://twitter.com/brunobat_


On 07/01/19 14:35, exabrial12 wrote:

> Thanks,
>
> Issue opened here:
> https://issues.apache.org/jira/projects/TOMEE/issues/TOMEE-2449
>
> We'll try and create a project that demonstrates the issue.
>
>
>
> --
> Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

exabrial12
Hey Bruno,

I'll contact you privately to get it to you. The heap is a little more
sensitive than the jstack :) You'll have to describe the process you're
doing with it though so we can all learn



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

doychin@dsoft-bg.com
Hi Jonathan,

You can also try this:
Download memory Analyzer Tool

Use it to process your heap dump and check the leak suspect report for biggest leak suspects.

You can provide the result of the reports here to.

Another tool you can use is JProfiler which is commercial product.

On 8.1.2019 г. 18:11, exabrial12 [via TomEE & OpenEJB] wrote:
Hey Bruno,

I'll contact you privately to get it to you. The heap is a little more
sensitive than the jstack :) You'll have to describe the process you're
doing with it though so we can all learn



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html



If you reply to this email, your message will be added to the discussion below:
http://tomee-openejb.979440.n4.nabble.com/Performance-issue-with-JMS-on-Tomee-7-0-5-tp4687296p4687341.html
To start a new topic under TomEE Dev, email [hidden email]
To unsubscribe from TomEE Dev, click here.
NAML


-- 
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116

doychin.vcf (280 bytes) Download Attachment
smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

brunobat
In reply to this post by exabrial12
That's fine.

I now use the Eclipse Analyzer: https://www.eclipse.org/mat/

I found it recently from Matthew Broadhead, here on the list. It's much
faster than jVisualVM.

Just sort by the retained memory values.

Cheers

Bruno Baptista
https://twitter.com/brunobat_


On 08/01/19 16:11, exabrial12 wrote:

> Hey Bruno,
>
> I'll contact you privately to get it to you. The heap is a little more
> sensitive than the jstack :) You'll have to describe the process you're
> doing with it though so we can all learn
>
>
>
> --
> Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

jgallimore
In reply to this post by exabrial12
One of the things on my todo list is to try and strip sensitive information
out of heap dumps using this library: https://github.com/eaftan/hprof-parser.
I'll have to try and get back to that.

Jon

On Tue, Jan 8, 2019 at 4:11 PM exabrial12 <[hidden email]> wrote:

> Hey Bruno,
>
> I'll contact you privately to get it to you. The heap is a little more
> sensitive than the jstack :) You'll have to describe the process you're
> doing with it though so we can all learn
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

exabrial12
I've got a project that can reproduce the issue in about 45s. From there, you
can take your own heap dump and see the issue. Follow the directions here:

https://github.com/exabrial/tomee-jms-perf

cheers,
-Jonathan [American version of Jonathan]





--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

exabrial12
Well... this might be a PEBKAC error


If you change this method:

        public void disposeSession(@Disposes @Any Session session) {
                try {
                        session.close();
                } catch (JMSException e) {
                        throw new RuntimeException(e);
                }
        }

to


        public void disposeSession(@Disposes Session session) {
                try {
                        session.close();
                } catch (JMSException e) {
                        throw new RuntimeException(e);
                }
        }


the problem goes away, which seems strange



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

jgallimore
In reply to this post by exabrial12
Cool, I managed to get a heap dump, taking a look.

Jon

On Tue, Jan 8, 2019 at 9:25 PM exabrial12 <[hidden email]> wrote:

> I've got a project that can reproduce the issue in about 45s. From there,
> you
> can take your own heap dump and see the issue. Follow the directions here:
>
> https://github.com/exabrial/tomee-jms-perf
>
> cheers,
> -Jonathan [American version of Jonathan]
>
>
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

Matthew Broadhead-2
In reply to this post by jgallimore
that is a good idea.  then i can submit more dumps.  does it work?

On 08/01/2019 17:27, Jonathan Gallimore wrote:

> One of the things on my todo list is to try and strip sensitive information
> out of heap dumps using this library: https://github.com/eaftan/hprof-parser.
> I'll have to try and get back to that.
>
> Jon
>
> On Tue, Jan 8, 2019 at 4:11 PM exabrial12 <[hidden email]> wrote:
>
>> Hey Bruno,
>>
>> I'll contact you privately to get it to you. The heap is a little more
>> sensitive than the jstack :) You'll have to describe the process you're
>> doing with it though so we can all learn
>>
>>
>>
>> --
>> Sent from:
>> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>>

Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

jgallimore
Its on my todo list currently - which means "I've had the idea, and I have
a rough idea of how to implement it" :-). I haven't done any more than that
yet. The hprof-parser library (which isn't mine) - yes, that does work.
I've even managed to look at heap dumps with that which have been corrupted
or otherwise unreadable by other tools. If you're interested in what goes
in a heap dump, the source code is worth a look.

Jon

On Wed, Jan 9, 2019 at 8:31 AM Matthew Broadhead
<[hidden email]> wrote:

> that is a good idea.  then i can submit more dumps.  does it work?
>
> On 08/01/2019 17:27, Jonathan Gallimore wrote:
> > One of the things on my todo list is to try and strip sensitive
> information
> > out of heap dumps using this library:
> https://github.com/eaftan/hprof-parser.
> > I'll have to try and get back to that.
> >
> > Jon
> >
> > On Tue, Jan 8, 2019 at 4:11 PM exabrial12 <[hidden email]> wrote:
> >
> >> Hey Bruno,
> >>
> >> I'll contact you privately to get it to you. The heap is a little more
> >> sensitive than the jstack :) You'll have to describe the process you're
> >> doing with it though so we can all learn
> >>
> >>
> >>
> >> --
> >> Sent from:
> >> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue with JMS on Tomee 7.0.5

Matthew Broadhead-2
maybe there should be a page in TomEE documentation about submitting
heap dumps to the team?

it could warn them to clean sensitive data using hprof-parser.  it could
list preferred methods of receiving the dump, e.g. google drive, dropbox
etc.
also i have found gdb faster than jmap or jstack for getting core
initially which could be mentioned

On 09/01/2019 10:26, Jonathan Gallimore wrote:

> Its on my todo list currently - which means "I've had the idea, and I have
> a rough idea of how to implement it" :-). I haven't done any more than that
> yet. The hprof-parser library (which isn't mine) - yes, that does work.
> I've even managed to look at heap dumps with that which have been corrupted
> or otherwise unreadable by other tools. If you're interested in what goes
> in a heap dump, the source code is worth a look.
>
> Jon
>
> On Wed, Jan 9, 2019 at 8:31 AM Matthew Broadhead
> <[hidden email]> wrote:
>
>> that is a good idea.  then i can submit more dumps.  does it work?
>>
>> On 08/01/2019 17:27, Jonathan Gallimore wrote:
>>> One of the things on my todo list is to try and strip sensitive
>> information
>>> out of heap dumps using this library:
>> https://github.com/eaftan/hprof-parser.
>>> I'll have to try and get back to that.
>>>
>>> Jon
>>>
>>> On Tue, Jan 8, 2019 at 4:11 PM exabrial12 <[hidden email]> wrote:
>>>
>>>> Hey Bruno,
>>>>
>>>> I'll contact you privately to get it to you. The heap is a little more
>>>> sensitive than the jstack :) You'll have to describe the process you're
>>>> doing with it though so we can all learn
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from:
>>>> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>>>>
>>