Old Gen Full of Annotation Finder Index

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Old Gen Full of Annotation Finder Index

PaulC.B
Hi,

I've been trying to lower the memory footprint of an EAR deployed to TomEE 8.0.0.

Here is a screenshot of the heap histogram. The total Old Gen is about 450MB (after forcing multiple GC's). If I boot TomEE without my EAR then the old gen is about 6MB.



The entire ear is 140MB zip, most of which is in the ears /lib directory which contains libs such as Kafka, hazelcast, apache POI, Google cloud APIs, AWS client APIs etc etc. In total our code has about 100 actual EJB's. If i remove files from the lib folder in the ear then I can see that the memory used by the annotation finder is lowered.

Is there any way I can tell TomEE that it need not bother scanning everything in the /lib folder of my EAR for annotations and fulling up the heap. I already set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to scan only the one jar in /lib which does have EJB's in it and it seems to obey this property but it doesn't seem to mean that annotation processing is skipped for all these other jars in /lib

Thanks!

Paul Carter-Brown
Director
Jini Guru
m:<a href="tel:+27834427179" value="+27832837000" style="color:rgb(17,85,204)" target="_blank">+27 (0) 83 442 7179
a:1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
 Johannesburg, South Africa
w:jini.guru  e: [hidden email]

Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.


Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

jgallimore
Hi Paul

Would you mind trying your application with a recent snapshot?
https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/.
I recently updated the exclude list.

Many thanks

Jon

On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
<[hidden email]> wrote:

> Hi,
>
> I've been trying to lower the memory footprint of an EAR deployed to TomEE
> 8.0.0.
>
> Here is a screenshot of the heap histogram. The total Old Gen is about
> 450MB (after forcing multiple GC's). If I boot TomEE without my EAR then
> the old gen is about 6MB.
>
> [image: Screenshot from 2019-12-11 12-53-12.png]
>
> The entire ear is 140MB zip, most of which is in the ears /lib directory
> which contains libs such as Kafka, hazelcast, apache POI, Google cloud
> APIs, AWS client APIs etc etc. In total our code has about 100 actual
> EJB's. If i remove files from the lib folder in the ear then I can see that
> the memory used by the annotation finder is lowered.
>
> Is there any way I can tell TomEE that it need not bother scanning
> everything in the /lib folder of my EAR for annotations and fulling up the
> heap. I already
> set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to scan
> only the one jar in /lib which does have EJB's in it and it seems to obey
> this property but it doesn't seem to mean that annotation processing is
> skipped for all these other jars in /lib
>
> Thanks!
>
> Paul Carter-Brown
> Director
> Jini Guru
> m: +27 (0) 83 442 7179 <+27834427179>
> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>   Johannesburg, South Africa
> w: jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain
> privileged, confidential and/or personal information. If you are not the
> intended recipient you may not disclose or distribute any of
> the information contained within this message. In such case you must
> destroy this message and inform the sender of the error. Jini Guru may not
> accept liability for any errors, omissions, information and viruses
> contained in the transmission of this message. Any opinions, conclusions
> and other information contained within this message not related to Jini
> Guru official business is deemed to be that of the individual only and is
> not endorsed by Jini Guru.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

PaulC.B
Hi Jon,

Unfortunately the snapshot behaves exactly the same way

Paul Carter-Brown
Director
Jini Guru
m: +27 (0) 83 442 7179 <+27834427179>
a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
  Johannesburg, South Africa
w: jini.guru  e: [hidden email]

Disclaimer: This message and/or attachment(s) may contain
privileged, confidential and/or personal information. If you are not the
intended recipient you may not disclose or distribute any of
the information contained within this message. In such case you must
destroy this message and inform the sender of the error. Jini Guru may not
accept liability for any errors, omissions, information and viruses
contained in the transmission of this message. Any opinions, conclusions
and other information contained within this message not related to Jini
Guru official business is deemed to be that of the individual only and is
not endorsed by Jini Guru.



On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
[hidden email]> wrote:

> Hi Paul
>
> Would you mind trying your application with a recent snapshot?
>
> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
> .
> I recently updated the exclude list.
>
> Many thanks
>
> Jon
>
> On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
> <[hidden email]> wrote:
>
> > Hi,
> >
> > I've been trying to lower the memory footprint of an EAR deployed to
> TomEE
> > 8.0.0.
> >
> > Here is a screenshot of the heap histogram. The total Old Gen is about
> > 450MB (after forcing multiple GC's). If I boot TomEE without my EAR then
> > the old gen is about 6MB.
> >
> > [image: Screenshot from 2019-12-11 12-53-12.png]
> >
> > The entire ear is 140MB zip, most of which is in the ears /lib directory
> > which contains libs such as Kafka, hazelcast, apache POI, Google cloud
> > APIs, AWS client APIs etc etc. In total our code has about 100 actual
> > EJB's. If i remove files from the lib folder in the ear then I can see
> that
> > the memory used by the annotation finder is lowered.
> >
> > Is there any way I can tell TomEE that it need not bother scanning
> > everything in the /lib folder of my EAR for annotations and fulling up
> the
> > heap. I already
> > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to scan
> > only the one jar in /lib which does have EJB's in it and it seems to obey
> > this property but it doesn't seem to mean that annotation processing is
> > skipped for all these other jars in /lib
> >
> > Thanks!
> >
> > Paul Carter-Brown
> > Director
> > Jini Guru
> > m: +27 (0) 83 442 7179 <+27834427179>
> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
> >   Johannesburg, South Africa
> > w: jini.guru  e: [hidden email]
> >
> > Disclaimer: This message and/or attachment(s) may contain
> > privileged, confidential and/or personal information. If you are not the
> > intended recipient you may not disclose or distribute any of
> > the information contained within this message. In such case you must
> > destroy this message and inform the sender of the error. Jini Guru may
> not
> > accept liability for any errors, omissions, information and viruses
> > contained in the transmission of this message. Any opinions, conclusions
> > and other information contained within this message not related to Jini
> > Guru official business is deemed to be that of the individual only and is
> > not endorsed by Jini Guru.
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

PaulC.B
Hi Jon,

I took a look at the source and worked out that I could add my own filters
using openejb.additional.exclude.

I set it to openejb.additional.exclude=com,net,io,org and the JVM heap now
sits at around 150MB whereas it would never go below 450MB previously!!!

Our EJB's and code is all in jars prefixed with guru.jini and hence they
are not filtered out and the system works 100%.

My sense is that there should be an easier way to specify what jars and/or
packages to process for annotations. I have come across the following
settings but can't really work out what fits where:

openejb.additional.exclude
openejb.additional.include
openejb.deployments.classpath.filter.descriptors
openejb.deployments.classpath.filter.systemapps
openejb.deployments.classpath.include
openejb.deployments.classpath.exclude
openejb.deployments.package.include
openejb.deployments.package.exclude


P.S. Perhaps also add these to the standard exclusion list as they are
common and yet won't need to be processed for annotations I guess?
com.amazonaws
com.fasterxml
com.google
com.hazelcast
io.grpc
io.netty
org.docx4j
org.mongodb
org.rocksdb
org.asynchttpclient
org.apache
org.aspectj

But then maybe some are too broad and I don't really understand what the
annotation index/cache is really needed for at runtime? Can it not be lazy
loaded or discarded if unused? Just thinking out loud here :-( Seems like
the cache uses a significant amount of heap when considering the drive
towards tiny micro services.

Paul Carter-Brown
Director
Jini Guru
m: +27 (0) 83 442 7179 <+27834427179>
a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
  Johannesburg, South Africa
w: jini.guru  e: [hidden email]

Disclaimer: This message and/or attachment(s) may contain
privileged, confidential and/or personal information. If you are not the
intended recipient you may not disclose or distribute any of
the information contained within this message. In such case you must
destroy this message and inform the sender of the error. Jini Guru may not
accept liability for any errors, omissions, information and viruses
contained in the transmission of this message. Any opinions, conclusions
and other information contained within this message not related to Jini
Guru official business is deemed to be that of the individual only and is
not endorsed by Jini Guru.



On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
<[hidden email]> wrote:

> Hi Jon,
>
> Unfortunately the snapshot behaves exactly the same way
>
> Paul Carter-Brown
> Director
> Jini Guru
> m: +27 (0) 83 442 7179 <+27834427179>
> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>   Johannesburg, South Africa
> w: jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain
> privileged, confidential and/or personal information. If you are not the
> intended recipient you may not disclose or distribute any of
> the information contained within this message. In such case you must
> destroy this message and inform the sender of the error. Jini Guru may not
> accept liability for any errors, omissions, information and viruses
> contained in the transmission of this message. Any opinions, conclusions
> and other information contained within this message not related to Jini
> Guru official business is deemed to be that of the individual only and is
> not endorsed by Jini Guru.
>
>
>
> On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
> [hidden email]> wrote:
>
>> Hi Paul
>>
>> Would you mind trying your application with a recent snapshot?
>>
>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>> .
>> I recently updated the exclude list.
>>
>> Many thanks
>>
>> Jon
>>
>> On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>> <[hidden email]> wrote:
>>
>> > Hi,
>> >
>> > I've been trying to lower the memory footprint of an EAR deployed to
>> TomEE
>> > 8.0.0.
>> >
>> > Here is a screenshot of the heap histogram. The total Old Gen is about
>> > 450MB (after forcing multiple GC's). If I boot TomEE without my EAR then
>> > the old gen is about 6MB.
>> >
>> > [image: Screenshot from 2019-12-11 12-53-12.png]
>> >
>> > The entire ear is 140MB zip, most of which is in the ears /lib directory
>> > which contains libs such as Kafka, hazelcast, apache POI, Google cloud
>> > APIs, AWS client APIs etc etc. In total our code has about 100 actual
>> > EJB's. If i remove files from the lib folder in the ear then I can see
>> that
>> > the memory used by the annotation finder is lowered.
>> >
>> > Is there any way I can tell TomEE that it need not bother scanning
>> > everything in the /lib folder of my EAR for annotations and fulling up
>> the
>> > heap. I already
>> > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to scan
>> > only the one jar in /lib which does have EJB's in it and it seems to
>> obey
>> > this property but it doesn't seem to mean that annotation processing is
>> > skipped for all these other jars in /lib
>> >
>> > Thanks!
>> >
>> > Paul Carter-Brown
>> > Director
>> > Jini Guru
>> > m: +27 (0) 83 442 7179 <+27834427179>
>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>> >   Johannesburg, South Africa
>> > w: jini.guru  e: [hidden email]
>> >
>> > Disclaimer: This message and/or attachment(s) may contain
>> > privileged, confidential and/or personal information. If you are not the
>> > intended recipient you may not disclose or distribute any of
>> > the information contained within this message. In such case you must
>> > destroy this message and inform the sender of the error. Jini Guru may
>> not
>> > accept liability for any errors, omissions, information and viruses
>> > contained in the transmission of this message. Any opinions, conclusions
>> > and other information contained within this message not related to Jini
>> > Guru official business is deemed to be that of the individual only and
>> is
>> > not endorsed by Jini Guru.
>> >
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

PaulC.B
FYI. Graph of the change in heap size on a prod instance after this change:


Paul Carter-Brown
Director
Jini Guru
m:<a href="tel:+27834427179" value="+27832837000" style="color:rgb(17,85,204)" target="_blank">+27 (0) 83 442 7179
a:1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
 Johannesburg, South Africa
w:jini.guru  e: [hidden email]

Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.




On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown <[hidden email]> wrote:
Hi Jon,

I took a look at the source and worked out that I could add my own filters using openejb.additional.exclude.

I set it to openejb.additional.exclude=com,net,io,org and the JVM heap now sits at around 150MB whereas it would never go below 450MB previously!!!

Our EJB's and code is all in jars prefixed with guru.jini and hence they are not filtered out and the system works 100%.

My sense is that there should be an easier way to specify what jars and/or packages to process for annotations. I have come across the following settings but can't really work out what fits where:

openejb.additional.exclude
openejb.additional.include
openejb.deployments.classpath.filter.descriptors
openejb.deployments.classpath.filter.systemapps
openejb.deployments.classpath.include
openejb.deployments.classpath.exclude
openejb.deployments.package.include
openejb.deployments.package.exclude


P.S. Perhaps also add these to the standard exclusion list as they are common and yet won't need to be processed for annotations I guess?
com.amazonaws
com.fasterxml
com.google
com.hazelcast
io.grpc
io.netty
org.docx4j
org.mongodb
org.rocksdb
org.asynchttpclient
org.apache
org.aspectj

But then maybe some are too broad and I don't really understand what the annotation index/cache is really needed for at runtime? Can it not be lazy loaded or discarded if unused? Just thinking out loud here :-( Seems like the cache uses a significant amount of heap when considering the drive towards tiny micro services. 

Paul Carter-Brown
Director
Jini Guru
m:<a href="tel:+27834427179" value="+27832837000" style="color:rgb(17,85,204)" target="_blank">+27 (0) 83 442 7179
a:1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
 Johannesburg, South Africa
w:jini.guru  e: [hidden email]

Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.




On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown <[hidden email]> wrote:
Hi Jon,

Unfortunately the snapshot behaves exactly the same way

Paul Carter-Brown
Director
Jini Guru
m:<a href="tel:+27834427179" value="+27832837000" style="color:rgb(17,85,204)" target="_blank">+27 (0) 83 442 7179
a:1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
 Johannesburg, South Africa
w:jini.guru  e: [hidden email]

Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.




On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <[hidden email]> wrote:
Hi Paul

Would you mind trying your application with a recent snapshot?
https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/.
I recently updated the exclude list.

Many thanks

Jon

On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
<[hidden email]> wrote:

> Hi,
>
> I've been trying to lower the memory footprint of an EAR deployed to TomEE
> 8.0.0.
>
> Here is a screenshot of the heap histogram. The total Old Gen is about
> 450MB (after forcing multiple GC's). If I boot TomEE without my EAR then
> the old gen is about 6MB.
>
> [image: Screenshot from 2019-12-11 12-53-12.png]
>
> The entire ear is 140MB zip, most of which is in the ears /lib directory
> which contains libs such as Kafka, hazelcast, apache POI, Google cloud
> APIs, AWS client APIs etc etc. In total our code has about 100 actual
> EJB's. If i remove files from the lib folder in the ear then I can see that
> the memory used by the annotation finder is lowered.
>
> Is there any way I can tell TomEE that it need not bother scanning
> everything in the /lib folder of my EAR for annotations and fulling up the
> heap. I already
> set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to scan
> only the one jar in /lib which does have EJB's in it and it seems to obey
> this property but it doesn't seem to mean that annotation processing is
> skipped for all these other jars in /lib
>
> Thanks!
>
> Paul Carter-Brown
> Director
> Jini Guru
> m: +27 (0) 83 442 7179 <+27834427179>
> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>   Johannesburg, South Africa
> w: jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain
> privileged, confidential and/or personal information. If you are not the
> intended recipient you may not disclose or distribute any of
> the information contained within this message. In such case you must
> destroy this message and inform the sender of the error. Jini Guru may not
> accept liability for any errors, omissions, information and viruses
> contained in the transmission of this message. Any opinions, conclusions
> and other information contained within this message not related to Jini
> Guru official business is deemed to be that of the individual only and is
> not endorsed by Jini Guru.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

Mark Struberg-2
Hi!

All this is just a workaround imo.

Afair we (Romain) introduced a temporary throw-away ClassLoader for scanning. That means after doing all the class scans we only keep the extracted information but do not keep the Classes in memory. Doesn't this work anymore?

LieGrue,
strub


> Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown <[hidden email]>:
>
> FYI. Graph of the change in heap size on a prod instance after this change:
>
>
> Paul Carter-Brown
> Director
> Jini Guru
> m: +27 (0) 83 442 7179
> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>   Johannesburg, South Africa
> w: jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.
>
>
>
>
> On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown <[hidden email]> wrote:
> Hi Jon,
>
> I took a look at the source and worked out that I could add my own filters using openejb.additional.exclude.
>
> I set it to openejb.additional.exclude=com,net,io,org and the JVM heap now sits at around 150MB whereas it would never go below 450MB previously!!!
>
> Our EJB's and code is all in jars prefixed with guru.jini and hence they are not filtered out and the system works 100%.
>
> My sense is that there should be an easier way to specify what jars and/or packages to process for annotations. I have come across the following settings but can't really work out what fits where:
>
> openejb.additional.exclude
> openejb.additional.include
> openejb.deployments.classpath.filter.descriptors
> openejb.deployments.classpath.filter.systemapps
> openejb.deployments.classpath.include
> openejb.deployments.classpath.exclude
> openejb.deployments.package.include
> openejb.deployments.package.exclude
>
>
> P.S. Perhaps also add these to the standard exclusion list as they are common and yet won't need to be processed for annotations I guess?
> com.amazonaws
> com.fasterxml
> com.google
> com.hazelcast
> io.grpc
> io.netty
> org.docx4j
> org.mongodb
> org.rocksdb
> org.asynchttpclient
> org.apache
> org.aspectj
>
> But then maybe some are too broad and I don't really understand what the annotation index/cache is really needed for at runtime? Can it not be lazy loaded or discarded if unused? Just thinking out loud here :-( Seems like the cache uses a significant amount of heap when considering the drive towards tiny micro services.
>
> Paul Carter-Brown
> Director
> Jini Guru
> m: +27 (0) 83 442 7179
> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>   Johannesburg, South Africa
> w: jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.
>
>
>
>
> On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown <[hidden email]> wrote:
> Hi Jon,
>
> Unfortunately the snapshot behaves exactly the same way
>
> Paul Carter-Brown
> Director
> Jini Guru
> m: +27 (0) 83 442 7179
> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>   Johannesburg, South Africa
> w: jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.
>
>
>
>
> On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <[hidden email]> wrote:
> Hi Paul
>
> Would you mind trying your application with a recent snapshot?
> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/.
> I recently updated the exclude list.
>
> Many thanks
>
> Jon
>
> On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
> <[hidden email]> wrote:
>
> > Hi,
> >
> > I've been trying to lower the memory footprint of an EAR deployed to TomEE
> > 8.0.0.
> >
> > Here is a screenshot of the heap histogram. The total Old Gen is about
> > 450MB (after forcing multiple GC's). If I boot TomEE without my EAR then
> > the old gen is about 6MB.
> >
> > [image: Screenshot from 2019-12-11 12-53-12.png]
> >
> > The entire ear is 140MB zip, most of which is in the ears /lib directory
> > which contains libs such as Kafka, hazelcast, apache POI, Google cloud
> > APIs, AWS client APIs etc etc. In total our code has about 100 actual
> > EJB's. If i remove files from the lib folder in the ear then I can see that
> > the memory used by the annotation finder is lowered.
> >
> > Is there any way I can tell TomEE that it need not bother scanning
> > everything in the /lib folder of my EAR for annotations and fulling up the
> > heap. I already
> > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to scan
> > only the one jar in /lib which does have EJB's in it and it seems to obey
> > this property but it doesn't seem to mean that annotation processing is
> > skipped for all these other jars in /lib
> >
> > Thanks!
> >
> > Paul Carter-Brown
> > Director
> > Jini Guru
> > m: +27 (0) 83 442 7179 <+27834427179>
> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
> >   Johannesburg, South Africa
> > w: jini.guru  e: [hidden email]
> >
> > Disclaimer: This message and/or attachment(s) may contain
> > privileged, confidential and/or personal information. If you are not the
> > intended recipient you may not disclose or distribute any of
> > the information contained within this message. In such case you must
> > destroy this message and inform the sender of the error. Jini Guru may not
> > accept liability for any errors, omissions, information and viruses
> > contained in the transmission of this message. Any opinions, conclusions
> > and other information contained within this message not related to Jini
> > Guru official business is deemed to be that of the individual only and is
> > not endorsed by Jini Guru.
> >
> >

Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

PaulC.B
Hi,

The code does seem to use org.apache.openejb.core.TempClassLoader. I can see one instance in the heap for every war in my ear. + 1.

The TempClassLoader however still has 100's of strong references to it. E.g:






Paul Carter-Brown
Director
Jini Guru
m:<a href="tel:+27834427179" value="+27832837000" style="color:rgb(17,85,204)" target="_blank">+27 (0) 83 442 7179
a:1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
 Johannesburg, South Africa
w:jini.guru  e: [hidden email]

Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.




On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <[hidden email]> wrote:
Hi!

All this is just a workaround imo.

Afair we (Romain) introduced a temporary throw-away ClassLoader for scanning. That means after doing all the class scans we only keep the extracted information but do not keep the Classes in memory. Doesn't this work anymore?

LieGrue,
strub


> Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown <[hidden email]>:
>
> FYI. Graph of the change in heap size on a prod instance after this change:
>
>
> Paul Carter-Brown
> Director
> Jini Guru
> m:    +27 (0) 83 442 7179
> a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>       Johannesburg, South Africa
> w:    jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.
>
>
>
>
> On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown <[hidden email]> wrote:
> Hi Jon,
>
> I took a look at the source and worked out that I could add my own filters using openejb.additional.exclude.
>
> I set it to openejb.additional.exclude=com,net,io,org and the JVM heap now sits at around 150MB whereas it would never go below 450MB previously!!!
>
> Our EJB's and code is all in jars prefixed with guru.jini and hence they are not filtered out and the system works 100%.
>
> My sense is that there should be an easier way to specify what jars and/or packages to process for annotations. I have come across the following settings but can't really work out what fits where:
>
> openejb.additional.exclude
> openejb.additional.include
> openejb.deployments.classpath.filter.descriptors
> openejb.deployments.classpath.filter.systemapps
> openejb.deployments.classpath.include
> openejb.deployments.classpath.exclude
> openejb.deployments.package.include
> openejb.deployments.package.exclude
>
>
> P.S. Perhaps also add these to the standard exclusion list as they are common and yet won't need to be processed for annotations I guess?
> com.amazonaws
> com.fasterxml
> com.google
> com.hazelcast
> io.grpc
> io.netty
> org.docx4j
> org.mongodb
> org.rocksdb
> org.asynchttpclient
> org.apache
> org.aspectj
>
> But then maybe some are too broad and I don't really understand what the annotation index/cache is really needed for at runtime? Can it not be lazy loaded or discarded if unused? Just thinking out loud here :-( Seems like the cache uses a significant amount of heap when considering the drive towards tiny micro services.
>
> Paul Carter-Brown
> Director
> Jini Guru
> m:    +27 (0) 83 442 7179
> a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>       Johannesburg, South Africa
> w:    jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.
>
>
>
>
> On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown <[hidden email]> wrote:
> Hi Jon,
>
> Unfortunately the snapshot behaves exactly the same way
>
> Paul Carter-Brown
> Director
> Jini Guru
> m:    +27 (0) 83 442 7179
> a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>       Johannesburg, South Africa
> w:    jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain privileged, confidential and/or personal information. If you are not the intended recipient you may not disclose or distribute any of the information contained within this message. In such case you must destroy this message and inform the sender of the error. Jini Guru may not accept liability for any errors, omissions, information and viruses contained in the transmission of this message. Any opinions, conclusions and other information contained within this message not related to Jini Guru official business is deemed to be that of the individual only and is not endorsed by Jini Guru.
>
>
>
>
> On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <[hidden email]> wrote:
> Hi Paul
>
> Would you mind trying your application with a recent snapshot?
> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/.
> I recently updated the exclude list.
>
> Many thanks
>
> Jon
>
> On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
> <[hidden email]> wrote:
>
> > Hi,
> >
> > I've been trying to lower the memory footprint of an EAR deployed to TomEE
> > 8.0.0.
> >
> > Here is a screenshot of the heap histogram. The total Old Gen is about
> > 450MB (after forcing multiple GC's). If I boot TomEE without my EAR then
> > the old gen is about 6MB.
> >
> > [image: Screenshot from 2019-12-11 12-53-12.png]
> >
> > The entire ear is 140MB zip, most of which is in the ears /lib directory
> > which contains libs such as Kafka, hazelcast, apache POI, Google cloud
> > APIs, AWS client APIs etc etc. In total our code has about 100 actual
> > EJB's. If i remove files from the lib folder in the ear then I can see that
> > the memory used by the annotation finder is lowered.
> >
> > Is there any way I can tell TomEE that it need not bother scanning
> > everything in the /lib folder of my EAR for annotations and fulling up the
> > heap. I already
> > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to scan
> > only the one jar in /lib which does have EJB's in it and it seems to obey
> > this property but it doesn't seem to mean that annotation processing is
> > skipped for all these other jars in /lib
> >
> > Thanks!
> >
> > Paul Carter-Brown
> > Director
> > Jini Guru
> > m: +27 (0) 83 442 7179 <+27834427179>
> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
> >   Johannesburg, South Africa
> > w: jini.guru  e: [hidden email]
> >
> > Disclaimer: This message and/or attachment(s) may contain
> > privileged, confidential and/or personal information. If you are not the
> > intended recipient you may not disclose or distribute any of
> > the information contained within this message. In such case you must
> > destroy this message and inform the sender of the error. Jini Guru may not
> > accept liability for any errors, omissions, information and viruses
> > contained in the transmission of this message. Any opinions, conclusions
> > and other information contained within this message not related to Jini
> > Guru official business is deemed to be that of the individual only and is
> > not endorsed by Jini Guru.
> >
> >

Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

jgallimore
The mailing list seems to be stripping off your images - can you resend and
include my email address as well? I'd be keen to dig into this a little
more.

Thanks

Jon

On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
<[hidden email]> wrote:

> Hi,
>
> The code does seem to use org.apache.openejb.core.TempClassLoader. I can
> see one instance in the heap for every war in my ear. + 1.
>
> The TempClassLoader however still has 100's of strong references to it.
> E.g:
>
> [image: image.png]
>
> [image: image.png]
>
>
> Paul Carter-Brown
> Director
> Jini Guru
> m: +27 (0) 83 442 7179 <+27834427179>
> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>   Johannesburg, South Africa
> w: jini.guru  e: [hidden email]
>
> Disclaimer: This message and/or attachment(s) may contain
> privileged, confidential and/or personal information. If you are not the
> intended recipient you may not disclose or distribute any of
> the information contained within this message. In such case you must
> destroy this message and inform the sender of the error. Jini Guru may not
> accept liability for any errors, omissions, information and viruses
> contained in the transmission of this message. Any opinions, conclusions
> and other information contained within this message not related to Jini
> Guru official business is deemed to be that of the individual only and is
> not endorsed by Jini Guru.
>
>
>
> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <[hidden email]>
> wrote:
>
>> Hi!
>>
>> All this is just a workaround imo.
>>
>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>> scanning. That means after doing all the class scans we only keep the
>> extracted information but do not keep the Classes in memory. Doesn't this
>> work anymore?
>>
>> LieGrue,
>> strub
>>
>>
>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>> <[hidden email]>:
>> >
>> > FYI. Graph of the change in heap size on a prod instance after this
>> change:
>> >
>> >
>> > Paul Carter-Brown
>> > Director
>> > Jini Guru
>> > m:    +27 (0) 83 442 7179
>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>> Leslie
>> >       Johannesburg, South Africa
>> > w:    jini.guru  e: [hidden email]
>> >
>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>> confidential and/or personal information. If you are not the intended
>> recipient you may not disclose or distribute any of the information
>> contained within this message. In such case you must destroy this message
>> and inform the sender of the error. Jini Guru may not accept liability for
>> any errors, omissions, information and viruses contained in the
>> transmission of this message. Any opinions, conclusions and other
>> information contained within this message not related to Jini Guru official
>> business is deemed to be that of the individual only and is not endorsed by
>> Jini Guru.
>> >
>> >
>> >
>> >
>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>> <[hidden email]> wrote:
>> > Hi Jon,
>> >
>> > I took a look at the source and worked out that I could add my own
>> filters using openejb.additional.exclude.
>> >
>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM heap
>> now sits at around 150MB whereas it would never go below 450MB previously!!!
>> >
>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>> they are not filtered out and the system works 100%.
>> >
>> > My sense is that there should be an easier way to specify what jars
>> and/or packages to process for annotations. I have come across the
>> following settings but can't really work out what fits where:
>> >
>> > openejb.additional.exclude
>> > openejb.additional.include
>> > openejb.deployments.classpath.filter.descriptors
>> > openejb.deployments.classpath.filter.systemapps
>> > openejb.deployments.classpath.include
>> > openejb.deployments.classpath.exclude
>> > openejb.deployments.package.include
>> > openejb.deployments.package.exclude
>> >
>> >
>> > P.S. Perhaps also add these to the standard exclusion list as they are
>> common and yet won't need to be processed for annotations I guess?
>> > com.amazonaws
>> > com.fasterxml
>> > com.google
>> > com.hazelcast
>> > io.grpc
>> > io.netty
>> > org.docx4j
>> > org.mongodb
>> > org.rocksdb
>> > org.asynchttpclient
>> > org.apache
>> > org.aspectj
>> >
>> > But then maybe some are too broad and I don't really understand what
>> the annotation index/cache is really needed for at runtime? Can it not be
>> lazy loaded or discarded if unused? Just thinking out loud here :-( Seems
>> like the cache uses a significant amount of heap when considering the drive
>> towards tiny micro services.
>> >
>> > Paul Carter-Brown
>> > Director
>> > Jini Guru
>> > m:    +27 (0) 83 442 7179
>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>> Leslie
>> >       Johannesburg, South Africa
>> > w:    jini.guru  e: [hidden email]
>> >
>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>> confidential and/or personal information. If you are not the intended
>> recipient you may not disclose or distribute any of the information
>> contained within this message. In such case you must destroy this message
>> and inform the sender of the error. Jini Guru may not accept liability for
>> any errors, omissions, information and viruses contained in the
>> transmission of this message. Any opinions, conclusions and other
>> information contained within this message not related to Jini Guru official
>> business is deemed to be that of the individual only and is not endorsed by
>> Jini Guru.
>> >
>> >
>> >
>> >
>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>> <[hidden email]> wrote:
>> > Hi Jon,
>> >
>> > Unfortunately the snapshot behaves exactly the same way
>> >
>> > Paul Carter-Brown
>> > Director
>> > Jini Guru
>> > m:    +27 (0) 83 442 7179
>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>> Leslie
>> >       Johannesburg, South Africa
>> > w:    jini.guru  e: [hidden email]
>> >
>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>> confidential and/or personal information. If you are not the intended
>> recipient you may not disclose or distribute any of the information
>> contained within this message. In such case you must destroy this message
>> and inform the sender of the error. Jini Guru may not accept liability for
>> any errors, omissions, information and viruses contained in the
>> transmission of this message. Any opinions, conclusions and other
>> information contained within this message not related to Jini Guru official
>> business is deemed to be that of the individual only and is not endorsed by
>> Jini Guru.
>> >
>> >
>> >
>> >
>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>> [hidden email]> wrote:
>> > Hi Paul
>> >
>> > Would you mind trying your application with a recent snapshot?
>> >
>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>> .
>> > I recently updated the exclude list.
>> >
>> > Many thanks
>> >
>> > Jon
>> >
>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>> > <[hidden email]> wrote:
>> >
>> > > Hi,
>> > >
>> > > I've been trying to lower the memory footprint of an EAR deployed to
>> TomEE
>> > > 8.0.0.
>> > >
>> > > Here is a screenshot of the heap histogram. The total Old Gen is about
>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my EAR
>> then
>> > > the old gen is about 6MB.
>> > >
>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>> > >
>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>> directory
>> > > which contains libs such as Kafka, hazelcast, apache POI, Google cloud
>> > > APIs, AWS client APIs etc etc. In total our code has about 100 actual
>> > > EJB's. If i remove files from the lib folder in the ear then I can
>> see that
>> > > the memory used by the annotation finder is lowered.
>> > >
>> > > Is there any way I can tell TomEE that it need not bother scanning
>> > > everything in the /lib folder of my EAR for annotations and fulling
>> up the
>> > > heap. I already
>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
>> scan
>> > > only the one jar in /lib which does have EJB's in it and it seems to
>> obey
>> > > this property but it doesn't seem to mean that annotation processing
>> is
>> > > skipped for all these other jars in /lib
>> > >
>> > > Thanks!
>> > >
>> > > Paul Carter-Brown
>> > > Director
>> > > Jini Guru
>> > > m: +27 (0) 83 442 7179 <+27834427179>
>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>> Leslie
>> > >   Johannesburg, South Africa
>> > > w: jini.guru  e: [hidden email]
>> > >
>> > > Disclaimer: This message and/or attachment(s) may contain
>> > > privileged, confidential and/or personal information. If you are not
>> the
>> > > intended recipient you may not disclose or distribute any of
>> > > the information contained within this message. In such case you must
>> > > destroy this message and inform the sender of the error. Jini Guru
>> may not
>> > > accept liability for any errors, omissions, information and viruses
>> > > contained in the transmission of this message. Any opinions,
>> conclusions
>> > > and other information contained within this message not related to
>> Jini
>> > > Guru official business is deemed to be that of the individual only
>> and is
>> > > not endorsed by Jini Guru.
>> > >
>> > >
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

jgallimore
Thanks for sending those over. I'm digging through this - I *think* I have
pinned down a specific reference that stops that classloader from being
released. The behaviour for a war/ear deployed in apps/ seems different to
deploying in webapps/ - I'm assuming that's what you're doing. If you can
confirm, that would help. I'll test out a patch with some bigger .ear files
today.

Thanks

Jon

On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
[hidden email]> wrote:

> The mailing list seems to be stripping off your images - can you resend
> and include my email address as well? I'd be keen to dig into this a little
> more.
>
> Thanks
>
> Jon
>
> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
> <[hidden email]> wrote:
>
>> Hi,
>>
>> The code does seem to use org.apache.openejb.core.TempClassLoader. I can
>> see one instance in the heap for every war in my ear. + 1.
>>
>> The TempClassLoader however still has 100's of strong references to it.
>> E.g:
>>
>> [image: image.png]
>>
>> [image: image.png]
>>
>>
>> Paul Carter-Brown
>> Director
>> Jini Guru
>> m: +27 (0) 83 442 7179 <+27834427179>
>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>>   Johannesburg, South Africa
>> w: jini.guru  e: [hidden email]
>>
>> Disclaimer: This message and/or attachment(s) may contain
>> privileged, confidential and/or personal information. If you are not the
>> intended recipient you may not disclose or distribute any of
>> the information contained within this message. In such case you must
>> destroy this message and inform the sender of the error. Jini Guru may not
>> accept liability for any errors, omissions, information and viruses
>> contained in the transmission of this message. Any opinions, conclusions
>> and other information contained within this message not related to Jini
>> Guru official business is deemed to be that of the individual only and is
>> not endorsed by Jini Guru.
>>
>>
>>
>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <[hidden email]>
>> wrote:
>>
>>> Hi!
>>>
>>> All this is just a workaround imo.
>>>
>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>> scanning. That means after doing all the class scans we only keep the
>>> extracted information but do not keep the Classes in memory. Doesn't this
>>> work anymore?
>>>
>>> LieGrue,
>>> strub
>>>
>>>
>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>> <[hidden email]>:
>>> >
>>> > FYI. Graph of the change in heap size on a prod instance after this
>>> change:
>>> >
>>> >
>>> > Paul Carter-Brown
>>> > Director
>>> > Jini Guru
>>> > m:    +27 (0) 83 442 7179
>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>> Leslie
>>> >       Johannesburg, South Africa
>>> > w:    jini.guru  e: [hidden email]
>>> >
>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>> confidential and/or personal information. If you are not the intended
>>> recipient you may not disclose or distribute any of the information
>>> contained within this message. In such case you must destroy this message
>>> and inform the sender of the error. Jini Guru may not accept liability for
>>> any errors, omissions, information and viruses contained in the
>>> transmission of this message. Any opinions, conclusions and other
>>> information contained within this message not related to Jini Guru official
>>> business is deemed to be that of the individual only and is not endorsed by
>>> Jini Guru.
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>> <[hidden email]> wrote:
>>> > Hi Jon,
>>> >
>>> > I took a look at the source and worked out that I could add my own
>>> filters using openejb.additional.exclude.
>>> >
>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM heap
>>> now sits at around 150MB whereas it would never go below 450MB previously!!!
>>> >
>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>> they are not filtered out and the system works 100%.
>>> >
>>> > My sense is that there should be an easier way to specify what jars
>>> and/or packages to process for annotations. I have come across the
>>> following settings but can't really work out what fits where:
>>> >
>>> > openejb.additional.exclude
>>> > openejb.additional.include
>>> > openejb.deployments.classpath.filter.descriptors
>>> > openejb.deployments.classpath.filter.systemapps
>>> > openejb.deployments.classpath.include
>>> > openejb.deployments.classpath.exclude
>>> > openejb.deployments.package.include
>>> > openejb.deployments.package.exclude
>>> >
>>> >
>>> > P.S. Perhaps also add these to the standard exclusion list as they are
>>> common and yet won't need to be processed for annotations I guess?
>>> > com.amazonaws
>>> > com.fasterxml
>>> > com.google
>>> > com.hazelcast
>>> > io.grpc
>>> > io.netty
>>> > org.docx4j
>>> > org.mongodb
>>> > org.rocksdb
>>> > org.asynchttpclient
>>> > org.apache
>>> > org.aspectj
>>> >
>>> > But then maybe some are too broad and I don't really understand what
>>> the annotation index/cache is really needed for at runtime? Can it not be
>>> lazy loaded or discarded if unused? Just thinking out loud here :-( Seems
>>> like the cache uses a significant amount of heap when considering the drive
>>> towards tiny micro services.
>>> >
>>> > Paul Carter-Brown
>>> > Director
>>> > Jini Guru
>>> > m:    +27 (0) 83 442 7179
>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>> Leslie
>>> >       Johannesburg, South Africa
>>> > w:    jini.guru  e: [hidden email]
>>> >
>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>> confidential and/or personal information. If you are not the intended
>>> recipient you may not disclose or distribute any of the information
>>> contained within this message. In such case you must destroy this message
>>> and inform the sender of the error. Jini Guru may not accept liability for
>>> any errors, omissions, information and viruses contained in the
>>> transmission of this message. Any opinions, conclusions and other
>>> information contained within this message not related to Jini Guru official
>>> business is deemed to be that of the individual only and is not endorsed by
>>> Jini Guru.
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>> <[hidden email]> wrote:
>>> > Hi Jon,
>>> >
>>> > Unfortunately the snapshot behaves exactly the same way
>>> >
>>> > Paul Carter-Brown
>>> > Director
>>> > Jini Guru
>>> > m:    +27 (0) 83 442 7179
>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>> Leslie
>>> >       Johannesburg, South Africa
>>> > w:    jini.guru  e: [hidden email]
>>> >
>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>> confidential and/or personal information. If you are not the intended
>>> recipient you may not disclose or distribute any of the information
>>> contained within this message. In such case you must destroy this message
>>> and inform the sender of the error. Jini Guru may not accept liability for
>>> any errors, omissions, information and viruses contained in the
>>> transmission of this message. Any opinions, conclusions and other
>>> information contained within this message not related to Jini Guru official
>>> business is deemed to be that of the individual only and is not endorsed by
>>> Jini Guru.
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>> [hidden email]> wrote:
>>> > Hi Paul
>>> >
>>> > Would you mind trying your application with a recent snapshot?
>>> >
>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>> .
>>> > I recently updated the exclude list.
>>> >
>>> > Many thanks
>>> >
>>> > Jon
>>> >
>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>> > <[hidden email]> wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > I've been trying to lower the memory footprint of an EAR deployed to
>>> TomEE
>>> > > 8.0.0.
>>> > >
>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>> about
>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my EAR
>>> then
>>> > > the old gen is about 6MB.
>>> > >
>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>> > >
>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>> directory
>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>> cloud
>>> > > APIs, AWS client APIs etc etc. In total our code has about 100 actual
>>> > > EJB's. If i remove files from the lib folder in the ear then I can
>>> see that
>>> > > the memory used by the annotation finder is lowered.
>>> > >
>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>> > > everything in the /lib folder of my EAR for annotations and fulling
>>> up the
>>> > > heap. I already
>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
>>> scan
>>> > > only the one jar in /lib which does have EJB's in it and it seems to
>>> obey
>>> > > this property but it doesn't seem to mean that annotation processing
>>> is
>>> > > skipped for all these other jars in /lib
>>> > >
>>> > > Thanks!
>>> > >
>>> > > Paul Carter-Brown
>>> > > Director
>>> > > Jini Guru
>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>> Leslie
>>> > >   Johannesburg, South Africa
>>> > > w: jini.guru  e: [hidden email]
>>> > >
>>> > > Disclaimer: This message and/or attachment(s) may contain
>>> > > privileged, confidential and/or personal information. If you are not
>>> the
>>> > > intended recipient you may not disclose or distribute any of
>>> > > the information contained within this message. In such case you must
>>> > > destroy this message and inform the sender of the error. Jini Guru
>>> may not
>>> > > accept liability for any errors, omissions, information and viruses
>>> > > contained in the transmission of this message. Any opinions,
>>> conclusions
>>> > > and other information contained within this message not related to
>>> Jini
>>> > > Guru official business is deemed to be that of the individual only
>>> and is
>>> > > not endorsed by Jini Guru.
>>> > >
>>> > >
>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

jgallimore
Further update - I think my changes work with respect to this issue, but I
do have a bunch of unit test failures which I need to look into. For info,
here's the diff:

diff --git
a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
index a6e2bdc4f5..c0d1a8c98f 100644
---
a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
+++
b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
@@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
 import org.apache.openejb.classloader.ClassLoaderConfigurer;
 import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
 import org.apache.openejb.component.ClassLoaderEnricher;
-import org.apache.openejb.config.ConfigurationFactory;
-import org.apache.openejb.config.NewLoaderLogic;
-import org.apache.openejb.config.QuickJarsTxtParser;
-import org.apache.openejb.config.TldScanner;
+import org.apache.openejb.config.*;
 import org.apache.openejb.core.ConnectorReference;
 import org.apache.openejb.core.CoreContainerSystem;
 import org.apache.openejb.core.CoreUserTransaction;
@@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
implements org.apache.openejb.spi.A

                 final AppContext appContext = new
AppContext(appInfo.appId, SystemInstance.get(), classLoader,
globalJndiContext, appJndiContext, appInfo.standaloneModule);
                 appContext.getProperties().putAll(appInfo.properties);
+
+                final Set<Object> keys =
appContext.getProperties().keySet();
+                for (final Object key : keys) {
+                    if (! Module.class.isInstance(key)) {
+                        appContext.getProperties().remove(key);
+                    }
+                }
+
                 appContext.getInjections().addAll(injections);
                 appContext.getBindings().putAll(globalBindings);
                 appContext.getBindings().putAll(appBindings);
diff --git
a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
index feff954d40..aed0fda941 100644
---
a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
+++
b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
@@ -384,12 +384,7 @@ public class JndiEncBuilder {
                 reference = new LinkRef(jndiName);

             } else if (BeanManager.class.equals(type)) {
-                reference = new LazyObjectReference<BeanManager>(new
Callable<BeanManager>() {
-                    @Override
-                    public BeanManager call() throws Exception {
-                        return new
InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
-                    }
-                });
+                reference = new LazyObjectReference<>(new
BeanManagerLazyReference());

             } else if (UserTransaction.class.equals(type)) {
                 reference = new
IntraVmJndiReference("comp/UserTransaction");
@@ -684,4 +679,11 @@ public class JndiEncBuilder {
             }
         }
     }
+
+    public static class BeanManagerLazyReference implements
Callable<BeanManager> {
+        @Override
+        public BeanManager call() throws Exception {
+            return new
InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
+        }
+    }
 }
diff --git
a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
index 2cf387e112..285007925f 100644
---
a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
+++
b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
@@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
     private final ClassLoader system;
     private final boolean embedded;
     private final boolean parentURLClassLoader;
+    private final StackTraceElement[] created;

     // 80% of class files are smaller then 6k
     private final ByteArrayOutputStream bout = new ByteArrayOutputStream(6
* 1024);
@@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
         this.system = ClassLoader.getSystemClassLoader();
         this.embedded = this.getClass().getClassLoader() == this.system;
         this.parentURLClassLoader =
URLClassLoader.class.isInstance(parent);
+        this.created = new Throwable().getStackTrace();
     }

     /*

In essence, this does 2 things:

1. Defines the Callable for the LazyObjectReference for BeanManager as a
static class. This stops that object from having a reference to
JndiEncBuilder and everything that has attached to it.
2. Strips off the EjbModule / WebModule from the SuperProperties on
AppContext. This is set when deploying through the apps folder and lives
forever. It has the TempClassLoader attached, and therefore all the
AnnotationFinder stuff.

I suspect (2) is too aggressive and is causing the test failures, but will
confirm.

Jon

On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
[hidden email]> wrote:

> Thanks for sending those over. I'm digging through this - I *think* I have
> pinned down a specific reference that stops that classloader from being
> released. The behaviour for a war/ear deployed in apps/ seems different to
> deploying in webapps/ - I'm assuming that's what you're doing. If you can
> confirm, that would help. I'll test out a patch with some bigger .ear files
> today.
>
> Thanks
>
> Jon
>
> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
> [hidden email]> wrote:
>
>> The mailing list seems to be stripping off your images - can you resend
>> and include my email address as well? I'd be keen to dig into this a little
>> more.
>>
>> Thanks
>>
>> Jon
>>
>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>> <[hidden email]> wrote:
>>
>>> Hi,
>>>
>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I can
>>> see one instance in the heap for every war in my ear. + 1.
>>>
>>> The TempClassLoader however still has 100's of strong references to it.
>>> E.g:
>>>
>>> [image: image.png]
>>>
>>> [image: image.png]
>>>
>>>
>>> Paul Carter-Brown
>>> Director
>>> Jini Guru
>>> m: +27 (0) 83 442 7179 <+27834427179>
>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>>>   Johannesburg, South Africa
>>> w: jini.guru  e: [hidden email]
>>>
>>> Disclaimer: This message and/or attachment(s) may contain
>>> privileged, confidential and/or personal information. If you are not the
>>> intended recipient you may not disclose or distribute any of
>>> the information contained within this message. In such case you must
>>> destroy this message and inform the sender of the error. Jini Guru may not
>>> accept liability for any errors, omissions, information and viruses
>>> contained in the transmission of this message. Any opinions, conclusions
>>> and other information contained within this message not related to Jini
>>> Guru official business is deemed to be that of the individual only and is
>>> not endorsed by Jini Guru.
>>>
>>>
>>>
>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <[hidden email]>
>>> wrote:
>>>
>>>> Hi!
>>>>
>>>> All this is just a workaround imo.
>>>>
>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>> scanning. That means after doing all the class scans we only keep the
>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>> work anymore?
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>>>
>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>> <[hidden email]>:
>>>> >
>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>> change:
>>>> >
>>>> >
>>>> > Paul Carter-Brown
>>>> > Director
>>>> > Jini Guru
>>>> > m:    +27 (0) 83 442 7179
>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>> Leslie
>>>> >       Johannesburg, South Africa
>>>> > w:    jini.guru  e: [hidden email]
>>>> >
>>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>>> confidential and/or personal information. If you are not the intended
>>>> recipient you may not disclose or distribute any of the information
>>>> contained within this message. In such case you must destroy this message
>>>> and inform the sender of the error. Jini Guru may not accept liability for
>>>> any errors, omissions, information and viruses contained in the
>>>> transmission of this message. Any opinions, conclusions and other
>>>> information contained within this message not related to Jini Guru official
>>>> business is deemed to be that of the individual only and is not endorsed by
>>>> Jini Guru.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>> <[hidden email]> wrote:
>>>> > Hi Jon,
>>>> >
>>>> > I took a look at the source and worked out that I could add my own
>>>> filters using openejb.additional.exclude.
>>>> >
>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>> previously!!!
>>>> >
>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>> they are not filtered out and the system works 100%.
>>>> >
>>>> > My sense is that there should be an easier way to specify what jars
>>>> and/or packages to process for annotations. I have come across the
>>>> following settings but can't really work out what fits where:
>>>> >
>>>> > openejb.additional.exclude
>>>> > openejb.additional.include
>>>> > openejb.deployments.classpath.filter.descriptors
>>>> > openejb.deployments.classpath.filter.systemapps
>>>> > openejb.deployments.classpath.include
>>>> > openejb.deployments.classpath.exclude
>>>> > openejb.deployments.package.include
>>>> > openejb.deployments.package.exclude
>>>> >
>>>> >
>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>> are common and yet won't need to be processed for annotations I guess?
>>>> > com.amazonaws
>>>> > com.fasterxml
>>>> > com.google
>>>> > com.hazelcast
>>>> > io.grpc
>>>> > io.netty
>>>> > org.docx4j
>>>> > org.mongodb
>>>> > org.rocksdb
>>>> > org.asynchttpclient
>>>> > org.apache
>>>> > org.aspectj
>>>> >
>>>> > But then maybe some are too broad and I don't really understand what
>>>> the annotation index/cache is really needed for at runtime? Can it not be
>>>> lazy loaded or discarded if unused? Just thinking out loud here :-( Seems
>>>> like the cache uses a significant amount of heap when considering the drive
>>>> towards tiny micro services.
>>>> >
>>>> > Paul Carter-Brown
>>>> > Director
>>>> > Jini Guru
>>>> > m:    +27 (0) 83 442 7179
>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>> Leslie
>>>> >       Johannesburg, South Africa
>>>> > w:    jini.guru  e: [hidden email]
>>>> >
>>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>>> confidential and/or personal information. If you are not the intended
>>>> recipient you may not disclose or distribute any of the information
>>>> contained within this message. In such case you must destroy this message
>>>> and inform the sender of the error. Jini Guru may not accept liability for
>>>> any errors, omissions, information and viruses contained in the
>>>> transmission of this message. Any opinions, conclusions and other
>>>> information contained within this message not related to Jini Guru official
>>>> business is deemed to be that of the individual only and is not endorsed by
>>>> Jini Guru.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>> <[hidden email]> wrote:
>>>> > Hi Jon,
>>>> >
>>>> > Unfortunately the snapshot behaves exactly the same way
>>>> >
>>>> > Paul Carter-Brown
>>>> > Director
>>>> > Jini Guru
>>>> > m:    +27 (0) 83 442 7179
>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>> Leslie
>>>> >       Johannesburg, South Africa
>>>> > w:    jini.guru  e: [hidden email]
>>>> >
>>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>>> confidential and/or personal information. If you are not the intended
>>>> recipient you may not disclose or distribute any of the information
>>>> contained within this message. In such case you must destroy this message
>>>> and inform the sender of the error. Jini Guru may not accept liability for
>>>> any errors, omissions, information and viruses contained in the
>>>> transmission of this message. Any opinions, conclusions and other
>>>> information contained within this message not related to Jini Guru official
>>>> business is deemed to be that of the individual only and is not endorsed by
>>>> Jini Guru.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>> [hidden email]> wrote:
>>>> > Hi Paul
>>>> >
>>>> > Would you mind trying your application with a recent snapshot?
>>>> >
>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>> .
>>>> > I recently updated the exclude list.
>>>> >
>>>> > Many thanks
>>>> >
>>>> > Jon
>>>> >
>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>> > <[hidden email]> wrote:
>>>> >
>>>> > > Hi,
>>>> > >
>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>> to TomEE
>>>> > > 8.0.0.
>>>> > >
>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>> about
>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my EAR
>>>> then
>>>> > > the old gen is about 6MB.
>>>> > >
>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>> > >
>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>> directory
>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>> cloud
>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>> actual
>>>> > > EJB's. If i remove files from the lib folder in the ear then I can
>>>> see that
>>>> > > the memory used by the annotation finder is lowered.
>>>> > >
>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>> > > everything in the /lib folder of my EAR for annotations and fulling
>>>> up the
>>>> > > heap. I already
>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
>>>> scan
>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>> to obey
>>>> > > this property but it doesn't seem to mean that annotation
>>>> processing is
>>>> > > skipped for all these other jars in /lib
>>>> > >
>>>> > > Thanks!
>>>> > >
>>>> > > Paul Carter-Brown
>>>> > > Director
>>>> > > Jini Guru
>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>> Leslie
>>>> > >   Johannesburg, South Africa
>>>> > > w: jini.guru  e: [hidden email]
>>>> > >
>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>> > > privileged, confidential and/or personal information. If you are
>>>> not the
>>>> > > intended recipient you may not disclose or distribute any of
>>>> > > the information contained within this message. In such case you must
>>>> > > destroy this message and inform the sender of the error. Jini Guru
>>>> may not
>>>> > > accept liability for any errors, omissions, information and viruses
>>>> > > contained in the transmission of this message. Any opinions,
>>>> conclusions
>>>> > > and other information contained within this message not related to
>>>> Jini
>>>> > > Guru official business is deemed to be that of the individual only
>>>> and is
>>>> > > not endorsed by Jini Guru.
>>>> > >
>>>> > >
>>>>
>>>>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

PaulC.B
Hi Jon

Excellent stuff. Yes it's an ear with about 10 skinny wars inside and all
shared libs in /libs in the ear . Each war is a microservice but we deploy
as an ear in environments where it's too much of a hassle to split the
services up and deploy individually.

Thanks so much for your effort on this!

On Mon, 16 Dec 2019, 16:48 Jonathan Gallimore, <[hidden email]>
wrote:

> Further update - I think my changes work with respect to this issue, but I
> do have a bunch of unit test failures which I need to look into. For info,
> here's the diff:
>
> diff --git
>
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> index a6e2bdc4f5..c0d1a8c98f 100644
> ---
>
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> +++
>
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>  import org.apache.openejb.classloader.ClassLoaderConfigurer;
>  import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>  import org.apache.openejb.component.ClassLoaderEnricher;
> -import org.apache.openejb.config.ConfigurationFactory;
> -import org.apache.openejb.config.NewLoaderLogic;
> -import org.apache.openejb.config.QuickJarsTxtParser;
> -import org.apache.openejb.config.TldScanner;
> +import org.apache.openejb.config.*;
>  import org.apache.openejb.core.ConnectorReference;
>  import org.apache.openejb.core.CoreContainerSystem;
>  import org.apache.openejb.core.CoreUserTransaction;
> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
> implements org.apache.openejb.spi.A
>
>                  final AppContext appContext = new
> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>                  appContext.getProperties().putAll(appInfo.properties);
> +
> +                final Set<Object> keys =
> appContext.getProperties().keySet();
> +                for (final Object key : keys) {
> +                    if (! Module.class.isInstance(key)) {
> +                        appContext.getProperties().remove(key);
> +                    }
> +                }
> +
>                  appContext.getInjections().addAll(injections);
>                  appContext.getBindings().putAll(globalBindings);
>                  appContext.getBindings().putAll(appBindings);
> diff --git
>
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> index feff954d40..aed0fda941 100644
> ---
>
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> +++
>
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>                  reference = new LinkRef(jndiName);
>
>              } else if (BeanManager.class.equals(type)) {
> -                reference = new LazyObjectReference<BeanManager>(new
> Callable<BeanManager>() {
> -                    @Override
> -                    public BeanManager call() throws Exception {
> -                        return new
>
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> -                    }
> -                });
> +                reference = new LazyObjectReference<>(new
> BeanManagerLazyReference());
>
>              } else if (UserTransaction.class.equals(type)) {
>                  reference = new
> IntraVmJndiReference("comp/UserTransaction");
> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>              }
>          }
>      }
> +
> +    public static class BeanManagerLazyReference implements
> Callable<BeanManager> {
> +        @Override
> +        public BeanManager call() throws Exception {
> +            return new
>
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> +        }
> +    }
>  }
> diff --git
>
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> index 2cf387e112..285007925f 100644
> ---
>
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> +++
>
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>      private final ClassLoader system;
>      private final boolean embedded;
>      private final boolean parentURLClassLoader;
> +    private final StackTraceElement[] created;
>
>      // 80% of class files are smaller then 6k
>      private final ByteArrayOutputStream bout = new ByteArrayOutputStream(6
> * 1024);
> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>          this.system = ClassLoader.getSystemClassLoader();
>          this.embedded = this.getClass().getClassLoader() == this.system;
>          this.parentURLClassLoader =
> URLClassLoader.class.isInstance(parent);
> +        this.created = new Throwable().getStackTrace();
>      }
>
>      /*
>
> In essence, this does 2 things:
>
> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
> static class. This stops that object from having a reference to
> JndiEncBuilder and everything that has attached to it.
> 2. Strips off the EjbModule / WebModule from the SuperProperties on
> AppContext. This is set when deploying through the apps folder and lives
> forever. It has the TempClassLoader attached, and therefore all the
> AnnotationFinder stuff.
>
> I suspect (2) is too aggressive and is causing the test failures, but will
> confirm.
>
> Jon
>
> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
> [hidden email]> wrote:
>
> > Thanks for sending those over. I'm digging through this - I *think* I
> have
> > pinned down a specific reference that stops that classloader from being
> > released. The behaviour for a war/ear deployed in apps/ seems different
> to
> > deploying in webapps/ - I'm assuming that's what you're doing. If you can
> > confirm, that would help. I'll test out a patch with some bigger .ear
> files
> > today.
> >
> > Thanks
> >
> > Jon
> >
> > On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
> > [hidden email]> wrote:
> >
> >> The mailing list seems to be stripping off your images - can you resend
> >> and include my email address as well? I'd be keen to dig into this a
> little
> >> more.
> >>
> >> Thanks
> >>
> >> Jon
> >>
> >> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
> >> <[hidden email]> wrote:
> >>
> >>> Hi,
> >>>
> >>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
> can
> >>> see one instance in the heap for every war in my ear. + 1.
> >>>
> >>> The TempClassLoader however still has 100's of strong references to it.
> >>> E.g:
> >>>
> >>> [image: image.png]
> >>>
> >>> [image: image.png]
> >>>
> >>>
> >>> Paul Carter-Brown
> >>> Director
> >>> Jini Guru
> >>> m: +27 (0) 83 442 7179 <+27834427179>
> >>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
> >>>   Johannesburg, South Africa
> >>> w: jini.guru  e: [hidden email]
> >>>
> >>> Disclaimer: This message and/or attachment(s) may contain
> >>> privileged, confidential and/or personal information. If you are not
> the
> >>> intended recipient you may not disclose or distribute any of
> >>> the information contained within this message. In such case you must
> >>> destroy this message and inform the sender of the error. Jini Guru may
> not
> >>> accept liability for any errors, omissions, information and viruses
> >>> contained in the transmission of this message. Any opinions,
> conclusions
> >>> and other information contained within this message not related to Jini
> >>> Guru official business is deemed to be that of the individual only and
> is
> >>> not endorsed by Jini Guru.
> >>>
> >>>
> >>>
> >>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
> <[hidden email]>
> >>> wrote:
> >>>
> >>>> Hi!
> >>>>
> >>>> All this is just a workaround imo.
> >>>>
> >>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
> >>>> scanning. That means after doing all the class scans we only keep the
> >>>> extracted information but do not keep the Classes in memory. Doesn't
> this
> >>>> work anymore?
> >>>>
> >>>> LieGrue,
> >>>> strub
> >>>>
> >>>>
> >>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
> >>>> <[hidden email]>:
> >>>> >
> >>>> > FYI. Graph of the change in heap size on a prod instance after this
> >>>> change:
> >>>> >
> >>>> >
> >>>> > Paul Carter-Brown
> >>>> > Director
> >>>> > Jini Guru
> >>>> > m:    +27 (0) 83 442 7179
> >>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
> >>>> Leslie
> >>>> >       Johannesburg, South Africa
> >>>> > w:    jini.guru  e: [hidden email]
> >>>> >
> >>>> > Disclaimer: This message and/or attachment(s) may contain
> privileged,
> >>>> confidential and/or personal information. If you are not the intended
> >>>> recipient you may not disclose or distribute any of the information
> >>>> contained within this message. In such case you must destroy this
> message
> >>>> and inform the sender of the error. Jini Guru may not accept
> liability for
> >>>> any errors, omissions, information and viruses contained in the
> >>>> transmission of this message. Any opinions, conclusions and other
> >>>> information contained within this message not related to Jini Guru
> official
> >>>> business is deemed to be that of the individual only and is not
> endorsed by
> >>>> Jini Guru.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
> >>>> <[hidden email]> wrote:
> >>>> > Hi Jon,
> >>>> >
> >>>> > I took a look at the source and worked out that I could add my own
> >>>> filters using openejb.additional.exclude.
> >>>> >
> >>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
> >>>> heap now sits at around 150MB whereas it would never go below 450MB
> >>>> previously!!!
> >>>> >
> >>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
> >>>> they are not filtered out and the system works 100%.
> >>>> >
> >>>> > My sense is that there should be an easier way to specify what jars
> >>>> and/or packages to process for annotations. I have come across the
> >>>> following settings but can't really work out what fits where:
> >>>> >
> >>>> > openejb.additional.exclude
> >>>> > openejb.additional.include
> >>>> > openejb.deployments.classpath.filter.descriptors
> >>>> > openejb.deployments.classpath.filter.systemapps
> >>>> > openejb.deployments.classpath.include
> >>>> > openejb.deployments.classpath.exclude
> >>>> > openejb.deployments.package.include
> >>>> > openejb.deployments.package.exclude
> >>>> >
> >>>> >
> >>>> > P.S. Perhaps also add these to the standard exclusion list as they
> >>>> are common and yet won't need to be processed for annotations I guess?
> >>>> > com.amazonaws
> >>>> > com.fasterxml
> >>>> > com.google
> >>>> > com.hazelcast
> >>>> > io.grpc
> >>>> > io.netty
> >>>> > org.docx4j
> >>>> > org.mongodb
> >>>> > org.rocksdb
> >>>> > org.asynchttpclient
> >>>> > org.apache
> >>>> > org.aspectj
> >>>> >
> >>>> > But then maybe some are too broad and I don't really understand what
> >>>> the annotation index/cache is really needed for at runtime? Can it
> not be
> >>>> lazy loaded or discarded if unused? Just thinking out loud here :-(
> Seems
> >>>> like the cache uses a significant amount of heap when considering the
> drive
> >>>> towards tiny micro services.
> >>>> >
> >>>> > Paul Carter-Brown
> >>>> > Director
> >>>> > Jini Guru
> >>>> > m:    +27 (0) 83 442 7179
> >>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
> >>>> Leslie
> >>>> >       Johannesburg, South Africa
> >>>> > w:    jini.guru  e: [hidden email]
> >>>> >
> >>>> > Disclaimer: This message and/or attachment(s) may contain
> privileged,
> >>>> confidential and/or personal information. If you are not the intended
> >>>> recipient you may not disclose or distribute any of the information
> >>>> contained within this message. In such case you must destroy this
> message
> >>>> and inform the sender of the error. Jini Guru may not accept
> liability for
> >>>> any errors, omissions, information and viruses contained in the
> >>>> transmission of this message. Any opinions, conclusions and other
> >>>> information contained within this message not related to Jini Guru
> official
> >>>> business is deemed to be that of the individual only and is not
> endorsed by
> >>>> Jini Guru.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
> >>>> <[hidden email]> wrote:
> >>>> > Hi Jon,
> >>>> >
> >>>> > Unfortunately the snapshot behaves exactly the same way
> >>>> >
> >>>> > Paul Carter-Brown
> >>>> > Director
> >>>> > Jini Guru
> >>>> > m:    +27 (0) 83 442 7179
> >>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
> >>>> Leslie
> >>>> >       Johannesburg, South Africa
> >>>> > w:    jini.guru  e: [hidden email]
> >>>> >
> >>>> > Disclaimer: This message and/or attachment(s) may contain
> privileged,
> >>>> confidential and/or personal information. If you are not the intended
> >>>> recipient you may not disclose or distribute any of the information
> >>>> contained within this message. In such case you must destroy this
> message
> >>>> and inform the sender of the error. Jini Guru may not accept
> liability for
> >>>> any errors, omissions, information and viruses contained in the
> >>>> transmission of this message. Any opinions, conclusions and other
> >>>> information contained within this message not related to Jini Guru
> official
> >>>> business is deemed to be that of the individual only and is not
> endorsed by
> >>>> Jini Guru.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
> >>>> [hidden email]> wrote:
> >>>> > Hi Paul
> >>>> >
> >>>> > Would you mind trying your application with a recent snapshot?
> >>>> >
> >>>>
> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
> >>>> .
> >>>> > I recently updated the exclude list.
> >>>> >
> >>>> > Many thanks
> >>>> >
> >>>> > Jon
> >>>> >
> >>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
> >>>> > <[hidden email]> wrote:
> >>>> >
> >>>> > > Hi,
> >>>> > >
> >>>> > > I've been trying to lower the memory footprint of an EAR deployed
> >>>> to TomEE
> >>>> > > 8.0.0.
> >>>> > >
> >>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
> >>>> about
> >>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
> EAR
> >>>> then
> >>>> > > the old gen is about 6MB.
> >>>> > >
> >>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
> >>>> > >
> >>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
> >>>> directory
> >>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
> >>>> cloud
> >>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
> >>>> actual
> >>>> > > EJB's. If i remove files from the lib folder in the ear then I can
> >>>> see that
> >>>> > > the memory used by the annotation finder is lowered.
> >>>> > >
> >>>> > > Is there any way I can tell TomEE that it need not bother scanning
> >>>> > > everything in the /lib folder of my EAR for annotations and
> fulling
> >>>> up the
> >>>> > > heap. I already
> >>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
> >>>> scan
> >>>> > > only the one jar in /lib which does have EJB's in it and it seems
> >>>> to obey
> >>>> > > this property but it doesn't seem to mean that annotation
> >>>> processing is
> >>>> > > skipped for all these other jars in /lib
> >>>> > >
> >>>> > > Thanks!
> >>>> > >
> >>>> > > Paul Carter-Brown
> >>>> > > Director
> >>>> > > Jini Guru
> >>>> > > m: +27 (0) 83 442 7179 <+27834427179>
> >>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
> >>>> Leslie
> >>>> > >   Johannesburg, South Africa
> >>>> > > w: jini.guru  e: [hidden email]
> >>>> > >
> >>>> > > Disclaimer: This message and/or attachment(s) may contain
> >>>> > > privileged, confidential and/or personal information. If you are
> >>>> not the
> >>>> > > intended recipient you may not disclose or distribute any of
> >>>> > > the information contained within this message. In such case you
> must
> >>>> > > destroy this message and inform the sender of the error. Jini Guru
> >>>> may not
> >>>> > > accept liability for any errors, omissions, information and
> viruses
> >>>> > > contained in the transmission of this message. Any opinions,
> >>>> conclusions
> >>>> > > and other information contained within this message not related to
> >>>> Jini
> >>>> > > Guru official business is deemed to be that of the individual only
> >>>> and is
> >>>> > > not endorsed by Jini Guru.
> >>>> > >
> >>>> > >
> >>>>
> >>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

jgallimore
In reply to this post by jgallimore
Hi Paul

I've pushed a new snapshot which (hopefully) should address this. The CI
build is running on it now. Would you mind giving it a try? If it looks
good, I'll finish rolling the release. Fingers crossed it looks ok for you.

Jon

On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
[hidden email]> wrote:

> Further update - I think my changes work with respect to this issue, but I
> do have a bunch of unit test failures which I need to look into. For info,
> here's the diff:
>
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> index a6e2bdc4f5..c0d1a8c98f 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>  import org.apache.openejb.classloader.ClassLoaderConfigurer;
>  import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>  import org.apache.openejb.component.ClassLoaderEnricher;
> -import org.apache.openejb.config.ConfigurationFactory;
> -import org.apache.openejb.config.NewLoaderLogic;
> -import org.apache.openejb.config.QuickJarsTxtParser;
> -import org.apache.openejb.config.TldScanner;
> +import org.apache.openejb.config.*;
>  import org.apache.openejb.core.ConnectorReference;
>  import org.apache.openejb.core.CoreContainerSystem;
>  import org.apache.openejb.core.CoreUserTransaction;
> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
> implements org.apache.openejb.spi.A
>
>                  final AppContext appContext = new
> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>                  appContext.getProperties().putAll(appInfo.properties);
> +
> +                final Set<Object> keys =
> appContext.getProperties().keySet();
> +                for (final Object key : keys) {
> +                    if (! Module.class.isInstance(key)) {
> +                        appContext.getProperties().remove(key);
> +                    }
> +                }
> +
>                  appContext.getInjections().addAll(injections);
>                  appContext.getBindings().putAll(globalBindings);
>                  appContext.getBindings().putAll(appBindings);
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> index feff954d40..aed0fda941 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>                  reference = new LinkRef(jndiName);
>
>              } else if (BeanManager.class.equals(type)) {
> -                reference = new LazyObjectReference<BeanManager>(new
> Callable<BeanManager>() {
> -                    @Override
> -                    public BeanManager call() throws Exception {
> -                        return new
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> -                    }
> -                });
> +                reference = new LazyObjectReference<>(new
> BeanManagerLazyReference());
>
>              } else if (UserTransaction.class.equals(type)) {
>                  reference = new
> IntraVmJndiReference("comp/UserTransaction");
> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>              }
>          }
>      }
> +
> +    public static class BeanManagerLazyReference implements
> Callable<BeanManager> {
> +        @Override
> +        public BeanManager call() throws Exception {
> +            return new
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> +        }
> +    }
>  }
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> index 2cf387e112..285007925f 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>      private final ClassLoader system;
>      private final boolean embedded;
>      private final boolean parentURLClassLoader;
> +    private final StackTraceElement[] created;
>
>      // 80% of class files are smaller then 6k
>      private final ByteArrayOutputStream bout = new
> ByteArrayOutputStream(6 * 1024);
> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>          this.system = ClassLoader.getSystemClassLoader();
>          this.embedded = this.getClass().getClassLoader() == this.system;
>          this.parentURLClassLoader =
> URLClassLoader.class.isInstance(parent);
> +        this.created = new Throwable().getStackTrace();
>      }
>
>      /*
>
> In essence, this does 2 things:
>
> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
> static class. This stops that object from having a reference to
> JndiEncBuilder and everything that has attached to it.
> 2. Strips off the EjbModule / WebModule from the SuperProperties on
> AppContext. This is set when deploying through the apps folder and lives
> forever. It has the TempClassLoader attached, and therefore all the
> AnnotationFinder stuff.
>
> I suspect (2) is too aggressive and is causing the test failures, but will
> confirm.
>
> Jon
>
> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
> [hidden email]> wrote:
>
>> Thanks for sending those over. I'm digging through this - I *think* I
>> have pinned down a specific reference that stops that classloader from
>> being released. The behaviour for a war/ear deployed in apps/ seems
>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>> If you can confirm, that would help. I'll test out a patch with some bigger
>> .ear files today.
>>
>> Thanks
>>
>> Jon
>>
>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>> [hidden email]> wrote:
>>
>>> The mailing list seems to be stripping off your images - can you resend
>>> and include my email address as well? I'd be keen to dig into this a little
>>> more.
>>>
>>> Thanks
>>>
>>> Jon
>>>
>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>> <[hidden email]> wrote:
>>>
>>>> Hi,
>>>>
>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>
>>>> The TempClassLoader however still has 100's of strong references to it.
>>>> E.g:
>>>>
>>>> [image: image.png]
>>>>
>>>> [image: image.png]
>>>>
>>>>
>>>> Paul Carter-Brown
>>>> Director
>>>> Jini Guru
>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>>>>   Johannesburg, South Africa
>>>> w: jini.guru  e: [hidden email]
>>>>
>>>> Disclaimer: This message and/or attachment(s) may contain
>>>> privileged, confidential and/or personal information. If you are not the
>>>> intended recipient you may not disclose or distribute any of
>>>> the information contained within this message. In such case you must
>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>> accept liability for any errors, omissions, information and viruses
>>>> contained in the transmission of this message. Any opinions, conclusions
>>>> and other information contained within this message not related to Jini
>>>> Guru official business is deemed to be that of the individual only and is
>>>> not endorsed by Jini Guru.
>>>>
>>>>
>>>>
>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> All this is just a workaround imo.
>>>>>
>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>> scanning. That means after doing all the class scans we only keep the
>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>> work anymore?
>>>>>
>>>>> LieGrue,
>>>>> strub
>>>>>
>>>>>
>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>> <[hidden email]>:
>>>>> >
>>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>>> change:
>>>>> >
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m:    +27 (0) 83 442 7179
>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> >       Johannesburg, South Africa
>>>>> > w:    jini.guru  e: [hidden email]
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>> <[hidden email]> wrote:
>>>>> > Hi Jon,
>>>>> >
>>>>> > I took a look at the source and worked out that I could add my own
>>>>> filters using openejb.additional.exclude.
>>>>> >
>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>> previously!!!
>>>>> >
>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>>> they are not filtered out and the system works 100%.
>>>>> >
>>>>> > My sense is that there should be an easier way to specify what jars
>>>>> and/or packages to process for annotations. I have come across the
>>>>> following settings but can't really work out what fits where:
>>>>> >
>>>>> > openejb.additional.exclude
>>>>> > openejb.additional.include
>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>> > openejb.deployments.classpath.include
>>>>> > openejb.deployments.classpath.exclude
>>>>> > openejb.deployments.package.include
>>>>> > openejb.deployments.package.exclude
>>>>> >
>>>>> >
>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>> > com.amazonaws
>>>>> > com.fasterxml
>>>>> > com.google
>>>>> > com.hazelcast
>>>>> > io.grpc
>>>>> > io.netty
>>>>> > org.docx4j
>>>>> > org.mongodb
>>>>> > org.rocksdb
>>>>> > org.asynchttpclient
>>>>> > org.apache
>>>>> > org.aspectj
>>>>> >
>>>>> > But then maybe some are too broad and I don't really understand what
>>>>> the annotation index/cache is really needed for at runtime? Can it not be
>>>>> lazy loaded or discarded if unused? Just thinking out loud here :-( Seems
>>>>> like the cache uses a significant amount of heap when considering the drive
>>>>> towards tiny micro services.
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m:    +27 (0) 83 442 7179
>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> >       Johannesburg, South Africa
>>>>> > w:    jini.guru  e: [hidden email]
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>> <[hidden email]> wrote:
>>>>> > Hi Jon,
>>>>> >
>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m:    +27 (0) 83 442 7179
>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> >       Johannesburg, South Africa
>>>>> > w:    jini.guru  e: [hidden email]
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>> [hidden email]> wrote:
>>>>> > Hi Paul
>>>>> >
>>>>> > Would you mind trying your application with a recent snapshot?
>>>>> >
>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>> .
>>>>> > I recently updated the exclude list.
>>>>> >
>>>>> > Many thanks
>>>>> >
>>>>> > Jon
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>> > <[hidden email]> wrote:
>>>>> >
>>>>> > > Hi,
>>>>> > >
>>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>>> to TomEE
>>>>> > > 8.0.0.
>>>>> > >
>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>> about
>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>> EAR then
>>>>> > > the old gen is about 6MB.
>>>>> > >
>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>> > >
>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>> directory
>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>> cloud
>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>> actual
>>>>> > > EJB's. If i remove files from the lib folder in the ear then I can
>>>>> see that
>>>>> > > the memory used by the annotation finder is lowered.
>>>>> > >
>>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>> fulling up the
>>>>> > > heap. I already
>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
>>>>> scan
>>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>>> to obey
>>>>> > > this property but it doesn't seem to mean that annotation
>>>>> processing is
>>>>> > > skipped for all these other jars in /lib
>>>>> > >
>>>>> > > Thanks!
>>>>> > >
>>>>> > > Paul Carter-Brown
>>>>> > > Director
>>>>> > > Jini Guru
>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > >   Johannesburg, South Africa
>>>>> > > w: jini.guru  e: [hidden email]
>>>>> > >
>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>> > > privileged, confidential and/or personal information. If you are
>>>>> not the
>>>>> > > intended recipient you may not disclose or distribute any of
>>>>> > > the information contained within this message. In such case you
>>>>> must
>>>>> > > destroy this message and inform the sender of the error. Jini Guru
>>>>> may not
>>>>> > > accept liability for any errors, omissions, information and viruses
>>>>> > > contained in the transmission of this message. Any opinions,
>>>>> conclusions
>>>>> > > and other information contained within this message not related to
>>>>> Jini
>>>>> > > Guru official business is deemed to be that of the individual only
>>>>> and is
>>>>> > > not endorsed by Jini Guru.
>>>>> > >
>>>>> > >
>>>>>
>>>>>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

PaulC.B
Thanks Jon.

I'm away at the moment but will give it a spin in the next few days and
let you know. Thanks again

On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, <[hidden email]>
wrote:

> Hi Paul
>
> I've pushed a new snapshot which (hopefully) should address this. The CI
> build is running on it now. Would you mind giving it a try? If it looks
> good, I'll finish rolling the release. Fingers crossed it looks ok for you.
>
> Jon
>
> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
> [hidden email]> wrote:
>
>> Further update - I think my changes work with respect to this issue, but
>> I do have a bunch of unit test failures which I need to look into. For
>> info, here's the diff:
>>
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> index a6e2bdc4f5..c0d1a8c98f 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>>  import org.apache.openejb.classloader.ClassLoaderConfigurer;
>>  import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>>  import org.apache.openejb.component.ClassLoaderEnricher;
>> -import org.apache.openejb.config.ConfigurationFactory;
>> -import org.apache.openejb.config.NewLoaderLogic;
>> -import org.apache.openejb.config.QuickJarsTxtParser;
>> -import org.apache.openejb.config.TldScanner;
>> +import org.apache.openejb.config.*;
>>  import org.apache.openejb.core.ConnectorReference;
>>  import org.apache.openejb.core.CoreContainerSystem;
>>  import org.apache.openejb.core.CoreUserTransaction;
>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
>> implements org.apache.openejb.spi.A
>>
>>                  final AppContext appContext = new
>> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
>> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>>                  appContext.getProperties().putAll(appInfo.properties);
>> +
>> +                final Set<Object> keys =
>> appContext.getProperties().keySet();
>> +                for (final Object key : keys) {
>> +                    if (! Module.class.isInstance(key)) {
>> +                        appContext.getProperties().remove(key);
>> +                    }
>> +                }
>> +
>>                  appContext.getInjections().addAll(injections);
>>                  appContext.getBindings().putAll(globalBindings);
>>                  appContext.getBindings().putAll(appBindings);
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> index feff954d40..aed0fda941 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>>                  reference = new LinkRef(jndiName);
>>
>>              } else if (BeanManager.class.equals(type)) {
>> -                reference = new LazyObjectReference<BeanManager>(new
>> Callable<BeanManager>() {
>> -                    @Override
>> -                    public BeanManager call() throws Exception {
>> -                        return new
>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>> -                    }
>> -                });
>> +                reference = new LazyObjectReference<>(new
>> BeanManagerLazyReference());
>>
>>              } else if (UserTransaction.class.equals(type)) {
>>                  reference = new
>> IntraVmJndiReference("comp/UserTransaction");
>> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>>              }
>>          }
>>      }
>> +
>> +    public static class BeanManagerLazyReference implements
>> Callable<BeanManager> {
>> +        @Override
>> +        public BeanManager call() throws Exception {
>> +            return new
>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>> +        }
>> +    }
>>  }
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> index 2cf387e112..285007925f 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>>      private final ClassLoader system;
>>      private final boolean embedded;
>>      private final boolean parentURLClassLoader;
>> +    private final StackTraceElement[] created;
>>
>>      // 80% of class files are smaller then 6k
>>      private final ByteArrayOutputStream bout = new
>> ByteArrayOutputStream(6 * 1024);
>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>>          this.system = ClassLoader.getSystemClassLoader();
>>          this.embedded = this.getClass().getClassLoader() == this.system;
>>          this.parentURLClassLoader =
>> URLClassLoader.class.isInstance(parent);
>> +        this.created = new Throwable().getStackTrace();
>>      }
>>
>>      /*
>>
>> In essence, this does 2 things:
>>
>> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
>> static class. This stops that object from having a reference to
>> JndiEncBuilder and everything that has attached to it.
>> 2. Strips off the EjbModule / WebModule from the SuperProperties on
>> AppContext. This is set when deploying through the apps folder and lives
>> forever. It has the TempClassLoader attached, and therefore all the
>> AnnotationFinder stuff.
>>
>> I suspect (2) is too aggressive and is causing the test failures, but
>> will confirm.
>>
>> Jon
>>
>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
>> [hidden email]> wrote:
>>
>>> Thanks for sending those over. I'm digging through this - I *think* I
>>> have pinned down a specific reference that stops that classloader from
>>> being released. The behaviour for a war/ear deployed in apps/ seems
>>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>>> If you can confirm, that would help. I'll test out a patch with some bigger
>>> .ear files today.
>>>
>>> Thanks
>>>
>>> Jon
>>>
>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>>> [hidden email]> wrote:
>>>
>>>> The mailing list seems to be stripping off your images - can you resend
>>>> and include my email address as well? I'd be keen to dig into this a little
>>>> more.
>>>>
>>>> Thanks
>>>>
>>>> Jon
>>>>
>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>>> <[hidden email]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>>
>>>>> The TempClassLoader however still has 100's of strong references to
>>>>> it. E.g:
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>>
>>>>> Paul Carter-Brown
>>>>> Director
>>>>> Jini Guru
>>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>>   Johannesburg, South Africa
>>>>> w: jini.guru  e: [hidden email]
>>>>>
>>>>> Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of
>>>>> the information contained within this message. In such case you must
>>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>>> accept liability for any errors, omissions, information and viruses
>>>>> contained in the transmission of this message. Any opinions, conclusions
>>>>> and other information contained within this message not related to Jini
>>>>> Guru official business is deemed to be that of the individual only and is
>>>>> not endorsed by Jini Guru.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
>>>>> <[hidden email]> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> All this is just a workaround imo.
>>>>>>
>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>>> scanning. That means after doing all the class scans we only keep the
>>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>>> work anymore?
>>>>>>
>>>>>> LieGrue,
>>>>>> strub
>>>>>>
>>>>>>
>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>>> <[hidden email]>:
>>>>>> >
>>>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>>>> change:
>>>>>> >
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m:    +27 (0) 83 442 7179
>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> >       Johannesburg, South Africa
>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru official
>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>>> <[hidden email]> wrote:
>>>>>> > Hi Jon,
>>>>>> >
>>>>>> > I took a look at the source and worked out that I could add my own
>>>>>> filters using openejb.additional.exclude.
>>>>>> >
>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>>> previously!!!
>>>>>> >
>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>>>> they are not filtered out and the system works 100%.
>>>>>> >
>>>>>> > My sense is that there should be an easier way to specify what jars
>>>>>> and/or packages to process for annotations. I have come across the
>>>>>> following settings but can't really work out what fits where:
>>>>>> >
>>>>>> > openejb.additional.exclude
>>>>>> > openejb.additional.include
>>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>>> > openejb.deployments.classpath.include
>>>>>> > openejb.deployments.classpath.exclude
>>>>>> > openejb.deployments.package.include
>>>>>> > openejb.deployments.package.exclude
>>>>>> >
>>>>>> >
>>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>>> > com.amazonaws
>>>>>> > com.fasterxml
>>>>>> > com.google
>>>>>> > com.hazelcast
>>>>>> > io.grpc
>>>>>> > io.netty
>>>>>> > org.docx4j
>>>>>> > org.mongodb
>>>>>> > org.rocksdb
>>>>>> > org.asynchttpclient
>>>>>> > org.apache
>>>>>> > org.aspectj
>>>>>> >
>>>>>> > But then maybe some are too broad and I don't really understand
>>>>>> what the annotation index/cache is really needed for at runtime? Can it not
>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-(
>>>>>> Seems like the cache uses a significant amount of heap when considering the
>>>>>> drive towards tiny micro services.
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m:    +27 (0) 83 442 7179
>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> >       Johannesburg, South Africa
>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru official
>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>>> <[hidden email]> wrote:
>>>>>> > Hi Jon,
>>>>>> >
>>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m:    +27 (0) 83 442 7179
>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> >       Johannesburg, South Africa
>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru official
>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>>> [hidden email]> wrote:
>>>>>> > Hi Paul
>>>>>> >
>>>>>> > Would you mind trying your application with a recent snapshot?
>>>>>> >
>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>>> .
>>>>>> > I recently updated the exclude list.
>>>>>> >
>>>>>> > Many thanks
>>>>>> >
>>>>>> > Jon
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>>> > <[hidden email]> wrote:
>>>>>> >
>>>>>> > > Hi,
>>>>>> > >
>>>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>>>> to TomEE
>>>>>> > > 8.0.0.
>>>>>> > >
>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>>> about
>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>>> EAR then
>>>>>> > > the old gen is about 6MB.
>>>>>> > >
>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>>> > >
>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>>> directory
>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>>> cloud
>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>>> actual
>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I
>>>>>> can see that
>>>>>> > > the memory used by the annotation finder is lowered.
>>>>>> > >
>>>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>>> fulling up the
>>>>>> > > heap. I already
>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.*
>>>>>> to scan
>>>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>>>> to obey
>>>>>> > > this property but it doesn't seem to mean that annotation
>>>>>> processing is
>>>>>> > > skipped for all these other jars in /lib
>>>>>> > >
>>>>>> > > Thanks!
>>>>>> > >
>>>>>> > > Paul Carter-Brown
>>>>>> > > Director
>>>>>> > > Jini Guru
>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > >   Johannesburg, South Africa
>>>>>> > > w: jini.guru  e: [hidden email]
>>>>>> > >
>>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>>> > > privileged, confidential and/or personal information. If you are
>>>>>> not the
>>>>>> > > intended recipient you may not disclose or distribute any of
>>>>>> > > the information contained within this message. In such case you
>>>>>> must
>>>>>> > > destroy this message and inform the sender of the error. Jini
>>>>>> Guru may not
>>>>>> > > accept liability for any errors, omissions, information and
>>>>>> viruses
>>>>>> > > contained in the transmission of this message. Any opinions,
>>>>>> conclusions
>>>>>> > > and other information contained within this message not related
>>>>>> to Jini
>>>>>> > > Guru official business is deemed to be that of the individual
>>>>>> only and is
>>>>>> > > not endorsed by Jini Guru.
>>>>>> > >
>>>>>> > >
>>>>>>
>>>>>>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

PaulC.B
Hi Jon

Latest snapshot worked a charm. Old gen went from 500MB to 110MB after
deploy completed. Thanks so much

On Tue, 17 Dec 2019, 11:33 Paul Carter-Brown, <[hidden email]>
wrote:

> Thanks Jon.
>
> I'm away at the moment but will give it a spin in the next few days and
> let you know. Thanks again
>
> On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, <
> [hidden email]> wrote:
>
>> Hi Paul
>>
>> I've pushed a new snapshot which (hopefully) should address this. The CI
>> build is running on it now. Would you mind giving it a try? If it looks
>> good, I'll finish rolling the release. Fingers crossed it looks ok for you.
>>
>> Jon
>>
>> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
>> [hidden email]> wrote:
>>
>>> Further update - I think my changes work with respect to this issue, but
>>> I do have a bunch of unit test failures which I need to look into. For
>>> info, here's the diff:
>>>
>>> diff --git
>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>> index a6e2bdc4f5..c0d1a8c98f 100644
>>> ---
>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>> +++
>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>>>  import org.apache.openejb.classloader.ClassLoaderConfigurer;
>>>  import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>>>  import org.apache.openejb.component.ClassLoaderEnricher;
>>> -import org.apache.openejb.config.ConfigurationFactory;
>>> -import org.apache.openejb.config.NewLoaderLogic;
>>> -import org.apache.openejb.config.QuickJarsTxtParser;
>>> -import org.apache.openejb.config.TldScanner;
>>> +import org.apache.openejb.config.*;
>>>  import org.apache.openejb.core.ConnectorReference;
>>>  import org.apache.openejb.core.CoreContainerSystem;
>>>  import org.apache.openejb.core.CoreUserTransaction;
>>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
>>> implements org.apache.openejb.spi.A
>>>
>>>                  final AppContext appContext = new
>>> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
>>> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>>>                  appContext.getProperties().putAll(appInfo.properties);
>>> +
>>> +                final Set<Object> keys =
>>> appContext.getProperties().keySet();
>>> +                for (final Object key : keys) {
>>> +                    if (! Module.class.isInstance(key)) {
>>> +                        appContext.getProperties().remove(key);
>>> +                    }
>>> +                }
>>> +
>>>                  appContext.getInjections().addAll(injections);
>>>                  appContext.getBindings().putAll(globalBindings);
>>>                  appContext.getBindings().putAll(appBindings);
>>> diff --git
>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>> index feff954d40..aed0fda941 100644
>>> ---
>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>> +++
>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>>>                  reference = new LinkRef(jndiName);
>>>
>>>              } else if (BeanManager.class.equals(type)) {
>>> -                reference = new LazyObjectReference<BeanManager>(new
>>> Callable<BeanManager>() {
>>> -                    @Override
>>> -                    public BeanManager call() throws Exception {
>>> -                        return new
>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>> -                    }
>>> -                });
>>> +                reference = new LazyObjectReference<>(new
>>> BeanManagerLazyReference());
>>>
>>>              } else if (UserTransaction.class.equals(type)) {
>>>                  reference = new
>>> IntraVmJndiReference("comp/UserTransaction");
>>> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>>>              }
>>>          }
>>>      }
>>> +
>>> +    public static class BeanManagerLazyReference implements
>>> Callable<BeanManager> {
>>> +        @Override
>>> +        public BeanManager call() throws Exception {
>>> +            return new
>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>> +        }
>>> +    }
>>>  }
>>> diff --git
>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>> index 2cf387e112..285007925f 100644
>>> ---
>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>> +++
>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>>>      private final ClassLoader system;
>>>      private final boolean embedded;
>>>      private final boolean parentURLClassLoader;
>>> +    private final StackTraceElement[] created;
>>>
>>>      // 80% of class files are smaller then 6k
>>>      private final ByteArrayOutputStream bout = new
>>> ByteArrayOutputStream(6 * 1024);
>>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>>>          this.system = ClassLoader.getSystemClassLoader();
>>>          this.embedded = this.getClass().getClassLoader() == this.system;
>>>          this.parentURLClassLoader =
>>> URLClassLoader.class.isInstance(parent);
>>> +        this.created = new Throwable().getStackTrace();
>>>      }
>>>
>>>      /*
>>>
>>> In essence, this does 2 things:
>>>
>>> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
>>> static class. This stops that object from having a reference to
>>> JndiEncBuilder and everything that has attached to it.
>>> 2. Strips off the EjbModule / WebModule from the SuperProperties on
>>> AppContext. This is set when deploying through the apps folder and lives
>>> forever. It has the TempClassLoader attached, and therefore all the
>>> AnnotationFinder stuff.
>>>
>>> I suspect (2) is too aggressive and is causing the test failures, but
>>> will confirm.
>>>
>>> Jon
>>>
>>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
>>> [hidden email]> wrote:
>>>
>>>> Thanks for sending those over. I'm digging through this - I *think* I
>>>> have pinned down a specific reference that stops that classloader from
>>>> being released. The behaviour for a war/ear deployed in apps/ seems
>>>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>>>> If you can confirm, that would help. I'll test out a patch with some bigger
>>>> .ear files today.
>>>>
>>>> Thanks
>>>>
>>>> Jon
>>>>
>>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>>>> [hidden email]> wrote:
>>>>
>>>>> The mailing list seems to be stripping off your images - can you
>>>>> resend and include my email address as well? I'd be keen to dig into this a
>>>>> little more.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Jon
>>>>>
>>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>>>> <[hidden email]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>>>
>>>>>> The TempClassLoader however still has 100's of strong references to
>>>>>> it. E.g:
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>>
>>>>>> Paul Carter-Brown
>>>>>> Director
>>>>>> Jini Guru
>>>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>>   Johannesburg, South Africa
>>>>>> w: jini.guru  e: [hidden email]
>>>>>>
>>>>>> Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of
>>>>>> the information contained within this message. In such case you must
>>>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>>>> accept liability for any errors, omissions, information and viruses
>>>>>> contained in the transmission of this message. Any opinions, conclusions
>>>>>> and other information contained within this message not related to Jini
>>>>>> Guru official business is deemed to be that of the individual only and is
>>>>>> not endorsed by Jini Guru.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
>>>>>> <[hidden email]> wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> All this is just a workaround imo.
>>>>>>>
>>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>>>> scanning. That means after doing all the class scans we only keep the
>>>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>>>> work anymore?
>>>>>>>
>>>>>>> LieGrue,
>>>>>>> strub
>>>>>>>
>>>>>>>
>>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>>>> <[hidden email]>:
>>>>>>> >
>>>>>>> > FYI. Graph of the change in heap size on a prod instance after
>>>>>>> this change:
>>>>>>> >
>>>>>>> >
>>>>>>> > Paul Carter-Brown
>>>>>>> > Director
>>>>>>> > Jini Guru
>>>>>>> > m:    +27 (0) 83 442 7179
>>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>> and Leslie
>>>>>>> >       Johannesburg, South Africa
>>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>>> >
>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>> information contained within this message. In such case you must destroy
>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>> Jini Guru.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>>>> <[hidden email]> wrote:
>>>>>>> > Hi Jon,
>>>>>>> >
>>>>>>> > I took a look at the source and worked out that I could add my own
>>>>>>> filters using openejb.additional.exclude.
>>>>>>> >
>>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>>>> previously!!!
>>>>>>> >
>>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and
>>>>>>> hence they are not filtered out and the system works 100%.
>>>>>>> >
>>>>>>> > My sense is that there should be an easier way to specify what
>>>>>>> jars and/or packages to process for annotations. I have come across the
>>>>>>> following settings but can't really work out what fits where:
>>>>>>> >
>>>>>>> > openejb.additional.exclude
>>>>>>> > openejb.additional.include
>>>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>>>> > openejb.deployments.classpath.include
>>>>>>> > openejb.deployments.classpath.exclude
>>>>>>> > openejb.deployments.package.include
>>>>>>> > openejb.deployments.package.exclude
>>>>>>> >
>>>>>>> >
>>>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>>>> > com.amazonaws
>>>>>>> > com.fasterxml
>>>>>>> > com.google
>>>>>>> > com.hazelcast
>>>>>>> > io.grpc
>>>>>>> > io.netty
>>>>>>> > org.docx4j
>>>>>>> > org.mongodb
>>>>>>> > org.rocksdb
>>>>>>> > org.asynchttpclient
>>>>>>> > org.apache
>>>>>>> > org.aspectj
>>>>>>> >
>>>>>>> > But then maybe some are too broad and I don't really understand
>>>>>>> what the annotation index/cache is really needed for at runtime? Can it not
>>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-(
>>>>>>> Seems like the cache uses a significant amount of heap when considering the
>>>>>>> drive towards tiny micro services.
>>>>>>> >
>>>>>>> > Paul Carter-Brown
>>>>>>> > Director
>>>>>>> > Jini Guru
>>>>>>> > m:    +27 (0) 83 442 7179
>>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>> and Leslie
>>>>>>> >       Johannesburg, South Africa
>>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>>> >
>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>> information contained within this message. In such case you must destroy
>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>> Jini Guru.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>>>> <[hidden email]> wrote:
>>>>>>> > Hi Jon,
>>>>>>> >
>>>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>>>> >
>>>>>>> > Paul Carter-Brown
>>>>>>> > Director
>>>>>>> > Jini Guru
>>>>>>> > m:    +27 (0) 83 442 7179
>>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>> and Leslie
>>>>>>> >       Johannesburg, South Africa
>>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>>> >
>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>> information contained within this message. In such case you must destroy
>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>> Jini Guru.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>>>> [hidden email]> wrote:
>>>>>>> > Hi Paul
>>>>>>> >
>>>>>>> > Would you mind trying your application with a recent snapshot?
>>>>>>> >
>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>>>> .
>>>>>>> > I recently updated the exclude list.
>>>>>>> >
>>>>>>> > Many thanks
>>>>>>> >
>>>>>>> > Jon
>>>>>>> >
>>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>>>> > <[hidden email]> wrote:
>>>>>>> >
>>>>>>> > > Hi,
>>>>>>> > >
>>>>>>> > > I've been trying to lower the memory footprint of an EAR
>>>>>>> deployed to TomEE
>>>>>>> > > 8.0.0.
>>>>>>> > >
>>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>>>> about
>>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>>>> EAR then
>>>>>>> > > the old gen is about 6MB.
>>>>>>> > >
>>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>>>> > >
>>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>>>> directory
>>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>>>> cloud
>>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>>>> actual
>>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I
>>>>>>> can see that
>>>>>>> > > the memory used by the annotation finder is lowered.
>>>>>>> > >
>>>>>>> > > Is there any way I can tell TomEE that it need not bother
>>>>>>> scanning
>>>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>>>> fulling up the
>>>>>>> > > heap. I already
>>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.*
>>>>>>> to scan
>>>>>>> > > only the one jar in /lib which does have EJB's in it and it
>>>>>>> seems to obey
>>>>>>> > > this property but it doesn't seem to mean that annotation
>>>>>>> processing is
>>>>>>> > > skipped for all these other jars in /lib
>>>>>>> > >
>>>>>>> > > Thanks!
>>>>>>> > >
>>>>>>> > > Paul Carter-Brown
>>>>>>> > > Director
>>>>>>> > > Jini Guru
>>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>>> Leslie
>>>>>>> > >   Johannesburg, South Africa
>>>>>>> > > w: jini.guru  e: [hidden email]
>>>>>>> > >
>>>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>>>> > > privileged, confidential and/or personal information. If you are
>>>>>>> not the
>>>>>>> > > intended recipient you may not disclose or distribute any of
>>>>>>> > > the information contained within this message. In such case you
>>>>>>> must
>>>>>>> > > destroy this message and inform the sender of the error. Jini
>>>>>>> Guru may not
>>>>>>> > > accept liability for any errors, omissions, information and
>>>>>>> viruses
>>>>>>> > > contained in the transmission of this message. Any opinions,
>>>>>>> conclusions
>>>>>>> > > and other information contained within this message not related
>>>>>>> to Jini
>>>>>>> > > Guru official business is deemed to be that of the individual
>>>>>>> only and is
>>>>>>> > > not endorsed by Jini Guru.
>>>>>>> > >
>>>>>>> > >
>>>>>>>
>>>>>>>
Reply | Threaded
Open this post in threaded view
|

Re: Old Gen Full of Annotation Finder Index

jgallimore
Hi Paul

Thanks for your feedback, that's great news. FYI, TomEE 8.0.1 is up for
vote (feel free to vote on it!), and this change is included.

Cheers

Jon

On Fri, Dec 20, 2019 at 7:10 AM Paul Carter-Brown
<[hidden email]> wrote:

> Hi Jon
>
> Latest snapshot worked a charm. Old gen went from 500MB to 110MB after
> deploy completed. Thanks so much
>
> On Tue, 17 Dec 2019, 11:33 Paul Carter-Brown, <[hidden email]>
> wrote:
>
>> Thanks Jon.
>>
>> I'm away at the moment but will give it a spin in the next few days and
>> let you know. Thanks again
>>
>> On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, <
>> [hidden email]> wrote:
>>
>>> Hi Paul
>>>
>>> I've pushed a new snapshot which (hopefully) should address this. The CI
>>> build is running on it now. Would you mind giving it a try? If it looks
>>> good, I'll finish rolling the release. Fingers crossed it looks ok for you.
>>>
>>> Jon
>>>
>>> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
>>> [hidden email]> wrote:
>>>
>>>> Further update - I think my changes work with respect to this issue,
>>>> but I do have a bunch of unit test failures which I need to look into. For
>>>> info, here's the diff:
>>>>
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> index a6e2bdc4f5..c0d1a8c98f 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>>>>  import org.apache.openejb.classloader.ClassLoaderConfigurer;
>>>>  import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>>>>  import org.apache.openejb.component.ClassLoaderEnricher;
>>>> -import org.apache.openejb.config.ConfigurationFactory;
>>>> -import org.apache.openejb.config.NewLoaderLogic;
>>>> -import org.apache.openejb.config.QuickJarsTxtParser;
>>>> -import org.apache.openejb.config.TldScanner;
>>>> +import org.apache.openejb.config.*;
>>>>  import org.apache.openejb.core.ConnectorReference;
>>>>  import org.apache.openejb.core.CoreContainerSystem;
>>>>  import org.apache.openejb.core.CoreUserTransaction;
>>>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
>>>> implements org.apache.openejb.spi.A
>>>>
>>>>                  final AppContext appContext = new
>>>> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
>>>> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>>>>                  appContext.getProperties().putAll(appInfo.properties);
>>>> +
>>>> +                final Set<Object> keys =
>>>> appContext.getProperties().keySet();
>>>> +                for (final Object key : keys) {
>>>> +                    if (! Module.class.isInstance(key)) {
>>>> +                        appContext.getProperties().remove(key);
>>>> +                    }
>>>> +                }
>>>> +
>>>>                  appContext.getInjections().addAll(injections);
>>>>                  appContext.getBindings().putAll(globalBindings);
>>>>                  appContext.getBindings().putAll(appBindings);
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> index feff954d40..aed0fda941 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>>>>                  reference = new LinkRef(jndiName);
>>>>
>>>>              } else if (BeanManager.class.equals(type)) {
>>>> -                reference = new LazyObjectReference<BeanManager>(new
>>>> Callable<BeanManager>() {
>>>> -                    @Override
>>>> -                    public BeanManager call() throws Exception {
>>>> -                        return new
>>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>>> -                    }
>>>> -                });
>>>> +                reference = new LazyObjectReference<>(new
>>>> BeanManagerLazyReference());
>>>>
>>>>              } else if (UserTransaction.class.equals(type)) {
>>>>                  reference = new
>>>> IntraVmJndiReference("comp/UserTransaction");
>>>> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>>>>              }
>>>>          }
>>>>      }
>>>> +
>>>> +    public static class BeanManagerLazyReference implements
>>>> Callable<BeanManager> {
>>>> +        @Override
>>>> +        public BeanManager call() throws Exception {
>>>> +            return new
>>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>>> +        }
>>>> +    }
>>>>  }
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> index 2cf387e112..285007925f 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>>>>      private final ClassLoader system;
>>>>      private final boolean embedded;
>>>>      private final boolean parentURLClassLoader;
>>>> +    private final StackTraceElement[] created;
>>>>
>>>>      // 80% of class files are smaller then 6k
>>>>      private final ByteArrayOutputStream bout = new
>>>> ByteArrayOutputStream(6 * 1024);
>>>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>>>>          this.system = ClassLoader.getSystemClassLoader();
>>>>          this.embedded = this.getClass().getClassLoader() ==
>>>> this.system;
>>>>          this.parentURLClassLoader =
>>>> URLClassLoader.class.isInstance(parent);
>>>> +        this.created = new Throwable().getStackTrace();
>>>>      }
>>>>
>>>>      /*
>>>>
>>>> In essence, this does 2 things:
>>>>
>>>> 1. Defines the Callable for the LazyObjectReference for BeanManager as
>>>> a static class. This stops that object from having a reference to
>>>> JndiEncBuilder and everything that has attached to it.
>>>> 2. Strips off the EjbModule / WebModule from the SuperProperties on
>>>> AppContext. This is set when deploying through the apps folder and lives
>>>> forever. It has the TempClassLoader attached, and therefore all the
>>>> AnnotationFinder stuff.
>>>>
>>>> I suspect (2) is too aggressive and is causing the test failures, but
>>>> will confirm.
>>>>
>>>> Jon
>>>>
>>>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
>>>> [hidden email]> wrote:
>>>>
>>>>> Thanks for sending those over. I'm digging through this - I *think* I
>>>>> have pinned down a specific reference that stops that classloader from
>>>>> being released. The behaviour for a war/ear deployed in apps/ seems
>>>>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>>>>> If you can confirm, that would help. I'll test out a patch with some bigger
>>>>> .ear files today.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Jon
>>>>>
>>>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> The mailing list seems to be stripping off your images - can you
>>>>>> resend and include my email address as well? I'd be keen to dig into this a
>>>>>> little more.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>>>>> <[hidden email]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>>>>
>>>>>>> The TempClassLoader however still has 100's of strong references to
>>>>>>> it. E.g:
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>>
>>>>>>> Paul Carter-Brown
>>>>>>> Director
>>>>>>> Jini Guru
>>>>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>>> Leslie
>>>>>>>   Johannesburg, South Africa
>>>>>>> w: jini.guru  e: [hidden email]
>>>>>>>
>>>>>>> Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of
>>>>>>> the information contained within this message. In such case you must
>>>>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>>>>> accept liability for any errors, omissions, information and viruses
>>>>>>> contained in the transmission of this message. Any opinions, conclusions
>>>>>>> and other information contained within this message not related to Jini
>>>>>>> Guru official business is deemed to be that of the individual only and is
>>>>>>> not endorsed by Jini Guru.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
>>>>>>> <[hidden email]> wrote:
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> All this is just a workaround imo.
>>>>>>>>
>>>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>>>>> scanning. That means after doing all the class scans we only keep the
>>>>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>>>>> work anymore?
>>>>>>>>
>>>>>>>> LieGrue,
>>>>>>>> strub
>>>>>>>>
>>>>>>>>
>>>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>>>>> <[hidden email]>:
>>>>>>>> >
>>>>>>>> > FYI. Graph of the change in heap size on a prod instance after
>>>>>>>> this change:
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m:    +27 (0) 83 442 7179
>>>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> >       Johannesburg, South Africa
>>>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>>>>> <[hidden email]> wrote:
>>>>>>>> > Hi Jon,
>>>>>>>> >
>>>>>>>> > I took a look at the source and worked out that I could add my
>>>>>>>> own filters using openejb.additional.exclude.
>>>>>>>> >
>>>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>>>>> previously!!!
>>>>>>>> >
>>>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and
>>>>>>>> hence they are not filtered out and the system works 100%.
>>>>>>>> >
>>>>>>>> > My sense is that there should be an easier way to specify what
>>>>>>>> jars and/or packages to process for annotations. I have come across the
>>>>>>>> following settings but can't really work out what fits where:
>>>>>>>> >
>>>>>>>> > openejb.additional.exclude
>>>>>>>> > openejb.additional.include
>>>>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>>>>> > openejb.deployments.classpath.include
>>>>>>>> > openejb.deployments.classpath.exclude
>>>>>>>> > openejb.deployments.package.include
>>>>>>>> > openejb.deployments.package.exclude
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > P.S. Perhaps also add these to the standard exclusion list as
>>>>>>>> they are common and yet won't need to be processed for annotations I guess?
>>>>>>>> > com.amazonaws
>>>>>>>> > com.fasterxml
>>>>>>>> > com.google
>>>>>>>> > com.hazelcast
>>>>>>>> > io.grpc
>>>>>>>> > io.netty
>>>>>>>> > org.docx4j
>>>>>>>> > org.mongodb
>>>>>>>> > org.rocksdb
>>>>>>>> > org.asynchttpclient
>>>>>>>> > org.apache
>>>>>>>> > org.aspectj
>>>>>>>> >
>>>>>>>> > But then maybe some are too broad and I don't really understand
>>>>>>>> what the annotation index/cache is really needed for at runtime? Can it not
>>>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-(
>>>>>>>> Seems like the cache uses a significant amount of heap when considering the
>>>>>>>> drive towards tiny micro services.
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m:    +27 (0) 83 442 7179
>>>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> >       Johannesburg, South Africa
>>>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>>>>> <[hidden email]> wrote:
>>>>>>>> > Hi Jon,
>>>>>>>> >
>>>>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m:    +27 (0) 83 442 7179
>>>>>>>> > a:    1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> >       Johannesburg, South Africa
>>>>>>>> > w:    jini.guru  e: [hidden email]
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>>>>> [hidden email]> wrote:
>>>>>>>> > Hi Paul
>>>>>>>> >
>>>>>>>> > Would you mind trying your application with a recent snapshot?
>>>>>>>> >
>>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>>>>> .
>>>>>>>> > I recently updated the exclude list.
>>>>>>>> >
>>>>>>>> > Many thanks
>>>>>>>> >
>>>>>>>> > Jon
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>>>>> > <[hidden email]> wrote:
>>>>>>>> >
>>>>>>>> > > Hi,
>>>>>>>> > >
>>>>>>>> > > I've been trying to lower the memory footprint of an EAR
>>>>>>>> deployed to TomEE
>>>>>>>> > > 8.0.0.
>>>>>>>> > >
>>>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen
>>>>>>>> is about
>>>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>>>>> EAR then
>>>>>>>> > > the old gen is about 6MB.
>>>>>>>> > >
>>>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>>>>> > >
>>>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>>>>> directory
>>>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI,
>>>>>>>> Google cloud
>>>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>>>>> actual
>>>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I
>>>>>>>> can see that
>>>>>>>> > > the memory used by the annotation finder is lowered.
>>>>>>>> > >
>>>>>>>> > > Is there any way I can tell TomEE that it need not bother
>>>>>>>> scanning
>>>>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>>>>> fulling up the
>>>>>>>> > > heap. I already
>>>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.*
>>>>>>>> to scan
>>>>>>>> > > only the one jar in /lib which does have EJB's in it and it
>>>>>>>> seems to obey
>>>>>>>> > > this property but it doesn't seem to mean that annotation
>>>>>>>> processing is
>>>>>>>> > > skipped for all these other jars in /lib
>>>>>>>> > >
>>>>>>>> > > Thanks!
>>>>>>>> > >
>>>>>>>> > > Paul Carter-Brown
>>>>>>>> > > Director
>>>>>>>> > > Jini Guru
>>>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > >   Johannesburg, South Africa
>>>>>>>> > > w: jini.guru  e: [hidden email]
>>>>>>>> > >
>>>>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> > > privileged, confidential and/or personal information. If you
>>>>>>>> are not the
>>>>>>>> > > intended recipient you may not disclose or distribute any of
>>>>>>>> > > the information contained within this message. In such case you
>>>>>>>> must
>>>>>>>> > > destroy this message and inform the sender of the error. Jini
>>>>>>>> Guru may not
>>>>>>>> > > accept liability for any errors, omissions, information and
>>>>>>>> viruses
>>>>>>>> > > contained in the transmission of this message. Any opinions,
>>>>>>>> conclusions
>>>>>>>> > > and other information contained within this message not related
>>>>>>>> to Jini
>>>>>>>> > > Guru official business is deemed to be that of the individual
>>>>>>>> only and is
>>>>>>>> > > not endorsed by Jini Guru.
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>>
>>>>>>>>