Buildbot slowness

classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Buildbot slowness

j4fm
Buildbot is slow with :cdi-tomee tests causing out of memory warnings.  The
PR409 (CXF 3.3.0 and JAXB 2.3.2) updates was the first merge to cause the
slowness.  The rolled up merges after show the same.


# Created on 2019-03-01T20:48:20.010
Corrupted STDOUT by directly writing to native stream in forked JVM 1.
Stream 'java.lang.OutOfMemoryError: Java heap space'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
after third character in command 'java.lang.OutOfMemoryError: Java heap
space'.
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
        at
org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
        at java.lang.Thread.run(Thread.java:745)



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

jgallimore
Seeing the same with the java11-experimentation build here:

# Created on 2019-03-01T18:01:43.414
Corrupted STDOUT by directly writing to native stream in forked JVM 1.
Stream 'java.lang.OutOfMemoryError: Java heap space'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
after third character in command 'java.lang.OutOfMemoryError: Java heap
space'.
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
        at
org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
        at java.lang.Thread.run(Thread.java:748)


# Created on 2019-03-01T18:01:43.414
Corrupted STDOUT by directly writing to native stream in forked JVM 1.
Stream 'Dumping heap to java_pid9348.hprof ...'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
after third character in command 'Dumping heap to java_pid9348.hprof ...'.
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
        at
org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
        at java.lang.Thread.run(Thread.java:748)


# Created on 2019-03-01T18:02:39.417
Corrupted STDOUT by directly writing to native stream in forked JVM 1.
Stream 'Heap dump file created [3085481793 bytes in 56.000 secs]'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
after third character in command 'Heap dump file created [3085481793 bytes
in 56.000 secs]'.
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
        at
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
        at
org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
        at java.lang.Thread.run(Thread.java:748)

If we're creating heap dumps, we ought to kill that build, and try and
figure out what's going on.

Thoughts?

Jon

On Fri, Mar 1, 2019 at 9:13 PM j4fm <[hidden email]> wrote:

> Getting this WARNING on :cdi-tomee on buildbot which is taking forever...
>
> # Created on 2019-03-01T20:48:20.010
> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
> Stream 'java.lang.OutOfMemoryError: Java heap space'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'java.lang.OutOfMemoryError: Java heap
> space'.
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> # Created on 2019-03-01T20:48:20.027
> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
> Stream 'Dumping heap to java_pid187.hprof ...'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'Dumping heap to java_pid187.hprof ...'.
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> # Created on 2019-03-01T20:49:04.600
> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
> Stream 'Heap dump file created [3106753056 bytes in 44.621 secs]'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'Heap dump file created [3106753056 bytes
> in 44.621 secs]'.
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>         at
>
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

jgallimore
On the plus side, I have a heap dump here, so we can take a look.

Jon

On Fri, Mar 1, 2019 at 9:28 PM Jonathan Gallimore <
[hidden email]> wrote:

> Seeing the same with the java11-experimentation build here:
>
> # Created on 2019-03-01T18:01:43.414
> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
> Stream 'java.lang.OutOfMemoryError: Java heap space'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'java.lang.OutOfMemoryError: Java heap
> space'.
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>         at java.lang.Thread.run(Thread.java:748)
>
>
> # Created on 2019-03-01T18:01:43.414
> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
> Stream 'Dumping heap to java_pid9348.hprof ...'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'Dumping heap to java_pid9348.hprof ...'.
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>         at java.lang.Thread.run(Thread.java:748)
>
>
> # Created on 2019-03-01T18:02:39.417
> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
> Stream 'Heap dump file created [3085481793 bytes in 56.000 secs]'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'Heap dump file created [3085481793 bytes
> in 56.000 secs]'.
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>         at
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>         at java.lang.Thread.run(Thread.java:748)
>
> If we're creating heap dumps, we ought to kill that build, and try and
> figure out what's going on.
>
> Thoughts?
>
> Jon
>
> On Fri, Mar 1, 2019 at 9:13 PM j4fm <[hidden email]> wrote:
>
>> Getting this WARNING on :cdi-tomee on buildbot which is taking forever...
>>
>> # Created on 2019-03-01T20:48:20.010
>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>> Stream 'java.lang.OutOfMemoryError: Java heap space'.
>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
>> after third character in command 'java.lang.OutOfMemoryError: Java heap
>> space'.
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>>
>> # Created on 2019-03-01T20:48:20.027
>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>> Stream 'Dumping heap to java_pid187.hprof ...'.
>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
>> after third character in command 'Dumping heap to java_pid187.hprof ...'.
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>>
>> # Created on 2019-03-01T20:49:04.600
>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>> Stream 'Heap dump file created [3106753056 bytes in 44.621 secs]'.
>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
>> after third character in command 'Heap dump file created [3106753056 bytes
>> in 44.621 secs]'.
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>         at
>>
>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> --
>> Sent from:
>> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

jgallimore
Waiting on my machine to crunch some numbers, but here's the summary:

       Total Bytes: 2,726,889,443
       Total Classes: 124,138
       Total Instances: 54,413,232
       Classloaders: 790
       GC Roots: 3,842
       Number of Objects Pending for Finalization: 0

That seems like a lot of classloaders to me...

Jon

On Fri, Mar 1, 2019 at 9:33 PM Jonathan Gallimore <
[hidden email]> wrote:

> On the plus side, I have a heap dump here, so we can take a look.
>
> Jon
>
> On Fri, Mar 1, 2019 at 9:28 PM Jonathan Gallimore <
> [hidden email]> wrote:
>
>> Seeing the same with the java11-experimentation build here:
>>
>> # Created on 2019-03-01T18:01:43.414
>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>> Stream 'java.lang.OutOfMemoryError: Java heap space'.
>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected
>> comma after third character in command 'java.lang.OutOfMemoryError: Java
>> heap space'.
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>>
>> # Created on 2019-03-01T18:01:43.414
>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>> Stream 'Dumping heap to java_pid9348.hprof ...'.
>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected
>> comma after third character in command 'Dumping heap to java_pid9348.hprof
>> ...'.
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>>
>> # Created on 2019-03-01T18:02:39.417
>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>> Stream 'Heap dump file created [3085481793 bytes in 56.000 secs]'.
>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected
>> comma after third character in command 'Heap dump file created [3085481793
>> bytes in 56.000 secs]'.
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>         at
>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>> If we're creating heap dumps, we ought to kill that build, and try and
>> figure out what's going on.
>>
>> Thoughts?
>>
>> Jon
>>
>> On Fri, Mar 1, 2019 at 9:13 PM j4fm <[hidden email]> wrote:
>>
>>> Getting this WARNING on :cdi-tomee on buildbot which is taking forever...
>>>
>>> # Created on 2019-03-01T20:48:20.010
>>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>>> Stream 'java.lang.OutOfMemoryError: Java heap space'.
>>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected
>>> comma
>>> after third character in command 'java.lang.OutOfMemoryError: Java heap
>>> space'.
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> # Created on 2019-03-01T20:48:20.027
>>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>>> Stream 'Dumping heap to java_pid187.hprof ...'.
>>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected
>>> comma
>>> after third character in command 'Dumping heap to java_pid187.hprof ...'.
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> # Created on 2019-03-01T20:49:04.600
>>> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
>>> Stream 'Heap dump file created [3106753056 bytes in 44.621 secs]'.
>>> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected
>>> comma
>>> after third character in command 'Heap dump file created [3106753056
>>> bytes
>>> in 44.621 secs]'.
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:511)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:209)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:176)
>>>         at
>>>
>>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>>
>>> --
>>> Sent from:
>>> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
Would like to get the stats at the end of the CI build if possible too.

PR409 was the first merge to start the issue but there are no code changes
in there except to examples and the bumps of CXF and JAXB.

Do you see it just being cdi-tomee that's slowed up?  I'm checking out it's
(indirect) dependencies on jaxb.



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
Not sure it's ever going to finish. Seems to be grinding to a halt. You're
right, no point letting it run.



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
I have a fix for the slow build (works locally without memory error now).  If
somebody wouldn't mind stopping the current buildbot task and I will create
a small PR.

Thanks in advance



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
False fix, it just got a lot further before an oom exception.



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

César Hernández Mendoza
Thanks for the update James.

I reverted locally:  https://github.com/apache/tomee/pull/409 , build tome
without tests and then:
mvn -pl tck/cdi-tomee --fail-at-end
-Djava.io.tmpdir=/tmp/tomee-trunk-ubuntu-jvm8 clean install -Pall-adapters
-Dsurefire.useFile=false -DdisableXmlReport=true
-Dopenejb.arquillian.debug=true

The module build with tests took ~30 min. [1]

The 409 PR, apart from the examples update, it just updating mp rest client
and cxf. The fix you are working on is related with this two artifacts?

[1]
ests run: 1571, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ cdi-tomee ---
[INFO] Building jar:
/Users/cesar/git/tomee/tck/cdi-tomee/target/cdi-tomee-8.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.7.1:attach-descriptor (attach-descriptor) @
cdi-tomee ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ cdi-tomee
---
[INFO] Installing
/Users/cesar/git/tomee/tck/cdi-tomee/target/cdi-tomee-8.0.0-SNAPSHOT.jar to
/Users/cesar/.m2/repository/org/apache/tomee/cdi-tomee/8.0.0-SNAPSHOT/cdi-tomee-8.0.0-SNAPSHOT.jar
[INFO] Installing /Users/cesar/git/tomee/tck/cdi-tomee/pom.xml to
/Users/cesar/.m2/repository/org/apache/tomee/cdi-tomee/8.0.0-SNAPSHOT/cdi-tomee-8.0.0-SNAPSHOT.pom
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 30:36 min
[INFO] Finished at: 2019-03-01T21:02:53-06:00
[INFO] Final Memory: 126M/1234M
[INFO]
------------------------------------------------------------------------





El vie., 1 mar. 2019 a las 18:37, j4fm (<[hidden email]>)
escribió:

> False fix, it just got a lot further before an oom exception.
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>


--
Atentamente:
César Hernández Mendoza.
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
Hey Jon, thanks for the test.  Do you still have all my PRs in there after
just removing the PR409?

Yeah, I was looking around all the dependency changes that CXF (and JAXB
which it depends on) brings in.
I do have something I am working on, instead of excluding the older javax
conflicting dependencies, am excluding the newer jakarta ones.

My machine is much slower, so have to wait for the tests much longer. :)



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

Daniel Cunha-2
Hi folks,

it is happening in a specif module?
I'm building tomee without samples and seems fine to me.

mvn clean install -Pquick

Em sáb, 2 de mar de 2019 às 00:36, j4fm <[hidden email]>
escreveu:

> Hey Jon, thanks for the test.  Do you still have all my PRs in there after
> just removing the PR409?
>
> Yeah, I was looking around all the dependency changes that CXF (and JAXB
> which it depends on) brings in.
> I do have something I am working on, instead of excluding the older javax
> conflicting dependencies, am excluding the newer jakarta ones.
>
> My machine is much slower, so have to wait for the tests much longer. :)
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>


--
Daniel "soro" Cunha
https://twitter.com/dvlc_
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
Building is not a problem, it's the CDI TCK test run causing out of memory
exception in tomee



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

Daniel Cunha-2
What is the mvn command which you are using to run it?

Em sáb, 2 de mar de 2019 às 04:27, j4fm <[hidden email]>
escreveu:

> Building is not a problem, it's the CDI TCK test run causing out of memory
> exception in tomee
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>


--
Daniel "soro" Cunha
https://twitter.com/dvlc_
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

jgallimore
In reply to this post by j4fm
Did anyone take a look at the heap dump (I can make it available if
needed)? If you did, what did you find?

I left visualvm computing sizes when  I went to bed last night. Not sure
where it's at. Hoping to get a look over the weekend.

Jon

On Sat, 2 Mar 2019, 07:27 j4fm, <[hidden email]> wrote:

> Building is not a problem, it's the CDI TCK test run causing out of memory
> exception in tomee
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
It's just changing CXF version to 3.3 on it's own that causes it.  Reverting
back to 3.2.7 solves it.
All the JAXB 2.8.2 and related dependency work seems to be fine.
Though I see the CXF mp client version bump adds in activation-api which
needs excluding.

I'm testing 3.3.1 out now just to check (Released yesterday but not yet in
maven).

I have a dump in front of me now.  I did look at it but not much yet.

Suspect 1.
144 instances of "org.apache.catalina.core.StandardContext", loaded by
"java.net.URLClassLoader @ 0x94d3de00" occupy 1,566,771,736 (76.22%) bytes.
These instances are referenced from one instance of
"java.util.HashMap$Node[]", loaded by "<system class loader>"

Suspect 2.
380 instances of "org.apache.tomee.catalina.TomEEWebappClassLoader", loaded
by "java.net.URLClassLoader @ 0x94d3de00" occupy 276,743,720 (13.46%) bytes.

Keywords
org.apache.tomee.catalina.TomEEWebappClassLoader
java.net.URLClassLoader @ 0x94d3de00

Also I got this stack trace:

02-Mar-2019 00:29:58.951 SEVERE [http-nio-41215-exec-983]
org.apache.openejb.core.transaction.EjbTransactionUtil.handleSystemException
EjbTransactionUtil.handleSystemException: Java heap space
 java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3236)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
        at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
        at org.apache.xbean.asm7.ClassReader.readStream(ClassReader.java:307)
        at org.apache.xbean.asm7.ClassReader.<init>(ClassReader.java:273)
        at
org.apache.xbean.finder.AnnotationFinder.readClassDef(AnnotationFinder.java:1167)
        at
org.apache.xbean.finder.AnnotationFinder.<init>(AnnotationFinder.java:145)
        at
org.apache.xbean.finder.AnnotationFinder.<init>(AnnotationFinder.java:158)
        at
org.apache.openejb.config.FinderFactory$OpenEJBAnnotationFinder.<init>(FinderFactory.java:546)
        at
org.apache.openejb.config.FinderFactory.newFinder(FinderFactory.java:267)
        at org.apache.openejb.config.FinderFactory.create(FinderFactory.java:80)
        at
org.apache.openejb.config.FinderFactory.createFinder(FinderFactory.java:69)
        at
org.apache.openejb.config.DeploymentLoader.addWebModule(DeploymentLoader.java:878)
        at
org.apache.openejb.config.DeploymentLoader.load(DeploymentLoader.java:233)
        at
org.apache.tomee.catalina.TomcatWebAppBuilder.loadApplication(TomcatWebAppBuilder.java:2352)
        at
org.apache.tomee.catalina.TomcatWebAppBuilder.startInternal(TomcatWebAppBuilder.java:1197)
        at
org.apache.tomee.catalina.TomcatWebAppBuilder.configureStart(TomcatWebAppBuilder.java:1130)
        at
org.apache.tomee.catalina.GlobalListenerSupport.lifecycleEvent(GlobalListenerSupport.java:134)
        at
org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
        at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5007)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:703)
        at
org.apache.tomee.catalina.TomcatWebAppBuilder.deployWar(TomcatWebAppBuilder.java:657)
        at
org.apache.tomee.catalina.TomcatWebAppBuilder.deployWebApps(TomcatWebAppBuilder.java:597)
        at
org.apache.tomee.catalina.deployment.TomcatWebappDeployer.deploy(TomcatWebappDeployer.java:47)
        at org.apache.openejb.assembler.DeployerEjb.deploy(DeployerEjb.java:177)
        at org.apache.openejb.assembler.DeployerEjb.deploy(DeployerEjb.java:140)
        at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

jgallimore
I was seeing the same, but didn't get to dig into it deeper yet - its
unlikely I'll be able to do much over the weekend, but we'll see.

I did bump up the heap on my build to 6G (from 1.5) to see how far it gets.

Jon

On Sat, 2 Mar 2019, 08:54 j4fm, <[hidden email]> wrote:

> It's just changing CXF version to 3.3 on it's own that causes it.
> Reverting
> back to 3.2.7 solves it.
> All the JAXB 2.8.2 and related dependency work seems to be fine.
> Though I see the CXF mp client version bump adds in activation-api which
> needs excluding.
>
> I'm testing 3.3.1 out now just to check (Released yesterday but not yet in
> maven).
>
> I have a dump in front of me now.  I did look at it but not much yet.
>
> Suspect 1.
> 144 instances of "org.apache.catalina.core.StandardContext", loaded by
> "java.net.URLClassLoader @ 0x94d3de00" occupy 1,566,771,736 (76.22%) bytes.
> These instances are referenced from one instance of
> "java.util.HashMap$Node[]", loaded by "<system class loader>"
>
> Suspect 2.
> 380 instances of "org.apache.tomee.catalina.TomEEWebappClassLoader", loaded
> by "java.net.URLClassLoader @ 0x94d3de00" occupy 276,743,720 (13.46%)
> bytes.
>
> Keywords
> org.apache.tomee.catalina.TomEEWebappClassLoader
> java.net.URLClassLoader @ 0x94d3de00
>
> Also I got this stack trace:
>
> 02-Mar-2019 00:29:58.951 SEVERE [http-nio-41215-exec-983]
>
> org.apache.openejb.core.transaction.EjbTransactionUtil.handleSystemException
> EjbTransactionUtil.handleSystemException: Java heap space
>  java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:3236)
>         at
> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
>         at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>         at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
>         at
> org.apache.xbean.asm7.ClassReader.readStream(ClassReader.java:307)
>         at org.apache.xbean.asm7.ClassReader.<init>(ClassReader.java:273)
>         at
>
> org.apache.xbean.finder.AnnotationFinder.readClassDef(AnnotationFinder.java:1167)
>         at
> org.apache.xbean.finder.AnnotationFinder.<init>(AnnotationFinder.java:145)
>         at
> org.apache.xbean.finder.AnnotationFinder.<init>(AnnotationFinder.java:158)
>         at
>
> org.apache.openejb.config.FinderFactory$OpenEJBAnnotationFinder.<init>(FinderFactory.java:546)
>         at
> org.apache.openejb.config.FinderFactory.newFinder(FinderFactory.java:267)
>         at
> org.apache.openejb.config.FinderFactory.create(FinderFactory.java:80)
>         at
> org.apache.openejb.config.FinderFactory.createFinder(FinderFactory.java:69)
>         at
>
> org.apache.openejb.config.DeploymentLoader.addWebModule(DeploymentLoader.java:878)
>         at
> org.apache.openejb.config.DeploymentLoader.load(DeploymentLoader.java:233)
>         at
>
> org.apache.tomee.catalina.TomcatWebAppBuilder.loadApplication(TomcatWebAppBuilder.java:2352)
>         at
>
> org.apache.tomee.catalina.TomcatWebAppBuilder.startInternal(TomcatWebAppBuilder.java:1197)
>         at
>
> org.apache.tomee.catalina.TomcatWebAppBuilder.configureStart(TomcatWebAppBuilder.java:1130)
>         at
>
> org.apache.tomee.catalina.GlobalListenerSupport.lifecycleEvent(GlobalListenerSupport.java:134)
>         at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
>         at
>
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5007)
>         at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
>         at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
>         at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
>         at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:703)
>         at
>
> org.apache.tomee.catalina.TomcatWebAppBuilder.deployWar(TomcatWebAppBuilder.java:657)
>         at
>
> org.apache.tomee.catalina.TomcatWebAppBuilder.deployWebApps(TomcatWebAppBuilder.java:597)
>         at
>
> org.apache.tomee.catalina.deployment.TomcatWebappDeployer.deploy(TomcatWebappDeployer.java:47)
>         at
> org.apache.openejb.assembler.DeployerEjb.deploy(DeployerEjb.java:177)
>         at
> org.apache.openejb.assembler.DeployerEjb.deploy(DeployerEjb.java:140)
>         at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
CXF 3.3.1 is also causing the problem.

I've submitted PR 429 just to make buildbot happy again for the time being.
Would you mind merging just to get the builds going again?  Will be good to
confirm it's happy with the other PRs.

Once a root cause/solution is found can bump CXF back up again.



--
Sent from: http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

jgallimore
Done

On Sat, 2 Mar 2019, 09:37 j4fm, <[hidden email]> wrote:

> CXF 3.3.1 is also causing the problem.
>
> I've submitted PR 429 just to make buildbot happy again for the time being.
> Would you mind merging just to get the builds going again?  Will be good to
> confirm it's happy with the other PRs.
>
> Once a root cause/solution is found can bump CXF back up again.
>
>
>
> --
> Sent from:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

Daniel Cunha-2
nice!

Em sáb, 2 de mar de 2019 às 06:47, Jonathan Gallimore <
[hidden email]> escreveu:

> Done
>
> On Sat, 2 Mar 2019, 09:37 j4fm, <[hidden email]> wrote:
>
> > CXF 3.3.1 is also causing the problem.
> >
> > I've submitted PR 429 just to make buildbot happy again for the time
> being.
> > Would you mind merging just to get the builds going again?  Will be good
> to
> > confirm it's happy with the other PRs.
> >
> > Once a root cause/solution is found can bump CXF back up again.
> >
> >
> >
> > --
> > Sent from:
> > http://tomee-openejb.979440.n4.nabble.com/TomEE-Dev-f982480.html
> >
>


--
Daniel "soro" Cunha
https://twitter.com/dvlc_
Reply | Threaded
Open this post in threaded view
|

Re: Buildbot slowness

j4fm
This post was updated on .
Slowness is fixed, so buildbot is happier but not quite happy.  Just for
awareness to avoid someone else looking at it too...

1. I forgot to update the CXF version in the examples\polling-parent.  So
polling-web failed.

2. Also, it turns out CXF didn't add support for mp rest client 1.2.0 until
3.3.0 either.

So I've updated those to dependency versions based on moving back to CXF
3.2.7 until that is fixed back up to 3.3.0.

Please would someone mind pushing PR430 to master so that buildbot is happy again.
(It's already failing and I don't want to wait 8 hours for a full test run
this same config locally again)

12