Investigating https://issues.apache.org/jira/browse/TOMEE-2403

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Investigating https://issues.apache.org/jira/browse/TOMEE-2403

doychin@dsoft-bg.com
Hi,

I would like to investigate this issue:
https://issues.apache.org/jira/browse/TOMEE-2403

Where I can get more detailed error log regarding this issue?

I tried to run the test locally but it works.

--
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116


doychin.vcf (280 bytes) Download Attachment
smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly

David Blevins-2
> On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]> wrote:
>
> I would like to investigate this issue: https://issues.apache.org/jira/browse/TOMEE-2403
>
> Where I can get more detailed error log regarding this issue?
>
> I tried to run the test locally but it works.

Here's a technique I use sometimes to check random failures:

    $ for n in {1..50}; do mvn clean install -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done

So far 3 out 17 have failed.  Debugging them will still be a pain in the butt, but at least there's some hope.


My shallow observation is that the approach taken in the test itself looks conspicuous.  Particularly, I'd say this test is not "correct" until it can pass without the System.gc() statements.

 - https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197

More importantly, this method is using thread sleeps and retries to wait for log output which *will* cause random failures.  We should have a discussion on if there is a better way to make this more deterministic.

 - https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216

Testing log output can work, however the test logic is ultimately "if you do or don't see this message in n seconds, we're good."  It needs to be "if you do or don't see this message *when the test is over*, we're good."

Since it's looking in log output, one way to do that might be to simply log a "test complete" statement (perhaps with a random number on it), and then wait for that in the log output.


-David





signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly

Carl Mosca
Pretty cool technique... never have done mvn in a loop like that...but I
think I will. ☺️

On Sun, Jan 6, 2019, 12:47 PM David Blevins <[hidden email] wrote:

> > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]>
> wrote:
> >
> > I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
> >
> > Where I can get more detailed error log regarding this issue?
> >
> > I tried to run the test locally but it works.
>
> Here's a technique I use sometimes to check random failures:
>
>     $ for n in {1..50}; do mvn clean install
> -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
>
> So far 3 out 17 have failed.  Debugging them will still be a pain in the
> butt, but at least there's some hope.
>
>
> My shallow observation is that the approach taken in the test itself looks
> conspicuous.  Particularly, I'd say this test is not "correct" until it can
> pass without the System.gc() statements.
>
>  -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
>
> More importantly, this method is using thread sleeps and retries to wait
> for log output which *will* cause random failures.  We should have a
> discussion on if there is a better way to make this more deterministic.
>
>  -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
>
> Testing log output can work, however the test logic is ultimately "if you
> do or don't see this message in n seconds, we're good."  It needs to be "if
> you do or don't see this message *when the test is over*, we're good."
>
> Since it's looking in log output, one way to do that might be to simply
> log a "test complete" statement (perhaps with a random number on it), and
> then wait for that in the log output.
>
>
> -David
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly

doychin@dsoft-bg.com
In reply to this post by David Blevins-2

I use IDE to run test multiple times. It failed only once here.

I also did some changes in the test code.

I removed the line

https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162

This made test many times faster. It is not needed. This tests non leaking business method so we expect everything to be cleared.

Also I removed the all the code that waits for log messages. I left only the line

assertEquals(message, times, logCapture.find(message).size());

There is no need to wait for anything. All the waiting is already done for the first and second tests in

runTest method

Regarding

System.gc(); 

It is needed only for the tests that leak so I removed it from all other tests.

Third and fourth test are run in the main thread and there is no need to wait before processing the log to.

You can see the change here :

In case you like it I can create PR.

On 6.1.2019 г. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote:
> On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]> wrote:
>
> I would like to investigate this issue: https://issues.apache.org/jira/browse/TOMEE-2403
>
> Where I can get more detailed error log regarding this issue?
>
> I tried to run the test locally but it works.

Here's a technique I use sometimes to check random failures:

    $ for n in {1..50}; do mvn clean install -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done

So far 3 out 17 have failed.  Debugging them will still be a pain in the butt, but at least there's some hope.


My shallow observation is that the approach taken in the test itself looks conspicuous.  Particularly, I'd say this test is not "correct" until it can pass without the System.gc() statements.

 - https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197

More importantly, this method is using thread sleeps and retries to wait for log output which *will* cause random failures.  We should have a discussion on if there is a better way to make this more deterministic.

 - https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216

Testing log output can work, however the test logic is ultimately "if you do or don't see this message in n seconds, we're good."  It needs to be "if you do or don't see this message *when the test is over*, we're good."

Since it's looking in log output, one way to do that might be to simply log a "test complete" statement (perhaps with a random number on it), and then wait for that in the log output.


-David





signature.asc (499 bytes) Download Attachment



To start a new topic under TomEE Dev, email [hidden email]
To unsubscribe from TomEE Dev, click here.
NAML


-- 
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116

doychin.vcf (280 bytes) Download Attachment
smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Investigating https://issues.apache.org/jira/browse/TOMEE-2403

César Hernández Mendoza
In reply to this post by doychin@dsoft-bg.com
Hi Doychin,
By looking at TomEE-2403, it seems the issue was occurring only on TomEE CI
[1].
If you click on a specific build you can have access to the entire log, see
for instance:
https://ci.apache.org/builders/tomee-trunk-ubuntu-jvm8/builds/987/steps/test/logs/stdio

As a side note, I think this CI information can be something we can improve
on the TomEE website.

[1] https://ci.apache.org/builders/tomee-trunk-ubuntu-jvm8

El dom., 6 ene. 2019 a las 4:25, Doychin Bondzhev (<[hidden email]>)
escribió:

> Hi,
>
> I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
>
> Where I can get more detailed error log regarding this issue?
>
> I tried to run the test locally but it works.
>
> --
> Doychin Bondzhev
> dSoft-Bulgaria Ltd.
> PowerPro - billing & provisioning solution for Service providers
> http://www.dsoft-bg.com/
> Mobile: +359888243116
>
>

--
Atentamente:
César Hernández Mendoza.
Reply | Threaded
Open this post in threaded view
|

Re: Investigating https://issues.apache.org/jira/browse/TOMEE-2403

doychin@dsoft-bg.com
Hi César,

I found the CI pages but looks like logs are not kept for long enough.

Any way I managed to reproduce the problem and I already implemented some changes in the test code and in the code that is tested.

You can check them here in case you are interested and let me know if you have comments.



On 7.1.2019 г. 16:44, César Hernández Mendoza [via TomEE & OpenEJB] wrote:
Hi Doychin,
By looking at TomEE-2403, it seems the issue was occurring only on TomEE CI
[1].
If you click on a specific build you can have access to the entire log, see
for instance:
https://ci.apache.org/builders/tomee-trunk-ubuntu-jvm8/builds/987/steps/test/logs/stdio

As a side note, I think this CI information can be something we can improve
on the TomEE website.

[1] https://ci.apache.org/builders/tomee-trunk-ubuntu-jvm8

El dom., 6 ene. 2019 a las 4:25, Doychin Bondzhev (<[hidden email]>)
escribió:

> Hi,
>
> I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
>
> Where I can get more detailed error log regarding this issue?
>
> I tried to run the test locally but it works.
>
> --
> Doychin Bondzhev
> dSoft-Bulgaria Ltd.
> PowerPro - billing & provisioning solution for Service providers
> http://www.dsoft-bg.com/
> Mobile: +359888243116
>
>

--
Atentamente:
César Hernández Mendoza.



To start a new topic under TomEE Dev, email [hidden email]
To unsubscribe from TomEE Dev, click here.
NAML


-- 
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116

doychin.vcf (280 bytes) Download Attachment
smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly

jgallimore
In reply to this post by doychin@dsoft-bg.com
This test was a pain to write, and I am grateful for the feedback. Please
do send a PR and I'll take a look.

Some background to the problem here is that the the AutoConnectionTracker
failed in some circumstances when servers were under heavy load with lots
of GC happening - there appeared to be a timing issue where a resource that
was actually in use was being forcefully closed by the auto connection
tracker, but because something else still had a hold of it, it ended up
going back in the pool and a broken resource being served up to the next
thing that requested it. The requests for GC were quite key in reproducing
the actual issue originally, so removing them might not actually be the
right approach.

It was particularly hard to reproduce and pin down and I ended up going
through something like 40GB of heap dumps and logs to figure out what was
happening. Previously there was no test, which made this even harder to pin
down. I'm definitely happy to explore some options to make this more solid.

Jon

On Sun, Jan 6, 2019 at 8:28 PM Doychin Bondzhev <[hidden email]>
wrote:

> I use IDE to run test multiple times. It failed only once here.
>
> I also did some changes in the test code.
>
> I removed the line
>
>
> https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162
>
> This made test many times faster. It is not needed. This tests non leaking
> business method so we expect everything to be cleared.
>
> Also I removed the all the code that waits for log messages. I left only
> the line
>
> assertEquals(message, times, logCapture.find(message).size());
>
>
> There is no need to wait for anything. All the waiting is already done for
> the first and second tests in
>
> runTest method
>
> Regarding
>
> System.gc();
>
>
> It is needed only for the tests that leak so I removed it from all other
> tests.
>
> Third and fourth test are run in the main thread and there is no need to
> wait before processing the log to.
> You can see the change here :
>
> https://github.com/doychin/tomee/commit/76e090b482825682a859d6a1637f572d8128f30a
>
> In case you like it I can create PR.
>
> On 6.1.2019 г. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote:
>
> > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=4687291&i=0>> wrote:
> >
> > I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
> >
> > Where I can get more detailed error log regarding this issue?
> >
> > I tried to run the test locally but it works.
>
> Here's a technique I use sometimes to check random failures:
>
>     $ for n in {1..50}; do mvn clean install
> -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
>
> So far 3 out 17 have failed.  Debugging them will still be a pain in the
> butt, but at least there's some hope.
>
>
> My shallow observation is that the approach taken in the test itself looks
> conspicuous.  Particularly, I'd say this test is not "correct" until it can
> pass without the System.gc() statements.
>
>  -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
>
> More importantly, this method is using thread sleeps and retries to wait
> for log output which *will* cause random failures.  We should have a
> discussion on if there is a better way to make this more deterministic.
>
>  -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
>
> Testing log output can work, however the test logic is ultimately "if you
> do or don't see this message in n seconds, we're good."  It needs to be "if
> you do or don't see this message *when the test is over*, we're good."
>
> Since it's looking in log output, one way to do that might be to simply
> log a "test complete" statement (perhaps with a random number on it), and
> then wait for that in the log output.
>
>
> -David
>
>
>
>
>
> *signature.asc* (499 bytes) Download Attachment
> <http://tomee-openejb.979440.n4.nabble.com/attachment/4687291/0/signature.asc>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687291.html
> To start a new topic under TomEE Dev, email
> [hidden email]
> To unsubscribe from TomEE Dev, click here
> <
> .
> NAML
> <
http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> Doychin Bondzhev
> dSoft-Bulgaria Ltd.
> PowerPro - billing & provisioning solution for Service providershttp://www.dsoft-bg.com/
> Mobile: +359888243116
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly

doychin@dsoft-bg.com
Hi Jon,

Please check this PR:

We can discuss the details there.

I also found this bug report which is still open:

Is that the problem you describe?

On 7.1.2019 г. 18:51, jgallimore [via TomEE & OpenEJB] wrote:
This test was a pain to write, and I am grateful for the feedback. Please
do send a PR and I'll take a look.

Some background to the problem here is that the the AutoConnectionTracker
failed in some circumstances when servers were under heavy load with lots
of GC happening - there appeared to be a timing issue where a resource that
was actually in use was being forcefully closed by the auto connection
tracker, but because something else still had a hold of it, it ended up
going back in the pool and a broken resource being served up to the next
thing that requested it. The requests for GC were quite key in reproducing
the actual issue originally, so removing them might not actually be the
right approach.

It was particularly hard to reproduce and pin down and I ended up going
through something like 40GB of heap dumps and logs to figure out what was
happening. Previously there was no test, which made this even harder to pin
down. I'm definitely happy to explore some options to make this more solid.

Jon

On Sun, Jan 6, 2019 at 8:28 PM Doychin Bondzhev <[hidden email]>
wrote:

> I use IDE to run test multiple times. It failed only once here.
>
> I also did some changes in the test code.
>
> I removed the line
>
>
> https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162
>
> This made test many times faster. It is not needed. This tests non leaking
> business method so we expect everything to be cleared.
>
> Also I removed the all the code that waits for log messages. I left only
> the line
>
> assertEquals(message, times, logCapture.find(message).size());
>
>
> There is no need to wait for anything. All the waiting is already done for
> the first and second tests in
>
> runTest method
>
> Regarding
>
> System.gc();
>
>
> It is needed only for the tests that leak so I removed it from all other
> tests.
>
> Third and fourth test are run in the main thread and there is no need to
> wait before processing the log to.
> You can see the change here :
>
> https://github.com/doychin/tomee/commit/76e090b482825682a859d6a1637f572d8128f30a
>
> In case you like it I can create PR.
>
> On 6.1.2019 г. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote:
>
> > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=4687291&i=0>> wrote:
> >
> > I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
> >
> > Where I can get more detailed error log regarding this issue?
> >
> > I tried to run the test locally but it works.
>
> Here's a technique I use sometimes to check random failures:
>
>     $ for n in {1..50}; do mvn clean install
> -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
>
> So far 3 out 17 have failed.  Debugging them will still be a pain in the
> butt, but at least there's some hope.
>
>
> My shallow observation is that the approach taken in the test itself looks
> conspicuous.  Particularly, I'd say this test is not "correct" until it can
> pass without the System.gc() statements.
>
>  -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
>
> More importantly, this method is using thread sleeps and retries to wait
> for log output which *will* cause random failures.  We should have a
> discussion on if there is a better way to make this more deterministic.
>
>  -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
>
> Testing log output can work, however the test logic is ultimately "if you
> do or don't see this message in n seconds, we're good."  It needs to be "if
> you do or don't see this message *when the test is over*, we're good."
>
> Since it's looking in log output, one way to do that might be to simply
> log a "test complete" statement (perhaps with a random number on it), and
> then wait for that in the log output.
>
>
> -David
>
>
>
>
>
> *signature.asc* (499 bytes) Download Attachment
> <http://tomee-openejb.979440.n4.nabble.com/attachment/4687291/0/signature.asc>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687291.html
> To start a new topic under TomEE Dev, email
> [hidden email]
> To unsubscribe from TomEE Dev, click here
> <
> .
> NAML
> <
http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> Doychin Bondzhev
> dSoft-Bulgaria Ltd.
> PowerPro - billing & provisioning solution for Service providershttp://www.dsoft-bg.com/
> Mobile: +359888243116
>
>



To start a new topic under TomEE Dev, email [hidden email]
To unsubscribe from TomEE Dev, click here.
NAML


-- 
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116

doychin.vcf (280 bytes) Download Attachment
smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly

jgallimore
Thanks. I'll set the rig up again and run some volume through with these
changes.

Jon

On Mon, Jan 7, 2019 at 8:46 PM Doychin Bondzhev <[hidden email]>
wrote:

> Hi Jon,
>
> Please check this PR:
> https://github.com/apache/tomee/pull/360
>
> We can discuss the details there.
>
> I also found this bug report which is still open:
> https://issues.apache.org/jira/browse/TOMEE-2257
>
> Is that the problem you describe?
>
> On 7.1.2019 г. 18:51, jgallimore [via TomEE & OpenEJB] wrote:
>
> This test was a pain to write, and I am grateful for the feedback. Please
> do send a PR and I'll take a look.
>
> Some background to the problem here is that the the AutoConnectionTracker
> failed in some circumstances when servers were under heavy load with lots
> of GC happening - there appeared to be a timing issue where a resource
> that
> was actually in use was being forcefully closed by the auto connection
> tracker, but because something else still had a hold of it, it ended up
> going back in the pool and a broken resource being served up to the next
> thing that requested it. The requests for GC were quite key in reproducing
> the actual issue originally, so removing them might not actually be the
> right approach.
>
> It was particularly hard to reproduce and pin down and I ended up going
> through something like 40GB of heap dumps and logs to figure out what was
> happening. Previously there was no test, which made this even harder to
> pin
> down. I'm definitely happy to explore some options to make this more
> solid.
>
> Jon
>
> On Sun, Jan 6, 2019 at 8:28 PM Doychin Bondzhev <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=4687328&i=0>>
> wrote:
>
> > I use IDE to run test multiple times. It failed only once here.
> >
> > I also did some changes in the test code.
> >
> > I removed the line
> >
> >
> >
> https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162
> >
> > This made test many times faster. It is not needed. This tests non
> leaking
> > business method so we expect everything to be cleared.
> >
> > Also I removed the all the code that waits for log messages. I left only
> > the line
> >
> > assertEquals(message, times, logCapture.find(message).size());
> >
> >
> > There is no need to wait for anything. All the waiting is already done
> for
> > the first and second tests in
> >
> > runTest method
> >
> > Regarding
> >
> > System.gc();
> >
> >
> > It is needed only for the tests that leak so I removed it from all other
> > tests.
> >
> > Third and fourth test are run in the main thread and there is no need to
> > wait before processing the log to.
> > You can see the change here :
> >
> >
> https://github.com/doychin/tomee/commit/76e090b482825682a859d6a1637f572d8128f30a
> >
> > In case you like it I can create PR.
> >
> > On 6.1.2019 г. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote:
> >
> > > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=4687291&i=0>> wrote:
> > >
> > > I would like to investigate this issue:
> > https://issues.apache.org/jira/browse/TOMEE-2403
> > >
> > > Where I can get more detailed error log regarding this issue?
> > >
> > > I tried to run the test locally but it works.
> >
> > Here's a technique I use sometimes to check random failures:
> >
> >     $ for n in {1..50}; do mvn clean install
> > -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
> >
> > So far 3 out 17 have failed.  Debugging them will still be a pain in the
> > butt, but at least there's some hope.
> >
> >
> > My shallow observation is that the approach taken in the test itself
> looks
> > conspicuous.  Particularly, I'd say this test is not "correct" until it
> can
> > pass without the System.gc() statements.
> >
> >  -
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
> >
> > More importantly, this method is using thread sleeps and retries to wait
> > for log output which *will* cause random failures.  We should have a
> > discussion on if there is a better way to make this more deterministic.
> >
> >  -
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
> >
> > Testing log output can work, however the test logic is ultimately "if
> you
> > do or don't see this message in n seconds, we're good."  It needs to be
> "if
> > you do or don't see this message *when the test is over*, we're good."
> >
> > Since it's looking in log output, one way to do that might be to simply
> > log a "test complete" statement (perhaps with a random number on it),
> and
> > then wait for that in the log output.
> >
> >
> > -David
> >
> >
> >
> >
> >
> > *signature.asc* (499 bytes) Download Attachment
> > <
> http://tomee-openejb.979440.n4.nabble.com/attachment/4687291/0/signature.asc>
>
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687291.html
> > To start a new topic under TomEE Dev, email
> > [hidden email] <http:///user/SendEmail.jtp?type=node&node=4687328&i=1>
> > To unsubscribe from TomEE Dev, click here
> > <
> > .
> > NAML
> > <
> http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> >
> >
> > --
> > Doychin Bondzhev
> > dSoft-Bulgaria Ltd.
> > PowerPro - billing & provisioning solution for Service providershttp://
> www.dsoft-bg.com/
> > Mobile: +359888243116
> >
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687328.html
> To start a new topic under TomEE Dev, email
> [hidden email]
> To unsubscribe from TomEE Dev, click here
> <
> .
> NAML
> <
http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> Doychin Bondzhev
> dSoft-Bulgaria Ltd.
> PowerPro - billing & provisioning solution for Service providershttp://www.dsoft-bg.com/
> Mobile: +359888243116
>
>