EJB timers stop working.

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

EJB timers stop working.

gmatej
Hi all.

In production under load, we experienced a case were a particular timer
stopped working.

I thought maybe a runstime exception stops a ScheduleExpression, but could
not simulate this. Glassfish I know was very sensible to this.

Could a low EJB timer thread seeting cause a timer to stop, working.

We use tomee 1.7

The timer is initialized like that.

@Singleton
@Lock(LockType.READ)
@Startup
public class CommEtaUpdateCacheTimer {

    @Inject
    private TimerScheduler scheduler;

    @PostConstruct
    public void doit() {
        final ScheduleExpression schedule = new ScheduleExpression()
                .hour("*")
                .minute("*")
                .second("*/5");

        scheduler.scheduleEvent(schedule, new
CommEtaUpdateCacheTimerEvent());
    }

    public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
/*
processing code.
*/

    }

    public static class CommEtaUpdateCacheTimerEvent {
    }
}

BR

Matej
Reply | Threaded
Open this post in threaded view
|

Re: EJB timers stop working.

Romain Manni-Bucau
Hi

how do you use TimerService? No log at all saying there is an issue?
What's @Timeout method?


Romain Manni-Bucau
@rmannibucau
http://www.tomitribe.com
http://rmannibucau.wordpress.com
https://github.com/rmannibucau


2015-02-11 15:48 GMT+01:00 Matej <[hidden email]>:

> Hi all.
>
> In production under load, we experienced a case were a particular timer
> stopped working.
>
> I thought maybe a runstime exception stops a ScheduleExpression, but could
> not simulate this. Glassfish I know was very sensible to this.
>
> Could a low EJB timer thread seeting cause a timer to stop, working.
>
> We use tomee 1.7
>
> The timer is initialized like that.
>
> @Singleton
> @Lock(LockType.READ)
> @Startup
> public class CommEtaUpdateCacheTimer {
>
>     @Inject
>     private TimerScheduler scheduler;
>
>     @PostConstruct
>     public void doit() {
>         final ScheduleExpression schedule = new ScheduleExpression()
>                 .hour("*")
>                 .minute("*")
>                 .second("*/5");
>
>         scheduler.scheduleEvent(schedule, new
> CommEtaUpdateCacheTimerEvent());
>     }
>
>     public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
> /*
> processing code.
> */
>
>     }
>
>     public static class CommEtaUpdateCacheTimerEvent {
>     }
> }
>
> BR
>
> Matej
Reply | Threaded
Open this post in threaded view
|

Re: EJB timers stop working.

gmatej
No log was noticed. But maybe we missed it. We will look further. But
definitely no periodic log.

Like that:

@Singleton
@Lock(LockType.READ)
public class TimerScheduler {

    @Resource
    private TimerService timerService;
    @Inject
    Logger logger;
    @Resource
    private BeanManager beanManager;

    public void scheduleEvent(ScheduleExpression schedule, Object event,
Annotation... qualifiers) {

        timerService.createCalendarTimer(schedule, new TimerConfig(new
EventConfig(event, qualifiers), false));
    }

    @Timeout
    private void timeout(Timer timer) {
        final EventConfig config = (EventConfig) timer.getInfo();

        beanManager.fireEvent(config.getEvent(), config.getQualifiers());
    }

    private final class EventConfig implements Serializable {

        private final Object event;
        private final Annotation[] qualifiers;

        private EventConfig(Object event, Annotation[] qualifiers) {
            this.event = event;
            this.qualifiers = qualifiers;
        }

        public Object getEvent() {
            return event;
        }

        public Annotation[] getQualifiers() {
            return qualifiers;
        }
    }
}

2015-02-11 15:54 GMT+01:00 Romain Manni-Bucau <[hidden email]>:

> Hi
>
> how do you use TimerService? No log at all saying there is an issue?
> What's @Timeout method?
>
>
> Romain Manni-Bucau
> @rmannibucau
> http://www.tomitribe.com
> http://rmannibucau.wordpress.com
> https://github.com/rmannibucau
>
>
> 2015-02-11 15:48 GMT+01:00 Matej <[hidden email]>:
> > Hi all.
> >
> > In production under load, we experienced a case were a particular timer
> > stopped working.
> >
> > I thought maybe a runstime exception stops a ScheduleExpression, but
> could
> > not simulate this. Glassfish I know was very sensible to this.
> >
> > Could a low EJB timer thread seeting cause a timer to stop, working.
> >
> > We use tomee 1.7
> >
> > The timer is initialized like that.
> >
> > @Singleton
> > @Lock(LockType.READ)
> > @Startup
> > public class CommEtaUpdateCacheTimer {
> >
> >     @Inject
> >     private TimerScheduler scheduler;
> >
> >     @PostConstruct
> >     public void doit() {
> >         final ScheduleExpression schedule = new ScheduleExpression()
> >                 .hour("*")
> >                 .minute("*")
> >                 .second("*/5");
> >
> >         scheduler.scheduleEvent(schedule, new
> > CommEtaUpdateCacheTimerEvent());
> >     }
> >
> >     public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
> > /*
> > processing code.
> > */
> >
> >     }
> >
> >     public static class CommEtaUpdateCacheTimerEvent {
> >     }
> > }
> >
> > BR
> >
> > Matej
>
Reply | Threaded
Open this post in threaded view
|

Re: EJB timers stop working.

Romain Manni-Bucau
Hmm, not having any log is weird if you dont have a JVM crash. If you
manage to reproduce it having at least a thread stack dump (using
jstack for instance) would help


Romain Manni-Bucau
@rmannibucau
http://www.tomitribe.com
http://rmannibucau.wordpress.com
https://github.com/rmannibucau


2015-02-11 16:04 GMT+01:00 Matej <[hidden email]>:

> No log was noticed. But maybe we missed it. We will look further. But
> definitely no periodic log.
>
> Like that:
>
> @Singleton
> @Lock(LockType.READ)
> public class TimerScheduler {
>
>     @Resource
>     private TimerService timerService;
>     @Inject
>     Logger logger;
>     @Resource
>     private BeanManager beanManager;
>
>     public void scheduleEvent(ScheduleExpression schedule, Object event,
> Annotation... qualifiers) {
>
>         timerService.createCalendarTimer(schedule, new TimerConfig(new
> EventConfig(event, qualifiers), false));
>     }
>
>     @Timeout
>     private void timeout(Timer timer) {
>         final EventConfig config = (EventConfig) timer.getInfo();
>
>         beanManager.fireEvent(config.getEvent(), config.getQualifiers());
>     }
>
>     private final class EventConfig implements Serializable {
>
>         private final Object event;
>         private final Annotation[] qualifiers;
>
>         private EventConfig(Object event, Annotation[] qualifiers) {
>             this.event = event;
>             this.qualifiers = qualifiers;
>         }
>
>         public Object getEvent() {
>             return event;
>         }
>
>         public Annotation[] getQualifiers() {
>             return qualifiers;
>         }
>     }
> }
>
> 2015-02-11 15:54 GMT+01:00 Romain Manni-Bucau <[hidden email]>:
>
>> Hi
>>
>> how do you use TimerService? No log at all saying there is an issue?
>> What's @Timeout method?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau
>> http://www.tomitribe.com
>> http://rmannibucau.wordpress.com
>> https://github.com/rmannibucau
>>
>>
>> 2015-02-11 15:48 GMT+01:00 Matej <[hidden email]>:
>> > Hi all.
>> >
>> > In production under load, we experienced a case were a particular timer
>> > stopped working.
>> >
>> > I thought maybe a runstime exception stops a ScheduleExpression, but
>> could
>> > not simulate this. Glassfish I know was very sensible to this.
>> >
>> > Could a low EJB timer thread seeting cause a timer to stop, working.
>> >
>> > We use tomee 1.7
>> >
>> > The timer is initialized like that.
>> >
>> > @Singleton
>> > @Lock(LockType.READ)
>> > @Startup
>> > public class CommEtaUpdateCacheTimer {
>> >
>> >     @Inject
>> >     private TimerScheduler scheduler;
>> >
>> >     @PostConstruct
>> >     public void doit() {
>> >         final ScheduleExpression schedule = new ScheduleExpression()
>> >                 .hour("*")
>> >                 .minute("*")
>> >                 .second("*/5");
>> >
>> >         scheduler.scheduleEvent(schedule, new
>> > CommEtaUpdateCacheTimerEvent());
>> >     }
>> >
>> >     public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
>> > /*
>> > processing code.
>> > */
>> >
>> >     }
>> >
>> >     public static class CommEtaUpdateCacheTimerEvent {
>> >     }
>> > }
>> >
>> > BR
>> >
>> > Matej
>>
Reply | Threaded
Open this post in threaded view
|

Re: EJB timers stop working.

romych
In reply to this post by gmatej


@Timeout
    private void timeout(Timer timer) {
        final EventConfig config = (EventConfig) timer.getInfo();

        beanManager.fireEvent(config.getEvent(), config.getQualifiers());
    } 
There may be problem in concurency management. May be previous work hangs inside timeout method. Then timer can not call timeout method, it works like synchronized. You may try to annotate singleton class with @ConcurrencyManagement(ConcurrencyManagementType.BEAN). And use something about on timeout method:
private static final Lock inTimer = new ReentrantLock();

@Timeout
private void timeout(Timer timer) {

    if (inTimer.tryLock(5000, TimeUnit.MILLISECONDS)) {
      try {
	doWork();
      } finally {
	inTimer.unlock();
      }
    } else {
	logger.warn("Can not get lock!");
    }

}
So you can see where problem is.
Reply | Threaded
Open this post in threaded view
|

Re: EJB timers stop working.

jieryn
In reply to this post by gmatej
I've seen this in production for quite a while, too, on Apache TomEE 1.7.1.

I've never been able to catch a log message which showed it, we end up
having to restart periodically in order to ensure the timers are
reactivated. I suspect this is a user error, but .. no log messages.
It's not easy to detect when something isn't running..

Is there some class name I can grep my logs for that would help?
Whatever the class that runs the @Scheduled @Singleton @Stateless
@Startup beans.

On Wed, Feb 11, 2015 at 9:48 AM, Matej <[hidden email]> wrote:

> Hi all.
>
> In production under load, we experienced a case were a particular timer
> stopped working.
>
> I thought maybe a runstime exception stops a ScheduleExpression, but could
> not simulate this. Glassfish I know was very sensible to this.
>
> Could a low EJB timer thread seeting cause a timer to stop, working.
>
> We use tomee 1.7
>
> The timer is initialized like that.
>
> @Singleton
> @Lock(LockType.READ)
> @Startup
> public class CommEtaUpdateCacheTimer {
>
>     @Inject
>     private TimerScheduler scheduler;
>
>     @PostConstruct
>     public void doit() {
>         final ScheduleExpression schedule = new ScheduleExpression()
>                 .hour("*")
>                 .minute("*")
>                 .second("*/5");
>
>         scheduler.scheduleEvent(schedule, new
> CommEtaUpdateCacheTimerEvent());
>     }
>
>     public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
> /*
> processing code.
> */
>
>     }
>
>     public static class CommEtaUpdateCacheTimerEvent {
>     }
> }
>
> BR
>
> Matej
Reply | Threaded
Open this post in threaded view
|

Re: EJB timers stop working.

Romain Manni-Bucau
Here the log I'd check: "Timer" (note: not sure it is on by default I
didnt check its default config in tomee)

Some additional checks: with jstack check threads "EjbTimerPool*"

Also check your config of the timer pool (here some basic
conf/system.properties properties for default setup):

EjbTimerPool.CorePoolSize=10
EjbTimerPool.RejectedExecutionHandlerClass=no default for it so it
uses next property
EjbTimerPool.OfferTimeout = 30 seconds


My guess is either these threads are stucked somewhere and we need to
know (can just be a standard queueing issue due to overload) or they
are rejecting tasks but this should be logged AFAIK



Romain Manni-Bucau
@rmannibucau
http://www.tomitribe.com
http://rmannibucau.wordpress.com
https://github.com/rmannibucau


2015-02-11 17:29 GMT+01:00 jieryn <[hidden email]>:

> I've seen this in production for quite a while, too, on Apache TomEE 1.7.1.
>
> I've never been able to catch a log message which showed it, we end up
> having to restart periodically in order to ensure the timers are
> reactivated. I suspect this is a user error, but .. no log messages.
> It's not easy to detect when something isn't running..
>
> Is there some class name I can grep my logs for that would help?
> Whatever the class that runs the @Scheduled @Singleton @Stateless
> @Startup beans.
>
> On Wed, Feb 11, 2015 at 9:48 AM, Matej <[hidden email]> wrote:
>> Hi all.
>>
>> In production under load, we experienced a case were a particular timer
>> stopped working.
>>
>> I thought maybe a runstime exception stops a ScheduleExpression, but could
>> not simulate this. Glassfish I know was very sensible to this.
>>
>> Could a low EJB timer thread seeting cause a timer to stop, working.
>>
>> We use tomee 1.7
>>
>> The timer is initialized like that.
>>
>> @Singleton
>> @Lock(LockType.READ)
>> @Startup
>> public class CommEtaUpdateCacheTimer {
>>
>>     @Inject
>>     private TimerScheduler scheduler;
>>
>>     @PostConstruct
>>     public void doit() {
>>         final ScheduleExpression schedule = new ScheduleExpression()
>>                 .hour("*")
>>                 .minute("*")
>>                 .second("*/5");
>>
>>         scheduler.scheduleEvent(schedule, new
>> CommEtaUpdateCacheTimerEvent());
>>     }
>>
>>     public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
>> /*
>> processing code.
>> */
>>
>>     }
>>
>>     public static class CommEtaUpdateCacheTimerEvent {
>>     }
>> }
>>
>> BR
>>
>> Matej
Reply | Threaded
Open this post in threaded view
|

Re: EJB timers stop working.

Romain Manni-Bucau
In reply to this post by romych
cause of @Lock(LockType.READ) it shouldn't be the case


Romain Manni-Bucau
@rmannibucau
http://www.tomitribe.com
http://rmannibucau.wordpress.com
https://github.com/rmannibucau


2015-02-11 16:32 GMT+01:00 romych <[hidden email]>:

> @Timeout    private void timeout(Timer timer) {        final EventConfig
> config = (EventConfig) timer.getInfo();
> beanManager.fireEvent(config.getEvent(), config.getQualifiers());    }
> There may be problem in concurency management. May be previous work hangs
> inside timeout method. Then timer can not call timeout method, it works like
> synchronized.You may try to annotate singleton class with
> @ConcurrencyManagement(ConcurrencyManagementType.BEAN).And use something
> about on timeout method:
> private static final Lock inTimer = new ReentrantLock();@Timeoutprivate void
> timeout(Timer timer) {    if (inTimer.tryLock(5000, TimeUnit.MILLISECONDS))
> {      try {    doWork();      } finally {      inTimer.unlock();      }    } else {
> logger.warn("Can not get lock!");    }}
> So you can see where problem is.
>
>
>
> --
> View this message in context: http://tomee-openejb.979440.n4.nabble.com/EJB-timers-stop-working-tp4673750p4673755.html
> Sent from the TomEE Users mailing list archive at Nabble.com.