SYMPTOM:
Users that have created some time-based scheduled subscriptions in a clustered environment do not see the subscriptions getting delivered correctly. When they review DSSErrors logs, they are not able to find any errors. Actually, they will regularly see that the Scheduler Trace is logging the line shown below:
Sample Code/Error
2011-03-08 11:00:00.014-05:00 MSISchedulerTimerEntryTask::Run(): TriggerName = <11amtest>: Reset schedule entry for next time.
However, if the users continue troubleshooting the issue, they will notice that Strategy should be also logging 2 more lines at the time a subscription is getting triggered. The correct logging traces should look like:
Sample Code/Error
2011-03-04 11:00:00.019-05:00 MSISchedulerTimerEntryTask::Run(): TriggerName = <11amtest>: Reset schedule entry for next time.
2011-03-04 11:00:00.644-05:00 Distribution Service Trigger: Retrieved 1 subscriptions for project 1 for Trigger 11amtest
2011-03-04 11:00:00.644-05:00 Distribution Service Trigger: remote, executing 1 subscriptions for project 1 for Trigger 11amtest
Another interesting symptom is that all the event-based subscriptions are getting delivered successfully.
If the users review IS_MESSAGE_STATS statistics table they will be able to track the behavior of this kind of subscriptions. The table below shows that the Intelligence Server was able to log information for all the days (since this test was done with a subscription running everyday) except for March 8, 9 and 13.
CAUSE:
It’s possible that all nodes in the Strategy clustered environment start up very close to the same time and there is a conflict in determining which one is the primary node. In this case, when the time-based schedule fires and no node is defined as primary, Strategy Intelligence Server would not run the subscriptions.
The issue should only affect time-based schedules since event-based schedules will run on the node where they are triggered.
ACTION:
Since the problem originated due to a conflict in determining the primary node in the cluster, the users should not start all Intelligence Servers at the same time - rather they should stagger their start-ups such that the Intelligence Servers start one after the other with a small time gap between them.
