Precisely Enterworks

On the weekends (from Friday evening to Sunday evening) our non-PROD environments are shut down, we have an automated process that handles turning the server off and back on, and we are leveraging a Scheduled Export/Import process that turns off the scheduled Imports/Exports/Promotions before the servers are shut off.

We are encountering an issue where Scheduled jobs become backed up post-restart and require clearing of stuck work items in EPX and potentially an additional restart to clear long processing Scheduled jobs; are there any best practices, recommendations or success stories for automating this process?

Matthew,

Can you elaborate on which workflows result in stuck work items that require an additional restart?  Is it just because many scheduled jobs are launched as soon as the services are restarted after the downtime period?  Or is there a backlog of work items that were created but not processed because of the way jobs are being disabled?

If the EnterWorks services are off for some period of time and you have scheduled jobs that run frequently (daily or multiple times per day), restarting the services will immediately launch all scheduled jobs that would have run at least once if the services had been left running.  This assumes you are disabling and enabling scheduled jobs solely by changing the Scheduled attribute for each job. 

A comprehensive shutdown/restart procedure would need to perform the following actions:

Shutdown:

1.          Optionally shut down the enable-web-server-service to prevent users from launching jobs.  It may be enough to notify users that they must be off the system, but stopping the service ensures that they are.
2.          Disable all Scheduled Import, Export, and Promotion jobs so no new jobs are launched.  This includes changing the <job> Scheduled flag to No and adding "_HOLD" to the end of each job name to prevent another job from launching it to get the next file, which mainly applies to imports.
3.         Flag any implementation-specific workflow that launches work items with a Scheduler BIC to skip processing for any new work items.  This requires each workflow to be modified to follow the Scheduler BIC starting-point activity, check for the flag, and route the work item to an endpoint if the flag is set.
4. Obtain the MonitorFlows lock. This ensures that errant work items won't be re-processed during shutdown. Note: The amount of time this will hold is defined in the Enterworks.properties file via the epim.timeoutMillis property.  This is only a concern if there are work items stuck with an error due to environmental conditions, and clearing the error will resume processing.
5.          Monitor the processing of scheduled jobs until all have a terminal status (Completed or Error).
6.          Monitor the Jobs processing in the Job Monitor for active jobs until there are no longer any jobs with a processing state.  This needs to be done after the scheduled jobs have stopped processing, as an active scheduled job may launch a job or another Scheduled Job.
7.          Monitor implementation-specific workflow activity until no workflows (except for the workflow that is shutting down the system) have active work items at automatic activities (that are not stuck with errors).
8.          Optionally, notify the system is safe to shut down or is being shut down, and optionally shut down all services.

Startup:

1.           Adjust the Next execution timestamp of all scheduled jobs to the next interval that is in the future, and then re-activate those scheduled jobs and remove the " HOLD" from their names.
2.          Clear the flag for implementation-specific workflows, so they resume processing work items at their scheduled intervals

I am putting together the necessary workflows to verify the feasibility of this approach.  If you see anything I might have overlooked, I would appreciate the feedback.

On the weekends (from Friday evening to Sunday evening) our non-PROD environments are shut down, we have an automated process that handles turning the server off and back on, and we are leveraging a Scheduled Export/Import process that turns off the scheduled Imports/Exports/Promotions before the servers are shut off.

We are encountering an issue where Scheduled jobs become backed up post-restart and require clearing of stuck work items in EPX and potentially an additional restart to clear long processing Scheduled jobs; are there any best practices, recommendations or success stories for automating this process?

Hello Brian,

The primary issue in our case is solely with Scheduled Exports/Imports/Promotions. We have a Scheduled process that handles turning off every scheduled Exports/Imports/Promotion prior to shutdown which also sets the Next Export Datetime into the future. And then a similar scheduled process that turns everything back on, but we are not checking for the current job monitor activity or scheduled framework activity before shut down.

As you can see from the screenshot I included, the conditions we encounter post restart is where the Job Monitor becomes inundated with jobs stuck in a Queued state.

Usually clearing any hung jobs or scheduled processes is sufficient to get things moving again, but occasionally a full restart is then also required.

Matthew,

Can you elaborate on which workflows result in stuck work items that require an additional restart?  Is it just because many scheduled jobs are launched as soon as the services are restarted after the downtime period?  Or is there a backlog of work items that were created but not processed because of the way jobs are being disabled?

If the EnterWorks services are off for some period of time and you have scheduled jobs that run frequently (daily or multiple times per day), restarting the services will immediately launch all scheduled jobs that would have run at least once if the services had been left running.  This assumes you are disabling and enabling scheduled jobs solely by changing the Scheduled attribute for each job. 

A comprehensive shutdown/restart procedure would need to perform the following actions:

Shutdown:

1.          Optionally shut down the enable-web-server-service to prevent users from launching jobs.  It may be enough to notify users that they must be off the system, but stopping the service ensures that they are.
2.          Disable all Scheduled Import, Export, and Promotion jobs so no new jobs are launched.  This includes changing the <job> Scheduled flag to No and adding "_HOLD" to the end of each job name to prevent another job from launching it to get the next file, which mainly applies to imports.
3.         Flag any implementation-specific workflow that launches work items with a Scheduler BIC to skip processing for any new work items.  This requires each workflow to be modified to follow the Scheduler BIC starting-point activity, check for the flag, and route the work item to an endpoint if the flag is set.
4. Obtain the MonitorFlows lock. This ensures that errant work items won't be re-processed during shutdown. Note: The amount of time this will hold is defined in the Enterworks.properties file via the epim.timeoutMillis property.  This is only a concern if there are work items stuck with an error due to environmental conditions, and clearing the error will resume processing.
5.          Monitor the processing of scheduled jobs until all have a terminal status (Completed or Error).
6.          Monitor the Jobs processing in the Job Monitor for active jobs until there are no longer any jobs with a processing state.  This needs to be done after the scheduled jobs have stopped processing, as an active scheduled job may launch a job or another Scheduled Job.
7.          Monitor implementation-specific workflow activity until no workflows (except for the workflow that is shutting down the system) have active work items at automatic activities (that are not stuck with errors).
8.          Optionally, notify the system is safe to shut down or is being shut down, and optionally shut down all services.

Startup:

1.           Adjust the Next execution timestamp of all scheduled jobs to the next interval that is in the future, and then re-activate those scheduled jobs and remove the " HOLD" from their names.
2.          Clear the flag for implementation-specific workflows, so they resume processing work items at their scheduled intervals

I am putting together the necessary workflows to verify the feasibility of this approach.  If you see anything I might have overlooked, I would appreciate the feedback.

On the weekends (from Friday evening to Sunday evening) our non-PROD environments are shut down, we have an automated process that handles turning the server off and back on, and we are leveraging a Scheduled Export/Import process that turns off the scheduled Imports/Exports/Promotions before the servers are shut off.

We are encountering an issue where Scheduled jobs become backed up post-restart and require clearing of stuck work items in EPX and potentially an additional restart to clear long processing Scheduled jobs; are there any best practices, recommendations or success stories for automating this process?

Hi Matthew,

In addition to Brian's answer, you could also take a staggered approach to turning the scheduled processes back on. This means you wouldn't even need to modify the Next Export Datetime. Instead of re-enabling all previously scheduled processes at the same time, you could turn them back on one at a time every x minutes (5–15 minutes is probably sufficient, though it depends on the machine's capacity, the heaviness of the jobs, and the number of scheduled jobs).

Thanks,
Oliver Stamkov
Pivotree, Inc.

Hello Brian,

The primary issue in our case is solely with Scheduled Exports/Imports/Promotions. We have a Scheduled process that handles turning off every scheduled Exports/Imports/Promotion prior to shutdown which also sets the Next Export Datetime into the future. And then a similar scheduled process that turns everything back on, but we are not checking for the current job monitor activity or scheduled framework activity before shut down.

As you can see from the screenshot I included, the conditions we encounter post restart is where the Job Monitor becomes inundated with jobs stuck in a Queued state.

Usually clearing any hung jobs or scheduled processes is sufficient to get things moving again, but occasionally a full restart is then also required.

Matthew,

Can you elaborate on which workflows result in stuck work items that require an additional restart?  Is it just because many scheduled jobs are launched as soon as the services are restarted after the downtime period?  Or is there a backlog of work items that were created but not processed because of the way jobs are being disabled?

If the EnterWorks services are off for some period of time and you have scheduled jobs that run frequently (daily or multiple times per day), restarting the services will immediately launch all scheduled jobs that would have run at least once if the services had been left running.  This assumes you are disabling and enabling scheduled jobs solely by changing the Scheduled attribute for each job. 

A comprehensive shutdown/restart procedure would need to perform the following actions:

Shutdown:

1.          Optionally shut down the enable-web-server-service to prevent users from launching jobs.  It may be enough to notify users that they must be off the system, but stopping the service ensures that they are.
2.          Disable all Scheduled Import, Export, and Promotion jobs so no new jobs are launched.  This includes changing the <job> Scheduled flag to No and adding "_HOLD" to the end of each job name to prevent another job from launching it to get the next file, which mainly applies to imports.
3.         Flag any implementation-specific workflow that launches work items with a Scheduler BIC to skip processing for any new work items.  This requires each workflow to be modified to follow the Scheduler BIC starting-point activity, check for the flag, and route the work item to an endpoint if the flag is set.
4. Obtain the MonitorFlows lock. This ensures that errant work items won't be re-processed during shutdown. Note: The amount of time this will hold is defined in the Enterworks.properties file via the epim.timeoutMillis property.  This is only a concern if there are work items stuck with an error due to environmental conditions, and clearing the error will resume processing.
5.          Monitor the processing of scheduled jobs until all have a terminal status (Completed or Error).
6.          Monitor the Jobs processing in the Job Monitor for active jobs until there are no longer any jobs with a processing state.  This needs to be done after the scheduled jobs have stopped processing, as an active scheduled job may launch a job or another Scheduled Job.
7.          Monitor implementation-specific workflow activity until no workflows (except for the workflow that is shutting down the system) have active work items at automatic activities (that are not stuck with errors).
8.          Optionally, notify the system is safe to shut down or is being shut down, and optionally shut down all services.

Startup:

1.           Adjust the Next execution timestamp of all scheduled jobs to the next interval that is in the future, and then re-activate those scheduled jobs and remove the " HOLD" from their names.
2.          Clear the flag for implementation-specific workflows, so they resume processing work items at their scheduled intervals

I am putting together the necessary workflows to verify the feasibility of this approach.  If you see anything I might have overlooked, I would appreciate the feedback.

On the weekends (from Friday evening to Sunday evening) our non-PROD environments are shut down, we have an automated process that handles turning the server off and back on, and we are leveraging a Scheduled Export/Import process that turns off the scheduled Imports/Exports/Promotions before the servers are shut off.

We are encountering an issue where Scheduled jobs become backed up post-restart and require clearing of stuck work items in EPX and potentially an additional restart to clear long processing Scheduled jobs; are there any best practices, recommendations or success stories for automating this process?

Matthew,

The screenshot shows the jobs were queued over a period of 1.5 hours, with times spaced 10-20 minutes apart for most of them.  In this particular case, were these jobs queued before the shutdown process was initiated, or after services were back up?  If they were from before, would your issue be addressed if the graceful shutdown process waited for them to complete before actually shutting down?

You mentioned that you are adjusting the Next <job> Datetime into the future – are you doing this at shutdown, or on startup?  If it is being done at shutdown, is the startup constantly known at that time, or are there occasions when the restart happens hours or even days later than anticipated?  Have the services ever been restarted sooner than anticipated?

You mentioned that after a restart, you often need to clear stuck work items in EPX.  If the Job Monitor screenshot is a typical result after restarting, what it shows doesn't suggest that any work items are stuck that need to be cleared.  Are there specific workflows or activities where work items get stuck and need to be cleared?

When shutting down the actual services, what is the order of service shutdown?

Oliver, staggering the restart of scheduled jobs is worth considering.  However, there may be cases where a general delay between services causes some jobs to run concurrently, which can be problematic and necessitate variable delays between jobs.  The scheduled intervals and times of day for the jobs already have this "baked in", so having those jobs fire at their next interval ensures no jobs run concurrently when they shouldn't.  This eliminates the need to set up an environment-specific staggering sequence.  In other words, the same solution would work in every environment.  The only area of concern I can see is for scheduled jobs that do not run frequently (e.g., once a week) and for which the graceful shutdown spans a scheduled run.  By resetting the Next <job> Datetime to the next future interval, an entire week would be skipped.  This scenario can be handled either by manually launching infrequent and missed jobs or by building interval consideration into the calculation logic, so that if the interval exceeds a specified number of days, the next execution time is the earliest future timestamp using the job's time component.  For example, if a job is scheduled to run every Friday at 11:30 pm and the software is shut down at 5 pm on Friday and restarted on Monday at 8:00 am, the job that didn't run on Friday would run on Monday at 11:30 pm.

Hi Matthew,

In addition to Brian's answer, you could also take a staggered approach to turning the scheduled processes back on. This means you wouldn't even need to modify the Next Export Datetime. Instead of re-enabling all previously scheduled processes at the same time, you could turn them back on one at a time every x minutes (5–15 minutes is probably sufficient, though it depends on the machine's capacity, the heaviness of the jobs, and the number of scheduled jobs).

Thanks,
Oliver Stamkov
Pivotree, Inc.

Hello Brian,

The primary issue in our case is solely with Scheduled Exports/Imports/Promotions. We have a Scheduled process that handles turning off every scheduled Exports/Imports/Promotion prior to shutdown which also sets the Next Export Datetime into the future. And then a similar scheduled process that turns everything back on, but we are not checking for the current job monitor activity or scheduled framework activity before shut down.

As you can see from the screenshot I included, the conditions we encounter post restart is where the Job Monitor becomes inundated with jobs stuck in a Queued state.

Usually clearing any hung jobs or scheduled processes is sufficient to get things moving again, but occasionally a full restart is then also required.

Matthew,

Can you elaborate on which workflows result in stuck work items that require an additional restart?  Is it just because many scheduled jobs are launched as soon as the services are restarted after the downtime period?  Or is there a backlog of work items that were created but not processed because of the way jobs are being disabled?

If the EnterWorks services are off for some period of time and you have scheduled jobs that run frequently (daily or multiple times per day), restarting the services will immediately launch all scheduled jobs that would have run at least once if the services had been left running.  This assumes you are disabling and enabling scheduled jobs solely by changing the Scheduled attribute for each job. 

A comprehensive shutdown/restart procedure would need to perform the following actions:

Shutdown:

1.          Optionally shut down the enable-web-server-service to prevent users from launching jobs.  It may be enough to notify users that they must be off the system, but stopping the service ensures that they are.
2.          Disable all Scheduled Import, Export, and Promotion jobs so no new jobs are launched.  This includes changing the <job> Scheduled flag to No and adding "_HOLD" to the end of each job name to prevent another job from launching it to get the next file, which mainly applies to imports.
3.         Flag any implementation-specific workflow that launches work items with a Scheduler BIC to skip processing for any new work items.  This requires each workflow to be modified to follow the Scheduler BIC starting-point activity, check for the flag, and route the work item to an endpoint if the flag is set.
4. Obtain the MonitorFlows lock. This ensures that errant work items won't be re-processed during shutdown. Note: The amount of time this will hold is defined in the Enterworks.properties file via the epim.timeoutMillis property.  This is only a concern if there are work items stuck with an error due to environmental conditions, and clearing the error will resume processing.
5.          Monitor the processing of scheduled jobs until all have a terminal status (Completed or Error).
6.          Monitor the Jobs processing in the Job Monitor for active jobs until there are no longer any jobs with a processing state.  This needs to be done after the scheduled jobs have stopped processing, as an active scheduled job may launch a job or another Scheduled Job.
7.          Monitor implementation-specific workflow activity until no workflows (except for the workflow that is shutting down the system) have active work items at automatic activities (that are not stuck with errors).
8.          Optionally, notify the system is safe to shut down or is being shut down, and optionally shut down all services.

Startup:

1.           Adjust the Next execution timestamp of all scheduled jobs to the next interval that is in the future, and then re-activate those scheduled jobs and remove the " HOLD" from their names.
2.          Clear the flag for implementation-specific workflows, so they resume processing work items at their scheduled intervals

I am putting together the necessary workflows to verify the feasibility of this approach.  If you see anything I might have overlooked, I would appreciate the feedback.

On the weekends (from Friday evening to Sunday evening) our non-PROD environments are shut down, we have an automated process that handles turning the server off and back on, and we are leveraging a Scheduled Export/Import process that turns off the scheduled Imports/Exports/Promotions before the servers are shut off.

We are encountering an issue where Scheduled jobs become backed up post-restart and require clearing of stuck work items in EPX and potentially an additional restart to clear long processing Scheduled jobs; are there any best practices, recommendations or success stories for automating this process?