Precisely Enterworks

Hello everyone,

I wonder if anyone has any insight on something we are seeing. We have a series of promotion jobs scheduled throughout the day with 180 min timeouts. If the server reboots while a job is processing, it stays stuck with status = processing. Future runs of the job complete, but process 0 records, thus changes are not reflected. We need to manually kill the job and retrigger it for everything to run correctly. 

Is there anything we can do to more proactively catch this or avoid it? Right now without any error being generated or notification I'm running a saved search on the Promotion Jobs to manually see if anything is stuck processing.

Thanks for any guidance.

------------------------------
Nick Lavely |
Thomson Reuters
------------------------------

Hi Nick,

We had a similar design and ran into the same issue. To proactively identify issues, we created a Job which can look for stuck jobs(Processing) for more than x minutes and have it email it to you.

Recently, we moved away from scheduled promotions and introduced a custom workflow which triggers the promotion jobs when records change in a repository.

Regards,
Ram

------------------------------
Ram Kumar Rajagopal | technical program manager
Starbucks Coffee Company |
------------------------------

Original Message:
Sent: 04-22-2021 15:18
From: Nick Lavely
Subject: Promotion jobs stuck in processing

Hello everyone,

I wonder if anyone has any insight on something we are seeing. We have a series of promotion jobs scheduled throughout the day with 180 min timeouts. If the server reboots while a job is processing, it stays stuck with status = processing. Future runs of the job complete, but process 0 records, thus changes are not reflected. We need to manually kill the job and retrigger it for everything to run correctly.

Is there anything we can do to more proactively catch this or avoid it? Right now without any error being generated or notification I'm running a saved search on the Promotion Jobs to manually see if anything is stuck processing.

Thanks for any guidance.

------------------------------
Nick Lavely |
Thomson Reuters
------------------------------

Nick,

There are several different things that need to be examined for the scenario that you describe.  First, what is the schedule interval for the promotion jobs?  When you say the server reboots - do you mean stopping and restarting the EnterWorks services, or do you mean actually rebooting the server(s) on which the application is running?  Either way, once the services are restarted, what is the status of the promotion jobs in the Job Monitor?  Do they show the job stopped with an Error?  Then what does the Promotion Jobs repository show?  Does it still show the job as Processing?  If it does, can you check the Scheduled Promotion Process workflow in Design Console and see if the work item is still processing? 

If running a new promotion job completes, but nothing gets promoted, can you check the record state (on the State tab) of some of the records that should  have been promoted to see what it shows?  If it shows "Sync", then the system will not attempt to promote the record.

Another thing to try is to manually promote an individual record, or the repository by opening the Promotion Configuration screen in the Classic UI (to take the Scheduled Promotions processing out of the equation). 

But as Ram had mentioned, the trend (best practice) is to set up an EPX workflow to monitor for changes and to launch promotion jobs only when there are changes that need to be promoted and after some configurable time interval has passed.  This is described in detail in the Brown Bag Session #58 - Batch-Promoting user-submitted repository records.  This recording shows how to handle user-initiated promotions but the same approach can be used to handle an automated check for changes.  With such an approach, the promotions are only run when there is something to promote and changes are run in batches to avoid trying to run promotions too frequently.

-Brian


-Brian

​

------------------------------
Brian Zupke | Senior Technical Support Engineer
Winshuttle North America |
------------------------------

Original Message:
Sent: 04-22-2021 15:18
From: Nick Lavely
Subject: Promotion jobs stuck in processing

Hello everyone,

I wonder if anyone has any insight on something we are seeing. We have a series of promotion jobs scheduled throughout the day with 180 min timeouts. If the server reboots while a job is processing, it stays stuck with status = processing. Future runs of the job complete, but process 0 records, thus changes are not reflected. We need to manually kill the job and retrigger it for everything to run correctly.

Is there anything we can do to more proactively catch this or avoid it? Right now without any error being generated or notification I'm running a saved search on the Promotion Jobs to manually see if anything is stuck processing.

Thanks for any guidance.

------------------------------
Nick Lavely |
Thomson Reuters
------------------------------

Thanks Brian! I'll check out that Brown Bag Session and figure out what we can do on our side.

------------------------------
Nick Lavely |
Thomson Reuters
------------------------------

Original Message:
Sent: 04-24-2021 16:32
From: Brian Zupke
Subject: Promotion jobs stuck in processing

Nick,

There are several different things that need to be examined for the scenario that you describe.  First, what is the schedule interval for the promotion jobs?  When you say the server reboots - do you mean stopping and restarting the EnterWorks services, or do you mean actually rebooting the server(s) on which the application is running?  Either way, once the services are restarted, what is the status of the promotion jobs in the Job Monitor?  Do they show the job stopped with an Error?  Then what does the Promotion Jobs repository show?  Does it still show the job as Processing?  If it does, can you check the Scheduled Promotion Process workflow in Design Console and see if the work item is still processing?

If running a new promotion job completes, but nothing gets promoted, can you check the record state (on the State tab) of some of the records that should  have been promoted to see what it shows?  If it shows "Sync", then the system will not attempt to promote the record.

Another thing to try is to manually promote an individual record, or the repository by opening the Promotion Configuration screen in the Classic UI (to take the Scheduled Promotions processing out of the equation).

But as Ram had mentioned, the trend (best practice) is to set up an EPX workflow to monitor for changes and to launch promotion jobs only when there are changes that need to be promoted and after some configurable time interval has passed.  This is described in detail in the Brown Bag Session #58 - Batch-Promoting user-submitted repository records.  This recording shows how to handle user-initiated promotions but the same approach can be used to handle an automated check for changes.  With such an approach, the promotions are only run when there is something to promote and changes are run in batches to avoid trying to run promotions too frequently.

-Brian


-Brian

​

------------------------------
Brian Zupke | Senior Technical Support Engineer
Winshuttle North America |

Original Message:
Sent: 04-22-2021 15:18
From: Nick Lavely
Subject: Promotion jobs stuck in processing

Hello everyone,

I wonder if anyone has any insight on something we are seeing. We have a series of promotion jobs scheduled throughout the day with 180 min timeouts. If the server reboots while a job is processing, it stays stuck with status = processing. Future runs of the job complete, but process 0 records, thus changes are not reflected. We need to manually kill the job and retrigger it for everything to run correctly.

Is there anything we can do to more proactively catch this or avoid it? Right now without any error being generated or notification I'm running a saved search on the Promotion Jobs to manually see if anything is stuck processing.

Thanks for any guidance.

------------------------------
Nick Lavely |
Thomson Reuters
------------------------------