Automate

 View Only
  • 1.  Several Daily Background Query Job started running TWICE at the same time. Others unaffected

    Posted 09-23-2025 12:53

    We have several background query jobs, submitted from Automate Studio 25.0, to run daily all at different times on Evolve 25.0.  They query SAP and populate a SQLServer database.  Most start and finish within 30 seconds to 2 minutes.  They are quick and work well.  However, we have noticed that several jobs have started running twice at their timeslot.  Once they started twice, they run concurrently and double-populate the database table, and both schedule themselves for the next daily run at the same time.  It has started with one job, then a few days later another one doubles (forever), then a few days another one.  Happens in development and production.   The jobs are set up the same way as they were on Studio/Foundation 12.1.   That server never behaved like this.   We have a ticket logged, but wanted to see if anyone has some insight.   We have deleted the jobs from production and resubmitted (once/job) the troubled jobs.  But now it is happening again.   Attached is a screenshot from one of our servers.    In the below image, you can see the duplicates started on 09/18/2025.   

    image


    ------------------------------
    Robert Zsedely
    Business Architect
    Seneca Foods Corporation
    Fairport NY
    ------------------------------


  • 2.  RE: Several Daily Background Query Job started running TWICE at the same time. Others unaffected

    Employee
    Posted 09-24-2025 13:08

    Hi Robert

    I have not run into this issue, so it's good you have a ticket opened.

    Is it possible it was scheduled multiple times in different apps?  I know that I scheduled one extract to SQL on my app and then another on a shared app. It took me a while to track that down.

    The only other thing I could think of is that it didn't complete properly and would reschedule itself, but it looks like the status of the jobs are green.  Have you selected the jobs and looked at the details?  Select row and click view details:

    I've found issues doing this, such as bad SQL credentials, etc.. in the job data section at the bottom:

        "AutoRunResponse": {
            "DataFileId": "0bf7b032-c110-4a28-aaf1-f7327d126ae8",
            "StoreRefId": "b3b3452f-1f4b-4aee-a5c5-39ca16bcc9a5",
            "ErrorCode": "2028The Windows credentials for SQL Server are invalid.",
            "ErrorMessage": "The Windows credentials for SQL Server are invalid.",
            "AutoRunStatus": 0
        }

    Hope that helps a little,

    Sigrid



    ------------------------------
    Sigrid Kok
    *Precisely Software Inc.
    ------------------------------



  • 3.  RE: Several Daily Background Query Job started running TWICE at the same time. Others unaffected

    Posted 09-24-2025 14:17
    Edited by Robert Zsedely 09-24-2025 14:23

    Hi Sigrid.   Thank you for the response.   We have not scheduled multiple instances of this query at this time.  I have checked the calendar listing of all scheduled jobs. The evolve server is doing it. The only "fix" is to delete the job, then resubmit it to background from Automate, and it will have one instance again.   That is, until it hiccups, and creates a duplicate eventually.   Out of 30 scheduled query jobs at different times, 3 have already "duplicated" in production in the past 2 weeks.    Looking at the first instance of "double jobs", it appears that a query background process on the server loses it's job-ID for updating the status and starts another.   For example, for the job in the original post, the EVOLVE log from 2025-09-18 at 12:25AM  (00:25:16) shows that it has "no inprogress or waiting job to update status with ID..."    This time matches EXACTLY TO THE SECOND of when the second job was created/started.

    Evolve job log when second job is created the first time

    Edit - I looked at the job data, and both had successful runs in the jobs->business jobs.  

     "AutoRunResponse": {
            "DataFileId": "a8cb5a31-e1bb-47aa-af6c-f604d7781b0c",
            "StoreRefId": "382de0f4-7e60-4571-bd33-fa910510c0b9",
            "DataFileName": "MDM_ZFGMATGRP 202509180025160008_1526968664.txt",
            "DataFileVersion": "2",
            "RunDetails": [
                {
                    "FileName": "IT Queries/MDM_ZFGMATGRP.Qsq",
                    "FileType": "Query",
                    "SuccessRows": 6,
                    "ErrorRows": 0,
                    "SapConnectionName": "ZQ3 - SAP RQ3 - Non SSO"
                }
            ],

    ...

    <LicenseUser>SFCMAIL\\WinshuttleSystemPost</LicenseUser>\r\n    (license id removed from this post by me) \r\n  </Common>\r\n</LogData>"
            ],
            "TotalRows": 6,
            "ErrorRows": 0,
            "AutoRunStatus": 1
        }
    }
    ------------------------------
    Robert Zsedely
    Business Architect
    Seneca Foods Corporation
    Fairport NY
    ------------------------------



  • 4.  RE: Several Daily Background Query Job started running TWICE at the same time. Others unaffected

    Employee
    Posted 09-24-2025 14:23

    Hi Robert

    Thanks for the reply. Please continue to work with support on this issue. It may be specific to your environment.

    If anyone else is on 25.0 and experiencing the same, please let us know.

    Best Regards

    Sigrid



    ------------------------------
    Sigrid Kok
    *Precisely Software Inc.
    ------------------------------



  • 5.  RE: Several Daily Background Query Job started running TWICE at the same time. Others unaffected

    Employee
    Posted 09-24-2025 14:36

    PS: One other tip if you haven't done so already - set the query output write records to clear all before running so that you don't get duplications in your SQL table..



    ------------------------------
    Sigrid Kok
    *Precisely Software Inc.
    ------------------------------



  • 6.  RE: Several Daily Background Query Job started running TWICE at the same time. Others unaffected

    Posted 09-24-2025 15:16

    Yes, we have Clear all mapped data set (screenshot below).   However, since the duplicate job kickoff, it schedules duplicate jobs to run at EXACTLY the same time.   This, ends up in duplicate data unfortunately, even with the "clear all mapped data from result file":

    Evolve running it twice:

    yet only scheduled once:

    but the double run (even with clear) still causes double load.



    ------------------------------
    Robert Zsedely
    Business Architect
    Seneca Foods Corporation
    Fairport NY
    ------------------------------



  • 7.  RE: Several Daily Background Query Job started running TWICE at the same time. Others unaffected

    Employee
    Posted 10-03-2025 11:32

    Hi Robert

    Did you open a ticket on this one?

    I'm wondering if there's something going on with the setup of the workers - if you have more than one?

    Do you have a DR setup or load balancer that may be asking two workers to do the same thing? 

    Just a thought. But I would definitely open a ticket to check it out.

    Best Regards,

    Sigrid



    ------------------------------
    Sigrid Kok
    *Precisely Software Inc.
    ------------------------------



  • 8.  RE: Several Daily Background Query Job started running TWICE at the same time. Others unaffected

    Posted 10-09-2025 15:43

    Yes, opened a ticket and the engineer quickly developed and sent us a patch for the  "Winshuttle.Finx.Scheduler.dll" and "Winshuttle.Finx.UG.dll" for the Evolve installation directory.  We updated and did an IISRESET.     Cancelled out the jobs that had "children".   Rescheduled them as daily in another timeslot.   Monitoring.   Non Production has been good for 2 weeks now.   Just updated production.  It is looking good so far!  



    ------------------------------
    Robert Zsedely
    Business Architect
    Seneca Foods Corporation
    Fairport NY
    ------------------------------