Spectrum Technology Platform

 View Only

  • 1.  API with Write to hub : how to improve performance ?

    Posted 10-22-2019 05:05

    Hello Spectrum Community

    To give some context regarding my needs : we have developped a REST Web service with Enterprise Designer
    This Web service is in production and is interfaced with our E-commerce site. The mechanism is simple : when a customer registers or updates his personal informations on the web site, this Web service is called in real time and basically we use a Write to hub stage to create or update the customer entities in Spectrum
    The web service also sends back to the E-commerce site a response code and some few ID informations

    My question is : we have noticed that the write to Hub is the bottleneck in term of perfomance => 90% of the whole treatment
    One solution we have in mind is to "parallelize" the process Write to Hub and the process "response to E-commerce site" so the web service will not have to wait for the full completion of the web service before sending a response to web site and customer is not blocked on the web site

    Is there a way to do this delegation or parellization (I don't know the good wording) in Enterprise Designer ?
    We are using Spectrum v12.2

    thanks
    Giang



    ------------------------------
    Giang Pham-Dac
    Data Manager
    L'Occitane International SA
    Plan-Les-Ouates
    ------------------------------


  • 2.  RE: API with Write to hub : how to improve performance ?

    Employee
    Posted 10-22-2019 11:56
    Edited by Luke Daugherty 10-22-2019 11:56
    Hi Giang,

    Currently there is no way to do this in a single spectrum service. All processing must complete before it returns and for Write to Hub that means committing the data across the cluster which is where the time is being spent. Running in standalone mode or in a cluster using Local Locks (directing data to a write master) would improve the performance of concurrent writes.

    ------------------------------
    Luke Daugherty
    Knowledge Community Shared Account
    Shelton CT
    ------------------------------

    Original Message


  • 3.  RE: API with Write to hub : how to improve performance ?

    Posted 10-23-2019 11:31
    Hi Luke

    thanks for this quick reply
    We are already running in a standalone mode and I am little bit surprise by the write to hub performance : 7s-10s to write 10 entities  and 20 relationships
    We are looking some solutions with the disk performance : turn from HDD to SSD disks

    thanks again
    Giang

    ------------------------------
    Giang Pham-Dac
    Data Manager
    L'Occitane International SA
    Plan-Les-Ouates
    ------------------------------

    Original Message


  • 4.  RE: API with Write to hub : how to improve performance ?

    Employee
    Posted 10-23-2019 12:59
    Going from a HDD to a SSD should give a sizable improvement in performance and is the suggested configuration. That along with a properly sized pagecache should allow for fast writes in the Data Hub, especially in standalone mode.

    ------------------------------
    Luke Daugherty
    Knowledge Community Shared Account
    Shelton CT
    ------------------------------

    Original Message