Data360 Analyze

 View Only
  • 1.  Paging on HTTP node

    Posted 03-18-2024 13:37
    Edited by Gerry Mullin 03-18-2024 17:45

    We currently using the requests package in a Python node rather than the HTTP node to pull back data from an API call that has an unknown number of pages. I would like to move to the HTTP node and it looks like there are Paging options but I can't seem to get it to work.

    Here is what the URL looks (I removed the actual URL we use):

    https://<URL>?Year=2023&endMonth=12&pageSize=100&pageNumber={pageNumber}

    If the response returns entries, we move onto pageNumber=2 and keep doing that until an empty response comes back, at which case I break the loop.

    This is what I tried in the Paging group of the HTTP node but it doesn't return anything. Any suggestions?

    PagingModel:

    Custom

    PagingSetupRequest:

    pageNumber = 1
    pageSize= 100
    PagingProcessResponses:
    if response.body != None:
        pageNumber += 1



    Update:

    Perhaps this might help somebody else. After having a second look at the documentation I got it to work.

    PagingSetupRequest:

    pageNumber = 1
    request.url.args['pageNumber'] = pageNumber
    PagingProcessResponses (Note that count is returned in my call):
    count = response.json['count']
    if count < 100:
        request = None
    else:
        pageNumber += 1
        request.url.args['pageNumber'] = pageNumber


  • 2.  RE: Paging on HTTP node

    Employee
    Posted 30 days ago

    Hi Gerry,

    I'm glad you managed to get it to work. 

    For the benefit of other people, Data360 Analyze ships with an example data flow that overviews the use of paging in the HTTP node.

    The data flow is included in the 'Data360 Samples' > 'Node Examples' directory and is named 'HTTP Integration with other services'

    We recommend copying the data flow into your own My Documents directory (using Save as ...) before running or modifying the example data flow.



    ------------------------------
    Adrian Williams
    Precisely Software Inc.
    ------------------------------