Articles in this section

Work with Paginated APIs Using the HTTP Connector

  • Updated
Follow

Overview

Many modern APIs return results in pages rather than a single response. Etlworks offers built-in support for paginated APIs directly in the HTTP connector, making it easier to work with large datasets without custom scripting.

This method is recommended for most use cases. It supports multiple pagination strategies including page-based, offset-based, cursor-based, next-link, and time-based pagination.

If for any reason your API does not work with the built-in pagination, you can fall back to a scripting-based technique.

See: Work with paginated APIs using scripting.

Supported Pagination Types

  • none: No pagination
  • page: Increments a page number in query string
  • offset: Increments an offset value in query string
  • cursor: Uses a cursor or token from the response to retrieve the next page
  • nextLink: Uses a full URL (from the response) for the next request
  • time: Increments time windows to fetch data across a defined range

General Configuration

⚠️ You no longer need to manually script pagination or modify URLs with tokens. The HTTP connector automatically handles pagination based on the selected settings.

To configure pagination in the HTTP connector, edit the connection and scroll to the General Pagination Settings section. The following parameters control pagination behavior:

  • Pagination Type – Select the pagination strategy used by the API: offset, page, cursor, nextLink, time, or none.
  • Max Pages – Maximum number of pages to fetch. Keep empty for unlimited.
  • Start Page or Offset – Starting page (for page-based) or offset (for offset-based).
  • Page Size – Records per page (if applicable).
  • Stop Condition JSON Path – (Optional) JSONPath that indicates when pagination should stop (e.g., empty array, done=false).
  • Max Pages Path in Response – JSONPath pointing to the total number of available pages.
  • End Pagination if Error Message Contains – Optional string or keyword in API error messages that signals the end of pagination.
  • Header Name for Next Page URL – (nextLink-based) Header name where the URL for the next page is stored.
  • Inject Pagination Parameters into Request Payload – Enable this if the API expects pagination controls in the payload (used by some AWS and Zoho APIs).

Additional settings are available depending on the selected pagination type.

Page-Based Pagination

This pagination technique increments a page number in query string.

Page-Based Pagination Settings

  • Page Parameter Name – Name of the query parameter for the page number (e.g., page).
  • Page Size Parameter Name – Name of the query parameter for page size (e.g., pageSize).

Page-Based Pagination Example: Rick and Morty API

Request: https://rickandmortyapi.com/api/character

Response:

{
   "info": {
      "pages": 42,
      "next": "https://rickandmortyapi.com/api/character?page=2"
   },
   "results": [ ... ]
}

Connection Settings:

Offset-Based Pagination

This pagination technique increments an offset value in query string.

Offset-Based Pagination Settings

Offset Parameter Name – Name of the offset parameter (e.g., offset, start, from).

Inject offset and page size into query string - If enabled, the offset and page size values will be injected directly into the query parameter (e.g., SQL-like queries used in platforms like QuickBooks). This applies only to pagination types Offset and Page. Use this option when the API expects pagination keywords such as STARTPOSITION and MAXRESULTS inside the query itself, rather than as separate URL parameters.

Offset-Based Pagination Example: Jira API

Request: https://api.atlassian.com/ex/jira/<cloudId>/rest/api/3/search

Response:

{
   "expand": "schema,names",
   "startAt": 0,
   "maxResults": 50,
   "total": 6,
   "issues": [...]
}

Connection Settings:

Cursor-Based Pagination

This pagination technique increments an offset value in query string.

Cursor-Based Pagination Settings

  • Cursor JSON Path – JSONPath to the cursor value in the response.
  • Cursor Parameter Name – Name of the query parameter to send the cursor token (e.g., after, nextPageToken).

Cursor-Based Pagination Example: HubSpot API

Connection Settings:

NextLink-Based Pagination

This pagination technique uses a full URL (from the response) for the next request.

NextLink-Based Pagination Settings

  • Next Link JSON Path – JSONPath to the full URL or relative path for the next page.

NextLink-Based Pagination Example: Microsoft Graph API

Request: https://graph.microsoft.com/v1.0/users

Response:

{
  "data": [ ... ],
  "@odata.nextLink": "https://graph.microsoft.com/v1.0/users?$skiptoken=XYZ"
}

Connection Settings:

Time-Based Pagination

Use when an API requires a date range or timestamp to define windows of data.

Time-Based Pagination Settings

  • Start Time (String) – The initial timestamp used in the first request. Must match the format defined in Time Format (Java). This value is incremented after each page to retrieve the next batch of records.

  • End Time (String) – Optional upper limit timestamp. When the incremented time exceeds this value, pagination stops. If left empty, pagination continues indefinitely (or until the API signals no more data).

  • Time Parameter Name – The name of the query or payload parameter to which the formatted time value is assigned. It is inserted automatically in each request. Example: start_time, from, timestamp.

  • Time Format (Java) – Format string used to format the timestamp. Follows Java’s DateTimeFormatter syntax. Example: yyyy-MM-dd'T'HH:mm:ss produces 2024-06-01T00:00:00.

Time-Based Pagination Example: Mixpanel API

Request: 

https://mixpanel.com/api/2.0/segmentation
     ?event=page_view
     &from_date=2024-06-01
     &to_date=2026-06-07
     &limit=100
     &unit=hour
     &interval=1
     &where=properties["country"]=="USA"
     &format=json

Response:

{
  "legend_size": 1,
  "meta": {
    "dates": ["2024-06-01T00:00:00", "2026-06-07T01:00:00"],
    "event_name": "page_view"

  },
  "data": {
       ...
    }
  }
}

Connection Settings:

  • Pagination Type: time

  • Start Time (String): 2024-06-01T00:00:00

  • End Time (String): 2024-06-01T23:59:59

  • Time Parameter Name: from_date (also mapped to to_date internally if needed)

  • Time Format (Java): yyyy-MM-dd'T'HH:mm:ss

 

Related articles

Comments

0 comments

Please sign in to leave a comment.