Articles in this section

See more

Work with Paginated APIs Using the HTTP Connector

  • Updated
Follow

Overview

Many modern APIs return results in pages rather than a single response. Etlworks offers built-in support for paginated APIs directly in the HTTP connector, making it easier to work with large datasets without custom scripting.

This method is recommended for most use cases. It supports multiple pagination strategies including page-based, offset-based, cursor-based, next-link, and time-based pagination.

If for any reason your API does not work with the built-in pagination, you can fall back to a scripting-based technique.

See: Work with paginated APIs using scripting.

Supported Pagination Types

  • none: No pagination
  • page: Increments a page number in query string
  • offset: Increments an offset value in query string
  • cursor: Uses a cursor or token from the response to retrieve the next page
  • nextLink: Uses a full URL (from the response) for the next request
  • time: Increments time windows to fetch data across a defined range

General Configuration

⚠️ You no longer need to manually script pagination or modify URLs with tokens. The HTTP connector automatically handles pagination based on the selected settings.

To configure pagination in the HTTP connector, edit the connection and scroll to the General Pagination Settings section. The following parameters control pagination behavior:

  • Pagination Type – Select the pagination strategy used by the API: offset, page, cursor, nextLink, time, or none.
  • Max Pages – Maximum number of pages to fetch. Keep empty for unlimited.
  • Start Page or Offset – Starting page (for page-based) or offset (for offset-based).
  • Page Size – Records per page (if applicable).
  • Stop Condition JSON Path – (Optional) JSONPath that indicates when pagination should stop (e.g., empty array, done=false).
  • Max Pages Path in Response – JSONPath pointing to the total number of available pages.
  • End Pagination if Error Message Contains – Optional string or keyword in API error messages that signals the end of pagination.
  • Header Name for Next Page URL – (nextLink-based) Header name where the URL for the next page is stored.
  • Inject Pagination Parameters into Request Payload – Enable this if the API expects pagination controls in the payload (used by some AWS and Zoho APIs).

Additional settings are available depending on the selected pagination type.

Page-Based Pagination

This pagination technique increments a page number in query string.

Page-Based Pagination Settings

  • Page Parameter Name – Name of the query parameter for the page number (e.g., page).
  • Page Size Parameter Name – Name of the query parameter for page size (e.g., pageSize).

Page-Based Pagination Example: Rick and Morty API

Request: https://rickandmortyapi.com/api/character

Response:

{
   "info": {
      "pages": 42,
      "next": "https://rickandmortyapi.com/api/character?page=2"
   },
   "results": [ ... ]
}

Connection Settings:

Offset-Based Pagination

This pagination technique increments an offset value in query string.

Offset-Based Pagination Settings

Offset Parameter Name – Name of the offset parameter (e.g., offset, start, from).

Inject offset and page size into query string - If enabled, the offset and page size values will be injected directly into the query parameter (e.g., SQL-like queries used in platforms like QuickBooks). This applies only to pagination types Offset and Page. Use this option when the API expects pagination keywords such as STARTPOSITION and MAXRESULTS inside the query itself, rather than as separate URL parameters.

Offset-Based Pagination Example: Jira API

Request: https://api.atlassian.com/ex/jira/<cloudId>/rest/api/3/search

Response:

{
   "expand": "schema,names",
   "startAt": 0,
   "maxResults": 50,
   "total": 6,
   "issues": [...]
}

Connection Settings:

Cursor-Based Pagination

This pagination technique increments an offset value in query string.

Cursor-Based Pagination Settings

  • Cursor JSON Path – JSONPath to the cursor value in the response.
  • Cursor Parameter Name – Name of the query parameter to send the cursor token (e.g., after, nextPageToken).

Cursor-Based Pagination Example: HubSpot API

Connection Settings:

NextLink-Based Pagination

This pagination technique uses a full URL (from the response) for the next request.

NextLink-Based Pagination Settings

  • Next Link JSON Path – JSONPath to the full URL or relative path for the next page.

NextLink-Based Pagination Example: Microsoft Graph API

Request: https://graph.microsoft.com/v1.0/users

Response:

{
  "data": [ ... ],
  "@odata.nextLink": "https://graph.microsoft.com/v1.0/users?$skiptoken=XYZ"
}

Connection Settings:

Derived URL Edge Cases (Using a Preprocessor)

In some APIs, most commonly with nextLink-based pagination, the value used for the next request is not a directly usable URL. Instead, the API may return a value that must be derived from the response, such as a link expression, a partial path, or other structured metadata. In these cases, the HTTP connector calculates the next request URL based on standard rules, but additional transformation may still be required. The Pagination URL Preprocessor has direct access to two string variables: url, which contains the URL calculated by the connector, and nextUrl, which contains the raw value returned by the API. The script can parse and modify either value and return a valid URL. If the script returns a string, that value is used as the URL for the next paginated request.

Example

If the calculated url value is returned in a non-standard format, such as:

<href://link; rel="link"/>

you can use the preprocessor to extract and return the actual URL (https://link) that will be used for the next request.

// Pagination URL Preprocessor

// Extract a usable URL from values like: <href://link; rel="link"/>

// You can parse either `url` (calculated) or `nextUrl` (raw). Here we prefer `nextUrl` if present.

var candidate = (nextUrl != null && String(nextUrl).trim().length() > 0) ? String(nextUrl) : String(url);

candidate = candidate == null ? "" : candidate.trim();

// If API returns something like: <href://link; rel="link"/>
// Capture the part after "<href:" up to ";" or ">".

var m = candidate.match(/<\s*href\s*:\s*([^;>]+)\s*(?:[;>])/i);

if (m && m[1]) {

  var extracted = m[1].trim();

  // If it looks like //host/path, add scheme

  if (extracted.indexOf("//") === 0) {
    extracted = "https:" + extracted;
  }

  // If it doesn't have a scheme, assume https (adjust if you need http)

  if (!/^[a-zA-Z][a-zA-Z0-9+.-]*:\/\//.test(extracted)) {
    extracted = "https://" + extracted;
  }

  extracted; // returning a string overrides the next request URL

} else {
  // Fallback: if it's already a normal URL, return it unchanged.

  candidate;
}

Time-Based Pagination

Use when an API requires a date range or timestamp to define windows of data.

Time-Based Pagination Settings

  • Start Time (String) – The initial timestamp used in the first request. Must match the format defined in Time Format (Java). This value is incremented after each page to retrieve the next batch of records.

  • End Time (String) – Optional upper limit timestamp. When the incremented time exceeds this value, pagination stops. If left empty, pagination continues indefinitely (or until the API signals no more data).

  • Time Parameter Name – The name of the query or payload parameter to which the formatted time value is assigned. It is inserted automatically in each request. Example: start_time, from, timestamp.

  • Time Format (Java) – Format string used to format the timestamp. Follows Java’s DateTimeFormatter syntax. Example: yyyy-MM-dd'T'HH:mm:ss produces 2024-06-01T00:00:00.

Time-Based Pagination Example: Mixpanel API

Request: 

https://mixpanel.com/api/2.0/segmentation
     ?event=page_view
     &from_date=2024-06-01
     &to_date=2026-06-07
     &limit=100
     &unit=hour
     &interval=1
     &where=properties["country"]=="USA"
     &format=json

Response:

{
  "legend_size": 1,
  "meta": {
    "dates": ["2024-06-01T00:00:00", "2026-06-07T01:00:00"],
    "event_name": "page_view"

  },
  "data": {
       ...
    }
  }
}

Connection Settings:

  • Pagination Type: time

  • Start Time (String): 2024-06-01T00:00:00

  • End Time (String): 2024-06-01T23:59:59

  • Time Parameter Name: from_date (also mapped to to_date internally if needed)

  • Time Format (Java): yyyy-MM-dd'T'HH:mm:ss

Related articles