Articles in this section

See more

Work with Paginated APIs Using the HTTP Connector

  • Updated
Follow

Overview

Many modern APIs return results in pages rather than a single response. Etlworks offers built-in support for paginated APIs directly in the HTTP connector, making it easier to work with large datasets without custom scripting.

This method is recommended for most use cases. It supports multiple pagination strategies including page-based, offset-based, cursor-based, next-link, and time-based pagination.

If for any reason your API does not work with the built-in pagination, you can fall back to a scripting-based technique.

See: Work with paginated APIs using scripting.

Supported Pagination Types

  • none: No pagination
  • page: Increments a page number in query string
  • offset: Increments an offset value in query string
  • cursor: Uses a cursor or token from the response to retrieve the next page
  • nextLink: Uses a full URL (from the response) for the next request
  • time: Increments time windows to fetch data across a defined range

General Configuration

⚠️ You no longer need to manually script pagination or modify URLs with tokens. The HTTP connector automatically handles pagination based on the selected settings.

To configure pagination in the HTTP connector, edit the connection and scroll to the General Pagination Settings section. The following parameters control pagination behavior:

  • Pagination Type – Select the pagination strategy used by the API: offset, page, cursor, nextLink, time, or none.
  • Max Pages – Maximum number of pages to fetch. Keep empty for unlimited.
  • Start Page or Offset – Starting page (for page-based) or offset (for offset-based).
  • Page Size – Records per page (if applicable).
  • Stop Condition JSON Path – (Optional) JSONPath that indicates when pagination should stop (e.g., empty array, done=false). When used with cursor pagination, this is treated as an additional stop signal on top of the standard "no next cursor" check — see Use Stop Condition JSON Path with cursor pagination.
  • Resume State Key – (Optional) A stable, unique key per endpoint / use case (e.g., orders-prod-cursor, hubspot-contacts-daily). When set, the connector persists the next pagination value at the end of a run and resumes from it on the next run. Leave empty for stateless pagination (default behavior). See Resume pagination between flow runs for details.
  • Max Pages Path in Response – JSONPath pointing to the total number of available pages.
  • End Pagination if Error Message Contains – Optional string or keyword in API error messages that signals the end of pagination.
  • Header Name for Next Page URL – (nextLink-based) Header name where the URL for the next page is stored.
  • Inject Pagination Parameters into Request Payload – Enable this if the API expects pagination controls in the payload (used by some AWS and Zoho APIs).

Additional settings are available depending on the selected pagination type.

Page-Based Pagination

This pagination technique increments a page number in query string.

Page-Based Pagination Settings

  • Page Parameter Name – Name of the query parameter for the page number (e.g., page).
  • Page Size Parameter Name – Name of the query parameter for page size (e.g., pageSize).

Page-Based Pagination Example: Rick and Morty API

Request: https://rickandmortyapi.com/api/character

Response:

{
   "info": {
      "pages": 42,
      "next": "https://rickandmortyapi.com/api/character?page=2"
   },
   "results": [ ... ]
}

Connection Settings:

Offset-Based Pagination

This pagination technique increments an offset value in query string.

Offset-Based Pagination Settings

Offset Parameter Name – Name of the offset parameter (e.g., offset, start, from).

Inject offset and page size into query string - If enabled, the offset and page size values will be injected directly into the query parameter (e.g., SQL-like queries used in platforms like QuickBooks). This applies only to pagination types Offset and Page. Use this option when the API expects pagination keywords such as STARTPOSITION and MAXRESULTS inside the query itself, rather than as separate URL parameters.

Offset-Based Pagination Example: Jira API

Request: https://api.atlassian.com/ex/jira/<cloudId>/rest/api/3/search

Response:

{
   "expand": "schema,names",
   "startAt": 0,
   "maxResults": 50,
   "total": 6,
   "issues": [...]
}

Connection Settings:

Cursor-Based Pagination

This pagination technique increments an offset value in query string.

Cursor-Based Pagination Settings

  • Cursor JSON Path – JSONPath to the cursor value in the response.
  • Cursor Parameter Name – Name of the query parameter to send the cursor token (e.g., after, nextPageToken).

Use Stop Condition JSON Path with cursor pagination

By default, cursor pagination continues as long as the API returns a next-cursor value at the Cursor JSON Path. When the cursor is missing or empty, pagination stops.

If you configure Stop Condition JSON Path (under General Configuration), it becomes an additional stop condition. Pagination stops as soon as either of these is true:

  • The cursor is missing or empty at Cursor JSON Path.
  • The value at Stop Condition JSON Path evaluates as "stop".

If the configured Stop Condition path is missing or null in the response, pagination stops.

Supported stop-path value types

Value type Continues while… Stops when… Example path
Boolean The value is true. The value is false. /pagination/hasNextPage
Number The value is greater than 0. The value is 0. /pagination/remaining
Array The array size is at least the configured page size. Fewer records are returned than the page size. /data

When to use it. This pairing is useful for APIs that return a cursor even after the last useful page (so the cursor-only check would keep paginating into empty pages), or for APIs that expose an explicit "more data available" flag alongside the cursor.

Example response

{
  "data": [
    { "id": 1 },
    { "id": 2 }
  ],
  "pagination": {
    "nextCursor": "abc123",
    "hasNextPage": true
  }
}

Example configuration

  • Pagination Type: cursor
  • Cursor JSON Path: /pagination/nextCursor
  • Cursor Parameter Name: cursor
  • Stop Condition JSON Path: /pagination/hasNextPage

With this configuration, the connector follows the cursor as long as hasNextPage is true. As soon as the API returns hasNextPage: false (or the field is missing / null, or the cursor itself disappears), pagination stops.

Cursor-Based Pagination Example: HubSpot API

Connection Settings:

NextLink-Based Pagination

This pagination technique uses a full URL (from the response) for the next request.

NextLink-Based Pagination Settings

  • Next Link JSON Path – JSONPath to the full URL or relative path for the next page.

NextLink-Based Pagination Example: Microsoft Graph API

Request: https://graph.microsoft.com/v1.0/users

Response:

{
  "data": [ ... ],
  "@odata.nextLink": "https://graph.microsoft.com/v1.0/users?$skiptoken=XYZ"
}

Connection Settings:

Derived URL Edge Cases (Using a Preprocessor)

In some APIs, most commonly with nextLink-based pagination, the value used for the next request is not a directly usable URL. Instead, the API may return a value that must be derived from the response, such as a link expression, a partial path, or other structured metadata. In these cases, the HTTP connector calculates the next request URL based on standard rules, but additional transformation may still be required. The Pagination URL Preprocessor has direct access to two string variables: url, which contains the URL calculated by the connector, and nextUrl, which contains the raw value returned by the API. The script can parse and modify either value and return a valid URL. If the script returns a string, that value is used as the URL for the next paginated request.

Example

If the calculated url value is returned in a non-standard format, such as:

<href://link; rel="link"/>

you can use the preprocessor to extract and return the actual URL (https://link) that will be used for the next request.

// Pagination URL Preprocessor

// Extract a usable URL from values like: <href://link; rel="link"/>

// You can parse either `url` (calculated) or `nextUrl` (raw). Here we prefer `nextUrl` if present.

var candidate = (nextUrl != null && String(nextUrl).trim().length() > 0) ? String(nextUrl) : String(url);

candidate = candidate == null ? "" : candidate.trim();

// If API returns something like: <href://link; rel="link"/>
// Capture the part after "<href:" up to ";" or ">".

var m = candidate.match(/<\s*href\s*:\s*([^;>]+)\s*(?:[;>])/i);

if (m && m[1]) {

  var extracted = m[1].trim();

  // If it looks like //host/path, add scheme

  if (extracted.indexOf("//") === 0) {
    extracted = "https:" + extracted;
  }

  // If it doesn't have a scheme, assume https (adjust if you need http)

  if (!/^[a-zA-Z][a-zA-Z0-9+.-]*:\/\//.test(extracted)) {
    extracted = "https://" + extracted;
  }

  extracted; // returning a string overrides the next request URL

} else {
  // Fallback: if it's already a normal URL, return it unchanged.

  candidate;
}

Time-Based Pagination

Use when an API requires a date range or timestamp to define windows of data.

Time-Based Pagination Settings

  • Start Time (String) – The initial timestamp used in the first request. Must match the format defined in Time Format (Java). This value is incremented after each page to retrieve the next batch of records.

  • End Time (String) – Optional upper limit timestamp. When the incremented time exceeds this value, pagination stops. If left empty, pagination continues indefinitely (or until the API signals no more data).

  • Time Parameter Name – The name of the query or payload parameter to which the formatted time value is assigned. It is inserted automatically in each request. Example: start_time, from, timestamp.

  • Time Format (Java) – Format string used to format the timestamp. Follows Java’s DateTimeFormatter syntax. Example: yyyy-MM-dd'T'HH:mm:ss produces 2024-06-01T00:00:00.

Time-Based Pagination Example: Mixpanel API

Request: 

https://mixpanel.com/api/2.0/segmentation
     ?event=page_view
     &from_date=2024-06-01
     &to_date=2026-06-07
     &limit=100
     &unit=hour
     &interval=1
     &where=properties["country"]=="USA"
     &format=json

Response:

{
  "legend_size": 1,
  "meta": {
    "dates": ["2024-06-01T00:00:00", "2026-06-07T01:00:00"],
    "event_name": "page_view"

  },
  "data": {
       ...
    }
  }
}

Connection Settings:

  • Pagination Type: time

  • Start Time (String): 2024-06-01T00:00:00

  • End Time (String): 2024-06-01T23:59:59

  • Time Parameter Name: from_date (also mapped to to_date internally if needed)

  • Time Format (Java): yyyy-MM-dd'T'HH:mm:ss

Resume pagination between flow runs

By default, the HTTP connector starts pagination fresh on every flow run — the first request uses the configured starting page, offset, time value, or no cursor at all. Resume State Key changes that: when set, the connector persists the next pagination value at the end of each run and uses it as the starting value on the next run. The same flow, scheduled to run every hour, can pick up exactly where it left off.

Resumable pagination applies to all five pagination types: page, offset, cursor, nextLink, and time.

When to use it. Incremental polling of an API where each run should continue from the last successfully processed page, offset, cursor, next-link URL, or time value — for example, daily HubSpot contact syncs, hourly order pulls, or rolling time-window ingestion from APIs that don't have a server-side cursor of their own.

Configure Resume State Key

  1. On the HTTP connection's General Pagination Settings, set Resume State Key to a stable, unique value that identifies this endpoint / use case.
  2. Save the connection and run the flow as normal.
  3. On the next run, the connector reads the saved pagination value and uses it as the starting point.

Choosing a Resume State Key

  • Pick a name that's stable across runs and unique to this endpoint and use case. Examples:
    • orders-prod-cursor
    • hubspot-contacts-daily
    • acme-orders-offset
  • Don't reuse the same key across unrelated flows or endpoints. Two flows that share the same key share the same saved pagination state — one flow will pick up where the other left off, almost certainly with wrong results.
  • Changing the key effectively starts a new pagination state. To restart from the beginning of the dataset, either change to a new key or clear the saved state.

The saved state is managed by Etlworks in the Integrator's internal metadata folder. You don't need to manage the file yourself — just pick the key.

What gets saved, per pagination type

Pagination type Stateless behavior (Resume State Key empty) Resumable behavior (Resume State Key set)
page Each run starts at the configured Start Page or Offset. Saves the next page number at end of run. Next run starts from that page.
offset Each run starts at the configured Start Page or Offset. Saves the next offset at end of run. Next run starts from that offset.
cursor Each run starts without a cursor parameter. Saves the next cursor at end of run. Next run starts from that cursor.
nextLink Each run starts at the configured base URL. Saves the next URL at end of run. Next run starts by requesting that URL.
time Each run starts at the configured starting time. Saves the next time value at end of run. Next run starts from that time.

In every case, leaving Resume State Key empty preserves the existing stateless behavior — nothing about your current configuration changes.

Example: resumable cursor pagination

Suppose you want to pull HubSpot CRM updates every 15 minutes and have each run start from where the previous one ended.

Configuration

  • Pagination Type: cursor
  • Cursor JSON Path: /pagination/nextCursor
  • Cursor Parameter Name: cursor
  • Resume State Key: orders-prod-status-cursor

First run. The connector starts normally (no cursor query parameter). As the run completes, Etlworks stores the last next-cursor value returned by the API under the key orders-prod-status-cursor.

Second run. The connector reads the saved cursor value and uses it as the starting cursor for the first request. Pagination continues from there until the API stops returning a next cursor; the new last cursor is then stored, ready for the third run.

Resetting the saved pagination state

If you need to re-process the API from the beginning — for example after a data-correction fix on the source side — you have two options:

  • Change the Resume State Key to a new value (e.g., append a date or version suffix: orders-prod-cursor-2026-q3). The connector treats this as a new state and starts fresh on the next run.
  • Clear the saved state in Etlworks so the existing key starts over from the configured starting position.

Either approach gives you a clean restart. Changing the key is the safer option when you want to keep the old state around as a fallback.

Related articles