Articles in this section

See more

How to debug HTTP requests

  • Updated
Follow

Overview

When an HTTP request misbehaves, the connection's static configuration rarely tells the whole story. The URL may be assembled from {tokens} and global variables, headers may be added by authentication or a preprocessor, the payload may be generated or transformed, and pagination may rewrite the request for every page. What matters for debugging is the final request Etlworks actually sends and the response metadata it receives.

Starting with Etlworks 9.8.5, the HTTP connector can log exactly that. Two new options — Log Request Details to Debug Log and Log Request Payload to Debug Log — join the existing Log URL to Debug Log. They are available on the generic HTTP connection and on all preconfigured HTTP API connectors, including Salesforce, HubSpot, Google, Microsoft, QuickBooks Online, Jira, Jira Service Management, Confluence, Amazon Marketplace, Etlworks APIs, and the Zoho connectors.

All three options are in the HTTP connection editor. The output goes to the debug log, which you can open from the Flows window or from the Audit Trail.

Which debugging option to use

Option What it logs Use it when
Log URL to Debug Log Only the final resolved request URL. You just need a quick check of a dynamic URL or resolved {tokens}.
Log Request Details to Debug Log The final HTTP method, resolved URL, request headers after authentication/signing, request configuration and entity metadata (content type, encoding, content length, chunked/repeatable/streaming state), response status, response headers, and any failure associated with the request. Authentication requests are labeled separately from data requests. Credential-bearing header values are redacted. Does not log the request payload. You need to see what was actually sent and what came back — the default choice for almost every HTTP problem.
Log Request Payload to Debug Log Independently logs the final outgoing request body — after Etlworks constructs or transforms it — for both data and authentication requests. If no body is sent, the log says so. Does not log the response body. The method, URL, and headers look right but the body may be wrong. Enable temporarily; payloads can contain secrets.

The two new options are available in Etlworks 9.8.5 and later. They complement each other: details give you the envelope, payload gives you the letter.

End-to-end troubleshooting workflow

  1. Enable Log Request Details to Debug Log on the HTTP connection.
  2. Run or rerun the flow.
  3. Open the Debug Log from Flows or the Audit Trail.
  4. Inspect authentication calls first, if present.
  5. Inspect the final data-request method, URL, headers, response status, and response headers.
  6. For paginated APIs, compare every generated page request in sequence.
  7. Enable Log Request Payload to Debug Log only if the URL, method, and headers look correct but the body may be wrong.
  8. Use Log Response on Exception or the response-file settings when you need the response body.
  9. Disable detailed and payload logging after troubleshooting — especially on high-volume or paginated flows.

Reading request and response details

With Log Request Details to Debug Log enabled, each request produces a request block and a response block. A sanitized example:

HTTP data request details
Method: GET
URL: https://api.example.com/v2/orders?updatedAfter=2026-08-01&page=1&limit=100
Headers:
  Authorization: Bearer <redacted>
  Accept: application/json
Configuration: [connectTimeout=30000, socketTimeout=120000, ...]
Entity: <none>

HTTP data response details
Status: HTTP/1.1 200 OK
Headers:
  Content-Type: application/json;charset=utf-8
  X-Request-Id: 4f7c...
Entity: contentType=application/json;charset=utf-8, contentEncoding=<none>, contentLength=-1, chunked=false, repeatable=false, streaming=true

How to read it:

  • The URL is fully resolved — {tokens}, global variables, and generated pagination parameters are already substituted.
  • The headers are the final set, after authentication and request signing. Header values whose names look credential-bearing (authorization, cookie, api-key, token, secret, password, credential, signature) are redacted; for Authorization headers the scheme stays visible, for example Bearer <redacted> or Basic <redacted>.
  • The Entity line describes the request or response body without printing it: content type, encoding, length, and chunked/repeatable/streaming state.
  • If the request fails, the failure is logged with the same data/authentication label, so you can tell exactly which call broke.

Because every retry passes through the same final-request logging path, a retried request produces one block per attempt — compare them to see whether anything changed between attempts.

Debugging authentication calls

The same options also trace the internal requests Etlworks makes to authenticate — token authentication, OAuth2, JWT assertion authentication, the authentication endpoints of applicable preconfigured connectors, and context-switch requests where applicable. Authentication blocks are labeled HTTP authentication request details / HTTP authentication response details, so they are easy to tell apart from data requests.

For an authentication call, the log can show:

  • the resolved authentication endpoint and HTTP method;
  • authentication request headers, configuration, and entity metadata;
  • the authentication request payload, when payload logging is enabled;
  • authentication response status and headers;
  • the subsequent data requests, with the resulting authorization header redacted.

Recommended reading order: inspect the authentication request and response first, then the data request. If the authentication call fails or returns an unexpected status, fix that before looking at the data request — this immediately separates token-acquisition failures from endpoint or data-request failures.

Debugging automatic pagination

When automatic pagination is configured, Etlworks generates a separate request for every page — and each generated request is logged separately, after it is constructed. This applies to all supported modes:

  • page-number pagination;
  • offset pagination;
  • cursor/token pagination;
  • next-link pagination, including absolute and relative next links;
  • time-based pagination.

It also applies to both injection methods: pagination values added to the query string, and pagination values injected into the request payload.

With request-detail logging enabled you can inspect the final URL, method, headers, response status, and response headers for every page. With payload logging also enabled, you can inspect the final generated body for each page when pagination is injected into the payload. A sanitized two-page example:

HTTP data request details
Method: GET
URL: https://api.example.com/v2/orders?page=1&limit=100
...
HTTP data response details
Status: HTTP/1.1 200 OK
...

HTTP data request details
Method: GET
URL: https://api.example.com/v2/orders?page=2&limit=100
...

Comparing the generated pages in sequence makes the classic pagination faults visible:

  • an incorrect starting page or offset;
  • a page-size parameter with the wrong name or value;
  • a cursor that is not advancing between pages;
  • an invalid or incorrectly resolved next link;
  • repeated or skipped pages;
  • an incorrect time window;
  • a pagination value injected into the wrong place (query string vs. payload);
  • an API returning an unexpected status or pagination header on one specific page.

Debugging request payloads

Log Request Payload to Debug Log logs the final outgoing request body — for regular data requests and for authentication requests — after Etlworks constructs or transforms it. That includes payloads built from templates with {tokens} and global variables, payloads modified by a connection preprocessor, and pagination values injected into the body.

HTTP authentication request payload
grant_type=client_credentials&client_id=example-app&client_secret=EXAMPLE-SANITIZED

If the request has no body, the log prints <empty> instead. (A streaming, non-repeatable body cannot be captured; the log says so explicitly.)

Warning: the payload is logged as sent — it can expose passwords, client secrets, JWT assertions, API data, and other sensitive information. There is no general secret redaction inside payloads. Enable this option only temporarily and disable it as soon as troubleshooting is done.

Note that this option logs the outgoing request body only. It never logs the response body — for that, see the next section.

Capturing response bodies, codes, and headers

Log Request Details to Debug Log includes response metadata: status, headers, and entity metadata. It does not log a successful response body. To capture response content, use the existing options on the HTTP connection:

Option What it does
Log Response on Exception Includes the response body in the log when an HTTP request fails.
Response File Name Saves the response body to a file. Supports {app.data} and other {tokens}.
Save Response Only on Error Saves the response file only for failed requests.
Save HTTP Code into Global Variable Stores the HTTP status code in the thread-safe global variable http_processor_last_http_code.
Save Response Headers into Global Variable Stores the response headers in the global variable http_processor_last_response_headers.

To read a captured value from JavaScript:

var httpCode = com.toolsverse.config.SystemConfig.instance().getContextProperties()
    .get("http_processor_last_http_code");

Handling nonstandard HTTP status codes

Some APIs use status codes in nonstandard ways. Two connection settings control how Etlworks interprets them:

  • Success HTTP Codes — a comma-separated list of codes Etlworks should consider successful when the API uses nonstandard behavior. When the list is set, only codes on it are treated as success. Example: an API that returns 202 for a successfully queued request.
  • Exception HTTP Codes — a comma-separated list of codes that are exempted from normal HTTP error handling. A code on this list does not raise an error even though it is normally an error code. Example: an API that returns 404 to mean "no rows found."

When a request completes, the debug log shows the actual status on the response-details block, which makes it easy to decide which of the two lists a misbehaving code belongs on.

Security and log-volume considerations

  • Credential-bearing header values are redacted in request details.
  • Resolved URLs can still contain sensitive query parameters — URL query values are not redacted.
  • Request payloads are logged without general secret redaction and can contain credentials or customer data.
  • Response bodies can also contain sensitive data when response logging or response-file options are enabled.
  • Pagination and retries multiply the output: every page and every attempt produces its own log blocks, which can generate a large volume of log records.
  • Treat these as temporary troubleshooting settings: enable, reproduce, read, disable — and protect access to debug logs.

Related articles