Overview
When a node is under CPU, memory, or disk pressure, the first operational question is always the same: which flows are contributing, and which executions deserve a closer look? The Resource Usage dashboard answers it with recorded per-flow CPU and allocation evidence, automatic incident windows, execution-impact comparison, and an AI assistant.
This guide is the operational walkthrough: how to investigate during an active incident, how to investigate after the fact, how to read each metric, and how to avoid the classic misreadings. For a short overview of the dashboard, see Account dashboard (Insights): Resource utilization.
Where: Insights → Dashboards → Resource Usage. The dashboard is available to Super Admins. It has three tabs — Overview, Incidents, and Ask (shown when AI is enabled) — plus a Configure button in the Overview tab's Capture status bar.
Read the evidence for what it is. The dashboard mixes measured contribution (recorded CPU time and allocation), estimates (the Est. heap columns), and correlation (executions that overlapped a pressure window). A high-ranked contributor is a candidate for investigation, not automatically the proven cause.
Investigate during an active incident
Step 1. Open Insights → Dashboards → Resource Usage → Incidents and select the open incident. Confirm the affected Node, the Resource (CPU, RAM, Disk, or a combined condition), the Opened time, and the Trigger value against the configured threshold (the current thresholds are shown in the Alerts row of the Capture status bar).
Step 2. Check capture status on the Overview tab: Health capture and Incident capture should be on, and Incident threads on if you want measured per-thread CPU and allocation during the incident. Make sure the selected From/To window includes the current period — click Refresh to re-follow the current time, or enable the Auto refresh switch to receive live updates while you watch.
Step 3. In Evidence for selected incident, sort by the metric that matches the pressure: CPU avg for CPU incidents, Alloc. or Peak/s for memory incidents. The table defaults to Alloc. descending. Evidence is sampled from the incident ring (which includes roughly the five minutes before the incident opened) and enriched from incident thread snapshots where available — distinguish pre-incident context from measurements taken during the incident by the Observed timestamps.
Note: flows that start after the incident opens appear in the evidence after they are sampled and the dashboard refreshes — they do not have to finish first.
Step 4. Click Analyze on the incident for the execution-impact comparison (baseline window versus incident window) and, when AI is enabled, the AI resource explanation. The two load independently.
Step 5. Switch to Overview with the same node and a matching time range to work the charts, Top contributing flows, Execution evidence, and per-execution thread trees.
Treat active-incident totals and impact comparisons as provisional — they grow as sampling continues, and reopening Analyze refreshes the comparison.
Investigate after an incident
Step 1. In Incidents, select the resolved (or interrupted) incident and note its node and exact time window (Opened/Resolved/Duration). An interrupted status means the open incident was closed because the node restarted — it does not mean the pressure cleared normally.
Step 2. Compare the infrastructure pressure (Overview charts for that node and window) with the ranked contributors and the individual executions in the incident evidence.
Step 3. Investigate suspicious executions through their links: the execution record, its logs and metrics, and the sampled Flow thread breakdown.
Step 4. Distinguish the flows contributing load (ranked contributors, measured CPU/allocation) from the flows suffering an impact (slower or failing executions in the Analyze comparison). They answer different questions, and the same flow can be both.
What survives, what does not: persisted incident evidence survives restarts, but historical investigation depends on what was captured and retained — samples that were never taken, or raw detail that has expired, cannot be reconstructed. See Capture and retention.
The Overview tab, tool by tool
Window, node, and refresh controls
- From/To dates with From time/To time select the window (times are shown in your browser timezone; the default window is the last 6 hours). Apply freezes the selected window; Refresh re-follows the current time keeping the same window length.
- Node selects a node for the charts and incidents (All nodes by default).
- The page-level Tenants selector narrows flow attribution and evidence to the selected tenants.
- Scope note: node health is node-wide — it is not a measurement of only the selected tenant or flow. The Node selector applies to the infrastructure/process charts and incidents; Top contributing flows and Execution evidence reflect the window and the tenant/flow filters.
- The summary cards show Peak CPU, Peak memory, Lowest system volume available, and Timeline samples for the selected window. Current-state summaries use the latest captured node snapshot.
Infrastructure utilization chart
- Series: CPU, RAM, Heap, System volume, and Data volume, as percentages over time, with incident overlays.
- Hover a point for absolute values (heap, threads, storage) and timestamps.
- Click a timeline point to switch Top contributing flows to the flows sampled as running around that point (where contributor evidence exists). A bar shows how many sampled flows were running at that time; use Clear chart-point filter to return to the broader window.
- Older parts of a wide window are served from coarser historical rollups rather than raw samples — expect smoother lines and fewer points at older ages.
Process resources
- Charts Open files and JVM threads as counts; on platforms that report a file-descriptor limit, a second mode shows Open files % against the process limit.
- Hover details include current and max open files, JVM threads, and peak JVM threads. The current-state text also shows the running-flow count.
- Missing platform metrics (for example, file descriptors on an OS/JVM that does not report them) are unavailable, not zero.
Top contributing flows
- Ranked table of recorded flow contribution for the selected window and filters. Default order is Alloc. descending; every metric column is sortable, and Top shown selects 25/50/100/200 rows (default 50).
- Columns: Flow, Tenant, Samples, CPU avg, Alloc., Peak/s, Alloc/s, Est. heap avg, Est. heap max.
- An attribution warning badge (medium confidence or low confidence) appears under the flow name when sampled activity could not be matched to an execution directly; hover it for the explanation. High-confidence attribution shows no badge.
- Flow names link to the flow. Select a row to focus Execution evidence on that flow; Clear flow filter in the toolbar removes the selection.
Execution evidence
- Actual sampled executions for the window (up to 200 rows), default sorted by CPU avg descending. Columns include the execution ID (linked), Started/Ended (running executions show "running"), the measured contribution columns, Samples, Nodes, and Evidence.
- Evidence distinguishes strong thread attribution (matched directly from sampled ETL thread activity) from sampled execution overlap (inferred from overlap with a running flow during the sampled window) — measured contribution versus correlation.
- Expand a row for the Flow thread breakdown: the latest sampled thread tree with its capture timestamp, root and worker threads, and both own values (CPU avg, Est. heap, Alloc.) and tree totals (Tree CPU, Tree heap, Tree alloc.). Tree totals already include the children — do not add parent tree totals to their child rows; that double-counts contribution.
How to read the metrics
| Metric | What it is | What it is not |
|---|---|---|
| CPU avg | Recorded thread CPU time divided by the represented window duration. 100% means one CPU core, so a flow with multiple workers can exceed 100%. Chart-point and thread-tree values use their own represented sampling intervals. | Not machine CPU utilization and not load average. |
| Alloc. | Recorded JVM heap-object allocation between valid consecutive observations, counting the root and its associated workers once each. | Allocation volume — not simultaneously retained heap, not native memory, not host RAM. JVM-reported object sizes are approximate. |
| Alloc/s | Recorded allocation divided by the displayed window duration. | Not necessarily the rate while the flow was actively working — a broad window dilutes short bursts. |
| Peak/s | The largest observed interval allocation rate for an execution and its associated workers. Useful for finding bursts. | Not an instantaneous peak between samples, and not a sum of independent execution peaks. |
| Est. heap avg / Est. heap max | Approximate heap attribution derived from JVM heap usage and the flow's relative lifetime thread CPU share. Secondary supporting evidence, useful when measured allocation is unavailable. | Not a direct measurement of retained memory owned by a flow. |
| Samples / Threads | How many samples and sampled thread roots the row is based on. Few samples mean weak evidence. | Not proof of complete temporal coverage. |
Confidence and coverage are different things:
- Attribution confidence describes how sampled activity was matched to a flow or execution. High confidence is not displayed; medium and low appear as a badge with hover detail.
- Counter coverage (shown when hovering an Alloc. value) describes how many observed threads had valid allocation counters.
- Neither proves complete temporal sampling, and neither proves causation.
Why recorded totals can undercount:
- The first observation of a thread establishes a baseline — nothing is measured before it.
- Short-lived workers can start and finish between captures and are never measured.
- Unsupported counters, execution/thread changes, and intervals crossing the selected window boundary leave gaps.
- Older history can show n/a for measured metrics while retaining the estimates. n/a means not measured — never interpret it as zero activity.
Worked example — allocation is not RAM: 6 GB allocated during an incident does not mean that the flow held 6 GB of RAM at once. Objects may have been repeatedly allocated and collected.
Worked example — burst versus steady (illustrative numbers): two flows each show Alloc. 3 GB over a one-hour window. Flow A shows Alloc/s ~0.85 MB/s and Peak/s 2 MB/s: steady, unremarkable allocation. Flow B shows the same Alloc/s but Peak/s 400 MB/s: it did almost all of its allocating in a short burst — exactly the profile that can drive a sudden heap spike and GC pressure. Same totals, very different suspects.
Choosing the ranking metric for the pressure you have
CPU pressure: sort contributors by CPU avg, compare against the machine CPU chart and the timing, then investigate the top candidates' executions and worker threads.
Heap / allocation pressure: start with Alloc. for total recorded contribution; use Peak/s to find sampled bursts and Alloc/s to compare sustained allocation across the same window. Correlate with absolute heap usage on the infrastructure chart. Treat the Est. heap columns as supporting evidence. High allocation churn suggests GC investigation — allocation alone does not prove retained heap or GC impact.
High host RAM with low JVM heap: do not automatically blame the largest allocator. Consider native/off-heap memory, other processes on the host, and OS memory accounting. The dashboard measures the JVM's view (heap, allocation, threads) and node-level utilization — it does not break down native memory or other processes.
Disk pressure: compare available space, the volume (the incident rule uses system-volume utilization; data-volume values are also charted), the node, the timing, and the executions running then. Allocation metrics do not measure disk writes.
Thread / file-descriptor pressure: use Process resources and the per-execution thread breakdown. A large thread count does not necessarily mean high CPU — waiting and blocked threads, connection pools, concurrency settings, and external-system delays need execution-level investigation.
Incidents
An incident opens when a tracked resource stays above a configured threshold for the configured sustain duration, and clears after the resource stays back below the warning threshold for the clear duration. Short spikes remain visible in the charts as sampling evidence without opening incidents.
Incident windows list one row per sustained condition: Status (open, resolved, or interrupted — closed by a node restart), Severity (warning or critical), Resource (CPU, RAM, Disk, or a combined condition such as CPU + RAM), Node, Opened/Resolved/Duration, Trigger (the value that fired), Sustained, Evidence, Thread details, and the Analyze action.
- Evidence counts distinct sampled flow executions — not health captures.
- Thread details (Requested, Enabled, or Not requested) reports the capture setting for the incident — it does not confirm that usable thread samples exist; the Evidence count tells you what was actually sampled.
Evidence for selected incident is a sortable table (default Alloc. descending) with the flow and execution links, Findings (from flow inspection, as additional diagnostic context — not proof of the pressure cause), execution Status, Observed bounds, Samples, Threads, and the contribution metrics. Observed timestamps are sampling evidence — not necessarily the execution's full start and end times. Evidence includes the pre-incident ring (about five minutes before open); distinguish that context from measurements taken during the incident.
Analyze: did the pressure coincide with impact?
The Analyze action opens the incident analysis: a calculated execution-impact comparison and, when AI is enabled, an AI resource explanation. The two load independently — if AI is unavailable, the calculated comparison still works.
- The comparison uses a baseline window of the same length ending at incident open versus the incident window: execution counts, average completed duration, failures, and the top impacted flows (shown as before / during).
- Counts group executions by start time; duration comparisons use completed executions only. Running executions can contribute pressure before any completed-duration evidence exists.
- The verdict is Likely, Possible, or No clear impact. "No clear impact" does not mean there was no pressure, or that no running flow was affected — it means completed durations and failure counts did not move materially.
- Scope: the calculated comparison covers the selected tenants across all nodes, while the resource evidence is scoped to the incident's node.
- Scheduling delay may be shown as Unavailable — the dashboard does not promise exact delay measurement.
- For an open incident the comparison is provisional; reopening Analyze refreshes it.
- Contributors and impacted flows answer different questions: who generated the load versus who suffered from it.
Capture, incident conditions, and retention
Capture switches
The Overview tab's Capture status bar exposes the switches (they mirror Global Settings → Insights):
- History — the master Resource Usage history/dashboard control. The other capture switches require it.
- Health capture — periodic node health snapshots (CPU, RAM, heap, volumes, threads, file descriptors).
- All ETL threads — continuous ETL thread sampling, including outside incidents. Requires Health capture. Useful for baselines and outside-incident comparison, but still periodic sampling.
- Incident capture — opens and persists incident windows and their evidence.
- Incident threads — measured per-thread CPU/allocation enrichment while incidents are active, sampled at the health cadence, with transitions captured immediately. Requires Incident capture. Incident evidence is collected independently of this switch — it only adds the measured enrichment.
- Auto refresh — live dashboard updates pushed while you watch.
Defaults: History, Health capture, Incident capture, and Incident threads are on; All ETL threads and Auto refresh are off. Health capture runs every 60 seconds and continuous ETL thread capture every 120 seconds by default. A prerequisite that is off disables the dependent switches. Changing capture settings affects future evidence, not the past.
Incident conditions and alerts (Configure)
The Configure button opens Configure Resource Usage alerts:
- Per-resource warning and critical thresholds (percent; warning must not exceed critical).
- Open incident when: Any metric (OR) — the default — or a combined AND condition (CPU and RAM, CPU and Disk, RAM and Disk, or CPU, RAM, and Disk) that opens one combined incident only when all selected metrics are sustained above threshold.
- Alert notification emails — comma- or space-separated recipients. Alerts are sent when a sustained incident opens, when an open warning incident escalates to critical, and when it clears.
- Advanced incident durations — per-resource warning/critical sustain and the clear-after-normal duration, in seconds (15–3600) in this dialog. The same values appear in Global Settings in milliseconds.
Current defaults:
| Resource | Warning | Critical | Warning sustain | Critical sustain |
|---|---|---|---|---|
| CPU | 80% | 95% | 5 minutes | 2 minutes |
| RAM | 80% | 90% | 2 minutes | 1 minute |
| Disk (system volume) | 85% | 95% | 5 minutes | 5 minutes |
Default clear duration: 2 minutes below the warning threshold. Default condition: Any metric (OR). Existing installations can retain configured overrides — check your actual settings in the dialog or in Global Settings → Insights.
A resource-usage webhook (Resource Usage.Threshold) also fires on threshold level transitions, with the resource, node, current value, and the previous/current level.
Retention
| Data | Default retention |
|---|---|
| Raw health samples | 14 days |
| Raw thread samples | 7 days |
| Execution samples | 30 days |
| 1-minute rollups | 30 days |
| 15-minute rollups | 180 days |
| 1-hour rollups | 365 days |
Broader history preserves aggregate evidence after raw thread detail expires — but not at the same granularity at every age. An investigation three weeks later can still see the pressure window and contributors; the per-thread trees may be gone.
Ask: AI-assisted investigation
The Ask tab (available when AI is enabled) answers questions against the captured evidence. Preset chips: Current state, Top bottleneck, Top contributors, Execution evidence.
- Current state uses the latest captured node snapshots — not the selected historical period, and not an instantaneous live measurement. Check the snapshot timestamps cited in the answer.
- Historical questions use the selected window and the active node/tenant/flow filters. Incident Analyze uses the incident's own authoritative node and time scope.
Questions that work well:
- Which flows allocated the most heap objects during this window?
- Which executions had the highest sampled allocation bursts?
- Which flows contributed the most CPU on this node during the incident?
- Did the largest contributors also have longer durations or more failures?
- Host RAM is high but JVM heap is low. What does the evidence support, and what should I check next?
- What evidence is missing, and how does that limit this conclusion?
AI answers use bounded captured evidence and may load separately. Treat them as a guided reading of the evidence, not proof of causation — check the cited times and execution IDs against the tables. If AI is disabled or unavailable, the charts, recorded evidence, and the calculated impact comparison remain fully usable.
Troubleshooting
| Symptom | Checks |
|---|---|
| Empty incident evidence | Was any ETL root thread active in the ring window? A flow that finished before the incident opened leaves no incident samples. Check that Health capture and Incident capture were on at the time. |
| A flow that is clearly running is missing from an open incident | It appears after it is sampled and the dashboard refreshes — use Refresh or Auto refresh and allow a capture cycle. |
| Missing allocation measurements (n/a) | n/a means not measured, not zero: first-sample baselines, short-lived workers, unsupported counters, window-boundary intervals, or raw detail older than the thread retention. Check the Est. heap columns as secondary evidence. |
| Missing execution or thread links | Low-confidence attribution may not resolve to a specific execution; the flow link and window timing still narrow the search. |
| Low sample counts | Short executions between captures, capture switches off, or a narrow window. Treat conclusions from one or two samples as weak. |
| Raw thread detail expired | Use execution samples and rollups — aggregate evidence outlives raw trees (see Retention). |
| Wrong or empty-looking data | Verify node selection, tenant filter, From/To window and times, timezone, and that a chart-point or flow filter is not still active; Refresh to re-follow current time. |
| Thread details says Requested/Enabled but no usable samples | The label reports the setting, not captured samples. Check the Evidence count and the Samples column. |
| High host pressure without corresponding ETL contribution | Consider native/off-heap memory, non-Etlworks processes, and OS accounting — the dashboard measures the JVM and node utilization, not other processes. Missing evidence does not prove no flow contributed. |
Cross-reference the execution records and logs, Flow Findings, the Running Flow Tasks dashboard, and the scheduling dashboards to confirm a diagnosis.
Appendix: CLI and API
The same evidence is available to Super Admins through the application CLI (read-only, no confirmation required) and the REST API.
| Command | Usage |
|---|---|
| resource-summary | resource-summary — latest node snapshots and highlights |
| resource-config | resource-config — capture, threshold, and retention configuration |
| resource-timeline | resource-timeline [from=<ISO-8601>] [to=<ISO-8601>] [nodeId=<node>] — timeline points (default lookback 2 hours) |
| resource-top-flows | resource-top-flows [from=…] [to=…] [tenants=<id1,id2>] [metric=<cpu|memory|allocation>] [top=<N>] — ranked contributors; metric=allocation ranks by recorded allocation |
| resource-executions | resource-executions [from=…] [to=…] [flowId=<id or name>] [tenants=…] [top=<N>] — sampled execution evidence |
| resource-threads | resource-threads [from=…] [to=…] [flowId=<id or name>] [tenants=…] [nodeId=<node>] [rootsOnly=<true|false>] [top=<N>] — sampled threads; rootsOnly defaults to true (set rootsOnly=false to include workers) |
| resource-bottlenecks | resource-bottlenecks [from=…] [to=…] [tenants=…] [top=<N>] — hottest CPU/memory windows, lowest-disk windows, top flows, related executions |
| resource-incidents | resource-incidents [from=…] [to=…] [nodeId=<node>] [status=<open|resolved|interrupted>] [resource=<cpu|ram|disk|cpu_ram|cpu_disk|ram_disk|cpu_ram_disk>] [top=<N>] — incident windows (default lookback 6 hours) |
| resource-incident-impact | resource-incident-impact incidentId=<id> [tenants=…] — the calculated execution-impact comparison for an incident |
Notes:
- Only resource-timeline, resource-threads, and resource-incidents accept nodeId; the other commands do not.
- Use explicit historical from/to bounds when investigating an incident, for example: resource-incidents from=2026-10-13T14:00:00Z to=2026-10-13T20:00:00Z status=resolved
- Output is JSON and includes a query block describing the effective scope.
REST routes (Super Admin; from/to required where shown in the UI workflows): /rest/v1/metrics/resource-usage/ summary, config, timeline, top-flows, executions, threads, execution-threads, incidents, and incidents/{incidentId}/impact.