Overview

Etlworks collects and aggregates usage, execution, and scheduling data across all Flows.

This data is available in the Insights section.

Insights helps you:

• Monitor system activity in real time 
• Analyze execution patterns and performance 
• Identify bottlenecks and gaps in schedules 
• Troubleshoot issues using execution-level visibility 
• Optimize workload distribution

Only Admin and SuperAdmin users can access Insights.

Insights

The Insights menu (previously called Statistics) provides two tabs:

• Statistics – high-level usage and account metrics 
• Dashboards – detailed operational views for executions, tasks, and schedules

Multi-tenant view (SuperAdmin)

SuperAdmin users can switch to multi-tenant mode.

In this mode:

• Metrics are aggregated across selected tenants 

• Dashboards reflect combined activity

• Each metric can be expanded to see per-tenant breakdown

Why this is useful:

• Monitor usage across all tenants 
• Compare activity between tenants 
• Identify heavy usage or anomalies

Statistics

The Statistics tab provides a summary of system usage and activity.

You can see:

• Total records processed 
• Total executions (scheduled, on-demand, listener) 
• Daily and monthly trends 
• Execution success vs error distribution 
• Total number of Flows, Connections, Schedules, Formats, and Listeners

Use the month selector to view historical data.

This view is useful for:

• Tracking overall usage 
• Monitoring growth trends 
• Identifying spikes in processing volume

Overage monitoring

Insights tracks usage against subscription limits.

Displays:

• Daily usage vs limit Monthly usage vs limit Potential overage

Notifications:

• Daily overage notice 
• Monthly overage notice

Sent to the account owner when limits are exceeded.

Dashboards

The Dashboards tab provides detailed, purpose-built views for monitoring and analysis.

Each dashboard focuses on a specific aspect of system activity.

Executions

The Executions dashboard shows all flow executions for a selected day.

Key capabilities:

• Real-time view of running and completed executions 
• Filter by: Name, Status, Records, Duration, Start and End time 
• Partial name matching supported 
• Combine filters for precise results

Click any execution to:

• View status and errors 
• Open logs 
• Inspect execution details

Export data:

• Download filtered executions as CSV 
• Useful for reporting and external analysis

Why this is useful:

• Troubleshoot failures quickly 
• Identify slow or large executions 
• Audit system activity

Running Flow Tasks

Shows currently running tasks across all flows.

Includes:

• Task type (extract, load, transform, etc.) 
• Flow name 
• Owner 
• Execution details

Why this is useful:

• Real-time visibility into what is running 
• Detect stuck or long-running tasks 
• Understand pipeline activity at step level

Executions Heatmap

Visualizes execution activity over time.

Features:

• Heatmap by hour, day, week, or month 
• Color intensity reflects execution volume 
• Shows: Average executions, Records processed, Average duration, Peak activity window 

Click any cell to drill down into executions.

Why this is useful:

• Identify busy time windows 
• Detect uneven workload distribution 
• Plan scheduling and scaling

Scheduled Activity

Analyzes schedules and predicts workload.

Includes multiple views:

Expected Timeline

• Simulated execution timeline based on schedules 
• Shows projected runs and durations

Gaps

• Identifies available time windows with no executions 
• Helps find capacity for new workloads

Scheduled vs Actual

• Compares planned executions vs actual runs 
• Highlights deviations

Ask (AI-assisted)

• Query schedules using natural language 
• Example: show me the gaps between midnight and noon 

Why this is useful:

• Optimize schedule distribution 
• Avoid overlaps and bottlenecks 
• Identify unused capacity 
• Improve overall system efficiency

Resource utilization

The Resource Usage dashboard correlates node pressure, ETL threads, and flow contribution over time: CPU, RAM, JVM heap, system and data volumes, JVM threads, and open file descriptors, per node, with recorded per-flow CPU and allocation evidence.

Available to Super Admin users. Three tabs — Overview, Incidents, and Ask (when AI is enabled) — plus a Configure button for thresholds and alerts.

For the full operational workflow — investigating during and after incidents, reading each metric, choosing the ranking metric, troubleshooting, and the CLI/API — see Find flows contributing to resource pressure with Resource Usage.

Overview

• Selected-window summary cards: Peak CPU, Peak memory, Lowest system volume available, and Timeline samples.
• Infrastructure utilization chart (CPU, RAM, Heap, System volume, Data volume) with hover details and incident overlays; click a timeline point to see the flows sampled as running at that moment.
• Process resources: open files and JVM threads, as counts or as a percentage of the process file limit where the platform reports one.
• Top contributing flows: recorded contribution ranked by allocation by default, with CPU avg, Alloc., Peak/s, Alloc/s, and estimated heap columns, attribution-confidence warnings, and flow links.
• Execution evidence with per-execution measured contribution and an expandable Flow thread breakdown (latest sampled thread tree with own and tree totals).

Reading the numbers: CPU avg is recorded thread CPU over the window (100% = one core; multi-worker flows can exceed it). Alloc. is JVM heap-object allocation volume — not retained RAM. Estimated heap columns are approximations. Metrics shown as n/a were not measured — that is not zero activity. See the metric interpretation guide.

Automatic incident tracking

An incident opens when a tracked resource (CPU, RAM, or disk system-volume utilization) stays above a configured threshold for a configured sustain duration, and clears after it stays below the warning threshold for the clear duration. Severity is warning or critical; an incident can also be interrupted when a node restart closes it. By default an incident opens on any sustained metric (OR); combined AND conditions (for example CPU and RAM) can be configured instead.

Each incident records the node, opened/resolved times, duration, trigger value, sustained duration, the sampled flow-execution evidence (ranked contributors with measured CPU and allocation where thread capture was on), and offers Analyze — a baseline-versus-incident execution-impact comparison with an optional AI explanation.

Default thresholds and sustains:

Default clear duration: 2 minutes. Override thresholds, the incident condition, sustain/clear durations, and alert email recipients through the dashboard's Configure button (Global Settings → Insights holds the same values; existing installations can retain older overrides — check yours). Alert emails are sent when a sustained incident opens, escalates from warning to critical, or clears, and a Resource Usage.Threshold webhook fires on threshold level transitions.

Incidents are persisted and survive restarts; they are queryable at GET /rest/v1/metrics/resource-usage/incidents. Capture switches (History, Health capture, All ETL threads, Incident capture, Incident threads, Auto refresh) and retention defaults are described in the investigation guide.

Ask (AI-assisted)

• Query the captured resource evidence in natural language, with presets: Current state, Top bottleneck, Top contributors, Execution evidence.
• Current state uses the latest captured node snapshot; historical questions use the selected window and filters.
• Example: which flows contributed the most CPU on this node during the incident?

Why this is useful:

• Detect node pressure before it impacts flows
• Correlate slow executions with resource contention
• Find which flows are the heaviest contributors
• Right-size capacity for the workload

AI usage

Tracks Simba activity, token consumption, billing, and knowledge-base sync.

Available to Admin and SuperAdmin users on tenants where AI features are enabled. SuperAdmins see an additional cross-tenant breakdown.

Tracked metrics (current month, resets at month start):

• Total tokens — prompt, completion, total
• Agentic vs non-agentic tokens — distinguishes state-changing tool calls from Q&A and knowledge-base search
• Conversations and total requests
• Daily usage chart for the current month

Billing visibility:

• Total cost, billable cost, and non-billable cost; agentic-only cost
• Cost figures account for cached prompt tokens, which are billed at a lower rate, so the displayed cost matches the provider invoice more closely
• Wallet balance, monthly drain, and auto-recharge status
• Monthly cap with current usage percentage
• Tenants on BYOK (their own OpenAI key) show zero billable cost — their OpenAI account is billed directly

Per-tenant breakdown (SuperAdmin only):

• All metrics aggregated by tenant
• Tenant display name, environment label, and billing flags

Leaderboard and drill-down:

• A per-user leaderboard ranks users by token consumption and cost; every column is sortable.
• Expand a user row to drill down into that user's own sessions and usage.
• Click a day in the daily-usage chart to scope the leaderboard to that day; clear the day to return to the full month.

Retrieve usage programmatically: AI usage is also available through the Etlworks CLI and API, so you can export or monitor consumption outside the dashboard.

RAG (knowledge base) status:

• Total indexed chunks across sources (knowledge-base articles, CLI commands, templates, marketing pages)
• Last sync time, duration, and status (success, failed, never)
• Next scheduled sync
• Trigger a full or incremental reindex from the dashboard, with progress polling

Why this is useful:

• Monitor monthly AI spend against the allowance and cap
• See which tenants and users are driving consumption
• Confirm the knowledge base is current — last sync, source coverage, scheduled refresh
• Trigger a reindex after major documentation or template changes

Flow Findings dashboard

The Flow Findings dashboard aggregates automatic flow-inspection reports across your instance. Whenever a flow runs, the engine writes an inspection report capturing observed issues — schema drift, performance anomalies, recurrent errors, data quality concerns. The dashboard reads those reports and surfaces the flows that need attention.

What's a finding?

A finding is a flow-level inspection report with one or more detected issues. Each report has a severity (ALL_GOOD, INFO, MINOR, MEDIUM, MAJOR, CRITICAL) computed from the worst issue in the report. Each issue has:

• Issue type — e.g., schema drift, slow step, recurrent error, data quality.
• Severity.
• Description — what was detected.
• Why — why it's flagged.
• Suggestion — what to do about it.

What the dashboard shows

The dashboard lists every flow that has an inspection report within the configured retention window (1–36 months). For each flow you see:

• Flow name, ID, and severity badge.
• Issue count.
• Last executed timestamp.
• Report last-updated timestamp.
• An indicator if the flow still exists (reports outlive deleted flows).

Click a flow to expand its report and see the per-issue detail.

Filters

Where the data lives

Inspection reports are written to {app.data}/errors/<tenant>/<flow-id>_<timestamp>.json by the flow runtime. The dashboard reads the files directly and aggregates them at query time — there's no separate findings table. Old reports are pruned per the retention policy.

API: GET /rest/v1/metrics/flow-findings?tenants=<csv>&months=<1-36>.