An Etlworks managed cluster runs the full Etlworks application and ETL engine in each application pod. Within your Cluster license, it adds application capacity when demand grows and safely drains pods when demand falls, and it manages application upgrades, pod sizing, backups and recovery as durable, reviewable operations.
You supply an existing Kubernetes cluster. The Etlworks installer deploys the application and the components required by the selected installation profile. You do not build container images, install Tomcat on the worker hosts or run a separate native Etlworks installation on every machine.
Installing on Amazon EKS, Azure AKS or Google GKE Standard? Provider preparation, cloud plans and cloud-specific troubleshooting are in Install Etlworks on cloud Kubernetes: Amazon EKS, Azure AKS and Google GKE; the downloads, licensing, installation and operations procedures in this guide apply to every profile. For native Linux/Windows installation and regular fixed clusters, see Running Etlworks on-premise: installation, upgrade, maintenance.
In this article
- Deployment options and terms
- What the installer deploys
- Before you start
- Prepare self-managed Kubernetes
- Download the installer and Cluster license
- Verify and extract your downloads
- Configure the installation
- Install and check progress
- Resume an interrupted installation
- Open Etlworks and finish setup
- Configure automatic scaling
- Understand pod and node scaling
- Monitor health, resources and cluster status
- View pods and Tomcat logs
- Upgrade the application
- Change the application pod size
- Back up, restore and migrate
- Security, certificates and network behavior
- Premium connectors on managed pods
- Troubleshooting
- After installation
Deployment options and terms
The managed installer selects a platform profile that determines where and how the installation runs:
| Platform profile | Target platform | Storage and network integration |
|---|---|---|
| local-nfs-v1 | Self-managed Linux Kubernetes. The default when no profile is set. | Installer-provisioned NFS, local worker disks or a retained disk plan, Calico network policy. |
| aws-eks-v1 | Amazon EKS with ordinary EC2 Linux managed node groups. | EFS shared storage, encrypted gp3 EBS disks, VPC CNI network policy, NLB entry. |
| azure-aks-v1 | Azure AKS with ordinary Linux VMSS node pools. | Premium Azure Files NFS with encryption in transit, Premium Azure Disk, Standard Load Balancer entry. |
| gcp-gke-standard-v1 | Google GKE Standard with ordinary Linux node pools. | Private Filestore, balanced persistent disks, passthrough load-balancer entry. |
All four profiles ship in the same release archive, for Linux AMD64 and ARM64 workers, starting with Etlworks 10.0.14. Select the profile with platformProfile in the installation configuration; omitting it selects the local profile, so existing local configurations keep working unchanged. There is no separate cloud download and no provider selection on the download pages.
Choose your path through this guide:
| You are installing on | Read |
|---|---|
| Your own Linux Kubernetes cluster | This guide, top to bottom. Prepare self-managed Kubernetes covers worker and storage preparation. |
| Amazon EKS, Azure AKS or Google GKE Standard | The cloud guide for provider preparation and the cloud plan, then return to this guide's downloads, configuration, installation and operations sections. |
An installation involves three separate choices:
- Cluster license — sizing only: pod CPU/RAM and the starting/maximum pod counts. The license's resource profile name is not a cloud selector, and there are no sizing keys in install.json.
- platformProfile in install.json — one of the profiles above.
- Platform administration — machines, zones, quotas and any node autoscaler remain your platform administrator's responsibility on every profile.
On every profile, PostgreSQL and Redis are independent choices between installer-provisioned containers and external services you supply. Provisioned PostgreSQL and Redis are single-instance components, and a multi-pod application does not by itself make every infrastructure component redundant: "managed" means Etlworks manages the application lifecycle, not that Etlworks creates or operates your Kubernetes cluster, worker machines, VPC, cloud accounts or DNS, and not that the supplied database, Redis, proxy or storage is a failover architecture.
This guide uses these terms:
| Term | Meaning |
|---|---|
| Application pod | The full Etlworks application and embedded ETL engine running together. The UI calls it a pod; lifecycle output can call it a member. |
| Kubernetes worker (node) | A machine on which Kubernetes places pods. Its size and count are managed by your Kubernetes/platform administrator. |
| Platform profile | The installation/storage/network integration selected by platformProfile: one of the four profiles above. |
| Licensed pod size (resource profile) | The authorized CPU and RAM per application pod, plus the signed member and aggregate limits. Separate from the platform profile. |
| Scaling settings | The operational minimum/maximum, CPU/RAM/heap targets and optional capacity schedules stored in the application, within license bounds. |
| Starting configuration | The signed initial pod count and size used during first installation. It is not reapplied as a capacity reset on every resume. |
What the installer deploys
Users, APIs and Integration Agents
|
HTTPS / HAProxy
|
Etlworks application pods
[application + ETL engine]
|
PostgreSQL / Redis / shared app data
KEDA + HPA ---> desired application pod count
|
Lifecycle adapter
creates / safely drains pods
cert-manager ---> internal certificates and trust
Kubernetes -----> pod placement and process recovery
Node autoscaler (if configured by your administrator) ---> machines| Component | Purpose |
|---|---|
| Application pods | Each runs the full app, Tomcat and embedded ETL engine. The license controls starting count, pod size and maximum capacity. Each pod invokes its own embedded engine locally. |
| Lifecycle adapter | A separate controller that creates application pods and waits for application drain/removal approval before planned deletion. It also executes upgrade, resize and recovery operations. |
| KEDA and Kubernetes HPA | Calculate the desired application pod count from published metrics and your scaling settings. Do not add a competing HPA or manually scale the generated member workloads. |
| HAProxy | The stable application traffic entry point. Participates in planned traffic withdrawal before a pod is removed. |
| PostgreSQL | Holds application metadata and shared application records. Provisioned by the installer as a Kubernetes container, or an external PostgreSQL 17 you supply. |
| Redis | Holds coordination and other application state. Provisioned by the installer as a Kubernetes container, or an external Redis 7 you supply. |
| Shared application storage | Persistent application data used across pods, plus a separate backup staging export and per-pod identity storage. |
| cert-manager and credential components | Issue, deliver and renew internal component certificates automatically. You do not manually issue a certificate for each added pod. |
One elected application scheduler coordinates schedules and license renewal. Scheduled executions are routed to eligible application pods: the election does not force all scheduled executions onto the scheduler's own pod. Manual, API/CLI, scheduled and listener-triggered execution paths keep their normal behavior on a managed cluster.
These components consume resources in addition to your licensed application pods. Two licensed application pods do not mean the namespace will contain only two pods, and a license allowing two 2-CPU/8-GiB pods does not mean two start immediately or that one lands on each worker: the starting count comes from the license, and placement follows Kubernetes resources, storage bindings and scheduling constraints. The installer process on your workstation exits after its work; the adapter stays in the cluster to manage the application lifecycle.
Before you start
Installation workstation
Use an administration workstation with access to the Kubernetes API and these tools:
- Java 25.
- Python 3.12 or newer.
- kubectl, compatible with the selected cluster.
- Helm 3.
- ZIP extraction and SHA-256 verification tools.
- Your cloud provider's CLI and authentication plugin, if your kubeconfig requires them.
On Linux and macOS use the etlworks-kubernetes.sh launcher shown throughout this guide. On Windows, deployment/installer/etlworks-kubernetes.ps1 accepts the same subcommands and options with PowerShell paths; a Windows workstation can install to Linux workers, but Windows worker containers are not a supported target. Check exact tool versions against the inventory shipped with your release.
You do not need a local Tomcat service or a Docker socket for the installation client, and you do not install Java or a native Etlworks application on each worker.
Kubernetes access and permissions
Obtain a kubeconfig and identify the exact context for the target cluster. The installer uses that explicit context; it does not switch your active context. Use a new namespace owned by this installation — the installer does not adopt an arbitrary existing namespace — and do not point a fresh installation at an existing Etlworks environment's data.
The installation identity needs platform permissions to create the selected profile's CRDs, admission policies, storage and certificate resources, as well as namespaced application resources. Have your Kubernetes administrator perform the installation or supply an appropriately authorized identity. Runtime service accounts use more limited permissions; no human kubeconfig is deployed into app pods.
Cluster capacity
The Cluster license supplies the starting application pod count and resource profile. You do not enter a second starting count or CPU/RAM size in install.json. Choose a trial size your workers can run, or obtain the agreed licensed capacity from Etlworks for your installation.
Each selected pod must fit on one eligible worker using that node's allocatable resources after Kubernetes/system overhead and other workloads. Unused RAM across several smaller nodes cannot be combined to fit one larger pod. Leave additional capacity for PostgreSQL, Redis, proxy, controller, metrics, certificates and storage. For two-pod operation, allow room for both pods; upgrades may require a separately licensed receiving/surge reservation.
For example, an 8 GiB/2 CPU trial starts one such application pod and permits up to two. A worker with exactly 8 GiB of physical RAM generally has less than 8 GiB allocatable and is not a sufficient sizing assumption for that pod and its overhead. There is no guaranteed one-pod-per-node placement: a single worker can host multiple application pods when its resources allow, and its loss then affects all of them.
Network and persistent state
Prepare:
- A hostname that users, API clients and Integration Agents can resolve and reach.
- DNS and a planned HTTPS access path. The local profile does not create public DNS records or a cloud load balancer. Cloud profiles create a TCP 443 load-balancer Service, but DNS and frontend certificate trust still remain yours.
- Registry access for the release's pinned application, adapter and platform images. Public managed images are on Docker Hub and the lifecycle adapter repository.
- Outbound HTTPS to the Etlworks download/licensing services as required by the selected license mode, plus connectivity to your flow sources and destinations.
- Retained storage and enough disk for application data, database growth, logs and backups. Store independent recovery copies outside the cluster's storage failure domain.
The local profile's NFS data path is not encrypted storage traffic, and application/control TLS does not make every storage hop encrypted. Keep worker and storage networks isolated with platform firewalls; see Security, certificates and network behavior.
Prepare self-managed Kubernetes
Prepare the Kubernetes cluster before running the Etlworks installer. Cluster creation, operating system installation, worker joins and machine provisioning belong to your platform administration workflow. Installing on EKS, AKS or GKE instead? Skip this section and follow the cloud guide.
The current release profile pins this platform stack:
| Item | Current profile |
|---|---|
| Kubernetes | 1.34.x server; use the release inventory for the exact baseline |
| Worker OS/CPU | Linux, matching the downloaded AMD64 or ARM64 release |
| Network policy | Calico 3.33, with at least two ready Calico instances |
| cert-manager | 1.21.3 |
| NFS CSI | 4.13.6, with the profile's ownership settings |
| KEDA | 2.17.2 |
| Member identity storage | Compatible local-path provisioner with the required private-directory ownership |
| Shared application data | Profile-managed NFSv4.1 storage |
| PostgreSQL / Redis | Provisioned by this installation profile, or external services you supply |
Use the inventory shipped with your release as the source for versions. Do not install arbitrary latest component versions or edit release.json to bypass a compatibility check.
For this local profile, prepare at least two ready Linux nodes and identify the storage workers. Ensure that workers can use NFSv4.1 and that the local storage provisioner is available. The installer can install absent pinned Calico, cert-manager, NFS CSI and KEDA assets; it does not bootstrap a missing local-path storage provisioner or turn an arbitrary existing cluster into the required platform. Existing compatible cert-manager, KEDA and other shared platform components are inspected and reused; the installer does not silently upgrade a customer's shared platform components. --reuse-platform-only prevents the installer from installing missing platform prerequisites, but requires the profile's prerequisites and ownership preparation to have already been completed; it does not bypass compatibility checks.
The profile creates retained local storage tied to the selected worker names and shared NFS claims. Local backing paths use /var/local/etlworks-NAMESPACE/. Ensure the underlying worker disks are durable and have sufficient space. The installer can create required local paths; path creation does not supply durable disks or make a disposable worker filesystem persistent. Do not use old data directories as a fresh-install shortcut.
PostgreSQL, Redis and the NFS server are single instances in this profile and are bound to local worker storage. Application pods can scale, but losing their storage worker can still make the application unavailable. Retained PVCs are not backups, and deleting a disposable test cluster or its worker containers can destroy the local data underneath them.
Inspect access, nodes and storage classes before proceeding:
kubectl --kubeconfig /absolute/path/kubeconfig config get-contexts
kubectl --kubeconfig /absolute/path/kubeconfig --context CONTEXT get nodes -o wide
kubectl --kubeconfig /absolute/path/kubeconfig --context CONTEXT get storageclass
kubectl --kubeconfig /absolute/path/kubeconfig --context CONTEXT get csidriverReplace uppercase placeholders and example absolute paths throughout this guide with your actual values. Do not run the installation against an unverified context.
Evaluation directories or a retained disk plan
By default, the local profile backs PostgreSQL, Redis, NFS and member identities with directories under /var/local/etlworks-NAMESPACE/ on the selected workers, as described above. That is the basic evaluation arrangement: simple, but only as durable as those directories.
For durable self-hosted storage, use a retained disk plan (Etlworks 10.0.14 and newer). The platform administrator prepares dedicated ext4 or XFS block-device mounts on the selected Linux workers. Root-filesystem directories, bind mounts of the root filesystem, overlay filesystems and tmpfs are not production disks. Configure mount persistence at boot and retain the underlying disks independently of Kubernetes workloads. The installer never formats disks, mounts filesystems, creates missing host paths, accepts an unmounted root-filesystem fallback, or recursively changes ownership of existing data.
Inspect each prepared mount on its worker and record the actual values for the plan:
findmnt -n -o TARGET,SOURCE,FSTYPE --target /srv/etlworks
stat -f -c %i /srv/etlworks
kubectl get node worker-a -o jsonpath='{.metadata.uid}'Create separate, non-overlapping roots with this ownership and capacity:
| Role | Ownership (UID:GID) | Mode | Minimum size | Roots needed |
|---|---|---|---|---|
| PostgreSQL | 999:999 | 2770 | 8 GiB | 1 (omit for external PostgreSQL) |
| Redis | 999:999 | 2770 | 1 GiB | 1 (omit for external Redis) |
| NFS shared application data | 0:0 | 750 | 8 GiB | 1 |
| Member identities | 1000:1000 | 700 | 1 GiB each | Licensed maximum + surge + 2 |
These are storage minima, not application RAM. Write a different administrator-generated UUID into a .etlworks-volume-id file in each root, readable by its owner. These are volume identity markers, not credentials — and a marker alone is insufficient, because the guard verifies the actual mount, device source, filesystem ID, node UID, ownership, capacity and free inodes.
Reference the plan from install.json alongside the worker names:
"storage": {
"workerA": "worker-a",
"workerB": "worker-b",
"retainedPlan": "/absolute/etlworks-private/retained-storage.json"
}The plan file is version 1 and has exactly schemaVersion, namespace, volumes and backup. One volume entry, with placeholders — this is deliberately not a complete installable plan; use the complete template shipped with your release, repeat the entry for every required role using your actual observations, and add redis, nfs and enough distinct identity roots:
{
"schemaVersion": 1,
"namespace": "etlworks",
"volumes": [
{
"role": "postgres",
"node": "worker-a",
"nodeUid": "REPLACE-WITH-ACTUAL-NODE-UID",
"path": "/srv/etlworks/postgres",
"mountPoint": "/srv/etlworks",
"mountSource": "/dev/mapper/etlworks-data",
"mountType": "ext4",
"filesystemId": "REPLACE-WITH-STAT-OUTPUT",
"volumeId": "REPLACE-WITH-A-DIFFERENT-UUID-FOR-EACH-ROOT",
"uid": 999,
"gid": 999,
"mode": "2770",
"capacityBytes": 8589934592,
"reserveBytes": 4294967296,
"minimumFreeBytes": 2147483648,
"minimumFreeInodes": 10000
}
],
"backup": {
"destinationReference": "operator-backup-vault",
"failureDomain": "independent-backup-host",
"platformRecoveryOwner": "platform-team",
"maximumAgeHours": 24
}
}The backup object declares where independent recovery copies live and who owns platform recovery; it is checked by backup verification. Capacity and reserve values must fit the actual backing filesystem, and declared capacities sharing one filesystem are summed once with a separate reserve. CPU/RAM and starting pod count remain signed license settings and do not belong in this file.
The normal preflight, install and resume commands validate the plan, and its content hash becomes part of the retained installation intent. Installation runs one read-only node probe at a time before credentials or infrastructure initialization, and every infrastructure start repeats the guard before directory preparation. A missing known PostgreSQL version file or Redis AOF manifest blocks restart instead of creating an empty replacement database. Do not clear markers, delete known claims, remove PV/PVC identity guards or recreate PVCs to get past a hold: missing or changed volume identities require investigation and the supported recovery procedure.
Retired PVCs and identity disks are not erased or silently recycled, so plan pool headroom for scale and upgrade churn; exhaustion leaves new pods Pending. Static local disks constrain placement to their prepared workers — provisioning another machine does not move those disks. Adding a plan to an already installed namespace is not an adoption or conversion. Shared NFS data and the backup staging export are not independently redundant backups, and multiple application pods do not make this single-instance infrastructure highly available.
Download the installer and Cluster license
You need two separate packages:
| Download | Contains | Does not contain |
|---|---|---|
| Managed installer/release ZIP | Installation client, charts, pinned platform assets, release descriptor and guides. References the exact images to pull. | Your license, kubeconfig or passwords. |
| Cluster license and setup ZIP | Signed environment, licensed capacity, starting configuration, execution authorization, public verification keys and the runtime-settings snapshot. | The installer, issuer private keys, cluster credentials or initial administrator password. |
The managed installation uses an environment-level Cluster license. A regular per-node .lic file or an Integration Agent .alic file is not a replacement for these signed Cluster setup files.
Download the installer ZIP for Linux AMD64 or ARM64 workers — the architecture of your target Kubernetes nodes, not of the laptop running the CLI. Both architectures use shared multi-architecture image references where the release supplies them. You do not choose a build candidate, build images or assemble a release; catalog releases are shown newest first, and older builds are available through the page's additional-build controls where provided.
New 14-day Kubernetes trial
- Open the installer request page.
- Enter your name, company and work email.
- Select My Kubernetes cluster (14-day cluster trial).
-
Choose a pod size that fits your workers:
RAM per application pod CPU cores per pod Starting pods Maximum pods 8 GiB 2 1 2 16 GiB 4 1 2 32 GiB 8 1 2 64 GiB 16 1 2 128 GiB 32 1 2 - Complete the form check and submit the request.
- Open the private email link within its validity period. Select the Linux AMD64 or ARM64 installer for your target Kubernetes workers, and download the Cluster license/setup files too. Retain the displayed SHA-256.
The trial runs 14 days, starts one application pod and permits two. Existing customers receive a separate evaluation environment, not a change to their production license. Repeating a request with a different size does not silently resize an existing trial; contact support to change it.
If the Kubernetes option is unavailable on the request page, installer-only downloads may still be available on the tools page; contact Etlworks about trial licensing. A regular trial license cannot be converted by renaming its files.
Installer only, with an existing Cluster license
Open Installer & CLI tools and use the Etlworks on Kubernetes download section for AMD64 or ARM64. This path needs no trial request form and includes no license. A valid Cluster license/setup is still required to install and run a managed cluster, and downloading tools does not install or upgrade your application.
Use the Cluster setup ZIP supplied by your Etlworks licensing administrator for the intended environment. If you have a private customer download link, it can provide both the selected release and that environment's setup files. Private links expire and are credentials: do not share them publicly, and obtain a new link through the normal route if needed.
Existing paid/customer environment
Ask the licensing administrator to supply current Cluster license/setup files for the environment. In the licensing portal, the environment's Cluster License workflow shows completed steps and the remaining action; once the saved state is ready, use Download Cluster license and setup files, or email the private download link to the person performing the installation. Existing per-node licenses are not converted or disabled by initializing Cluster licensing.
The portal's guided steps identify missing capacity, starting configuration or authorization before setup download. If signed inputs are rejected later, obtain corrected current setup files rather than editing their contents. Those portal actions are for the licensing administrator; the person operating the Kubernetes installer does not need license-server administrator access. Downloading files does not change a running deployment.
Online and offline license authorization
Online versus offline authorization is a licensing choice, not a cloud-provider choice:
- Online installations need outbound verified HTTPS access to the Etlworks license server and the separately issued renewal credential. A short connectivity failure is governed by the signed cached-authorization policy; it is not an unlimited offline license.
- Offline authorization carries its own signed validity and expiry rules. Trial policies use fixed-term offline authorization and need no renewal credential; internet access may still be needed to download the release and pull images.
Do not switch modes as a troubleshooting shortcut, and renewal does not extend a trial or commercial expiry. If authorization is rejected, the licensing administrator should obtain corrected signed setup through the portal's guided steps.
Current setup packages also contain a runtime-settings snapshot and its separate refresh credential. These deliver environment business settings — including enabled AI configuration — to every application pod; they are not a different capacity license. Each pod loads the initial verified snapshot and refreshes settings approximately hourly over verified HTTPS, independently of the elected scheduler and of the online/offline license mode; a manual license refresh also requests a settings refresh across pods. A failed refresh keeps the last verified settings and reports its state, and refreshing settings never changes licensed capacity or extends license expiry. Older setup packages still install, but cluster health reports missing runtime settings until you obtain current setup files and resume with the existing installation configuration.
Verify and extract your downloads
Use a separate working directory. Keep the release files unchanged and keep all private installation inputs outside the extracted release.
Compare the installer's SHA-256 with the value supplied through your trusted Etlworks download page/channel before extraction:
# Linux
sha256sum RELEASE.zip
# macOS
shasum -a 256 RELEASE.zipOn Windows PowerShell, use Get-FileHash -Algorithm SHA256 RELEASE.zip and Expand-Archive into a new folder. Replace RELEASE.zip with your actual installer filename. Do not use a checksum copied from an unrelated release or architecture.
mkdir etlworks-release
unzip RELEASE.zip -d etlworks-release
cd etlworks-release
python3 deployment/installer/release_archive.py verify --root .
mkdir -p ../etlworks-private
chmod 700 ../etlworks-privateExtract the separate setup ZIP into ../etlworks-private, not into the release directory. Substitute its downloaded filename:
unzip /absolute/downloads/CLUSTER-SETUP.zip -d ../etlworks-private
chmod 600 ../etlworks-private/*.jsonThe setup package contains:
| Filename | Purpose |
|---|---|
| installation-bundle.json | Signed environment binding, capacity and starting configuration. |
| license-authorization.json | Signed execution authorization and its matching grant. |
| environment-public.der | Environment identity verification key. |
| capacity-public.der | Capacity/authorization verification key. |
| bootstrap-public.der | Starting-configuration verification key. |
| runtime-settings.json, runtime-settings-credential | Current environment settings snapshot and its scoped refresh credential, in current setup packages. The installer discovers them beside installation-bundle.json — keep the extracted setup files together. |
| README.txt | Package instructions and public-key fingerprints. |
An online installation additionally uses the separately supplied renewal credential and its documented installation step; the setup ZIP alone does not contain it. Public verification keys are not the license server's private signing keys: never generate issuer private keys or copy private signing material into your cluster.
The release verifier checks inventory consistency. Do not add your kubeconfig, install.json, password, logs or backups to the release directory, or mix files from different releases. Retain the original archives and expected checksums. For a later release, its shipped verifier can also extract another archive after checking its independently obtained digest, without overwriting any existing path:
python3 deployment/installer/release_archive.py extract \
--archive /absolute/downloads/NEXT-RELEASE.zip \
--sha256 EXPECTED_SHA256 --output /absolute/releases/NEXT-RELEASEConfigure the installation
Copy the configuration example supplied with your release:
cp deployment/installer/example.json ../etlworks-private/install.json
chmod 600 ../etlworks-private/install.jsonEdit it using absolute paths for your workstation. The following shows the local profile's input shape; replace the example values with your own:
{
"schemaVersion": 1,
"kubeconfig": "/absolute/etlworks-private/kubeconfig",
"context": "YOUR_CONTEXT",
"namespace": "etlworks",
"clusterName": "etlworks",
"releaseDescriptor": "/absolute/etlworks-release/release.json",
"bundle": "/absolute/etlworks-private/installation-bundle.json",
"authorization": "/absolute/etlworks-private/license-authorization.json",
"identityKey": "/absolute/etlworks-private/environment-public.der",
"capacityKey": "/absolute/etlworks-private/capacity-public.der",
"bootstrapKey": "/absolute/etlworks-private/bootstrap-public.der",
"hostname": "etlworks.example.com",
"administrator": {
"email": "owner@example.com",
"passwordFile": "/absolute/etlworks-private/initial-password"
},
"platformProfile": "local-nfs-v1",
"storage": {
"workerA": "YOUR_FIRST_STORAGE_WORKER",
"workerB": "YOUR_SECOND_STORAGE_WORKER"
},
"postgres": { "mode": "provisioned" },
"redis": { "mode": "provisioned" }
}| Setting | How to choose it |
|---|---|
| kubeconfig, context | Exact credentials/context for the intended cluster. |
| namespace, clusterName | Stable installation names. Use lowercase letters, numbers and hyphens; choose before installation. The namespace must be new and exclusive to this installation. |
| releaseDescriptor | The release.json at the root of the verified extracted release. |
| bundle, authorization | The matching signed setup files for this environment. |
| identityKey, capacityKey, bootstrapKey | The trusted public verifier files from the setup package. |
| hostname | DNS hostname only, without https://, port or URL path. It must match your chosen access path/certificate. |
| administrator.email | Email for the initial administrator account. |
| administrator.passwordFile | Protected file containing only the initial password. Omit this setting to use a hidden interactive prompt in a suitable terminal. |
| platformProfile | One of the four platform profiles. Omitting it selects local-nfs-v1. |
| storage.workerA, storage.workerB | Actual Linux node names from kubectl get nodes, matching the release CPU architecture. These are storage bindings, not a guarantee of application pod placement. Optionally add storage.retainedPlan for dedicated prepared disks (see retained disk plans). A cloud profile instead uses storage.cloudPlan; do not combine it with workerA/workerB or retainedPlan. |
| postgres, redis | Independent choices between provisioned (the default) and external; see external PostgreSQL and Redis. |
Create the password file with a local editor/password manager and protect it:
chmod 600 ../etlworks-private/initial-password
chmod 600 ../etlworks-private/kubeconfigThe initial password must be 8–32 characters, include a letter and a digit, and use only letters, digits, _, !, @, or #. Do not put a password literal in install.json or a shell command. SMTP is not needed to create the initial account: the initial administrator is established before public application admission.
Do not add sizing keys such as replicas, CPU, RAM or heap to install.json. The installer verifies the signed starting configuration and rejects unsupported fields. The resource profile named by your Cluster license is a sizing selection — pod CPU/RAM and counts, under whatever name your license uses — not a platform or cloud selector; the same license sizing applies whichever platform profile you install. Contact Etlworks if the supplied license does not match your release.
If registry authentication is needed, the configuration can refer to existing namespace-local image-pull Secret names through imagePullSecrets. Arrange their creation with the platform administrator; never put registry passwords in this JSON. Public images do not require a private repository credential, although registry rate limits and network policy can still affect pulls.
External PostgreSQL and Redis
On every platform profile, PostgreSQL and Redis are independent choices between provisioned and external, under the same archive and license — these are service modes, not extra profiles, and no license-portal or website control is involved. Provisioned means installer-managed containers inside Kubernetes on retained storage; the databases remain persistent infrastructure, not disposable pod-local databases. For a provisioned dependency, use only {"mode":"provisioned"} or omit the service object — there is no automatic or fallback mode. For external services, replace the service objects:
"postgres": {
"mode": "external",
"host": "postgres.example.internal",
"port": 5432,
"database": "etlworks",
"credentialSecret": "customer-postgres-login",
"trustSecret": "customer-postgres-ca",
"connectionBudget": 80
},
"redis": {
"mode": "external",
"host": "redis.example.internal",
"port": 6379,
"databaseIndex": 0,
"credentialSecret": "customer-redis-login",
"trustSecret": "customer-redis-ca"
}The connectionBudget (80 is illustrative) is the connection capacity you reserve for Etlworks on your server — the application's reserved share, not the server's entire connection maximum. The installer derives bounded application and engine pools from the signed licensed maximum, surge and a failed-incarnation reserve, and rejects an insufficient assigned budget. Reserve the budget explicitly so it accommodates the licensed maximum, surge and maintenance; do not plan around raising server connection limits indefinitely or killing sessions after crashes.
External PostgreSQL requirements: PostgreSQL 17, one writable primary endpoint and an Etlworks-dedicated database owned by its login role, using the ordinary application schemas public and history owned by that login (PostgreSQL's standard pg_database_owner ownership of public is accepted when the login owns the database). Other non-system schemas or foreign schema owners are refused. The role needs database connect/create/temp and ownership of its application objects; server superuser, database creation, server configuration changes and session termination are not required. Other clients of the dedicated database must be stopped during bootstrap and offline recovery.
External Redis requirements: Redis 7, a dedicated instance with ordinary single-endpoint semantics and database 0 only — a nonzero index, Sentinel or sharded Cluster is rejected. Use AOF with appendfsync always and maxmemory-policy noeviction. ACLs for the supported default user must permit actual Etlworks coordination, all-key logical backup and offline recovery, including PING, INFO server, CONFIG GET, CLIENT ID/LIST, SCAN, DUMP, expiry inspection, logical data and Lua operations, and database-scoped FLUSHDB/RESTORE — a read-only Redis user is insufficient. No FLUSHALL, server restart or persistence repair is used. Other database-0 clients block the offline helper.
Both services require verified TLS with a server certificate matching the configured DNS hostname. The application keeps TLS verification on: a TLS error, missing Secret, wrong password, hostname or CA, or an unavailable endpoint holds startup, and the installer does not create a local replacement or overwrite supplied credentials. A managed database product with restricted commands or different endpoint semantics does not meet this contract merely because its version matches.
Credentials live in namespace-local Kubernetes Secrets that you create at the installation checkpoint: a login Secret (PostgreSQL: username and password keys; Redis: password, used with the default ACL user) and a trust Secret with a currently valid public CA certificate in ca.crt. Secret names are at most 54 characters with separate login/trust names; names starting with etlworks- or infra- are reserved. Keep passwords and private keys out of install.json, shell history, license documents and source control.
Endpoint, port, database/index, assigned budget and Secret references become retained installation intent, and supplied Secret UIDs are recorded. Changing the service mode or endpoint during resume is rejected; moving an installed database to or from an external service is a migration/cutover, not a mode toggle. Rotating a supplied login needs inspected runtime-secret reconciliation — changing the server credential while runtime credentials are stale results in a hold. External service maintenance, backups, credentials and provider availability remain your administrators' responsibilities.
Install and check progress
Run from the verified release root. Review preflight output first:
sh deployment/installer/etlworks-kubernetes.sh preflight \
--config ../etlworks-private/install.json --jsonPreflight validates inputs, signatures, tools, cluster version and selected nodes. It is not proof that every storage, certificate, image pull or runtime startup step will succeed. Resolve reported prerequisites before proceeding.
Start installation:
sh deployment/installer/etlworks-kubernetes.sh install \
--config ../etlworks-private/install.json --wait 30m --jsonThe installer records the installation intent, prepares compatible platform dependencies, retained storage and credentials, starts the provisioned infrastructure, bootstraps the first application member, completes administrator onboarding and reconciles the licensed starting fleet. Installation advances through retained checkpoints — credentials, storage, dependencies, first application initialization and the remaining licensed starting members — and short-lived bootstrap Jobs are expected. Application pods and the adapter remain running afterward.
Check progress with the same configuration:
sh deployment/installer/etlworks-kubernetes.sh status \
--config ../etlworks-private/install.json --jsonInspect state, phase, reason and the actual counts. Do not equate a Running pod or a completed Helm command with successful Etlworks installation; wait for confirmed installation/admitting capacity and successful onboarding.
| Exit code | Meaning |
|---|---|
| 0 | The command completed; for installation, the initial fleet was confirmed. Read the command's reported state. |
| 2 | Progress is safely pending; inspect status and resume the same installation. |
| 3 | The operation is blocked; resolve the reported condition. |
| 1 | Invalid input or an unconfirmed/error condition; inspect the error before retrying. |
A --wait value limits the client's wait, not the lifetime of running executions or the authority of a blocked operation: a wait that ends is not a failed or canceled installation. On Windows, run deployment/installer/etlworks-kubernetes.ps1 with the same subcommands and options and PowerShell paths.
If the native Etlworks CLI is already installed on the workstation, it can delegate to this same client — point it at the matching extracted release first:
export ETLWORKS_MANAGED_HOME=/absolute/etlworks-release
./etlworks-cli.sh kubernetes status --config /absolute/etlworks-private/install.json --jsonThe standalone release launcher is sufficient; installing the native host CLI is not a prerequisite. PowerShell uses etlworks-cli.ps1 kubernetes ... and $env:ETLWORKS_MANAGED_HOME.
External-service Secrets: the installation checkpoint
When either service is external, the install sequence gains one checkpoint. Do not pre-create the namespace — the installer refuses to adopt one. Let the installer create its namespace and record the installation intent first, using a short wait:
sh deployment/installer/etlworks-kubernetes.sh install \
--config ../etlworks-private/install.json --wait 5s --jsonA pending checkpoint or a specific missing-Secret hold is expected until the supplied Secrets exist; resolve any other reported hold first. Then create the named Opaque Secrets in the installer-owned namespace through your platform's protected secret workflow, for example:
kubectl --kubeconfig /absolute/etlworks-private/kubeconfig --context CONTEXT -n NAMESPACE \
create secret generic customer-postgres-login \
--from-file=username=/protected/pg-username --from-file=password=/protected/pg-password
kubectl --kubeconfig /absolute/etlworks-private/kubeconfig --context CONTEXT -n NAMESPACE \
create secret generic customer-postgres-ca --from-file=ca.crt=/protected/pg-ca.crt
kubectl --kubeconfig /absolute/etlworks-private/kubeconfig --context CONTEXT -n NAMESPACE \
create secret generic customer-redis-login --from-file=password=/protected/redis-password
kubectl --kubeconfig /absolute/etlworks-private/kubeconfig --context CONTEXT -n NAMESPACE \
create secret generic customer-redis-ca --from-file=ca.crt=/protected/redis-ca.crtCreate only the Secrets for services you configured as external, with the names from your install.json. The credential files must not contain trailing newlines. Verify the service DNS names against the server certificates. Then resume with the same configuration and signed inputs:
sh deployment/installer/etlworks-kubernetes.sh resume \
--config ../etlworks-private/install.json --wait 30m --jsonA fully provisioned installation does not stop at this checkpoint. Resume cannot silently change a service mode, endpoint or retained Secret identity. Never replace the retained intent or start a new installation to get past a hold.
Resume an interrupted installation
If the terminal disconnects, the workstation restarts or the client wait expires, retain the same release directory, configuration, license inputs, namespace and cluster. First inspect status, then resume:
sh deployment/installer/etlworks-kubernetes.sh status \
--config ../etlworks-private/install.json --json
sh deployment/installer/etlworks-kubernetes.sh resume \
--config ../etlworks-private/install.json --wait 30m --jsonThe installation's retained intent and receipts are in Kubernetes, not only on your workstation. Resume reconciles the same operation rather than starting a second fleet or resetting administrator credentials, and an unknown command acceptance is reconciled rather than repeated. A concurrent or interrupted installer lease may need to expire before another client can proceed.
If blocked, fix the reported dependency and resume. Do not delete Secrets, installation receipts, member operations, PVCs or the namespace to clear a hold. Do not change cluster/context, storage worker bindings, hostname or release as a resume shortcut. Missing or changed identities require investigation/recovery. There is no automatic uninstall/reset/force command that safely discards uncertain installation state.
If signed inputs expired, obtain current setup files and follow the reported licensing/recovery path. Once a first pod is allocated, arbitrary replacement of its bootstrap files is not an accepted renewal mechanism.
Open Etlworks and finish setup
Use the configured HTTPS hostname after installation reports readiness. Ensure DNS/client routing and certificate trust are correct for that hostname. A status URL does not mean the installer configured your public DNS or ingress.
For local evaluation, forward the proxy Service. Keep the command running:
kubectl --kubeconfig /absolute/etlworks-private/kubeconfig --context CONTEXT \
-n NAMESPACE port-forward service/CLUSTER_NAME-proxy 8443:443For this access path, map the configured hostname to 127.0.0.1 on your workstation and open https://CONFIGURED_HOSTNAME:8443/. Use your operating system's approved hosts/DNS and certificate-trust procedure. The local frontend certificate is signed by the installation's internal CA; obtain and verify that public CA certificate with the administrator. In this profile it is available as ca.crt in the etlworks-infra-trust Secret; an authorized administrator can export only that public field. Do not export the entire Secret, its private keys or application credentials. Keep TLS verification enabled; do not bypass certificate warnings.
Port forwarding is a workstation access method, not a public ingress or HA endpoint. Other users and Integration Agents need a stable reachable address.
On a cloud profile, the installation instead creates a TCP 443 load-balancer Service named CLUSTER_NAME-public (internal by default). Point application DNS at its advertised endpoint and establish frontend certificate trust separately; see network access in the cloud guide.
Sign in using the administratorLogin reported by installation status (normally admin for a fresh installation) and the password you supplied. The installer completes its one-time account setup before exposing normal traffic. Resume does not reset a password changed after successful onboarding, and a migration-based installation preserves the imported administrator instead of creating a fresh one.
Finish the application configuration:
- Confirm application/engine version and health, then run a representative test flow.
- Set Settings → General → Home URL to the stable public URL users and Integration Agents will actually reach, including a nondefault port if applicable. Every Agent host/container must be able to resolve and connect to it.
- Configure email settings for later invitations, password recovery and notifications.
- Review Settings → Elastic cluster and configure automatic scaling within the license bounds.
- Enable appropriate metrics/alerts, retained logs and independent backups before relying on the installation for important workloads.
Each application pod invokes its own embedded engine locally. Do not route ETL Engine Proxy URL through the public HAProxy or configure the managed pods as an unmanaged engine farm: the normal managed path needs no custom legacy engine proxy URL. Verify the current Network settings before changing any value there, and change networking settings only through the documented profile or with Etlworks guidance.
Configure automatic scaling
A SuperAdmin manages cluster capacity in Settings → Elastic cluster. The page has four tabs: Scaling settings, Pod size, Upgrades and Change history. On the Scaling settings tab:
- Read Cluster status at the top: the status badge, Running app pods (with starting/draining counts), Requested pods, Licensed maximum and Resources per pod. Open Configuration and status details for the saved/applied/license revisions, and use Refresh status for a fresh read.
- Choose your pod range: set Minimum pods and Maximum pods. The maximum cannot exceed the licensed capacity. Equal values keep a fixed application pod count.
- Set the load targets: CPU target, RAM target and JVM heap target, each a percentage. The three targets have equal standing — CPU is not a privileged leading metric. The RAM target compares total pod container memory with the pod limit; the heap target compares used JVM heap with the maximum JVM heap. Optionally open Scaling speed and stabilization to adjust Add up to / Remove up to (pods per minute) and the Scale-up stabilization / Scale-down stabilization periods.
- Plan extra capacity (optional): under Recurring schedules, add schedules with a Schedule name, Time zone, Repeat (Monday–Friday, Every day, Saturday–Sunday or a custom cron schedule), start/end times and a Minimum pods floor. Overlapping windows use the largest required minimum. These are capacity schedules — they keep pods ready; they do not schedule flows.
- Optionally open Temporary minimum and enable Keep extra pods ready temporarily: choose Keep at least and a Duration (1/4/8/24 hours, or an end time within the next 24 hours). The editor shows your local time and stores UTC. This is a temporary floor, not a fixed replica override — demand can still grow the fleet above it, up to the maximum.
- Review and save: optionally add a Change note, then click Save scaling settings.
- Use the Change history tab to review saved revisions (Scaling change history): each row shows the revision, when it was saved, who changed it and the note, and Review these settings loads old values into the editor for review. Old values take effect only when saved as a new revision.
How scaling behaves:
- Each resource target produces an independent recommendation. Any one of them can request growth; reducing capacity requires that the lower count satisfy all relevant metrics, the schedule/temporary floors and the configured stabilization periods.
- Missing or stale metrics never authorize removal — unavailable observations hold scale-in rather than counting as zero load.
- An added pod receives future eligible executions; it cannot split a flow already running on another pod. Memory-based scaling adds pods; it does not expand a running JVM.
- For planned scale-in, new work is withdrawn from the selected pod and existing work is accounted for. Ordinary executions finish. Eligible healthy continuous executions may be stopped and relocated through the managed lifecycle, and a CDC snapshot in progress can delay that step. Actual work must stop before removal: elapsed time, a canceled audit row or an expired lease is not proof, and there is no force-kill after a grace period.
- Kubernetes replacing an unavailable or crashed pod is process recovery; it is not an automatic restart or replay of the business flows that were running there.
On a regular (non-cluster) installation, the same page runs as a permanent Interactive demo (banner: Explore managed clusters) with sample data. Demo changes stay on the page — they do not scale, upgrade or resize that installation. To use the actual capability, obtain a Cluster license and install the managed cluster as described in this guide.
Understand pod and node scaling
The Cluster license sets hard capacity limits, allowed pod sizes and the starting fleet. The scaling settings govern demand-based behavior within those bounds. Four concepts stay distinct:
- Starting count/size — signed license values used at first installation.
- Runtime scaling range — the minimum/maximum and targets you configure in the application.
- Pod size — CPU/RAM per pod, changed only by the pod size replacement operation.
- Worker machines — VM size, node count, node groups and any node autoscaler, configured by your platform administrator outside Etlworks.
KEDA/HPA requests a desired application count; the lifecycle adapter creates pods or drains selected pods safely; Kubernetes places pods where requested CPU/RAM and storage allow. HPA does not directly delete a busy application pod, so the actual count can remain above the desired count while a drain is blocked by real work.
With fixed workers, additional application pods may remain Pending until capacity is provided. Reducing application pods does not delete worker machines. A cloud node autoscaler, where your platform administrator configures one, can add or remove machines separately — node autoscaling is not inherently limited to EKS/AKS/GKE, but it requires an appropriate platform integration, and these Etlworks profiles do not enable one. Voluntary node disruption (consolidation, node upgrades, maintenance) must honor Etlworks drain protection; PodDisruptionBudgets and eviction hints are not a guarantee against forced provider shutdown, so coordinate node maintenance with application drain rather than enabling blanket nonempty-node consolidation.
Monitor health, resources and cluster status
System Health shows each current application pod individually — CPU limit and utilization, container RAM, JVM heap used/maximum and JVM threads — under Application pods, and the shared PostgreSQL, Redis, shared application data and system filesystem once under Shared services and storage. Do not read a worker's RAM as each pod's allocation. An unknown shared-storage capacity remains unknown; it is not a usable disk quota.
Resource Usage uses Pod terminology on a managed cluster (Node on regular installations). The selector offers All pods plus the current pods only; historical measurements remain in the charts for troubleshooting, so a past pod's identifier in a chart or peak card is history, not another running pod. A draining pod remains current until it is retired. With All pods selected, the charts present averages across members — not summed capacity and not the hottest member — while the peak cards (Peak CPU, Peak memory, Lowest system volume available) identify the individual pod and time of each extreme observation.
The cluster status badge on the Scaling settings tab distinguishes Up to date, Applying changes, Autoscaling not ready, Needs attention, Status out of date, Status unavailable, Setup needed and Changing pod size. Saved settings are not proof that fresh KEDA/HPA observations are healthy: when the badge is not green, open Configuration and status details and read the reported reason instead of repeatedly saving or refreshing — a browser refresh cannot repair a backend or controller hold.
Execution and cluster state updates are pushed to the browser over the application's existing notification transport (long polling, streaming or upgraded WebSocket connections). Proxies and load balancers in front of the application must allow that long-lived traffic; no separate WebSocket-only service is required. Flow started/finished events produce one toast per event in current releases, and every event still updates the execution state. If flow status only updates after a page refresh, see troubleshooting — do not restart or re-run a business flow just to refresh the screen.
In the license server, the environment's Pods / members view reports current members with their state, resources, app/engine versions, process starts, health links and login history. It is passive reporting fed by the installation about once a minute — not a direct Kubernetes inventory. A report goes stale after a few minutes; a stale row alone proves neither a healthy nor a stopped pod. Do not add per-node licenses to make Cluster members appear there.
View pods and Tomcat logs
Use the release client:
sh deployment/installer/etlworks-kubernetes.sh pods \
--config ../etlworks-private/install.json
sh deployment/installer/etlworks-kubernetes.sh pods \
--config ../etlworks-private/install.json --json
sh deployment/installer/etlworks-kubernetes.sh logs \
--config ../etlworks-private/install.json --pod APP_POD --tail 200 --follow
sh deployment/installer/etlworks-kubernetes.sh logs \
--config ../etlworks-private/install.json --pod APP_POD --previous
sh deployment/installer/etlworks-kubernetes.sh logs \
--config ../etlworks-private/install.json --pod CONTROLLER_POD \
--container controller --tail 200 --followThe application container is app-engine, and Tomcat's console output is that container's output. pods lists all pods in the namespace, including the lifecycle controller, HAProxy and infrastructure components — discover their actual pod and container names from that listing, and select a different component's container with --container. Defaults are the last 200 lines with timestamps; --tail -1 requests all available lines. --follow streams until Ctrl+C, staying with the named pod — select a replacement pod explicitly. --previous selects the previous container instance within an existing pod; it does not retrieve deleted-pod history, so arrange central log collection to retain logs beyond pod lifetime. These commands need only Kubernetes read permissions, not an Etlworks login.
Direct kubectl is also available:
kubectl --kubeconfig /absolute/etlworks-private/kubeconfig --context CONTEXT \
-n NAMESPACE get pods -o wide
kubectl --kubeconfig /absolute/etlworks-private/kubeconfig --context CONTEXT \
-n NAMESPACE logs APP_POD -c app-engine --tail=200 --follow
kubectl --kubeconfig /absolute/etlworks-private/kubeconfig --context CONTEXT \
-n NAMESPACE get events --sort-by=.metadata.creationTimestampCreate a bounded, read-only support snapshot outside the release directory:
sh deployment/installer/etlworks-kubernetes.sh diagnostics \
--config ../etlworks-private/install.json \
--output ../etlworks-private/support.zipThe diagnostics ZIP contains typed public status — not Secrets, kubeconfigs, raw plans or unrestricted logs — but it can still include sensitive configuration detail and paths, so share it only through the protected support channel. When contacting support, retain and provide the release identity, profile/architecture, the relevant request IDs, status/events and the first error — never plaintext passwords or private signing material.
Upgrade the application
Application upgrades are durable managed operations with two modes:
- Rolling — compatible release pairs replace pods one at a time through safe drain, keeping the required receiving capacity. Busy ordinary work can delay a replacement, and a bounded licensed upgrade reservation may be needed even at the normal maximum.
- Maintenance — for pairs that require an outage: the operation closes new execution admission and scheduling, settles work, stops the old application processes, creates and verifies a complete offline recovery package, starts the permitted target, runs the app-driven schema migration, restores the target fleet and reopens only after the required checks. The application UI can be unavailable during the outage; the CLI remains the control path.
An application upgrade updates the application and embedded engine and retains the operational count, size, policy and data intent. It is not a rebalance and not an infrastructure upgrade: Kubernetes, the lifecycle controller and CRDs, PostgreSQL/Redis server binaries, CNI/CSI, KEDA and cert-manager require separately planned compatible maintenance.
Make releases available
The platform operator registers the exact release pair in the installation's release catalog and image allowlist through the supported release process shipped with the release. Publishing an image to a registry does not by itself register it in an existing installation, and two builds with the same version label are not interchangeable — the current installed identity must match the selected source exactly. Schema migration authority comes from the registered pair, not from a version string.
Upgrade from the application
- As a SuperAdmin, open Settings → Elastic cluster → Upgrades.
- Under Select releases, choose the Current release (it must match the application actually running in this cluster) and the Upgrade to target. Only the newest build of each version is shown, newest first; More builds exposes earlier registered builds when the installed build is older.
- Click Review upgrade plan. The plan establishes the upgrade mode (Rolling or Maintenance) and verifies compatibility, schema/shared formats and receiving capacity.
- For a maintenance upgrade, tick I acknowledge the maintenance outage.
- Click Request upgrade, then monitor the operation on the same tab: its phase, reason and last-observed time, with Pause and Resume where offered. Open Request and diagnostic details for the request ID, and record that ID.
The backend records a durable operation for the lifecycle adapter. It does not run the CLI, hold a human kubeconfig or install an updater service in application pods.
Upgrade from the CLI
The CLI addresses the same durable operation using Kubernetes RBAC — no Etlworks account or SuperAdmin login:
sh deployment/installer/etlworks-kubernetes.sh update plan \
--config ../etlworks-private/install.json \
--source SOURCE_MANIFEST_DIGEST --target TARGET_MANIFEST_DIGEST --json
sh deployment/installer/etlworks-kubernetes.sh update start \
--config ../etlworks-private/install.json --request REQUEST_UUID \
--source SOURCE_MANIFEST_DIGEST --target TARGET_MANIFEST_DIGEST \
--plan-digest RETURNED_PLAN_DIGEST --json
sh deployment/installer/etlworks-kubernetes.sh update status \
--config ../etlworks-private/install.json --request REQUEST_UUID --json
sh deployment/installer/etlworks-kubernetes.sh update pause \
--config ../etlworks-private/install.json --request REQUEST_UUID --json
sh deployment/installer/etlworks-kubernetes.sh update resume \
--config ../etlworks-private/install.json --request REQUEST_UUID --jsonWhen the plan reports that an outage is required, update start additionally requires --acknowledge-outage. Manifest digests come from the registered releases and the plan digest from the returned plan — not from a version label, the container tag or an invented hash. Generate the request UUID once and retain it: acceptance means requested, not finished, and an interrupted or unknown reply is reconciled by inspecting and resuming the same request rather than creating another.
During and after an upgrade
- The first target pod receives the migration permit and runs the app-driven schema migration; ordinary managed startups only validate schema. Do not run SQL or schema-repair tooling, edit schema history, or use a separate migration job as a default procedure.
- Pause stops new source selections; an already committed safe drain continues. There is no automatic image or database rollback and no healthy-flow kill deadline.
- Status can retain a historical completed result independently of current connectivity; "Status needs verification" and similar freshness labels mean read the actual state, not that the operation failed.
- After a completed upgrade, use the release distribution matching the installed version for backup and recovery; the original installation's release is historical.
A failed target or an unknown migration result requires the specific recovery procedure — see recover a failed upgrade — not switching an old image onto a potentially newer schema.
Update the Java runtime
The Java runtime ships inside the managed release's application image, and the managed release owns the JVM settings inside the application pods: updating Java is an ordinary application upgrade to a release whose image contains the patched JVM. The standard installer does not expose a general add-Java-argument command. The steps below use the interim context of CVE-2026-114938 (Java4Shell) in Etlworks; the mechanics apply to any JVM patch.
- Inventory the application pods, their actual worker architectures, and their running Etlworks image digests, using the target kubeconfig and context. Repeat the Java check on each application pod:
Reading node information and executing a Java version check require their respective Kubernetes permissions. When assessing exposure on x86-64 workers, determine whether the actual worker exposes AVX-512 VBMI to the workload; the architecture of the operator's workstation is irrelevant to runtime exposure.kubectl --context CONTEXT get nodes -L kubernetes.io/arch kubectl --context CONTEXT -n NAMESPACE get pods -o wide kubectl --context CONTEXT -n NAMESPACE get pod APP_POD \ -o jsonpath='{.status.containerStatuses[?(@.name=="app-engine")].imageID}' kubectl --context CONTEXT -n NAMESPACE exec APP_POD -c app-engine -- java -version - Obtain the matched managed release for the deployment's architecture and confirm that its application image contains the patched JVM, using the verified release descriptor and image reference. A downstream product build can reuse a digest-pinned Java base, so inspect that release's actual runtime, not the base image's label. Register the release pair through the existing release-registration procedure for the exact source/target pair, record the installed build, and have the documented recovery package or backup plan appropriate to the upgrade mode.
- From the UI, a SuperAdmin opens Settings → Elastic cluster → Upgrades, selects the exact installed source and the patched target, clicks Review upgrade plan, verifies the required receiving capacity and outage requirements, and requests the upgrade, exactly as in Upgrade from the application. A maintenance plan requires acknowledgment of the outage. Record the request ID.
- From the CLI, a platform administrator with the required Kubernetes RBAC runs the same managed operation with the complete CLI steps above, using the original kubeconfig/context and the exact config path — update plan, then update start (add --acknowledge-outage when the plan requires it and the outage is approved), then update status, preserving the same request ID throughout. A pending request is an operation in progress.
- Wait for managed completion, check health, repeat the Java and image checks of step 1 on every current application pod, and verify a flow. For a held or failed upgrade, follow the existing status and recovery steps for that request. Preserve backup and status evidence.
- Do not edit individual pods, replace an image outside the managed operation, add arbitrary JAVA_TOOL_OPTIONS, or patch the worker nodes' JDK as a substitute — the managed release owns the Java arguments in the application pods. If a verified patched release for your deployment is unavailable, contact Etlworks Support for an approved path.
Change the application pod size
Changing CPU/RAM per pod is a distinct operation from adding pods (scaling) and from resizing worker machines (platform administration). Authorizing a new size in the license server does not resize running pods, and the signed initial bootstrap values are a record of the original installation, not a runtime resize command.
Pod size replacement is available on Etlworks 10.0.16 and newer installations whose application, installer and lifecycle-controller releases were installed or upgraded together with the resize capability enabled. On an installation without the matching enabled capability, the cluster administrator must supply those platform components first — upgrading only the application image or WAR does not add Kubernetes CRDs, RBAC or controller configuration.
The flow:
- The licensing administrator authorizes the target resource profile and sufficient aggregate/overlap capacity, keeping the current profile licensed until the replacement finishes.
- Apply the updated signed capacity/authorization through the supported workflow (below) and refresh until the target profile is visible. Do not edit signed tokens or Kubernetes resource limits to work around an entitlement.
- The platform administrator verifies worker CPU/RAM, placement and storage capacity for the target size. This step does not enlarge a VM and does not start a node autoscaler.
- As a SuperAdmin, open Settings → Elastic cluster → Pod size. Read the Applied size, choose a New pod size and a Replacement strategy, then click Review size change. Check the target CPU, RAM, maximum heap, final pod count and licensed ceiling, then click Start pod size change.
- Monitor the same request on that tab; Pause and Resume are available, and only an unclaimed pending request can be canceled. Mixed sizes can exist while the operation progresses.
The two strategies:
- Keep capacity during replacement (default) — the controller creates and verifies one replacement before draining its source, which requires room for one extra pod in both the license and the cluster. A replacement that cannot start leaves the source running.
- Allow one fewer pod temporarily — requires at least two healthy pods and the explicit acknowledgement I understand that the cluster will temporarily run with one fewer application pod. A source retires before its replacement starts, so a failed or pending replacement can leave reduced capacity until resolved. The last receiving pod is never shut down.
The CLI uses the same durable operation with Kubernetes RBAC:
sh deployment/installer/etlworks-kubernetes.sh resize plan \
--config ../etlworks-private/install.json --profile AUTHORIZED_PROFILE --json
sh deployment/installer/etlworks-kubernetes.sh resize start \
--config ../etlworks-private/install.json --profile AUTHORIZED_PROFILE \
--request REQUEST_UUID --plan-digest RETURNED_PLAN_DIGEST --json
sh deployment/installer/etlworks-kubernetes.sh resize status \
--config ../etlworks-private/install.json --request REQUEST_UUID --json
sh deployment/installer/etlworks-kubernetes.sh resize pause \
--config ../etlworks-private/install.json --request REQUEST_UUID --json
sh deployment/installer/etlworks-kubernetes.sh resize resume \
--config ../etlworks-private/install.json --request REQUEST_UUID --jsonFor the reduced-capacity strategy, add --strategy reduce-first and --acknowledge-reduced-capacity to both plan and start. A changed fleet, license, policy or placement invalidates an unstarted plan — review a fresh plan. If a start response is lost, inspect and retry the same UUID and digest; do not generate a new UUID to resolve an uncertain result.
If the licensing administrator supplied a signed capacity grant and matching authorization as separate files, the plan command can import them first:
sh deployment/installer/etlworks-kubernetes.sh resize plan \
--config ../etlworks-private/install.json --profile AUTHORIZED_PROFILE \
--capacity-grant /absolute/etlworks-private/capacity-grant.txt \
--authorization /absolute/etlworks-private/authorization.txt --jsonUse protected files containing the signed token strings — never token literals on a command line. A RESIZE_CAPACITY_IMPORT_PENDING_REPEAT_PLAN response means repeat the same command after the controller processes the import; a rejected import's reason is published in the etlworks-resize-capacity-result ConfigMap, and an administrator removes the rejected etlworks-resize-capacity Secret before submitting corrected inputs. The import must preserve the license mode and expiry and continue authorizing the current size; it does not modify the license server or extend a license.
The operation replaces pods one at a time, verifies each replacement's actual CPU/RAM and heap against the bound reservation, updates the calculated maximum heap, and preserves the application version, desired count and scaling policy — normal scaling resumes after the durable applied size is confirmed. A running JVM is not resized in place. Ordinary work finishes and eligible continuous work follows planned relocation; the operation does not force-delete busy pods, change Kubernetes node sizes, or roll back automatically after a partial replacement. Upgrades, backup/recovery and conflicting scaling edits cannot run concurrently with a size change. On a regular installation, the Pod size tab runs as an interactive demo with sample data.
Back up, restore and migrate
Pick the operation that matches the need:
| Need | Correct path |
|---|---|
| Diagnose a problem | Status, logs, events and the diagnostics ZIP — not a recovery checkpoint. |
| Consistent complete backup | Stop and settle all writers under exclusive maintenance, capture the full archive, retain an independent protected copy (below). |
| Restore the same installation | Inspect, explicit destructive restore, verify, then reconcile authority and reopening (below). Data restored does not mean service reopened. |
| Recover a failed maintenance upgrade before reopening | Upgrade-request-specific restore, then explicit source reactivation (below). |
| Move between a regular installation and a managed cluster | Explicit migration plan, separate valid destination license/configuration, fenced source and reviewed cutover (below). |
| Rebuild an arbitrary lost cluster or provider | Separate platform disaster-recovery planning. A same-installation restore is not an automatic clean-cluster rebuild. |
Offline backup and held restore
The complete archive includes the dedicated PostgreSQL database, all logical Redis contents with expiries, the full shared application data and protected installation metadata/credentials — including when PostgreSQL or Redis is external. It is sensitive and not encrypted: store independent copies in encrypted, access-controlled storage. The NFS /backups staging export is separate from application data (so a backup cannot recursively capture its own output) but shares the storage failure domain — it is staging, not a disaster-recovery copy.
An offline backup or restore requires exclusive maintenance control. The transport verifies these conditions; it does not stop anything itself:
- Settle executions and owned work through the normal application barriers, then stop the application, proxy and metrics workloads and the adapter's reconciliation.
- Pause KEDA and remove its HPA, suspend scheduled/Job writers, and exclude external clients and GitOps/automation restarts for the window.
- Keep the required PostgreSQL, Redis and storage infrastructure running. Stop other writers on supplied external services too — Etlworks never terminates their sessions.
Use the exact sequence in the operations and recovery instructions shipped with your release. Then:
sh deployment/installer/etlworks-kubernetes.sh backup \
--config ../etlworks-private/install.json \
--archive /absolute/backups/checkpoint.zip
sh deployment/installer/etlworks-kubernetes.sh restore inspect \
--config ../etlworks-private/install.json \
--archive /absolute/backups/checkpoint.zip
sh deployment/installer/etlworks-kubernetes.sh restore start \
--config ../etlworks-private/install.json \
--archive /absolute/backups/checkpoint.zip
sh deployment/installer/etlworks-kubernetes.sh restore status \
--config ../etlworks-private/install.json
sh deployment/installer/etlworks-kubernetes.sh restore resume \
--config ../etlworks-private/install.json \
--archive /absolute/backups/checkpoint.ziprestore inspect only reads. restore start is destructive: it replaces the selected installation's PostgreSQL database, dedicated Redis database and full application data. A normal restore requires the same namespace, cluster, installation, PVC identities, environment, credentials and exact release image — after a completed upgrade, back up and restore with the currently installed release, not the original installer. Restore workers must actually terminate before another offline operation: if a helper pod remains after a lost connection, inspect that exact pod, confirm its command is no longer active and let it terminate normally — never force-remove an unreachable possible writer, and the tool refuses to adopt an unknown live helper.
A successful restore finishes as DATA_RESTORED_RECONCILIATION_REQUIRED with execution still held: restored Redis contents are data and evidence, not permission to revive old leases, replay due queues or retry uncertain executions. Reopening requires the separately reviewed ownership reconciliation, and the recovery fence (etlworks-recovery ConfigMap and the hold marker in application data) must not be removed to bypass it.
Recover a failed upgrade
For an interrupted rolling or maintenance upgrade, first inspect and resume the same request with update status / update resume. Two further operations cover specific failures:
Rolling: a failed migration target before business admission. When the registered release pair supports migration-target recovery, use the failed target and owner revision shown by update status:
sh deployment/installer/etlworks-kubernetes.sh update recover-target \
--config ../etlworks-private/install.json \
--request UPGRADE_UUID --command COMMAND_UUID --expected-owner-revision NGenerate one command UUID and retain it for retries after an uncertain response — ordinary update resume grants no authority to replace a migration pod. In the application UI, Recover migration target submits the same request (SuperAdmin). Recovery waits for actual container termination and verified database/session evidence before creating a replacement with the same immutable target image; no timeout force-removes a member or repairs database history.
Maintenance: a failure before business reopening. Restore the verified complete recovery package against that specific upgrade request, using the release whose application image is the original source image:
sh deployment/installer/etlworks-kubernetes.sh restore inspect \
--config ../etlworks-private/install.json \
--archive /absolute/backups/checkpoint.zip --upgrade-request REQUEST_UUID
sh deployment/installer/etlworks-kubernetes.sh restore start \
--config ../etlworks-private/install.json \
--archive /absolute/backups/checkpoint.zip --upgrade-request REQUEST_UUID
sh deployment/installer/etlworks-kubernetes.sh restore status \
--config ../etlworks-private/install.json --upgrade-request REQUEST_UUIDThis path pauses the selected upgrade, stops only positively observed targets and known proxy/metrics workloads, and finishes held as DATA_RESTORED_RECONCILIATION_REQUIRED. It does not complete the failed upgrade or reopen service. Reopening on the original source is a separate, explicit continuation of that same recovery:
sh deployment/installer/etlworks-kubernetes.sh restore reactivate inspect \
--config ../etlworks-private/install.json \
--archive /absolute/backups/checkpoint.zip --upgrade-request REQUEST_UUID
sh deployment/installer/etlworks-kubernetes.sh restore reactivate start \
--config ../etlworks-private/install.json \
--archive /absolute/backups/checkpoint.zip --upgrade-request REQUEST_UUID \
--recovery-operation RECOVERY_UUID
sh deployment/installer/etlworks-kubernetes.sh restore reactivate status \
--config ../etlworks-private/install.json \
--upgrade-request REQUEST_UUID --recovery-operation RECOVERY_UUIDUse the recovery UUID reported by the held restore; after an interrupted start, use restore reactivate resume with the same UUIDs and unchanged archive. Source reactivation requires a source release that supports it — the capability is checked before a supported maintenance upgrade closes source admission — and keeps business admission closed until the durable reopening decision. Successful recovery records RECOVERED_TO_SOURCE for the failed upgrade; it never reports that the target upgrade succeeded, and a later upgrade needs a new request. None of this condenses to "roll back the image": an image swap cannot undo a migrated database, and unknown migration results stay held for investigation.
Migrate between regular and managed deployments
Moving between a regular (native) installation and a managed cluster is an explicit migration with its own plan file — not a restore of one deployment's archive into the other. Both directions require a valid destination license and prepared destination infrastructure, a matching release/schema contract, all source writers fenced (including other nodes, Agents and external consumers), administrator and connection-decryption checks, and an explicit cutover. The source fence is your operator attestation; keep the original immutable archive as evidence.
Regular → managed: capture a full offline checkpoint of the stopped regular installation with the release's native transport (backup --native-config source.json --archive ... through the native CLI delegation), then add a migration object — archiveSha256, administratorLogin and schedulePolicy — to the normal managed installation configuration. Installation pauses at MIGRATION_IMPORT with capacity reserved and no application pod created. Write the full migration plan using the returned installation identity, then:
sh deployment/installer/etlworks-kubernetes.sh restore inspect \
--config ../etlworks-private/install.json \
--archive /absolute/backups/source.zip --migration-plan /absolute/etlworks-private/migration.json
sh deployment/installer/etlworks-kubernetes.sh restore start \
--config ../etlworks-private/install.json \
--archive /absolute/backups/source.zip --migration-plan /absolute/etlworks-private/migration.json
sh deployment/installer/etlworks-kubernetes.sh restore cutover \
--config ../etlworks-private/install.json \
--archive /absolute/backups/source.zip --migration-plan /absolute/etlworks-private/migration.json
sh deployment/installer/etlworks-kubernetes.sh resume \
--config ../etlworks-private/install.json --wait 20m --jsonManaged → regular: take the normal full offline managed backup, prepare an empty regular target with its valid NODE license and managed mode disabled, and run the same restore inspect|start|resume|cutover sequence with --native-config target.json in place of --config. A successful cutover reports MIGRATION_READY_FOR_NATIVE_START; start the host service through the normal host procedure.
The migration plan's exact fields, the native source/target configuration and the supported source environments are specified in the recovery instructions shipped with your release — use its complete template rather than a partial example. After an interruption, resume with exactly the same inputs. Import validates the packaged schema history, an existing enabled administrator and representative connection decryption; it preserves users, passwords, business IDs, files, logs, keys and checkpoints, and archives the source's cluster records as inactive evidence. All imported schedules and maintenance tasks remain paused for review — review intended activation and missed schedules, and review Home URL and ETL Engine Proxy URL for the new topology, before enabling traffic and re-enabling schedules. A successful data import remains held until explicit cutover and the normal license/storage/startup checks pass. Migration is not a cross-version conversion: use matching releases on both sides.
Verify independent backups and monitor storage
Copy each complete archive to an independent storage failure domain and verify recovery on a supported isolated target, saving the actual restore command's JSON result. The bundled verifier checks source/copy identity and hashes, complete archive sections and credentials, capture age and the matching successful held-restore evidence:
python3 deployment/recovery/checkpoint.py \
--source /protected/staging/checkpoint.zip \
--copy /independent-backups/checkpoint.zip \
--declaration ../etlworks-private/checkpoint-location.json \
--restore-receipt ../etlworks-private/restore-result.json \
> ../etlworks-private/verified-checkpoint.jsonThe declaration names your actual independent domains and the owner's backup-age policy:
{
"schemaVersion": 1,
"sourceFailureDomain": "etlworks-cluster-storage",
"copyFailureDomain": "independent-backup-vault",
"maximumAgeHours": 24,
"platformRecoveryOwner": "platform-team"
}VERIFIED_INDEPENDENT_CHECKPOINT means the complete matching copy and restore evidence passed. Physical hosting independence remains your declaration — a different filename cannot prove a different storage failure domain. A stale, partial or unconfirmed checkpoint is not healthy, and copying a ZIP beside the source data is not disaster recovery. Back up Kubernetes/platform configuration, volume state and private issuer roots separately through the platform's own protected recovery procedure.
The status command reports retained storage observations where the guard observer is available; missing or stale samples mean unavailable, not zero usage, and disks used only by private identities may need a platform collector. An optional helper exports saved status/checkpoint information for an existing Prometheus textfile collector:
sh deployment/installer/etlworks-kubernetes.sh status \
--config ../etlworks-private/install.json --json > ../etlworks-private/status.json
python3 deployment/operations/storage_metrics.py \
--status ../etlworks-private/status.json \
--checkpoint ../etlworks-private/verified-checkpoint.json \
--maximum-age-hours 24 > ../etlworks-private/etlworks-storage.promYour monitoring operator schedules fresh collection, publishes the output atomically to the real collector and tests alert delivery — the helper installs no monitoring server, daemon or notification receiver. Alert on unavailable or failed guards, insufficient disk or inodes and a stale independent checkpoint. Do not add these storage metrics to HPA replica-demand calculations.
Security, certificates and network behavior
Certificates. cert-manager runs headlessly as a cluster component, and ordinary leaf-certificate renewal is automatic in the supplied configuration — you do not issue a certificate for each added pod. An existing compatible cert-manager installation is reused where supported. Public DNS, browser-trusted frontend TLS, platform issuer-root recovery and external-service credentials remain separate responsibilities.
Transport. Private lifecycle and proxy-administration traffic uses mutual TLS; PostgreSQL and Redis connections use verified server TLS plus credentials. This does not make every network path encrypted: the HAProxy-to-application business backend is HTTP inside the cluster network, and shared-storage encryption depends on the profile (local AUTH_SYS NFS is unencrypted storage traffic; cloud profiles state their storage transport in the cloud guide). Keep worker and storage networks isolated with platform firewalls, expose only the application entry point, and never configure trust-all TLS.
Long-running requests and idle connections. The managed HAProxy profile uses 24-hour inactivity limits for business client/server connections and upgraded tunnels such as WebSockets — an idle limit, not a total-duration limit. TCP keepalives run independently on both HAProxy legs: 240-second idle, 10-second probe interval, 3 failed probes. Keepalives do not reset HTTP inactivity timers or keep a disconnected client alive. Short bounds remain separate: 5-second connect, 2-second health-check response, 30 seconds for incomplete request headers and 10 seconds for idle HTTP connection reuse — the last two do not cap time spent processing an already received request.
Application-to-embedded-engine execution is a loopback call inside each pod; it does not traverse HAProxy or the node network. External synchronous listeners, responses, uploads and notification connections do traverse the proxy — and every other hop you add (reverse proxy, cloud load balancer, NAT, firewall) applies its own idle policy, so match those settings to your intended request lifetime. Health probes are separate connections and cannot keep a silent business connection alive. Node and cloud network settings (for example EC2 ENI tracking timeouts) are platform configuration: inspect the hops actually on the path instead of applying one provider's setting everywhere.
Premium connectors on managed pods
Etlworks 10.0.14 and newer includes application-managed premium connector administration: a SuperAdmin uses Settings → Premium connectors to inspect versions and status and to manage supported packages and licensing. See Managing premium connectors for the full guide. A Cluster license's pod capacity does not itself grant premium connector licensing.
Shared application data holds the staged connector packages. Each pod keeps its own working copy and per-server runtime/license state; the managed-pod default working folder is /var/cache/etlworks/managed-drivers, on the disposable cache volume. Do not mount a shared .lic file or a shared working folder across pods, do not copy JARs into running pods as an installation procedure, and do not treat the per-pod cache as an authoritative backup. A new pod reconciles the desired managed packages from application state and staging.
For managed clusters, production premium connector use requires the supported runtime key (RTK). Node-bound license-key activation is not offered in managed mode, because pod replacement would consume activations. Connector trials follow their own per-connector policy with per-pod activation and are distinct from the 14-day Etlworks Cluster trial; pod recreation does not reset an agreed commercial trial term. Configure keys through the protected application settings and licensing mechanism, never through installation JSON or public examples.
Version operations report per-server results and can leave a member needing attention: review the Servers results view rather than assuming a selected version completed on every member. An application image upgrade and a premium connector version change are separate operations; follow the premium connector guide for preflight, switching and rollback.
Troubleshooting
| Symptom | Check and next action |
|---|---|
| No Kubernetes trial option, or an expired private link | Use the correct route: the request page for a trial, the tools page for installer-only downloads. Request a fresh link or contact Etlworks if unavailable; a replacement link does not restart a trial, and credentials are never generated locally as a workaround. |
| Signed setup/license rejected | Check the environment, starting configuration, current authorization/expiry and trusted keys, then obtain corrected signed files through the portal's guided steps. Do not edit their contents or manage entitlement timestamps manually. |
| AI or business settings missing although the app starts | Check the setup snapshot and refresh-credential state in cluster health. Keep the extracted setup files together, obtain current setup files, and resume with the existing configuration to add the missing entries. Settings delivery never changes licensed capacity. |
| Online licensing healthy on one pod only | Check each member's current verified authorization and runtime state, outbound licensing connectivity and matching app/controller releases. Do not switch to offline mode or reset credentials as a shortcut. |
| Checksum/inventory mismatch | Stop; obtain the correct archive/checksum. Do not edit release.json or mix extracted releases. Keep private inputs outside the release root. |
| JAVA_25_REQUIRED / HOST_TOOL_REQUIRED | Verify Java 25, Python, kubectl and Helm on the installation workstation, including PATH/JAVA_HOME. |
| KUBERNETES_PROFILE_UNSUPPORTED / UNSUPPORTED_RELEASE_PROFILE | Verify the actual release profile and Kubernetes version. Do not change version strings to bypass the check. |
| LOCAL_STORAGE_PROFILE_REQUIRED / network-policy profile error | The selected cluster lacks the required local provisioner or compatible Calico setup. Ask the platform administrator to prepare the supported profile; do not overwrite a hosted cluster's CNI. |
| Pod Pending / insufficient CPU or memory | Read the pod's events for the CPU/RAM, storage, identity-pool, network/CSI or image-pull cause. Providing larger or additional workers is platform work, separate from changing license limits; a larger license does not add machines. |
| Pending PVC or volume/mount error | Inspect PVC/PV bindings, CSI availability, storage worker identity and NFS reachability/permissions. Retain existing volumes and data while investigating. |
| ImagePullBackOff | Check registry access, architecture, exact digest and any image-pull Secret references. Do not substitute latest. |
| Certificate/webhook pending | Inspect cert-manager readiness and certificate events. Wait/resume when dependencies settle; do not regenerate retained signing/trust state manually. |
| Browser TLS/DNS error | Verify hostname resolution, proxy access and the approved public CA trust. Port forwarding does not configure DNS. Keep verification enabled. |
| Install times out or the client disconnects | Check status, fix the reported hold, resume the same configuration and release. A timeout does not prove failure and never authorizes a competing installation. |
| Namespace or retained-intent conflict | Use the original inputs and identities. Do not rename resources, overwrite receipts or delete the namespace to make an installation appear fresh. |
| External service TLS/auth/ACL/ownership/budget hold | Check endpoint DNS, CA/server certificate validity, login Secrets, privileges, the assigned connection budget and durability settings (AOF, noeviction). Repair through the supported procedure — no unencrypted fallback, and do not silently switch to a local service. |
| External Secrets missing at first installation | Expected at the checkpoint: confirm the installer-owned namespace, create only the named supplied Secrets, then resume with unchanged inputs. |
| Missing, unmounted or full retained disk | Restore the exact original mount and identity; inspect free bytes/inodes against the declared reserve, not just a marker. Do not initialize empty directories or delete identity guards. |
| Application inaccessible | Inspect application readiness, HAProxy, the Service/load balancer, allowed client networks, DNS and frontend trust. Do not expose private lifecycle or database ports. |
| Flow status stuck until a page refresh, or repeated duplicate toasts | Inspect the browser's notification connection, proxy buffering/idle behavior, session/authentication and Redis messaging health, and use a current release — these were corrected in recent releases. Do not restart or replay the business flow just to refresh the screen. |
| Applying changes persists | Compare saved/applied revisions in Configuration and status details and check autoscaler readiness and fresh observations; inspect the reported blocker. A browser refresh cannot repair a backend or controller hold. |
| Scale-down, resize or upgrade waits | Inspect actual ordinary/continuous work, open requests, scheduler handoff, licensed capacity and pending replacements. Long-running ordinary work legitimately delays these operations; do not force-delete healthy busy pods. |
| No releases on the Upgrades tab, or the exact source does not match | The operator must register the matching source/target release metadata in this installation; the newest version label alone is insufficient. Use More builds to select the actual installed source build. |
| Upgrade interrupted or failed | Inspect the same request through update status, preserve the archive and request IDs, and follow the supported resume, target-recovery or source-reactivation path. No blind image rollback or schema repair. |
| Redis process identity changed | The coordination recovery hold is deliberate. Follow the protected recovery path; a reconnect is not automatic flow replay, and the run-ID witness must not be rewritten. |
| More pod identifiers in historical charts than running pods | Retained history versus the current-only selector — see monitoring. Do not delete historical attribution. |
| A connection closes during a long flow | Check timeouts and keepalives at every frontend/LB/network hop, not only HAProxy (see network behavior). Connection loss is not proof the engine stopped; check execution state before retrying. |
| Login password-change request errors after setup | Configure SMTP and verify whether the password already changed before repeating an ambiguous request. Initial onboarding itself does not require SMTP. |
| Checkpoint verification unconfirmed or stale | Produce the complete independent copy and the matching real restore evidence. Investigate the gap; do not construct a success receipt manually. |
| Premium connector reports per-server attention | Inspect that connector's Servers results view, runtime-key/trial mode and pod cache reconciliation. Do not use node license keys or share per-pod license files. |
For support, provide the release identity, profile/architecture, installation state and reason, relevant request IDs, pod events/logs and the bounded diagnostics ZIP. Share sensitive material only through an approved protected channel.
After installation
Keep the verified release, protected configuration, current license/setup files and independent backups available for maintenance. Do not modify application workloads directly to perform upgrades or force a scale-in.
Day-two operations are covered above: automatic scaling, health and status monitoring, application upgrades, pod size changes and backup, restore and migration. Upgrading the application does not upgrade Kubernetes, cert-manager, KEDA, the lifecycle adapter, PostgreSQL/Redis server binaries or the storage platform — coordinate those platform changes separately with your administrator and Etlworks, using the operations instructions shipped with the release.
On cloud profiles, see also cloud operations and problem solving.
For assistance, contact Etlworks support.