A few patterns worth knowing when working with databases in Etlworks:
- Use a temporary database as a staging area — in-memory SQLite-based staging that's cleaned up automatically.
- Performance tips when working with databases — bulk load patterns, parallel execution, batch sizing.
- Load one source dataset into multiple destination tables — normalize a denormalized source into parent / child tables.
- Automatic partitioning — extract large tables in parallel chunks.
- Use global variables in database connections — parametrize the URL, username, password, or other fields.
- Transaction management — coordinated commits, manual commits, and rollback control.