Data & Databases

The database operators share a common shape: a query action for reads, an execute action for writes, and (where the engine supports it) bulk operations. Connection strings and pool settings live on the connection, never in the flow.

SQL databases

Operator	invoke:	Typical actions
PostgreSQL	postgresql.*	query, execute
MS-SQL	mssql.*	query, execute, bulk-insert
MySQL	mysql.*	query, execute
Oracle	oracle.*	query, execute
Redshift	redshift.*	query, execute, Spectrum/COPY workflows

using:
  - zenvara/postgresql

steps:
  - $orders:
      invoke: postgresql.query
      on: warehouse
      with:
        Query: "SELECT id, customer, total FROM orders WHERE order_date = '${date}'"

Output: rows (the result set, a typed list) and metadata. Reference as ${orders.rows}. Parameterise queries through the flow’s typed input: rather than string-concatenating untrusted values.

NoSQL

Operator	invoke:	Notes
MongoDB	mongodb.*	Document find / insert / update / aggregate. Filter inputs are validated to prevent injection.

The runtime-native transformers

Three of the most common data operations are not operators — they are built into the runtime and need no using: import. They are documented in full under Transformers:

• join: — relational join of two sources (inner / left / right / full).
• delta: — change detection across runs (compare, hash, per-row by: hash).
• filter: — SQL-like filtering on payload arrays (where:, order-by:, limit:, fields:).

A common pattern is to query a database, delta: against a persisted snapshot to find only what changed, and write back just the delta — incremental sync without an external CDC system.

Spill-to-disk for large result sets

The engine streams and spills large datasets to disk rather than holding everything in memory, so a query that returns millions of rows does not exhaust the process. This is automatic; you author the flow the same way regardless of size.