Transformers

Three first-class transformers cover the data-shaping work flows do most often. They are not operators — they are runtime-native, so they need no using: import.

join: — relational join

A relational join of two data sources. Four types (inner is the default, plus left, right, full), keyed by on: (an explicit { left: right } map) or using: (same-name shorthand).

- $with-stock:
    join: "${stock.rows}"
    from: "${products.rows}"
    using: sku
    type: left

For file sources, pre-load via an explicit load: step before joining — the planner fuses the chain at zero runtime cost. For category-based enrichment (e.g. Akeneo reference entities, attribute groups), join: also supports lookup-by:.

delta: — change detection across runs

Detects what changed since a previous run. Three modes, inferred from the keywords you use:

Mode	Keyword	Result
compare	(default)	Full dataset diff into add / remove / update / equal.
hash	mode: hash	A single content hash — “did anything change at all?“.
per-row	by: hash	Keeps only the rows that changed.

Persistence is explicit — your flow declares the persist: variable that holds the previous state:

persist:
  stored-hashes: !obj
steps:
  - $changed:
      delta: "${batch.items}"
      by: hash
      key: "id"
      from: "${stored-hashes}"
  - $stored-hashes: "${changed.hashes}"

Because the previous state lives in a persist: variable, change detection survives restarts and runs incrementally.

filter: — SQL-like array filtering

Array slicing and SQL-like filtering on payload data. where: accepts the usual SQL operators (=, <>, IN, LIKE, BETWEEN, IS NULL); order-by:, limit:, offset:, and fields: round it out.

- $hot-tickets:
    filter: "${jira.issues}"
    where: "priority IN ('High', 'Critical') AND status != 'Closed'"
    order-by: "created DESC"
    limit: 25
    fields: [key, summary, priority, assignee]

fields: projects to a subset of columns — useful for trimming a wide result set before it leaves the flow or feeds a downstream step.