Skip to Content
Projections

Tables and effects

Write each unit of input to your own SQL tables in the checkpoint transaction, or run external effects at least once.

Projection.process runs an Effect once for each unit. Use it to write rows to your own SQL tables, or to call another system. For the full signature, see Projection.process.

Process

process(batch, unit, context) gets the whole unit and returns an Effect of void or a WakeAt request. batch has one Slice for each input, with its items in stream order. context is an object:

  • context.prepared is the result of prepare, or undefined when the declaration has none. See Prepare.
  • context.attempt is the binding run attempt, starting at 1 and increasing after failed attached runs. Direct run and follow always supply 1.
  • context.outputs holds the bound addresses of the declared outputs. It exists only when the declaration lists outputs. See Declared outputs.

Destructure the fields that the handler uses, as in process: (batch, unit, { prepared }) => .... A handler that uses none can take (batch, unit).

To handle one entry at a time, wrap an entry handler in Projection.each. It runs the handler for every entry in order, one after another: all entries of the first declared input, then those of the next. It discards the results.

process has no default Commit. Choose one by what the handler writes:

CommitUse it forGuarantee
Commit.transactionSQL rows on the same storage as the checkpointThe rows and the checkpoint commit together
Commit.atLeastOnceCalls to other systemsThe handler can run more than once for a unit

Commit.checkpoint and Commit.producer do not fit process, and the compiler rejects them. See Commits.

Timers

Return Projection.wakeAt(time) from a process handler to replace the binding's one recorded timer. time is epoch milliseconds or an Effect DateTime. It must convert to a non-negative safe integer; fractional, negative or unsafe times throw RangeError. Returning nothing clears the timer. Ordinary return values are discarded as before.

Timers require Commit.atLeastOnce or Commit.transaction, including Commit.withCompensation around either. The compiler rejects a typed wake request paired with a custom Commit<void>. Custom timer Commits are not supported in this release. The context has no timer member.

import { Effect, Option } from "effect";
import { Streamsy } from "@streamsy/core";
import { Commit, Projection } from "@streamsy/projection";
import { Schema } from "effect";

class Deadlines extends Streamsy.json<Deadlines>()("Timers/Deadlines", "deadlines", {
  params: {},
  schema: Schema.Finite,
}) {}

const deadlines = Projection.process({
  id: "deadlines",
  params: {},
  inputs: { deadlines: Deadlines },
  commit: Commit.atLeastOnce,
  process: (batch, unit) =>
    Effect.gen(function* () {
      if (Option.isSome(unit.due)) {
        yield* Effect.log(`Deadline ${unit.due.value} is due`);
      }
      const next = batch.deadlines.items.at(-1);
      return next === undefined ? undefined : Projection.wakeAt(next);
    }),
});

A due unit has unit.due: Some(recordedTime). With no new input, every slice is empty and unit.ranges is {}. If input arrives after the deadline, one unit contains both the items and unit.due. Input before the deadline runs normally, with unit.due: None, and its result replaces or clears the timer. A deadline can still run after every input has closed. A due unit contributes no accepted row.

Use the recorded unit.due to decide what is due. Reading Clock is useful for a late policy, but must not change which effects a pinned retry repeats. The unit key includes the recorded due time, and Commit.atLeastOnce pins that time with the ranges. Commit.transaction saves the next timer with the handler's SQL and checkpoint; a failed write rolls them all back.

Projection.run, pass and follow make a due unit only when they run at or after the time. They do not wait for the deadline. follow can discover it on an input change or repair cycle. A cycle commits at most one due unit without input, even when the handler returns a time already passed. Return a time after unit.due for the next wake. Projection.attach and ProjectionWakeup.attach wake at the recorded deadline.

Projection.attach wakes a sleeping binding at its recorded deadline. Its process wait re-reads the wall clock at least every 30 seconds while a deadline exists, so after host suspend a passed wake is discovered within 30 seconds of resume. ProjectionWakeup.attach exposes that deadline to the Durable Object alarm host, which arms its alarm for the wake time. Wakes can be late; handlers use unit.due to apply their late policy. A wake during backoff waits for backoff to end. A stopped binding keeps its checkpoint timer and fires late after retry or restart. With durable: false, pending work and timers live in private memory: restart or eviction loses a timer's wake until the binding's next input. Use durable marks on persistent storage for restart recovery.

Return a time after unit.due for the next wake. Repeatedly returning a passed time leaves the mark due immediately and can run the handler at pass speed.

SQL tables

With Commit.transaction, the handler runs inside the SQL transaction that saves the checkpoint. The Commit provides the transaction's SqlClient, so the rows and the new offsets commit together or roll back together.

import { Effect } from "effect";
import { SqlClient } from "effect/sql";
import { Commit, Projection } from "@streamsy/projection";
import { Orders } from "./streams.ts";

const rows = Projection.process({
  id: "order-rows",
  params: {},
  inputs: { orders: Orders },
  commit: Commit.transaction,
  process: Projection.each(({ item }) =>
    Effect.gen(function* () {
      const sql = yield* SqlClient.SqlClient;
      yield* sql`INSERT INTO order_rows (order_id, amount) VALUES (${item.orderId}, ${item.amount})`;
    }),
  ),
});

The application creates and migrates its tables. Streamsy does not create them.

To write rows and also keep a value, use a fold step under Commit.transaction. The step can return an Effect that uses the SqlClient:

import { SqlClient } from "effect/sql";

const orderTotals = Projection.fold({
  id: "order-totals",
  commit: Commit.transaction,
  params: {},
  inputs: { input: Orders },
  state: { schema: Schema.Finite, initial: 0 },
  fold: (sum, { item }) =>
    Effect.gen(function* () {
      const sql = yield* SqlClient.SqlClient;
      yield* sql`INSERT INTO order_totals (order_id, total)
        VALUES (${item.orderId}, ${item.amount})
        ON CONFLICT (order_id) DO UPDATE SET total = excluded.total`;
      return sum + item.amount;
    }),
});

Commit.transaction needs a SQL host with the CheckpointStore and Transaction services. A memory host has no Transaction. See Layers.

Before the handler runs, the Commit loads the record again inside the transaction. When another writer changed the record after the unit was read, the unit fails with a ProjectionFault with phase checkpoint and reason token-conflict, and nothing commits. A failed handler rolls back its rows.

An append that the handler makes to a stream on the same storage commits with the rows. An append to a stream on other storage fails with phase process and reason unsupported-composition. To write declared output streams in the same transaction, see Transaction outputs.

Declared outputs

Under Commit.transaction, a process declaration can list outputs, as emit does. Each output is a stream declaration or a fixed address, and each declaration must take a subset of params. The handler gets the bound address of each output in context.outputs, and it appends to them itself.

import { Array, Effect } from "effect";
import { Streams, StreamsWriter } from "@streamsy/core/protocol";
import { Commit, Projection } from "@streamsy/projection";
import { Orders, OrderTotals } from "./streams.ts";

const copyTotals = Projection.process({
  id: "copy-totals",
  params: {},
  inputs: { orders: Orders },
  outputs: { totals: OrderTotals },
  commit: Commit.transaction,
  process: (batch, _unit, { outputs }) =>
    Effect.gen(function* () {
      const rows = Array.map(batch.orders.items, (order) => ({
        orderId: order.orderId,
        total: order.amount,
      }));
      if (!Array.isReadonlyArrayNonEmpty(rows)) return;
      const { wire } = yield* outputs.totals.stream;
      yield* Streams.append(outputs.totals, rows).pipe(Effect.provideService(StreamsWriter, wire));
    }),
});
  • Before prepare and the handler run, the Commit checks that every output shares the storage of the Transaction service. An output on other storage fails the unit with phase load and reason unsupported-composition, and the handler does not run.
  • The appends commit with the checkpoint, or roll back with it.
  • The Commit does not create the outputs. Create each output first, or create it in the handler.
  • prepare gets the same addresses in its context.outputs.
  • Commit.atLeastOnce does not provide SqlClient, so the compiler rejects outputs with it.

FoldByKey.process builds a handler of this shape, which folds rows of a State stream.

Prepare

prepare(batch, unit) runs before the transaction opens. Use it for work that must not hold the transaction open, such as a slow read from another system. Its result reaches process as context.prepared. When the declaration lists outputs, prepare also gets { outputs } as its third argument.

  • prepare needs Commit.transaction, or Commit.transaction wrapped with Commit.withCompensation. With any other Commit, Projection.process throws a RangeError with the message prepare requires Commit.transaction.
  • prepare runs at least once for each unit. A retry runs it again. It must not make effects that a second run would repeat.
  • When prepare fails, the run fails with its error and the transaction does not open.
  • The runner's caller provides the services that prepare needs.

prepare and process see the same unit.key within an attempt. A transaction retry reads the inputs again, and the key changes when new input extends a range. Do not use unit.key as an idempotency key that must hold across retries of Commit.transaction.

External effects

Commit.atLeastOnce saves the input ranges of the unit in the checkpoint record, then runs the handler, then saves the new offsets. When a pass stops between the two saves, the next pass reads the same ranges again and runs the handler again with the same unit.key. Pass unit.key to the other system as an idempotency key, so that it can drop the repeated call.

Commit.atLeastOnce needs only the CheckpointStore service, so it works on a memory host.

When the handler fails, the pass fails, the offsets stay, and the next pass runs the unit again. Commit.atLeastOnce keeps no delivery queue and has no retry policy of its own. Use unit.key as the external call's idempotency key, set the declaration's binding retry policy for a retry budget and onExhausted, and return Projection.wakeAt for a deferred retry or a snooze. See the effects guide for the complete pattern.

On this page