Skip to Content
Projections

How projections work

A projection reads input streams in checkpointed units and writes derived streams, values, or rows.

A projection reads one or more input streams and writes data derived from them. It works in units. Each unit holds the new items of each input, and a Commit runs the handler over the unit and saves the new input offsets as a checkpoint. A restart continues from the last checkpoint.

A pass of a projection makes at most one unit:

  1. The Commit loads the checkpoint record of the projection.
  2. The runner reads one page of each input after its checkpoint offset, in declaration order. When the record holds the saved ranges of an unfinished unit, the runner reads those ranges again.
  3. When no input has new items, no input has newly closed or gone, and no process timer is due, the pass ends. The handler does not run.
  4. Otherwise the Commit runs the handler over the unit and saves the new offsets with the handler result.

To declare a projection and its handler, see Define a projection. To run passes, see Run a projection.

Choose a constructor

The constructor sets what the projection writes and the shape of its handler.

You wantConstructorPage
A value, such as a total or a small setfoldValues
New stream items for each input entryemitOutput streams
A value and output itemsmapAccum, mapAccumBatchValues, Output streams
Rows in your own SQL tables, or other effectsprocessTables and effects
A wake row for each source streamwakeWake streams

For the full signatures, see Constructors.

For patterns built from these constructors, see Effects and Command streams. The timer stream example keeps many deadlines in one binding.

Choose a Commit

The Commit sets the delivery guarantee of a unit.

CommitWhat it does
Commit.checkpointSaves the encoded value and the offsets in one compare-and-set
Commit.transactionCommits the SQL writes of the handler, the output appends, and the checkpoint in one transaction on the SQL storage owner
Commit.producerSaves the input ranges and the output positions, then appends the outputs at those positions. A retry answers Duplicate and writes nothing twice
Commit.atLeastOnceSaves the input ranges before the handler runs, and the offsets after it. After a crash, the handler runs again on the same ranges

fold uses Commit.checkpoint when the declaration has no commit. wake always uses Commit.producer. The other constructors require commit, and the compiler rejects a Commit that does not fit the constructor. See Commits for which Commit fits which constructor, and for custom Commits.

Example

import { Effect, Schema } from "effect";
import { Streamsy } from "@streamsy/core";
import { Commit, Projection } from "@streamsy/projection";

export class Numbers extends Streamsy.json<Numbers>()("Minimal/Numbers", "numbers", {
  params: {},
  schema: Schema.Finite,
}) {}
export class Doubled extends Streamsy.json<Doubled>()("Minimal/Doubled", "doubled", {
  params: {},
  schema: Schema.Finite,
}) {}

export const minimal = Effect.gen(function* () {
  const numbers = yield* Numbers;
  const doubled = yield* Doubled;
  yield* numbers.create({});
  yield* doubled.create({});
  yield* numbers.append({}, [1, 2, 3]);
  return yield* Projection.run(
    Projection.emit({
      id: "doubled",
      params: {},
      inputs: { input: Numbers },
      outputs: { output: Doubled },
      commit: Commit.transaction,
      emit: ({ item }) => ({ output: [item * 2] }),
    }).at({}),
    { limit: 10 },
  );
});

Projection.emit declares the projection, .at({}) binds it, and Projection.run runs passes until the input is caught up. limit: 10 caps the run at 10 units. A single unit reads [1, 2, 3] and appends [2, 4, 6] to doubled in the same transaction as the checkpoint. The run returns status: "caught-up", units: 1, and items: 3.

Commit.transaction needs a SQL storage Layer with CheckpointStore.layer and Transaction.layer. A memory host has no Transaction. See Layers. The source is packages/projection/src/examples/minimal.ts, and a test in packages/projection/test/run.test.ts runs it on SQLite.

Fact streams and state streams

A fact stream records domain events, declared with Streamsy.json. A state stream holds keyed rows as Durable State change events, declared with Streamsy.state. To write rows to a state stream from a projection, see State outputs.

On this page