How projections work
A projection reads input streams in checkpointed units and writes derived streams, values, or rows.
A projection reads one or more input streams and writes data derived from them. It works in units. Each unit holds the new items of each input, and a Commit runs the handler over the unit and saves the new input offsets as a checkpoint. A restart continues from the last checkpoint.
A pass of a projection makes at most one unit:
- The Commit loads the checkpoint record of the projection.
- The runner reads one page of each input after its checkpoint offset, in declaration order. When the record holds the saved ranges of an unfinished unit, the runner reads those ranges again.
- When no input has new items, no input has newly closed or gone, and no process timer is due, the pass ends. The handler does not run.
- Otherwise the Commit runs the handler over the unit and saves the new offsets with the handler result.
To declare a projection and its handler, see Define a projection. To run passes, see Run a projection.
Choose a constructor
The constructor sets what the projection writes and the shape of its handler.
| You want | Constructor | Page |
|---|---|---|
| A value, such as a total or a small set | fold | Values |
| New stream items for each input entry | emit | Output streams |
| A value and output items | mapAccum, mapAccumBatch | Values, Output streams |
| Rows in your own SQL tables, or other effects | process | Tables and effects |
| A wake row for each source stream | wake | Wake streams |
For the full signatures, see Constructors.
For patterns built from these constructors, see Effects and Command streams. The timer stream example keeps many deadlines in one binding.
Choose a Commit
The Commit sets the delivery guarantee of a unit.
| Commit | What it does |
|---|---|
Commit.checkpoint | Saves the encoded value and the offsets in one compare-and-set |
Commit.transaction | Commits the SQL writes of the handler, the output appends, and the checkpoint in one transaction on the SQL storage owner |
Commit.producer | Saves the input ranges and the output positions, then appends the outputs at those positions. A retry answers Duplicate and writes nothing twice |
Commit.atLeastOnce | Saves the input ranges before the handler runs, and the offsets after it. After a crash, the handler runs again on the same ranges |
fold uses Commit.checkpoint when the declaration has no commit. wake always uses Commit.producer. The other constructors require commit, and the compiler rejects a Commit that does not fit the constructor. See Commits for which Commit fits which constructor, and for custom Commits.
Example
import { Effect, Schema } from "effect";
import { Streamsy } from "@streamsy/core";
import { Commit, Projection } from "@streamsy/projection";
export class Numbers extends Streamsy.json<Numbers>()("Minimal/Numbers", "numbers", {
params: {},
schema: Schema.Finite,
}) {}
export class Doubled extends Streamsy.json<Doubled>()("Minimal/Doubled", "doubled", {
params: {},
schema: Schema.Finite,
}) {}
export const minimal = Effect.gen(function* () {
const numbers = yield* Numbers;
const doubled = yield* Doubled;
yield* numbers.create({});
yield* doubled.create({});
yield* numbers.append({}, [1, 2, 3]);
return yield* Projection.run(
Projection.emit({
id: "doubled",
params: {},
inputs: { input: Numbers },
outputs: { output: Doubled },
commit: Commit.transaction,
emit: ({ item }) => ({ output: [item * 2] }),
}).at({}),
{ limit: 10 },
);
});Projection.emit declares the projection, .at({}) binds it, and Projection.run runs passes until the input is caught up. limit: 10 caps the run at 10 units. A single unit reads [1, 2, 3] and appends [2, 4, 6] to doubled in the same transaction as the checkpoint. The run returns status: "caught-up", units: 1, and items: 3.
Commit.transaction needs a SQL storage Layer with CheckpointStore.layer and Transaction.layer. A memory host has no Transaction. See Layers. The source is packages/projection/src/examples/minimal.ts, and a test in packages/projection/test/run.test.ts runs it on SQLite.
Fact streams and state streams
A fact stream records domain events, declared with Streamsy.json. A state stream holds keyed rows as Durable State change events, declared with Streamsy.state. To write rows to a state stream from a projection, see State outputs.