Skip to Content
Projections

Values

Fold input entries into one value, read the value with its offsets, and decide what to append from it.

A value projection keeps one value, such as a total or a small set, and saves it with the input offsets in the checkpoint record. Projection.fold builds one. Projection.mapAccum and Projection.mapAccumBatch keep a value too, and also write output items. See Output streams. For the full signatures, see Constructors and Reads.

Fold a value

import { Effect, Schema } from "effect";
import { Projection } from "@streamsy/projection";
import { Orders } from "./streams.ts";

const revenue = Projection.fold({
  id: "revenue",
  params: {},
  inputs: { orders: Orders },
  state: { schema: Schema.Finite, initial: 0 },
  fold: (sum, { item }) => sum + item.amount,
});

const program = Effect.gen(function* () {
  const orders = yield* Orders;
  yield* orders.create({});
  yield* orders.append({}, [
    { orderId: "o1", amount: 5 },
    { orderId: "o2", amount: 7 },
  ]);
  const total = revenue.at({});
  yield* Projection.run(total);
  const { state } = yield* Projection.read(total);
  return state; // 12 on a fresh host
});

fold(state, entry, unit) gets the current value and an input entry, and returns the next value. It can return the value, or an Effect of the value. A unit folds all entries of the first declared input, then those of the next input, and the Commit then saves the result.

fold uses Commit.checkpoint when the declaration has no commit. Commit.checkpoint writes the encoded value and the new offsets in one compare-and-set, so the value never counts an entry twice. Keep external effects out of the step. A retry runs the step again. To write SQL rows from the step, use Commit.transaction. See SQL tables.

The three constructors that keep a value differ in their step:

ConstructorStep
Projection.foldfold(state, entry, unit) returns the next value
Projection.mapAccummapAccum(state, entry, unit) returns [state, emits]
Projection.mapAccumBatchmapAccumBatch(state, batch, unit) returns [state, emits]

State

state: { schema, initial } declares the value. A bound projection keeps one value for each binding, so revenue.at(params) with other parameters keeps its own value.

FieldMeaning
schemaThe Schema codec of the value. The record stores the value as the JSON string of its encoded form
initialThe value before the first unit. A function of the decoded parameters is called only when no value exists

The record holds the encoded value and the offsets together, so a value always matches its offsets. Each save replaces the value, and the record keeps no history. See StateDefinition.

A value that does not encode fails the unit with a ProjectionFault with phase process and reason invalid-output, and nothing is saved. A stored value that does not decode fails the load with phase load and reason invalid-record. To change the schema of a stored value, declare a new generation. See Generations.

Read

Projection.read(projection) loads the checkpoint record through the Commit and returns { state, offsets }. offsets has one entry for each input, the offset that the value has folded up to. read runs no step and reads no input.

Recordstateoffsets
No record yetinitialThe start of each input
A saved recordThe decoded valueThe offsets saved with it

A declaration with no state reads as { state: undefined, offsets }.

read needs the services of the Commit's load, for example CheckpointStore. It fails with a ProjectionFault, for example identity-mismatch when the stored inputs differ from the declaration.

run returns a Progress with the same state and offsets. After run returns, read gives the same offsets, unless another runner committed a unit between the two calls. See Run a projection.

Size

Each unit decodes the whole value, runs the step, and encodes and saves the whole value again. Keep the value small, for example a counter, a summary or a few hundred rows. The hex board example measured an encoded value of 21,599 B after 3,005 events.

To keep many rows, write them to your own SQL tables with Projection.process or a fold step under Commit.transaction. See SQL tables.

Read, decide, append

A command can use a value projection as its view, then append to one of its inputs. Projection.decide reads the value with Projection.current, calls your decide, and appends the items with that input's offset as expectedOffset.

import { Effect, Schema } from "effect";
import { Streamsy } from "@streamsy/core";
import { Projection } from "@streamsy/projection";

class Seats extends Streamsy.json<Seats>()("Venue/Seats", "seats", {
  params: {},
  schema: Schema.Struct({ seat: Schema.String }),
}) {}

const taken = Projection.fold({
  id: "taken-seats",
  params: {},
  inputs: { input: Seats },
  state: { schema: Schema.Array(Schema.String), initial: [] },
  fold: (seats, { item }) => [...seats, item.seat],
}).at({});

const reserve = (seat: string) =>
  Projection.decide(taken, {
    input: "input",
    decide: ({ state }) =>
      Effect.succeed(
        state.includes(seat)
          ? { value: "rejected" as const, items: [] }
          : { value: "reserved" as const, items: [{ seat }] },
      ),
  });

Projection.current(projection) loads the checkpoint record, then reads every input from the saved offsets and folds the entries in memory. It returns { state, offsets } and includes every entry appended before the call. It takes no lock, and it writes no checkpoint and no outputs. For mapAccum and mapAccumBatch, it keeps the value and discards the output items. A unit that a Commit pinned but did not settle is folded again from the saved offsets.

The steps of Projection.decide(projection, spec):

  1. It calls current for the value and the offsets.
  2. It calls spec.decide with that snapshot. spec.decide must not write.
  3. When items is empty, it returns at once, with no append and no tail check. A rejection is an answer at the offsets that it read.
  4. Otherwise it appends the items to the input that spec.input names, with that input's offset as expectedOffset. spec.input must be a key of inputs, and the items must have that input's item type.
  5. When its own append fails with OffsetMismatch, it goes back to step 1.

spec.retry is the Schedule for step 5. The default is Schedule.recurs(3), so one call makes at most four appends. When the schedule ends, decide fails with the last OffsetMismatch. It does not retry an OffsetMismatch that spec.decide itself fails with, and it does not retry any other failure.

decide returns { value, offset, appended }. offset is the new tail after an append, or the offset that it read when there was none. appended is true when it wrote the items. See DecideSpec.

current and decide need a declaration with state. The compiler rejects a projection without it. Both calls work inside or outside a caller's SQL transaction, and alongside a follower. Concurrent calls fold in memory and do not conflict with the follower, because they write no checkpoint. Only the append can lose a race.

Without a follower, the checkpoint stays where the last runner left it, so current folds a longer tail. With no checkpoint, it folds from the start of each input. A growing input can delay its return. The step must be replayable and must give the same value for any split of the entries into units.

A projection with no state, such as a SQL table projection or an emit projection, cannot serve as the view. Run it with one follower, and have the request wait for the output that its own write produces.

The Hacker News newest example uses this pattern. To fold a stream directly, without a projection, see Read, fold, append.

Serve a value

To serve a value over HTTP, read it with Projection.read in a route. See Projection outputs.

On this page