Skip to Content
Projections

Define a projection

Declare the inputs, outputs, parameters, and handler of a projection, then bind its parameters.

A projection declaration names the inputs, the outputs, the parameters, the Commit, and the handler of a projection. For the full signatures, see Constructors and Declaration.

Declare

import { Commit, Projection } from "@streamsy/projection";
import { Orders, OrderTotals } from "./streams.ts";

export const orderTotals = Projection.emit({
  id: "order-totals",
  params: {},
  inputs: { input: Orders },
  outputs: { totals: OrderTotals },
  commit: Commit.producer,
  emit: ({ item }) => ({ totals: [{ orderId: item.orderId, total: item.amount }] }),
});
export const orderTotalsProjection = orderTotals.at({});

A declaration is inert. It reads and writes nothing. orderTotals.at({}) binds the parameters and returns a bound projection, and a runner takes the bound projection. See Run a projection.

FieldValue
idThe projection name. It is part of the checkpoint record key
versionA whole number, default 1. It is part of the record key and of the producer id of the outputs, so a new version starts a new record from the start of each input
generationA whole number, default 1. It is part of the record key and is the producer epoch of the outputs. See Checkpoints and generations
paramsThe parameter codecs, keyed by parameter name. {} for none. See Parameters and binding
inputsA record of stream classes or fixed addresses. See Inputs
outputsRequired for emit, mapAccum, and mapAccumBatch; optional for transactional process. JSON or State stream classes or fixed addresses. See Output streams
stateRequired for fold, mapAccum, and mapAccumBatch. { schema, initial }. See Values
commitThe Commit. Optional for fold, where the default is Commit.checkpoint. Required for process, emit, mapAccum, and mapAccumBatch. See Commits
acceptedOptional wake class for publishing accepted input offsets with Commit.transaction or Commit.producer. See Accepted stream
prepareOptional, for process with Commit.transaction only. See Tables and effects
The handlerA field named after the constructor, such as emit above. See Handler

Projection.wake takes input and output in place of inputs, outputs, commit, and a handler. See Wake streams.

A declaration that the compiler accepts can still throw a RangeError:

  • The constructor throws prepare requires Commit.transaction when prepare has another Commit.
  • at throws when a parameter value does not encode with its codec.
  • at throws when two outputs name the same stream, when an output names an input stream, or when an output is not a JSON or State stream.

Inputs

The examples name a single input input: inputs: { input: Orders }. An entry handler then reads entry.item, and a batch handler reads batch.input.items.

With more inputs, give each one a name, for example inputs: { orders: Orders, refunds: Refunds }. entry.input is the name of the input that an entry came from, and a batch has one slice for each name.

The order of the record is the declaration order. A pass reads the inputs in declaration order, and the entries of a unit come in declaration order, then in stream order within each input. Append order across two inputs is not kept.

An input is a stream class, such as Orders, or a fixed address. at binds a stream class with the projection parameters, and leaves a fixed address as it is.

The checkpoint record stores the name, stream id, and content type of each input. A change to any of them makes the next pass fail with identity-mismatch. See Checkpoints and generations.

Parameters and binding

import { Schema } from "effect";
import { Streamsy } from "@streamsy/core";
import { Commit, Projection } from "@streamsy/projection";
import { OrderEvent, OrderRow } from "./streams.ts";

class Facts extends Streamsy.json<Facts>()("Shop/Facts", "facts/:workspaceId", {
  params: { workspaceId: Schema.String },
  schema: OrderEvent,
}) {}
class Totals extends Streamsy.json<Totals>()("Shop/Totals", "totals/:workspaceId", {
  params: { workspaceId: Schema.String },
  schema: OrderRow,
}) {}

const workspaceTotals = Projection.emit({
  id: "workspace-totals",
  params: { workspaceId: Schema.String },
  inputs: { facts: Facts },
  outputs: { totals: Totals },
  commit: Commit.producer,
  emit: ({ item }) => ({ totals: [{ orderId: item.orderId, total: item.amount }] }),
});

const bound = workspaceTotals.at({ workspaceId: "ws-42" });
// Option.some({ workspaceId: "ws-42" })
const params = workspaceTotals.parse("facts/ws-42");

at(params) encodes each parameter with its codec and binds every stream class of the declaration with the parameters that the stream class declares. bound reads facts/ws-42 and writes totals/ws-42.

Parameter rules:

  • Each input and output stream class declares a subset of the projection parameter names, with the same types. The compiler checks this.
  • A stream class that declares fewer parameters is shared by every binding with the same values for those parameters. A stream class with no parameters is shared by every binding.
  • parse(id) tries the input stream classes in declaration order and skips fixed addresses. It returns Option.some with the decoded parameters of the first stream class that matches the id, or Option.none(). A shared input gives only the parameters that it declares.
  • Projection.pass, Projection.run, and Projection.follow take one bound projection. The package keeps no list of bindings for them, so bind and trigger each one.
  • Projection.attach takes the declaration. It binds the stream id of each local commit to an input with parse, so each input must declare every projection parameter. See Run a projection.

To keep a set of bindings that changes, keep a registry stream of parameter records. A projection over that stream can start and stop the triggers. See Triggers.

Handler

Each constructor takes its handler in a field with the name of the constructor.

ConstructorHandlerRunsReturns
foldfold(state, entry, unit)Once for each entryThe next value
emitemit(entry, unit)Once for each entryThe output items, keyed by output
mapAccummapAccum(state, entry, unit)Once for each entry[value, outputs]
mapAccumBatchmapAccumBatch(state, batch, unit)Once for each unit[value, outputs]
processprocess(batch, unit, context)Once for each unitAn Effect of void or WakeAt

process must return an Effect. The other handlers return the result, or an Effect of it. An entry handler runs over the entries of the unit in order, and each call gets the value that the call before it returned.

ArgumentContents
entry{ input, item, index }. input is the input name, item is the decoded item, and index is the position of the entry in the whole unit
batchOne slice for each input, with from, items, optional offsets, nextOffset, upToDate, and closed. A slice of an input that is gone also has gone: true
unitprojectionId, version, generation, params, ranges, due, and key
stateThe current value. The first call of a unit gets the stored value, or initial when no value is stored
context{ prepared, attempt }: the prepared result and run attempt. With outputs, also the bound output addresses. See Tables and effects

For local inputs, batch.input.offsets[i] and entry.offset are the storage offset of the corresponding item. A read from entry.offset resumes after that entry; the last offset of a non-empty local slice equals its nextOffset. Non-empty HTTP reads have offsets undefined and omit entry.offset, because HTTP provides page boundaries only. All empty slices have offsets: [], including skipped inputs and timer-only units.

unit.params holds the decoded parameter values. unit.ranges has a { from, nextOffset } range for each input that gave items, closed, or is gone in this unit. Inputs with nothing new have no range.

unit.key is a string built from the projection id, the generation, the parameters, the ranges, and the recorded due time when present. A unit with the same ranges and due time has the same key. Commit.producer and Commit.atLeastOnce retry a unit with its saved ranges, so a retry keeps the key. A Commit.transaction retry reads the inputs again, so a longer range gives a new key. See Commits.

When the handler fails, the runner fails with the error of the handler, and the Commit saves no offsets. The next pass reads the same items again.

The host provides the services that the handler needs. Commit.transaction also provides SqlClient to the handler. A runner needs the handler services, the services of the Commit, and the services of each input and output stream.

Helpers

import { Effect } from "effect";
import { CheckpointStore, Commit, Projection } from "@streamsy/projection";
import { Orders } from "./streams.ts";

const logOrders = Projection.process({
  id: "log-orders",
  params: {},
  inputs: { input: Orders },
  commit: Commit.atLeastOnce,
  process: Projection.each(({ item }, unit) => Effect.log(`${item.orderId} in ${unit.key}`)),
}).at({});

const token = Effect.gen(function* () {
  const store = yield* CheckpointStore;
  const loaded = yield* store.load(Projection.key(logOrders));
  return loaded.token;
});
HelperDoes
Projection.items(batch)Returns the entries of a batch as { input, item, index }, in declaration order, then in stream order within each input
Projection.each(handle)Makes a process handler that runs handle(entry, unit) for each entry, one at a time and in order. It stops at the first failure
Projection.key(bound)Returns { id, version, generation, params } of a bound projection, with the encoded parameters. CheckpointStore takes it as the key

Use Projection.key to load or remove the checkpoint record of a binding, or to name the binding in logs. It is not unit.key. See Projection.key and Unit.

Next

Run a projection, and choose a Commit on Commits.

On this page