Define a projection
Declare the inputs, outputs, parameters, and handler of a projection, then bind its parameters.
A projection declaration names the inputs, the outputs, the parameters, the Commit, and the handler of a projection. For the full signatures, see Constructors and Declaration.
Declare
import { Commit, Projection } from "@streamsy/projection";
import { Orders, OrderTotals } from "./streams.ts";
export const orderTotals = Projection.emit({
id: "order-totals",
params: {},
inputs: { input: Orders },
outputs: { totals: OrderTotals },
commit: Commit.producer,
emit: ({ item }) => ({ totals: [{ orderId: item.orderId, total: item.amount }] }),
});
export const orderTotalsProjection = orderTotals.at({});A declaration is inert. It reads and writes nothing. orderTotals.at({}) binds the parameters and returns a bound projection, and a runner takes the bound projection. See Run a projection.
| Field | Value |
|---|---|
id | The projection name. It is part of the checkpoint record key |
version | A whole number, default 1. It is part of the record key and of the producer id of the outputs, so a new version starts a new record from the start of each input |
generation | A whole number, default 1. It is part of the record key and is the producer epoch of the outputs. See Checkpoints and generations |
params | The parameter codecs, keyed by parameter name. {} for none. See Parameters and binding |
inputs | A record of stream classes or fixed addresses. See Inputs |
outputs | Required for emit, mapAccum, and mapAccumBatch; optional for transactional process. JSON or State stream classes or fixed addresses. See Output streams |
state | Required for fold, mapAccum, and mapAccumBatch. { schema, initial }. See Values |
commit | The Commit. Optional for fold, where the default is Commit.checkpoint. Required for process, emit, mapAccum, and mapAccumBatch. See Commits |
accepted | Optional wake class for publishing accepted input offsets with Commit.transaction or Commit.producer. See Accepted stream |
prepare | Optional, for process with Commit.transaction only. See Tables and effects |
| The handler | A field named after the constructor, such as emit above. See Handler |
Projection.wake takes input and output in place of inputs, outputs, commit, and a handler. See Wake streams.
A declaration that the compiler accepts can still throw a RangeError:
- The constructor throws
prepare requires Commit.transactionwhenpreparehas another Commit. atthrows when a parameter value does not encode with its codec.atthrows when two outputs name the same stream, when an output names an input stream, or when an output is not a JSON or State stream.
Inputs
The examples name a single input input: inputs: { input: Orders }. An entry handler then reads entry.item, and a batch handler reads batch.input.items.
With more inputs, give each one a name, for example inputs: { orders: Orders, refunds: Refunds }. entry.input is the name of the input that an entry came from, and a batch has one slice for each name.
The order of the record is the declaration order. A pass reads the inputs in declaration order, and the entries of a unit come in declaration order, then in stream order within each input. Append order across two inputs is not kept.
An input is a stream class, such as Orders, or a fixed address. at binds a stream class with the projection parameters, and leaves a fixed address as it is.
The checkpoint record stores the name, stream id, and content type of each input. A change to any of them makes the next pass fail with identity-mismatch. See Checkpoints and generations.
Parameters and binding
import { Schema } from "effect";
import { Streamsy } from "@streamsy/core";
import { Commit, Projection } from "@streamsy/projection";
import { OrderEvent, OrderRow } from "./streams.ts";
class Facts extends Streamsy.json<Facts>()("Shop/Facts", "facts/:workspaceId", {
params: { workspaceId: Schema.String },
schema: OrderEvent,
}) {}
class Totals extends Streamsy.json<Totals>()("Shop/Totals", "totals/:workspaceId", {
params: { workspaceId: Schema.String },
schema: OrderRow,
}) {}
const workspaceTotals = Projection.emit({
id: "workspace-totals",
params: { workspaceId: Schema.String },
inputs: { facts: Facts },
outputs: { totals: Totals },
commit: Commit.producer,
emit: ({ item }) => ({ totals: [{ orderId: item.orderId, total: item.amount }] }),
});
const bound = workspaceTotals.at({ workspaceId: "ws-42" });
// Option.some({ workspaceId: "ws-42" })
const params = workspaceTotals.parse("facts/ws-42");at(params) encodes each parameter with its codec and binds every stream class of the declaration with the parameters that the stream class declares. bound reads facts/ws-42 and writes totals/ws-42.
Parameter rules:
- Each input and output stream class declares a subset of the projection parameter names, with the same types. The compiler checks this.
- A stream class that declares fewer parameters is shared by every binding with the same values for those parameters. A stream class with no parameters is shared by every binding.
parse(id)tries the input stream classes in declaration order and skips fixed addresses. It returnsOption.somewith the decoded parameters of the first stream class that matches the id, orOption.none(). A shared input gives only the parameters that it declares.Projection.pass,Projection.run, andProjection.followtake one bound projection. The package keeps no list of bindings for them, so bind and trigger each one.Projection.attachtakes the declaration. It binds the stream id of each local commit to an input withparse, so each input must declare every projection parameter. See Run a projection.
To keep a set of bindings that changes, keep a registry stream of parameter records. A projection over that stream can start and stop the triggers. See Triggers.
Handler
Each constructor takes its handler in a field with the name of the constructor.
| Constructor | Handler | Runs | Returns |
|---|---|---|---|
fold | fold(state, entry, unit) | Once for each entry | The next value |
emit | emit(entry, unit) | Once for each entry | The output items, keyed by output |
mapAccum | mapAccum(state, entry, unit) | Once for each entry | [value, outputs] |
mapAccumBatch | mapAccumBatch(state, batch, unit) | Once for each unit | [value, outputs] |
process | process(batch, unit, context) | Once for each unit | An Effect of void or WakeAt |
process must return an Effect. The other handlers return the result, or an Effect of it. An entry handler runs over the entries of the unit in order, and each call gets the value that the call before it returned.
| Argument | Contents |
|---|---|
entry | { input, item, index }. input is the input name, item is the decoded item, and index is the position of the entry in the whole unit |
batch | One slice for each input, with from, items, optional offsets, nextOffset, upToDate, and closed. A slice of an input that is gone also has gone: true |
unit | projectionId, version, generation, params, ranges, due, and key |
state | The current value. The first call of a unit gets the stored value, or initial when no value is stored |
context | { prepared, attempt }: the prepared result and run attempt. With outputs, also the bound output addresses. See Tables and effects |
For local inputs, batch.input.offsets[i] and entry.offset are the storage offset of the corresponding item. A read from entry.offset resumes after that entry; the last offset of a non-empty local slice equals its nextOffset. Non-empty HTTP reads have offsets undefined and omit entry.offset, because HTTP provides page boundaries only. All empty slices have offsets: [], including skipped inputs and timer-only units.
unit.params holds the decoded parameter values. unit.ranges has a { from, nextOffset } range for each input that gave items, closed, or is gone in this unit. Inputs with nothing new have no range.
unit.key is a string built from the projection id, the generation, the parameters, the ranges, and the recorded due time when present. A unit with the same ranges and due time has the same key. Commit.producer and Commit.atLeastOnce retry a unit with its saved ranges, so a retry keeps the key. A Commit.transaction retry reads the inputs again, so a longer range gives a new key. See Commits.
When the handler fails, the runner fails with the error of the handler, and the Commit saves no offsets. The next pass reads the same items again.
The host provides the services that the handler needs. Commit.transaction also provides SqlClient to the handler. A runner needs the handler services, the services of the Commit, and the services of each input and output stream.
Helpers
import { Effect } from "effect";
import { CheckpointStore, Commit, Projection } from "@streamsy/projection";
import { Orders } from "./streams.ts";
const logOrders = Projection.process({
id: "log-orders",
params: {},
inputs: { input: Orders },
commit: Commit.atLeastOnce,
process: Projection.each(({ item }, unit) => Effect.log(`${item.orderId} in ${unit.key}`)),
}).at({});
const token = Effect.gen(function* () {
const store = yield* CheckpointStore;
const loaded = yield* store.load(Projection.key(logOrders));
return loaded.token;
});| Helper | Does |
|---|---|
Projection.items(batch) | Returns the entries of a batch as { input, item, index }, in declaration order, then in stream order within each input |
Projection.each(handle) | Makes a process handler that runs handle(entry, unit) for each entry, one at a time and in order. It stops at the first failure |
Projection.key(bound) | Returns { id, version, generation, params } of a bound projection, with the encoded parameters. CheckpointStore takes it as the key |
Use Projection.key to load or remove the checkpoint record of a binding, or to name the binding in logs. It is not unit.key. See Projection.key and Unit.
Next
Run a projection, and choose a Commit on Commits.