Skip to Content
Reference@streamsy/projection

StateTable and FoldByKey

Reference for StateTable, FoldByKey and FoldByKeyError in @streamsy/projection. A State table keeps the rows of a Durable State stream in SQL, and a keyed fold writes rows into it.

import { FoldByKey, FoldByKeyError, StateTable } from "@streamsy/projection";

StateTable keeps the rows of a Durable State stream in SQL tables, and FoldByKey builds process handlers that fold records into those rows. See State tables, Keyed folds and Operate State tables.

The checks on this page compare each shown signature with the real export for one State class, Workspace, and its issue collection.

StateTable

StateTable.define

// A Streamsy.state class with param codecs Codecs and collections C, served by service R.
type StateClass<
  Codecs extends StreamRoute.ParamCodecs,
  C extends StreamAddress.Collections,
  R,
> = StreamRoute.StateRoute<StreamRoute.Params<Codecs>, C> &
  Streamsy.Any & {
    readonly paramCodecs: Codecs;
    readonly at: (
      params: StreamRoute.Params<Codecs>,
    ) => StreamAddress.StateAddress<C> &
      Streamsy.Address<
        StreamAddress.CollectionsEvent<C>,
        R,
        StreamAddress.CollectionsDecodingServices<C>,
        StreamAddress.CollectionsEncodingServices<C>
      >;
  };

declare function define<
  Codecs extends StreamRoute.ParamCodecs,
  C extends StreamAddress.Collections,
  R,
>(
  state: StateClass<Codecs, C, R>,
  options?: { readonly accepted?: undefined },
): StateTable.Declaration<Codecs, C, R>;

define takes a Streamsy.state class and returns the declaration of its table. It reads the param codecs from the paramCodecs of the class. The optional second argument is { accepted: WakeOutput }, with wake parameters drawn from the State class parameters. It adds the wake service requirement to the table declaration. See Accepted stream. The signature above shows the default without an accepted class.

StateTable.Declaration

interface Declaration<
  Codecs extends StreamRoute.ParamCodecs,
  C extends StreamAddress.Collections,
  R,
> {
  readonly id: string;
  readonly version: number;
  readonly generation: number;
  readonly params: Codecs;
  readonly parse: (id: string) => Option.Option<StreamRoute.Params<Codecs>>;
  readonly at: (params: StreamRoute.Params<Codecs>) => StateTable.Bound<C, R>;
  readonly retry?: import("@streamsy/projection").Declaration<
    Codecs,
    StateTable.Bound<C, R>
  >["retry"];
  // Also `inputs`, with the State class as `state`.
}

A runner, Projection.attach and ProjectionWakeup.attach take the declaration like any declaration.

FieldValue
idstate-table/ followed by the service name of the class
version1
generation1
paramsThe param codecs of the State stream, by name
parseThe parse of the State class. It gives the params of a stream id
atBinds the params and gives the bound table

at throws a RangeError when a param does not encode.

StateTable.Bound

interface Bound<C extends StreamAddress.Collections, R> {
  readonly id: string;
  readonly version: number;
  readonly generation: number;
  readonly params: Record<string, string>;
  readonly get: <K extends keyof C & string>(
    collection: K,
    key: string,
  ) => Effect.Effect<
    { readonly offset: string; readonly value: C[K]["schema"]["Type"] | undefined },
    Schema.SchemaError | SqlError.SqlError,
    SqlClient.SqlClient | C[K]["schema"]["DecodingServices"]
  >;
  readonly getMany: <K extends keyof C & string>(
    collection: K,
    keys: ReadonlyArray<string>,
  ) => Effect.Effect<
    { readonly offset: string; readonly rows: ReadonlyMap<string, C[K]["schema"]["Type"]> },
    Schema.SchemaError | SqlError.SqlError,
    SqlClient.SqlClient | C[K]["schema"]["DecodingServices"]
  >;
  readonly list: <K extends keyof C & string>(
    collection: K,
    options?: StateTable.ListOptions,
  ) => Effect.Effect<
    {
      readonly offset: string;
      readonly rows: ReadonlyArray<{
        readonly key: string;
        readonly value: C[K]["schema"]["Type"];
      }>;
      readonly next: string | undefined;
    },
    ProjectionFault | Schema.SchemaError | SqlError.SqlError,
    SqlClient.SqlClient | C[K]["schema"]["DecodingServices"]
  >;
  readonly offset: () => Effect.Effect<string, SqlError.SqlError, SqlClient.SqlClient>;
  readonly sync: () => Effect.Effect<void, ProjectionFault, Transaction | R>;
  // The members that make it a bound projection.
  readonly inputs: { readonly state: Streamsy.Address<string, R> };
  readonly accepted?: Streamsy.Address<Streamsy.WakeEvent, R>;
  readonly commit: typeof Commit.transaction;
  readonly process: (
    batch: Slices<{ readonly state: Streamsy.Address<string, R> }>,
  ) => Effect.Effect<void, ProjectionFault, Transaction | R>;
}

Bound is the bound table. It is also a bound projection, so Projection.run and Projection.follow take it. It uses Commit.transaction. Its one input, state, reads the State stream as stored JSON text, so the table never decodes a stored value with a collection schema.

MemberMeaning
get(collection, key)The row of key. value is undefined when the row is absent
getMany(collection, keys)The rows of keys that exist, by key. It reads 50 keys for each SQL query
list(collection, options?)The rows in key order, after options.after, at most options.limit. next is the last key of a full page, else undefined
offset()The stream offset that the rows reflect. The zero offset when the table has no head
sync()Applies the stream from the head to the tail in the caller's transaction

Each read takes offset first and then the rows, so the rows reflect at least offset. Each read decodes the stored values with the collection schema and fails with a SchemaError when a value does not decode. list fails with a ProjectionFault with phase load and reason invalid-options when limit is not a positive safe integer.

sync deletes the rows of the stream when the head is past the tail. The reads need SqlClient. sync needs Transaction and the service of the State stream. sync fails only with a ProjectionFault:

Causephasereason
The stream does not exist or is gone, or a read does not advancereadhistory-unavailable
The stream rejects the read requestreadinvalid-source
A storage or transport fault of the streamreadstorage-failure
A stored message is not a State messageprocessinvalid-record
A SQL statement on the row tables failsprocessstorage-failure

A storage-failure fault is retryable unless its cause is a stream fault that is not. The cause field holds the original error.

StateTable.migrate

declare function migrate(
  sql: SqlClient.SqlClient,
): Effect.Effect<void, ProjectionFault | SqlError.SqlError>;

migrate creates streamsy_state_heads and streamsy_state_rows with CREATE TABLE IF NOT EXISTS. Run it once at host start, outside storage migrations. On a dialect other than SQLite it fails with a ProjectionFault with phase load and reason unsupported-composition.

StateTable.rebuild

declare function rebuild<C extends StreamAddress.Collections, R>(
  table: StateTable.Bound<C, R>,
): Effect.Effect<void, ProjectionFault, R | Transaction>;

rebuild takes a bound table. In one transaction, it deletes the rows and the head of the stream, then calls sync. The checkpoint of the table binding does not change. A failure rolls back, and the old rows and head stay. It fails with the faults of sync.

StateTable.reset

declare function reset<C extends StreamAddress.Collections, R>(
  table: StateTable.Bound<C, R>,
): Effect.Effect<AppendResult, AppendFailure<{}>, R>;

reset takes a bound table and appends a reset control message to its State stream. It does not touch the rows. The next table run, or the next sync, deletes every row of the stream. See Fix a fold.

StateTable.ListOptions

interface ListOptions {
  readonly after?: string | undefined;
  readonly limit?: number | undefined;
}
FieldMeaning
afterRead rows whose key sorts after this key
limitThe most rows to read. With no limit, list reads them all

FoldByKey

FoldByKey.process

interface Spec<C extends StreamAddress.Collections, K extends keyof C & string, O, A, P, E, R> {
  readonly output: O;
  readonly collection: K;
  readonly key: (record: A, unit: Unit<P>) => string | undefined;
  readonly fold: (
    previous: C[K]["schema"]["Type"] | undefined,
    record: A,
    unit: Unit<P>,
  ) => Step<C[K]["schema"]["Type"] | FoldByKey.Remove, E, R>;
}

// With records: each input item gives records of type A.
declare function process<
  Codecs extends StreamRoute.ParamCodecs,
  C extends StreamAddress.Collections,
  TR,
  K extends keyof C & string,
  O extends string,
  I extends InputMap,
  A,
  P,
  E = never,
  R = never,
>(
  table: StateTable.Declaration<Codecs, C, TR>,
  spec: Spec<C, K, O, A, P, E, R> & {
    readonly records: (item: ItemOf<I[keyof I]>, unit: Unit<P>) => ReadonlyArray<A>;
  },
): FoldByKey.Handler<C, O, I, P, E, R, TR>;
// Without records: each input item is one record.
declare function process<
  Codecs extends StreamRoute.ParamCodecs,
  C extends StreamAddress.Collections,
  TR,
  K extends keyof C & string,
  O extends string,
  I extends InputMap,
  P,
  E = never,
  R = never,
>(
  table: StateTable.Declaration<Codecs, C, TR>,
  spec: Spec<C, K, O, ItemOf<I[keyof I]>, P, E, R> & { readonly records?: never },
): FoldByKey.Handler<C, O, I, P, E, R, TR>;

process builds the process handler of a Projection.process declaration with Commit.transaction. The declaration must list the State stream of table in outputs, under the key spec.output. The handler writes one collection, spec.collection.

FieldMeaning
outputThe key in outputs of the State stream
collectionThe collection that the fold writes. It must be a collection of the State class
recordsOptional. Gives the records of one input item. Without it, each item is one record
keyGives the row key of a record. undefined skips the record
foldGives the next row from the previous row and a record, or FoldByKey.remove to delete the row. It can return an Effect

In each unit the handler creates the output if it is missing, calls sync on the table, groups the records by key in first-seen order, loads the stored rows of those keys, and folds each key's records in order. It drops a key whose new stored JSON equals its stored row, and appends one change for each other key in one append. It does not apply its own append to the table. See What a unit does.

FoldByKey.Handler

// The bound address of the State stream, as `outputs` gives it.
type Output<C extends StreamAddress.Collections, TR> = StreamAddress.StateAddress<C> &
  Streamsy.Address<
    StreamAddress.CollectionsEvent<C>,
    TR,
    StreamAddress.CollectionsDecodingServices<C>,
    StreamAddress.CollectionsEncodingServices<C>
  >;

type Handler<
  C extends StreamAddress.Collections,
  O extends string,
  I extends InputMap,
  P,
  E,
  R,
  TR,
> = (
  batch: Slices<I>,
  unit: Unit<P>,
  context: { readonly outputs: Readonly<Record<O, Output<C, TR>>> },
) => Effect.Effect<
  void,
  E | FoldByKeyError | ProjectionFault,
  | R
  | TR
  | SqlClient.SqlClient
  | Transaction
  | StreamAddress.CollectionsDecodingServices<C>
  | StreamAddress.CollectionsEncodingServices<C>
>;

Handler is the type that FoldByKey.process returns: a process handler for a declaration that lists the State stream in outputs under the key O. E and R are the error and the services of a fold that returns an Effect, and TR is the service of the State stream.

The handler fails with:

FailureWhen
FoldByKeyErrorA folded row's key field differs from key
EA fold Effect fails. The error passes through unchanged
ProjectionFaultsync faults, or the handler cannot read the rows or write the output. See the table below
Causephasereason
A storage or transport fault on the create or the appendprocessstorage-failure
The output is busyprocessoutput-busy
The output otherwise refuses the create or the appendprocessinvalid-output
The SQL read of the stored rows failsprocessstorage-failure
A stored row does not decode with the collection schemaprocessinvalid-record
A folded row does not encode with the collection schemaprocessinvalid-output

A records, key or fold callback that throws is a defect, as in every other handler kind. The handler does not catch it.

FoldByKey.remove

declare const remove: unique symbol;
type Remove = typeof remove;

Return FoldByKey.remove from fold to delete the row of the key. The next record of that key in the unit gets undefined as previous. A key with no stored row that ends the unit removed appends nothing.

FoldByKeyError

interface FoldByKeyError extends Error {
  readonly _tag: "FoldByKeyError";
  readonly message: string;
}

FoldByKeyError is a Schema.TaggedError in the error channel of the handler. It is not a ProjectionFault, so it has no phase and no reason. The handler fails with it when a folded row carries a key that differs from the key that key returned. message names both keys. Import it from the top level of @streamsy/projection.

The runners treat it like any handler error. Under attach in either durability mode the binding defers on its own backoff. See Errors.

On this page