Skip to Content
Reference@streamsy/projection

Projection constructors

Reference for the constructors of the Projection namespace in @streamsy/projection. A constructor declares a projection over input streams, and its at method binds the params.

import {
  Projection,
  type Accumulated,
  type CheckpointStore,
  type Commit,
  type Declaration,
  type Emits,
  type Emitted,
  type Entry,
  type Folded,
  type Slices,
  type StateDefinition,
  type Step,
  type Unit,
} from "@streamsy/projection";
import type { StreamRoute } from "@streamsy/core/protocol";
import type { Effect } from "effect";

A constructor returns an inert Declaration. Its at(params) binds the params, and a runner runs the bound projection. See Define and run.

The signatures use types that have no exported name, such as Common, Addresses, and RunningDeclaration. Types with no exported name shows each one.

Every constructor takes the fields of Common:

FieldMeaning
idThe projection name. It is part of the checkpoint key
versionThe declared version. The default is 1
generationThe checkpoint generation. The default is 1. A new generation has no checkpoint, so it reads each input from the start
paramsThe param codecs, by name. Use {} for a projection with no params
inputsThe input streams, by name: stream declarations, fixed addresses, or both
acceptedOptional wake class for accepted input offsets. Requires Commit.transaction or Commit.producer. See Accepted stream

Each stream declaration in inputs and outputs must declare a subset of params, with the same types. When one does not, the type of its field becomes never, and the compiler rejects the definition. A fixed address takes no params from the projection.

Constructors

Projection.fold

declare function fold<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  A,
  SI,
  E = never,
  R = never,
  P = never,
  Q = CheckpointStore,
  W extends WakeOutput | undefined = undefined,
  HE = never,
  HR = never,
>(
  definition: Common<C, I, W, HE, HR> &
    ([W] extends [undefined]
      ? { readonly commit?: Commit.Commit<AcceptedResult<Folded<NoInfer<A>>, W>, P, Q> }
      : { readonly commit: Commit.Commit<AcceptedResult<Folded<NoInfer<A>>, W>, P, Q> }) & {
      readonly state: StateDefinition<A, SI, StreamRoute.Params<C>>;
      readonly fold: (
        state: NoInfer<A>,
        entry: Entry<Addresses<I>>,
        unit: Unit<StreamRoute.Params<C>>,
      ) => Step<A, E, R>;
    },
): RunningDeclaration<C, I, E, R, Folded<A>, P, Q, A, AcceptedServices<W>, HE, HR>;

fold declares a projection that folds each entry of a unit into one state value. Each local input entry has entry.offset, the storage cursor just after that item; reading from it resumes after the entry. HTTP input entries omit it. The runner calls the fold handler once for each entry, in Projection.items order, with the value that the previous call returned. The handler returns the next value, or an Effect of it.

  • state sets the schema and the initial value. See StateDefinition.
  • commit is required when accepted is set; use Commit.transaction or Commit.producer. Otherwise it is optional. The default is Commit.checkpoint, which saves the encoded value and the input offsets in one compare-and-set write. Commit.transaction also fits fold.
  • After each unit, the runner encodes the value with state.schema. When the value does not encode, the unit fails with a ProjectionFault with phase process and reason invalid-output.
  • When the runner retries a unit, it calls the handler again for each entry. Keep the handler free of external effects. With Commit.transaction, the SQL of the handler commits with the checkpoint.

Projection.process

The process context includes attempt: 1 for a first binding run, increasing after failed runs under either attach host. Every unit in that run sees the same attempt. Projection.run and Projection.follow always supply 1.

// The third argument of `process`, and the declared outputs of `prepare` and `process`.
interface ProcessContext<Prepared = void> {
  readonly prepared: Prepared;
  readonly attempt: number;
}
interface OutputsContext<O extends RoutedMap> {
  readonly outputs: Addresses<O>;
}
type TimerResult<A> = 0 extends 1 & A
  ? void
  : [Extract<A, import("@streamsy/projection").WakeAt>] extends [never]
    ? void
    : void | import("@streamsy/projection").WakeAt;
// Without `outputs`.
declare function process<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  E = never,
  R = never,
  P = never,
  Q = never,
  Prepared = void,
  PE = never,
  PR = never,
  W extends WakeOutput | undefined = undefined,
  HE = never,
  HR = never,
  A = void,
>(
  definition: Common<C, I, W, HE, HR> & {
    readonly commit: Commit.Commit<AcceptedResult<TimerResult<A>, W>, P, Q>;
    readonly outputs?: never;
    readonly prepare?: (
      batch: Slices<Addresses<I>>,
      unit: Unit<StreamRoute.Params<C>>,
    ) => Effect.Effect<Prepared, PE, PR>;
    readonly process: (
      batch: Slices<Addresses<I>>,
      unit: Unit<StreamRoute.Params<C>>,
      context: ProcessContext<Prepared>,
    ) => Effect.Effect<A, E, R>;
  },
): RunningDeclaration<
  C,
  I,
  E | PE,
  R,
  TimerResult<A>,
  P,
  Q | PR,
  never,
  AcceptedServices<W>,
  HE,
  HR
>;
// With `outputs`.
declare function process<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  const O extends RoutedMap,
  E = never,
  R = never,
  Q = never,
  Prepared = void,
  PE = never,
  PR = never,
  W extends WakeOutput | undefined = undefined,
  HE = never,
  HR = never,
  A = void,
>(
  definition: Common<C, I, W, HE, HR> & {
    readonly commit: Commit.Commit<AcceptedResult<TimerResult<A>, W>, SqlClient.SqlClient, Q>;
    readonly outputs: O & OutputsWithin<C, O>;
    readonly prepare?: (
      batch: Slices<Addresses<I>>,
      unit: Unit<StreamRoute.Params<C>>,
      context: OutputsContext<O>,
    ) => Effect.Effect<Prepared, PE, PR>;
    readonly process: (
      batch: Slices<Addresses<I>>,
      unit: Unit<StreamRoute.Params<C>>,
      context: ProcessContext<Prepared> & OutputsContext<O>,
    ) => Effect.Effect<A, E, R>;
  },
): OutputDeclaration<
  C,
  I,
  O,
  E | PE,
  R,
  TimerResult<A>,
  SqlClient.SqlClient,
  Q | PR,
  never,
  AcceptedServices<W>,
  HE,
  HR
>;

process declares a projection that runs an effect for each unit. Local slices carry one storage offset per item in offsets, with the last equal to nextOffset when non-empty. Non-empty HTTP slices have offsets undefined. Every empty slice has offsets: [], including skipped inputs and timer-only units. The runner calls the process handler once for each unit, with one Slice for each input. The projection keeps no state. To handle one entry at a time, wrap the handler with Projection.each.

  • commit is required. Commit.transaction runs process in the SQL transaction of the checkpoint, and Commit.atLeastOnce runs it at least once.
  • prepare is optional and needs Commit.transaction, alone or wrapped by Commit.withCompensation. With another Commit, the process call throws a RangeError.
  • Commit.transaction runs prepare before it opens the transaction, and passes its result to process as context.prepared. A retry of the unit runs prepare again.
  • outputs is optional. It names the streams that the handler appends to, by name, with the same param rule as inputs. Only a Commit that provides SqlClient, such as Commit.transaction, accepts it. The Commit checks every output before the handler runs. prepare and process get the bound addresses as context.outputs. The Commit appends nothing itself, so the handler writes the outputs. See Declared outputs.

Projection.wakeAt

interface WakeAt {
  readonly _tag: "WakeAt";
  readonly wakeAt: number;
}
declare function wakeAt(time: number | DateTime.DateTime): WakeAt;

Return this request from process to replace its recorded wake; returning void clears it. The time must convert to non-negative safe integer epoch milliseconds, otherwise wakeAt throws RangeError. Timers require Commit.atLeastOnce or Commit.transaction, including compensation wrappers. A typed wake request does not fit a custom Commit<void>. Other handler results are discarded. See Timers for due units and runner timing.

Projection.emit

declare function emit<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  const O extends RoutedMap,
  X extends Emits<O>,
  E = never,
  R = never,
  P = never,
  Q = never,
  W extends WakeOutput | undefined = undefined,
  HE = never,
  HR = never,
>(
  definition: Common<C, I, W, HE, HR> & {
    readonly commit: Commit.Commit<Emitted<NoInfer<O>>, P, Q>;
    readonly outputs: O & OutputsWithin<C, O>;
    readonly emit: ((
      entry: Entry<Addresses<I>>,
      unit: Unit<StreamRoute.Params<C>>,
    ) => Step<X, E, R>) &
      (Exclude<keyof X, keyof O> extends never
        ? unknown
        : { readonly undeclaredOutput: Exclude<keyof X, keyof O> });
  },
): OutputDeclaration<C, I, O, E, R, Emitted<O>, P, Q, never, AcceptedServices<W>, HE, HR>;

emit declares a projection that maps each entry to output items. The runner calls the emit handler once for each entry, in Projection.items order. The handler returns an Emits object, or an Effect of it. The projection keeps no state.

  • outputs names the output streams: stream declarations, fixed addresses, or both. Each output must be a JSON or State stream.
  • The Emits object holds an array of items for each output name. A missing name emits nothing. For a name that outputs does not declare, the compiler reports an error on undeclaredOutput.
  • commit is required. Commit.producer appends each output with its own producer position. Commit.transaction appends in the SQL transaction of the checkpoint.
  • at throws a RangeError when two outputs name the same stream, when an output names an input stream, or when an output is not a JSON or State stream.
  • When the value of an output name is not an array, or a change does not fit its collection, the unit fails with a ProjectionFault with phase process and reason invalid-output.

The bound projection also has streams, which holds the address of each output after at binds the params.

Projection.mapAccum

declare function mapAccum<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  const O extends RoutedMap,
  A,
  SI,
  X extends Emits<O>,
  E = never,
  R = never,
  P = never,
  Q = never,
  W extends WakeOutput | undefined = undefined,
  HE = never,
  HR = never,
>(
  definition: Common<C, I, W, HE, HR> & {
    readonly commit: Commit.Commit<Accumulated<NoInfer<A>, NoInfer<O>>, P, Q>;
    readonly outputs: O & OutputsWithin<C, O>;
    readonly state: StateDefinition<A, SI, StreamRoute.Params<C>>;
  } & {
    readonly mapAccum: ((
      state: NoInfer<A>,
      entry: Entry<Addresses<I>>,
      unit: Unit<StreamRoute.Params<C>>,
    ) => Step<readonly [A, X], E, R>) &
      (Exclude<keyof X, keyof O> extends never
        ? unknown
        : { readonly undeclaredOutput: Exclude<keyof X, keyof O> });
  },
): AccumulatingDeclaration<C, I, O, A, E, R, P, Q, AcceptedServices<W>, HE, HR>;

mapAccum declares a projection that keeps a state value and emits output items for each entry. The runner calls the mapAccum handler once for each entry, with the state that the previous call returned. The handler returns [state, emits], or an Effect of it.

state follows the rules of Projection.fold, and outputs and emits follow the rules of Projection.emit. commit is required: Commit.producer or Commit.transaction.

Projection.mapAccumBatch

declare function mapAccumBatch<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  const O extends RoutedMap,
  A,
  SI,
  X extends Emits<O>,
  E = never,
  R = never,
  P = never,
  Q = never,
  W extends WakeOutput | undefined = undefined,
  HE = never,
  HR = never,
>(
  definition: Common<C, I, W, HE, HR> & {
    readonly commit: Commit.Commit<Accumulated<NoInfer<A>, NoInfer<O>>, P, Q>;
    readonly outputs: O & OutputsWithin<C, O>;
    readonly state: StateDefinition<A, SI, StreamRoute.Params<C>>;
  } & {
    readonly mapAccumBatch: ((
      state: NoInfer<A>,
      batch: Slices<Addresses<I>>,
      unit: Unit<StreamRoute.Params<C>>,
    ) => Step<readonly [A, X], E, R>) &
      (Exclude<keyof X, keyof O> extends never
        ? unknown
        : { readonly undeclaredOutput: Exclude<keyof X, keyof O> });
  },
): AccumulatingDeclaration<C, I, O, A, E, R, P, Q, AcceptedServices<W>, HE, HR>;

mapAccumBatch declares a projection that computes the next state value and the output items once for each unit. The runner calls the mapAccumBatch handler with the Slices of the unit. The handler returns [state, emits], or an Effect of it.

The rules of Projection.mapAccum apply. A unit can hold any number of entries, so the handler must give the same final rows for any split of the input into units. The emitted items can differ.

Projection.wake

declare function wake<
  C extends StreamRoute.ParamCodecs,
  const I extends Input,
  const O extends WakeOutput,
  W extends WakeOutput | undefined = undefined,
>(definition: WakeDefinition<C, I, O, W>): WakeDeclaration<C, I, O, W>;

wake declares a wake projection, which writes one row to a wake stream for each unit of a source stream. input is the source, and output is a stream class from Streamsy.wake. Both must declare a subset of params.

  • Each unit appends one upsert { stream, offset, closed? } with Commit.producer. Several source writes in one unit give one row, and the server answers a retry of the append with Duplicate.
  • offset is the nextOffset of the source slice. closed is true when the source is closed.
  • When the source is removed or expires, the projection appends a delete of its row, and the runner then forgets the binding.
  • Commit.producer creates a missing wake stream on its own storage owner. A wake stream on another owner must exist before the first append.
  • The definition has no commit field. The declaration needs CheckpointStore, and its state is always null.
import { Schema } from "effect";
import { Projection } from "@streamsy/projection";
import { Files, UserWake } from "./streams.ts";

const filesWake = Projection.wake({
  id: "files-wake",
  params: { user: Schema.String },
  input: Files,
  output: UserWake,
});

Types with no exported name

The constructor signatures use these types. Apart from WakeOutput, the package does not export them. This section shows them so that you can read the signatures.

// A stream declaration. `at` resolves it with the projection params.
type StreamDeclaration = StreamRoute.StreamRoute<any, any> &
  Streamsy.Any & { readonly at: (params: any) => Streamsy.Address<any, any> };

// One input or output: a stream declaration, or a fixed address.
type Input = StreamDeclaration | Streamsy.Address<any, any>;
type Inputs = Readonly<Record<string, Input>>;
type RoutedMap = Inputs;

// The address of each input after `at` binds the params.
type AddressOf<X> = X extends StreamDeclaration
  ? ReturnType<X["at"]>
  : X extends Streamsy.Address<any, any>
    ? X
    : never;
type Addresses<I extends Inputs> = { readonly [K in keyof I]: AddressOf<I[K]> };

// The fields that every constructor takes. `Within` makes an input `never` when
// its params are not projection params of the same type.
type AcceptedServices<W> = W extends StreamDeclaration ? Services<{ accepted: W }> : never;
type AcceptedResult<Result, W> = W extends WakeOutput
  ? Result extends Folded<infer A>
    ? Accumulated<A, {}>
    : Result extends import("@streamsy/projection").WakeAt
      ? Emitted<{}> & Pick<import("@streamsy/projection").WakeAt, "wakeAt">
      : Emitted<{}>
  : Result;
interface Common<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  W = undefined,
  HE = never,
  HR = never,
> {
  readonly retry?: {
    readonly schedule?: import("effect").Schedule.Schedule<unknown>;
    readonly maxAttempts?: number;
    readonly onExhausted?: (
      unit: Unit<StreamRoute.Params<C>>,
      failure: import("@streamsy/projection").RunnerFailure,
    ) => Effect.Effect<void, HE, HR>;
  };
  readonly id: string;
  readonly version?: number;
  readonly generation?: number;
  readonly params: C;
  readonly inputs: I & Within<C, I>;
  readonly accepted?: W & OneWithin<C, W>;
}
type Within<C extends StreamRoute.ParamCodecs, I extends Inputs> = {
  readonly [K in keyof I]: InvalidInput<C, I[K]> extends never ? I[K] : never;
};
type OutputsWithin<C extends StreamRoute.ParamCodecs, D extends RoutedMap> = Within<C, D>;
type OneWithin<C extends StreamRoute.ParamCodecs, X> = InvalidInput<C, X> extends never ? X : never;
type InvalidInput<C extends StreamRoute.ParamCodecs, X> = X extends StreamDeclaration
  ? Invalid<C, DeclaredParams<X>>
  : never;
type DeclaredParams<X> = X extends StreamRoute.StreamRoute<infer P, any, any, any> ? P : {};
type Invalid<C extends StreamRoute.ParamCodecs, P> = {
  readonly [Name in keyof P]: Name extends keyof StreamRoute.Params<C>
    ? P[Name] extends StreamRoute.Params<C>[Name]
      ? StreamRoute.Params<C>[Name] extends P[Name]
        ? never
        : Name
      : Name
    : Name;
}[keyof P];

// The item type of one output, and the services of all outputs.
type Item<T> =
  AddressOf<T> extends { readonly collections: infer C extends StreamAddress.Collections }
    ? DurableState.Entry<C>
    : AddressOf<T> extends { readonly codec: Schema.Codec<infer A, any, any, any> }
      ? A
      : never;
type Services<O extends RoutedMap> = {
  [K in keyof O]: AddressOf<O[K]> extends Streamsy.Address<any, infer R> ? R : never;
}[keyof O];

// What a constructor returns. `at` gives the bound projection that runners take.
type RunningDeclaration<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  E,
  R,
  Result,
  P,
  Q,
  S = never,
  OR = never,
  HE = never,
  HR = never,
> = Declaration<
  C,
  Omit<
    Executable<Addresses<I>, E | ProjectionFault, R, Q, Result, P, S, OR>,
    "process" | "prepare" | "state"
  > & {
    readonly state: [S] extends [never] ? undefined : BoundState<S>;
    readonly process: (
      batch: Slices<Addresses<I>>,
      unit: Unit,
      encoded?: string,
    ) => Effect.Effect<Result, E | ProjectionFault, R>;
  },
  HE,
  HR
>;
type OutputDeclaration<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  O extends RoutedMap,
  E,
  R,
  Result,
  P,
  Q,
  A = never,
  OR = never,
  HE = never,
  HR = never,
> = Omit<RunningDeclaration<C, I, E, R, Result, P, Q, A, Services<O> | OR, HE, HR>, "at"> & {
  readonly at: (params: StreamRoute.Params<C>) => ReturnType<
    RunningDeclaration<C, I, E, R, Result, P, Q, A, Services<O> | OR, HE, HR>["at"]
  > & {
    readonly streams: Readonly<Record<string, Streamsy.Address<unknown, Services<O> | OR>>>;
  };
};
type AccumulatingDeclaration<
  C extends StreamRoute.ParamCodecs,
  I extends Inputs,
  O extends RoutedMap,
  A,
  E,
  R,
  P,
  Q,
  OR = never,
  HE = never,
  HR = never,
> = OutputDeclaration<C, I, O, E, R, Accumulated<A, O>, P, Q, A, OR, HE, HR>;

// The bound projection that the runners take, and its bound state.
interface Executable<
  I extends InputMap,
  E,
  R,
  Q,
  Result,
  P = never,
  S = unknown,
  OR = never,
> extends Identity {
  readonly inputs: I;
  readonly state?: BoundState<S>;
  readonly streams?: Readonly<Record<string, Streamsy.Address<unknown, OR>>>;
  readonly accepted?: Streamsy.Address<Streamsy.WakeEvent, OR>;
  readonly commit: Commit.Commit<Result, P, Q>;
  readonly prepare?: (batch: Slices<I>, unit: Unit) => Effect.Effect<unknown, E, Q>;
  readonly process: (
    batch: Slices<I>,
    unit: Unit,
    encoded?: string,
    prepared?: unknown,
    attempt?: number,
  ) => Effect.Effect<Result, E, R | P>;
}
interface BoundState<A> {
  readonly schema: StateSchema<A>;
  readonly initial: () => A;
}

// The definition and result of `Projection.wake`.
type WakeOutput<P = any> = StreamDeclaration &
  StreamRoute.StateRoute<P, Streamsy.WakeCollections> & {
    readonly paramCodecs: StreamRoute.ParamCodecs;
  };
interface WakeDefinition<C extends StreamRoute.ParamCodecs, I extends Input, O, W = undefined> {
  readonly accepted?: W & OneWithin<C, W>;
  readonly id: string;
  readonly version?: number;
  readonly generation?: number;
  readonly params: C;
  readonly input: I & OneWithin<C, I>;
  readonly output: O & OneWithin<C, O>;
}
type WakeDeclaration<
  C extends StreamRoute.ParamCodecs,
  I extends Input,
  O extends Input,
  W = undefined,
> = AccumulatingDeclaration<
  C,
  { readonly source: I },
  { readonly wake: O },
  null,
  never,
  never,
  never,
  CheckpointStore,
  AcceptedServices<W>
>;
TypeMeaning
Input, Inputs, RoutedMapA stream declaration or a fixed address, and a record of them by name
AddressOf, AddressesThe address of each input after at binds the params
CommonThe fields that every constructor takes
Within, OutputsWithin, OneWithinThe param check: a stream whose params are not projection params of the same type is never
Item, ServicesThe item type of one output, and the stream services of all outputs
RunningDeclarationThe declaration that fold and process return
OutputDeclaration, AccumulatingDeclarationThe declarations with outputs. Their bound projection also has streams
Executable, BoundStateThe bound projection that the runners take, and its state with initial bound to the params
WakeOutput, WakeDefinition, WakeDeclarationThe output, definition, and result of Projection.wake

WakeOutput<P> is the exported type for a Streamsy.wake class. Its type parameter is the decoded params; omitting it accepts any wake class. WakeOutput<{}> is the fixed-address variant used by AttachOptions.triggers.

On this page