Projection constructors
Reference for the constructors of the Projection namespace in @streamsy/projection. A constructor declares a projection over input streams, and its at method binds the params.
import {
Projection,
type Accumulated,
type CheckpointStore,
type Commit,
type Declaration,
type Emits,
type Emitted,
type Entry,
type Folded,
type Slices,
type StateDefinition,
type Step,
type Unit,
} from "@streamsy/projection";
import type { StreamRoute } from "@streamsy/core/protocol";
import type { Effect } from "effect";A constructor returns an inert Declaration. Its at(params) binds the params, and a runner runs the bound projection. See Define and run.
The signatures use types that have no exported name, such as Common, Addresses, and RunningDeclaration. Types with no exported name shows each one.
Every constructor takes the fields of Common:
| Field | Meaning |
|---|---|
id | The projection name. It is part of the checkpoint key |
version | The declared version. The default is 1 |
generation | The checkpoint generation. The default is 1. A new generation has no checkpoint, so it reads each input from the start |
params | The param codecs, by name. Use {} for a projection with no params |
inputs | The input streams, by name: stream declarations, fixed addresses, or both |
accepted | Optional wake class for accepted input offsets. Requires Commit.transaction or Commit.producer. See Accepted stream |
Each stream declaration in inputs and outputs must declare a subset of params, with the same types. When one does not, the type of its field becomes never, and the compiler rejects the definition. A fixed address takes no params from the projection.
Constructors
Projection.fold
declare function fold<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
A,
SI,
E = never,
R = never,
P = never,
Q = CheckpointStore,
W extends WakeOutput | undefined = undefined,
HE = never,
HR = never,
>(
definition: Common<C, I, W, HE, HR> &
([W] extends [undefined]
? { readonly commit?: Commit.Commit<AcceptedResult<Folded<NoInfer<A>>, W>, P, Q> }
: { readonly commit: Commit.Commit<AcceptedResult<Folded<NoInfer<A>>, W>, P, Q> }) & {
readonly state: StateDefinition<A, SI, StreamRoute.Params<C>>;
readonly fold: (
state: NoInfer<A>,
entry: Entry<Addresses<I>>,
unit: Unit<StreamRoute.Params<C>>,
) => Step<A, E, R>;
},
): RunningDeclaration<C, I, E, R, Folded<A>, P, Q, A, AcceptedServices<W>, HE, HR>;
fold declares a projection that folds each entry of a unit into one state value. Each local input entry has entry.offset, the storage cursor just after that item; reading from it resumes after the entry. HTTP input entries omit it. The runner calls the fold handler once for each entry, in Projection.items order, with the value that the previous call returned. The handler returns the next value, or an Effect of it.
statesets the schema and the initial value. SeeStateDefinition.commitis required whenacceptedis set; useCommit.transactionorCommit.producer. Otherwise it is optional. The default isCommit.checkpoint, which saves the encoded value and the input offsets in one compare-and-set write.Commit.transactionalso fitsfold.- After each unit, the runner encodes the value with
state.schema. When the value does not encode, the unit fails with aProjectionFaultwith phaseprocessand reasoninvalid-output. - When the runner retries a unit, it calls the handler again for each entry. Keep the handler free of external effects. With
Commit.transaction, the SQL of the handler commits with the checkpoint.
Projection.process
The process context includes attempt: 1 for a first binding run, increasing after
failed runs under either attach host. Every unit in that run sees the same attempt.
Projection.run and Projection.follow always supply 1.
// The third argument of `process`, and the declared outputs of `prepare` and `process`.
interface ProcessContext<Prepared = void> {
readonly prepared: Prepared;
readonly attempt: number;
}
interface OutputsContext<O extends RoutedMap> {
readonly outputs: Addresses<O>;
}
type TimerResult<A> = 0 extends 1 & A
? void
: [Extract<A, import("@streamsy/projection").WakeAt>] extends [never]
? void
: void | import("@streamsy/projection").WakeAt;
// Without `outputs`.
declare function process<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
E = never,
R = never,
P = never,
Q = never,
Prepared = void,
PE = never,
PR = never,
W extends WakeOutput | undefined = undefined,
HE = never,
HR = never,
A = void,
>(
definition: Common<C, I, W, HE, HR> & {
readonly commit: Commit.Commit<AcceptedResult<TimerResult<A>, W>, P, Q>;
readonly outputs?: never;
readonly prepare?: (
batch: Slices<Addresses<I>>,
unit: Unit<StreamRoute.Params<C>>,
) => Effect.Effect<Prepared, PE, PR>;
readonly process: (
batch: Slices<Addresses<I>>,
unit: Unit<StreamRoute.Params<C>>,
context: ProcessContext<Prepared>,
) => Effect.Effect<A, E, R>;
},
): RunningDeclaration<
C,
I,
E | PE,
R,
TimerResult<A>,
P,
Q | PR,
never,
AcceptedServices<W>,
HE,
HR
>;
// With `outputs`.
declare function process<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
const O extends RoutedMap,
E = never,
R = never,
Q = never,
Prepared = void,
PE = never,
PR = never,
W extends WakeOutput | undefined = undefined,
HE = never,
HR = never,
A = void,
>(
definition: Common<C, I, W, HE, HR> & {
readonly commit: Commit.Commit<AcceptedResult<TimerResult<A>, W>, SqlClient.SqlClient, Q>;
readonly outputs: O & OutputsWithin<C, O>;
readonly prepare?: (
batch: Slices<Addresses<I>>,
unit: Unit<StreamRoute.Params<C>>,
context: OutputsContext<O>,
) => Effect.Effect<Prepared, PE, PR>;
readonly process: (
batch: Slices<Addresses<I>>,
unit: Unit<StreamRoute.Params<C>>,
context: ProcessContext<Prepared> & OutputsContext<O>,
) => Effect.Effect<A, E, R>;
},
): OutputDeclaration<
C,
I,
O,
E | PE,
R,
TimerResult<A>,
SqlClient.SqlClient,
Q | PR,
never,
AcceptedServices<W>,
HE,
HR
>;
process declares a projection that runs an effect for each unit. Local slices carry one storage offset per item in offsets, with the last equal to nextOffset when non-empty. Non-empty HTTP slices have offsets undefined. Every empty slice has offsets: [], including skipped inputs and timer-only units. The runner calls the process handler once for each unit, with one Slice for each input. The projection keeps no state. To handle one entry at a time, wrap the handler with Projection.each.
commitis required.Commit.transactionrunsprocessin the SQL transaction of the checkpoint, andCommit.atLeastOnceruns it at least once.prepareis optional and needsCommit.transaction, alone or wrapped byCommit.withCompensation. With another Commit, theprocesscall throws aRangeError.Commit.transactionrunspreparebefore it opens the transaction, and passes its result toprocessascontext.prepared. A retry of the unit runsprepareagain.outputsis optional. It names the streams that the handler appends to, by name, with the same param rule asinputs. Only a Commit that providesSqlClient, such asCommit.transaction, accepts it. The Commit checks every output before the handler runs.prepareandprocessget the bound addresses ascontext.outputs. The Commit appends nothing itself, so the handler writes the outputs. See Declared outputs.
Projection.wakeAt
interface WakeAt {
readonly _tag: "WakeAt";
readonly wakeAt: number;
}
declare function wakeAt(time: number | DateTime.DateTime): WakeAt;
Return this request from process to replace its recorded wake; returning void
clears it. The time must convert to non-negative safe integer epoch milliseconds,
otherwise wakeAt throws RangeError. Timers require Commit.atLeastOnce or
Commit.transaction, including compensation wrappers. A typed wake request does
not fit a custom Commit<void>. Other handler results are discarded. See
Timers for due units and runner timing.
Projection.emit
declare function emit<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
const O extends RoutedMap,
X extends Emits<O>,
E = never,
R = never,
P = never,
Q = never,
W extends WakeOutput | undefined = undefined,
HE = never,
HR = never,
>(
definition: Common<C, I, W, HE, HR> & {
readonly commit: Commit.Commit<Emitted<NoInfer<O>>, P, Q>;
readonly outputs: O & OutputsWithin<C, O>;
readonly emit: ((
entry: Entry<Addresses<I>>,
unit: Unit<StreamRoute.Params<C>>,
) => Step<X, E, R>) &
(Exclude<keyof X, keyof O> extends never
? unknown
: { readonly undeclaredOutput: Exclude<keyof X, keyof O> });
},
): OutputDeclaration<C, I, O, E, R, Emitted<O>, P, Q, never, AcceptedServices<W>, HE, HR>;
emit declares a projection that maps each entry to output items. The runner calls the emit handler once for each entry, in Projection.items order. The handler returns an Emits object, or an Effect of it. The projection keeps no state.
outputsnames the output streams: stream declarations, fixed addresses, or both. Each output must be a JSON or State stream.- The
Emitsobject holds an array of items for each output name. A missing name emits nothing. For a name thatoutputsdoes not declare, the compiler reports an error onundeclaredOutput. commitis required.Commit.producerappends each output with its own producer position.Commit.transactionappends in the SQL transaction of the checkpoint.atthrows aRangeErrorwhen two outputs name the same stream, when an output names an input stream, or when an output is not a JSON or State stream.- When the value of an output name is not an array, or a change does not fit its collection, the unit fails with a
ProjectionFaultwith phaseprocessand reasoninvalid-output.
The bound projection also has streams, which holds the address of each output after at binds the params.
Projection.mapAccum
declare function mapAccum<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
const O extends RoutedMap,
A,
SI,
X extends Emits<O>,
E = never,
R = never,
P = never,
Q = never,
W extends WakeOutput | undefined = undefined,
HE = never,
HR = never,
>(
definition: Common<C, I, W, HE, HR> & {
readonly commit: Commit.Commit<Accumulated<NoInfer<A>, NoInfer<O>>, P, Q>;
readonly outputs: O & OutputsWithin<C, O>;
readonly state: StateDefinition<A, SI, StreamRoute.Params<C>>;
} & {
readonly mapAccum: ((
state: NoInfer<A>,
entry: Entry<Addresses<I>>,
unit: Unit<StreamRoute.Params<C>>,
) => Step<readonly [A, X], E, R>) &
(Exclude<keyof X, keyof O> extends never
? unknown
: { readonly undeclaredOutput: Exclude<keyof X, keyof O> });
},
): AccumulatingDeclaration<C, I, O, A, E, R, P, Q, AcceptedServices<W>, HE, HR>;
mapAccum declares a projection that keeps a state value and emits output items for each entry. The runner calls the mapAccum handler once for each entry, with the state that the previous call returned. The handler returns [state, emits], or an Effect of it.
state follows the rules of Projection.fold, and outputs and emits follow the rules of Projection.emit. commit is required: Commit.producer or Commit.transaction.
Projection.mapAccumBatch
declare function mapAccumBatch<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
const O extends RoutedMap,
A,
SI,
X extends Emits<O>,
E = never,
R = never,
P = never,
Q = never,
W extends WakeOutput | undefined = undefined,
HE = never,
HR = never,
>(
definition: Common<C, I, W, HE, HR> & {
readonly commit: Commit.Commit<Accumulated<NoInfer<A>, NoInfer<O>>, P, Q>;
readonly outputs: O & OutputsWithin<C, O>;
readonly state: StateDefinition<A, SI, StreamRoute.Params<C>>;
} & {
readonly mapAccumBatch: ((
state: NoInfer<A>,
batch: Slices<Addresses<I>>,
unit: Unit<StreamRoute.Params<C>>,
) => Step<readonly [A, X], E, R>) &
(Exclude<keyof X, keyof O> extends never
? unknown
: { readonly undeclaredOutput: Exclude<keyof X, keyof O> });
},
): AccumulatingDeclaration<C, I, O, A, E, R, P, Q, AcceptedServices<W>, HE, HR>;
mapAccumBatch declares a projection that computes the next state value and the output items once for each unit. The runner calls the mapAccumBatch handler with the Slices of the unit. The handler returns [state, emits], or an Effect of it.
The rules of Projection.mapAccum apply. A unit can hold any number of entries, so the handler must give the same final rows for any split of the input into units. The emitted items can differ.
Projection.wake
declare function wake<
C extends StreamRoute.ParamCodecs,
const I extends Input,
const O extends WakeOutput,
W extends WakeOutput | undefined = undefined,
>(definition: WakeDefinition<C, I, O, W>): WakeDeclaration<C, I, O, W>;
wake declares a wake projection, which writes one row to a wake stream for each unit of a source stream. input is the source, and output is a stream class from Streamsy.wake. Both must declare a subset of params.
- Each unit appends one upsert
{ stream, offset, closed? }withCommit.producer. Several source writes in one unit give one row, and the server answers a retry of the append withDuplicate. offsetis thenextOffsetof the source slice.closedistruewhen the source is closed.- When the source is removed or expires, the projection appends a delete of its row, and the runner then forgets the binding.
Commit.producercreates a missing wake stream on its own storage owner. A wake stream on another owner must exist before the first append.- The definition has no
commitfield. The declaration needsCheckpointStore, and its state is alwaysnull.
import { Schema } from "effect";
import { Projection } from "@streamsy/projection";
import { Files, UserWake } from "./streams.ts";
const filesWake = Projection.wake({
id: "files-wake",
params: { user: Schema.String },
input: Files,
output: UserWake,
});Types with no exported name
The constructor signatures use these types. Apart from WakeOutput, the package does not export them. This section shows them so that you can read the signatures.
// A stream declaration. `at` resolves it with the projection params.
type StreamDeclaration = StreamRoute.StreamRoute<any, any> &
Streamsy.Any & { readonly at: (params: any) => Streamsy.Address<any, any> };
// One input or output: a stream declaration, or a fixed address.
type Input = StreamDeclaration | Streamsy.Address<any, any>;
type Inputs = Readonly<Record<string, Input>>;
type RoutedMap = Inputs;
// The address of each input after `at` binds the params.
type AddressOf<X> = X extends StreamDeclaration
? ReturnType<X["at"]>
: X extends Streamsy.Address<any, any>
? X
: never;
type Addresses<I extends Inputs> = { readonly [K in keyof I]: AddressOf<I[K]> };
// The fields that every constructor takes. `Within` makes an input `never` when
// its params are not projection params of the same type.
type AcceptedServices<W> = W extends StreamDeclaration ? Services<{ accepted: W }> : never;
type AcceptedResult<Result, W> = W extends WakeOutput
? Result extends Folded<infer A>
? Accumulated<A, {}>
: Result extends import("@streamsy/projection").WakeAt
? Emitted<{}> & Pick<import("@streamsy/projection").WakeAt, "wakeAt">
: Emitted<{}>
: Result;
interface Common<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
W = undefined,
HE = never,
HR = never,
> {
readonly retry?: {
readonly schedule?: import("effect").Schedule.Schedule<unknown>;
readonly maxAttempts?: number;
readonly onExhausted?: (
unit: Unit<StreamRoute.Params<C>>,
failure: import("@streamsy/projection").RunnerFailure,
) => Effect.Effect<void, HE, HR>;
};
readonly id: string;
readonly version?: number;
readonly generation?: number;
readonly params: C;
readonly inputs: I & Within<C, I>;
readonly accepted?: W & OneWithin<C, W>;
}
type Within<C extends StreamRoute.ParamCodecs, I extends Inputs> = {
readonly [K in keyof I]: InvalidInput<C, I[K]> extends never ? I[K] : never;
};
type OutputsWithin<C extends StreamRoute.ParamCodecs, D extends RoutedMap> = Within<C, D>;
type OneWithin<C extends StreamRoute.ParamCodecs, X> = InvalidInput<C, X> extends never ? X : never;
type InvalidInput<C extends StreamRoute.ParamCodecs, X> = X extends StreamDeclaration
? Invalid<C, DeclaredParams<X>>
: never;
type DeclaredParams<X> = X extends StreamRoute.StreamRoute<infer P, any, any, any> ? P : {};
type Invalid<C extends StreamRoute.ParamCodecs, P> = {
readonly [Name in keyof P]: Name extends keyof StreamRoute.Params<C>
? P[Name] extends StreamRoute.Params<C>[Name]
? StreamRoute.Params<C>[Name] extends P[Name]
? never
: Name
: Name
: Name;
}[keyof P];
// The item type of one output, and the services of all outputs.
type Item<T> =
AddressOf<T> extends { readonly collections: infer C extends StreamAddress.Collections }
? DurableState.Entry<C>
: AddressOf<T> extends { readonly codec: Schema.Codec<infer A, any, any, any> }
? A
: never;
type Services<O extends RoutedMap> = {
[K in keyof O]: AddressOf<O[K]> extends Streamsy.Address<any, infer R> ? R : never;
}[keyof O];
// What a constructor returns. `at` gives the bound projection that runners take.
type RunningDeclaration<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
E,
R,
Result,
P,
Q,
S = never,
OR = never,
HE = never,
HR = never,
> = Declaration<
C,
Omit<
Executable<Addresses<I>, E | ProjectionFault, R, Q, Result, P, S, OR>,
"process" | "prepare" | "state"
> & {
readonly state: [S] extends [never] ? undefined : BoundState<S>;
readonly process: (
batch: Slices<Addresses<I>>,
unit: Unit,
encoded?: string,
) => Effect.Effect<Result, E | ProjectionFault, R>;
},
HE,
HR
>;
type OutputDeclaration<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
O extends RoutedMap,
E,
R,
Result,
P,
Q,
A = never,
OR = never,
HE = never,
HR = never,
> = Omit<RunningDeclaration<C, I, E, R, Result, P, Q, A, Services<O> | OR, HE, HR>, "at"> & {
readonly at: (params: StreamRoute.Params<C>) => ReturnType<
RunningDeclaration<C, I, E, R, Result, P, Q, A, Services<O> | OR, HE, HR>["at"]
> & {
readonly streams: Readonly<Record<string, Streamsy.Address<unknown, Services<O> | OR>>>;
};
};
type AccumulatingDeclaration<
C extends StreamRoute.ParamCodecs,
I extends Inputs,
O extends RoutedMap,
A,
E,
R,
P,
Q,
OR = never,
HE = never,
HR = never,
> = OutputDeclaration<C, I, O, E, R, Accumulated<A, O>, P, Q, A, OR, HE, HR>;
// The bound projection that the runners take, and its bound state.
interface Executable<
I extends InputMap,
E,
R,
Q,
Result,
P = never,
S = unknown,
OR = never,
> extends Identity {
readonly inputs: I;
readonly state?: BoundState<S>;
readonly streams?: Readonly<Record<string, Streamsy.Address<unknown, OR>>>;
readonly accepted?: Streamsy.Address<Streamsy.WakeEvent, OR>;
readonly commit: Commit.Commit<Result, P, Q>;
readonly prepare?: (batch: Slices<I>, unit: Unit) => Effect.Effect<unknown, E, Q>;
readonly process: (
batch: Slices<I>,
unit: Unit,
encoded?: string,
prepared?: unknown,
attempt?: number,
) => Effect.Effect<Result, E, R | P>;
}
interface BoundState<A> {
readonly schema: StateSchema<A>;
readonly initial: () => A;
}
// The definition and result of `Projection.wake`.
type WakeOutput<P = any> = StreamDeclaration &
StreamRoute.StateRoute<P, Streamsy.WakeCollections> & {
readonly paramCodecs: StreamRoute.ParamCodecs;
};
interface WakeDefinition<C extends StreamRoute.ParamCodecs, I extends Input, O, W = undefined> {
readonly accepted?: W & OneWithin<C, W>;
readonly id: string;
readonly version?: number;
readonly generation?: number;
readonly params: C;
readonly input: I & OneWithin<C, I>;
readonly output: O & OneWithin<C, O>;
}
type WakeDeclaration<
C extends StreamRoute.ParamCodecs,
I extends Input,
O extends Input,
W = undefined,
> = AccumulatingDeclaration<
C,
{ readonly source: I },
{ readonly wake: O },
null,
never,
never,
never,
CheckpointStore,
AcceptedServices<W>
>;
| Type | Meaning |
|---|---|
Input, Inputs, RoutedMap | A stream declaration or a fixed address, and a record of them by name |
AddressOf, Addresses | The address of each input after at binds the params |
Common | The fields that every constructor takes |
Within, OutputsWithin, OneWithin | The param check: a stream whose params are not projection params of the same type is never |
Item, Services | The item type of one output, and the stream services of all outputs |
RunningDeclaration | The declaration that fold and process return |
OutputDeclaration, AccumulatingDeclaration | The declarations with outputs. Their bound projection also has streams |
Executable, BoundState | The bound projection that the runners take, and its state with initial bound to the params |
WakeOutput, WakeDefinition, WakeDeclaration | The output, definition, and result of Projection.wake |
WakeOutput<P> is the exported type for a Streamsy.wake class. Its type parameter is the decoded params; omitting it accepts any wake class. WakeOutput<{}> is the fixed-address variant used by AttachOptions.triggers.
Testing
Reference for the @streamsy/core/testing entry point. It has the storage contract tests, fault injection, and a protocol Layer for tests.
Projection runners
Reference for the runners and reads of the Projection namespace in @streamsy/projection. A runner reads the inputs of a bound projection, runs its handler, and commits each unit.