Skip to Content
Reference@streamsy/projection

Units and progress

Reference for the unit, batch, and result types in @streamsy/projection. A unit is one slice of each input that a projection reads, handles, and commits together.

import type {
  AttachOptions,
  Attachable,
  Entry,
  FollowOptions,
  InputMap,
  ItemOf,
  Offsets,
  PassResult,
  Progress,
  Range,
  RunOptions,
  Slice,
  Slices,
  Snapshot,
  Unit,
} from "@streamsy/projection";
import type { Streamsy, StreamAddress } from "@streamsy/core";
import type { StreamRoute } from "@streamsy/core/protocol";
import type { Schedule } from "effect";

A runner reads one page of each input after its checkpoint offset. The pages make one unit: the handler sees them as Slices, and the Commit saves the unit with its Range values. See Handler and Running.

Batches

InputMap

type InputMap = Record<string, Streamsy.Address<unknown, unknown>>;

InputMap holds the bound inputs of a projection: one address for each input name. The order of the names is the read order of every pass, and the order of the entries in Projection.items.

ItemOf

type ItemOf<Address> = Address extends StreamAddress.StreamAddress<infer A> ? A : never;

ItemOf is the decoded item type of an address.

Slice

interface Slice<A> {
  readonly from: string;
  readonly items: ReadonlyArray<A>;
  readonly offsets?: ReadonlyArray<string>;
  readonly nextOffset: string;
  readonly upToDate: boolean;
  readonly closed: boolean;
  readonly gone?: boolean;
}

A Slice holds the items of an input in a unit, and the read position around them.

For local storage inputs, offsets has one storage offset for each item, in the same order. Each offset is also the read cursor just after that item, and offsets.at(-1) === nextOffset for a non-empty local slice. Non-empty inputs read over HTTP have offsets undefined: the protocol supplies page boundaries only. Every empty slice has offsets: [], including skipped inputs, timer-only units, gone inputs and empty pinned ranges.

FieldMeaning
fromThe offset that the read started after. It equals nextOffset when the slice is empty
itemsThe decoded items, in stream order
offsetsOne storage offset for each item on local inputs; undefined for non-empty HTTP reads; [] when empty
nextOffsetThe offset after the last item. The checkpoint moves the input to it
upToDatetrue when the read reached the tail of the input
closedtrue when the input is closed
gonetrue when the input was removed or expired. The runner then forgets the binding

A closed input with no new items still makes one unit, with closed on its slice. The checkpoint records the close, so the next pass does not repeat the empty unit. A removed input with a checkpoint makes one final unit with gone. When an input is missing and has no checkpoint, the pass fails with reason history-unavailable.

Slices

type Slices<Inputs extends InputMap> = {
  readonly [K in keyof Inputs]: Slice<ItemOf<Inputs[K]>>;
};

Slices is the batch of a unit: one Slice for each input, by input name. A batch handler, such as process or mapAccumBatch, takes it. An input with no new items has an empty slice.

Entry

type Entry<Inputs extends InputMap> = {
  readonly [K in keyof Inputs]: {
    readonly input: K;
    readonly item: ItemOf<Inputs[K]>;
    readonly index: number;
    readonly offset?: string;
  };
}[keyof Inputs];

An Entry is an item of a unit, tagged with the name of its input. For a local input, offset is the storage offset of that item. A read from entry.offset resumes after that entry. Entries read over HTTP omit the offset key. index is the position of the item in the whole unit, from 0. Check input to narrow item to the item type of that input. An entry handler, such as fold, emit, or mapAccum, takes one entry at a time.

Units

Unit

interface Unit<P = Record<string, string>> {
  readonly projectionId: string;
  readonly version: number;
  readonly generation: number;
  readonly params: P;
  readonly ranges: Record<string, Range>;
  readonly due: import("effect").Option.Option<number>;
  readonly key: string;
}

A Unit identifies the unit that a handler runs. A handler gets the decoded params as params.

FieldMeaning
projectionIdThe id of the projection
versionThe declared version
generationThe declared generation
paramsThe params that the declaration was bound with
rangesThe range of each input that gave items, a close, or a removal. Other inputs have no entry
dueSome(recordedTime) for a due process unit, otherwise None
keyA string made from the id, the generation, the encoded params, ranges, and the due time when present

A due unit without new input has empty slices and ranges. due is the recorded deadline, not the clock time at execution. Its key includes that time; units without a due time keep their existing key encoding.

A Commit that pins the unit, such as Commit.producer or Commit.atLeastOnce, reads the same ranges on a retry, so key stays the same. A Commit.transaction retry reads the inputs again, and a longer tail changes key. In an entry handler, ranges describes the whole unit, not the entry.

Range

interface Range {
  readonly from: string;
  readonly nextOffset: string;
  readonly closed?: boolean;
  readonly gone?: true;
}

A Range is the part of an input that a unit read: from after from up to nextOffset. closed and gone record a close or a removal of the input. The root entry point exports only the type.

Offsets

type Offsets<Inputs extends InputMap> = { readonly [K in keyof Inputs]: string };

Offsets holds the accepted offset of each input, by input name. The next pass reads after these offsets.

Snapshot

interface Snapshot<S, Inputs extends InputMap> {
  readonly state: S;
  readonly offsets: Offsets<Inputs>;
}

A Snapshot is a state value and the input offsets that it is at. Projection.read and Projection.current return one. state is undefined when the declaration has no state:.

Results

PassResult

interface PassResult<Inputs extends InputMap = InputMap> {
  readonly status: "progress" | "caught-up" | "source-closed";
  readonly units: number;
  readonly items: number;
  readonly offsets: Offsets<Inputs>;
}

Projection.pass returns a PassResult. units is 1 when the pass committed a unit, and 0 otherwise. items counts the input items of that unit. offsets holds the offsets after the pass.

StatusMeaning
progressThe pass committed a unit
caught-upThe pass found no new input
source-closedEvery input is closed and read, or the pass committed the unit of a removed input

Progress

interface Progress<S, Inputs extends InputMap = InputMap> extends Snapshot<S, Inputs> {
  readonly status: "caught-up" | "limit-reached" | "source-closed";
  readonly wakeAt: import("effect").Option.Option<number>;
  readonly units: number;
  readonly items: number;
}

Progress is the result of a cycle of Projection.run, and an element of Projection.follow. state and offsets are the ones that the last pass of the cycle left. units counts the units that the cycle committed, and items counts their input items. wakeAt is the next recorded process wake in epoch milliseconds, or None. A cycle makes at most one due unit without input; return a time after unit.due for the next wake.

StatusMeaning
caught-upThe last pass found no new input
limit-reachedThe cycle committed limit units
source-closedEvery input is closed and read, or an input is gone

Options

RunOptions

interface RunOptions {
  readonly limit?: number;
}

RunOptions holds the options of Projection.run. limit is the most units that a call commits. It has no default: with no limit, run continues until the inputs are caught up or closed. A limit that is not a positive safe integer fails the call with reason invalid-options.

FollowOptions

interface FollowOptions extends RunOptions {
  readonly repairIntervalMs?: number;
  readonly retry?: Schedule.Schedule<unknown>;
}

FollowOptions holds the options of Projection.follow. limit applies to each cycle.

FieldMeaning
repairIntervalMsThe longest wait for a change on an input, in milliseconds. The default is 30000
retryThe schedule for a retry after a retryable fault. The default is ProjectionRetry

repairIntervalMs must be a positive safe integer.

AttachOptions

interface AttachOptions {
  readonly triggers?: ReadonlyArray<Real.WakeOutput<{}>>;
  readonly limit?: number;
  readonly passRetry?: Schedule.Schedule<unknown>;
  readonly durable?: boolean;
  readonly batch?: number;
  readonly concurrency?: number;
}

AttachOptions holds the options of Projection.attach and ProjectionWakeup.attach.

FieldMeaning
limitThe most units for each binding in a run. The default is 100
passRetryTiming only. The runner calls the schedule once per retryable failed pass with no input. The default is ProjectionRetry
durableWhen true, the runner keeps a pending mark for each binding with work in the storage owner. The default is false for Projection.attach and true for ProjectionWakeup.attach
triggersFixed-address wake classes whose rows request binding runs. Remote triggers require the process host. Default []; see Trigger streams
batchThe most due marks that a pass reads. The default is 25
concurrencyThe most bindings run at once. The default is 1; must be a positive safe integer

An attach options object that still contains retry throws a RangeError naming passRetry.

See Binding concurrency for transaction, pool and handler rules.

With durable: false, pending work and timers reset on restart; a recorded timer then waits for its next input.

In both modes, a retryable failed binding uses the declaration's binding retry policy, defaulting to 1 s doubling to 1 hour. Handler errors and defects follow the same rule. A non-retryable projection or storage fault stops its binding until retry, schedule or restart. The passRetry schedule paces retryable failures of the runner's own marks, cursor or trigger operations.

In durable mode, each commit that writes an input also writes a pending mark for each binding that it touches, in the same transaction. A binding that catches up keeps a sleeping mark when its checkpoint records a wake; otherwise it deletes its mark, unless a write during the run marked it again. After a restart, an eviction, or a deploy, the runner reads the marks, runs due bindings and retains future timer deadlines. When a binding fails, the runner backs off or stops its mark and logs a warning, and the other bindings still run. A start makes stopped marks due once. By default a retryable failure starts a 1 s wait, doubling with consecutive failures up to 1 hour. The declaration's retry can change its timing and budget. The runner removes a mark whose params no longer decode, and logs a warning.

Durable mode needs a local memory or SQL storage Layer, which provides the projection marks. Over other storage, such as remote storage, acquiring the runner dies. The runner throws a RangeError when batch is not a positive safe integer.

Attachable

interface Attachable<B, HE = never, HR = never> {
  readonly retry?:
    | {
        readonly schedule?: import("effect").Schedule.Schedule<unknown>;
        readonly maxAttempts?: number;
        onExhausted?(
          this: void,
          unit: import("@streamsy/projection").Unit<unknown>,
          failure: import("@streamsy/projection").RunnerFailure,
        ): import("effect").Effect.Effect<void, HE, HR>;
      }
    | undefined;
  readonly id: string;
  readonly version: number;
  readonly generation: number;
  readonly params: StreamRoute.ParamCodecs;
  readonly inputs: Inputs;
  at(params: Readonly<Record<string, unknown>>): B;
}

// Inputs has no exported name. It is the record of input stream declarations and
// fixed addresses, by name. See /docs/reference/projection/constructors.
type Inputs = Readonly<Record<string, Input>>;

Attachable is the part of a declaration that attach reads. A Declaration from any constructor fits. at is a method, so a declaration with typed params fits too.

On this page