Units and progress
Reference for the unit, batch, and result types in @streamsy/projection. A unit is one slice of each input that a projection reads, handles, and commits together.
import type {
AttachOptions,
Attachable,
Entry,
FollowOptions,
InputMap,
ItemOf,
Offsets,
PassResult,
Progress,
Range,
RunOptions,
Slice,
Slices,
Snapshot,
Unit,
} from "@streamsy/projection";
import type { Streamsy, StreamAddress } from "@streamsy/core";
import type { StreamRoute } from "@streamsy/core/protocol";
import type { Schedule } from "effect";A runner reads one page of each input after its checkpoint offset. The pages make one unit: the handler sees them as Slices, and the Commit saves the unit with its Range values. See Handler and Running.
Batches
InputMap
type InputMap = Record<string, Streamsy.Address<unknown, unknown>>;
InputMap holds the bound inputs of a projection: one address for each input name. The order of the names is the read order of every pass, and the order of the entries in Projection.items.
ItemOf
type ItemOf<Address> = Address extends StreamAddress.StreamAddress<infer A> ? A : never;
ItemOf is the decoded item type of an address.
Slice
interface Slice<A> {
readonly from: string;
readonly items: ReadonlyArray<A>;
readonly offsets?: ReadonlyArray<string>;
readonly nextOffset: string;
readonly upToDate: boolean;
readonly closed: boolean;
readonly gone?: boolean;
}
A Slice holds the items of an input in a unit, and the read position around them.
For local storage inputs, offsets has one storage offset for each item, in the same order. Each offset is also the read cursor just after that item, and offsets.at(-1) === nextOffset for a non-empty local slice. Non-empty inputs read over HTTP have offsets undefined: the protocol supplies page boundaries only. Every empty slice has offsets: [], including skipped inputs, timer-only units, gone inputs and empty pinned ranges.
| Field | Meaning |
|---|---|
from | The offset that the read started after. It equals nextOffset when the slice is empty |
items | The decoded items, in stream order |
offsets | One storage offset for each item on local inputs; undefined for non-empty HTTP reads; [] when empty |
nextOffset | The offset after the last item. The checkpoint moves the input to it |
upToDate | true when the read reached the tail of the input |
closed | true when the input is closed |
gone | true when the input was removed or expired. The runner then forgets the binding |
A closed input with no new items still makes one unit, with closed on its slice. The checkpoint records the close, so the next pass does not repeat the empty unit. A removed input with a checkpoint makes one final unit with gone. When an input is missing and has no checkpoint, the pass fails with reason history-unavailable.
Slices
type Slices<Inputs extends InputMap> = {
readonly [K in keyof Inputs]: Slice<ItemOf<Inputs[K]>>;
};
Slices is the batch of a unit: one Slice for each input, by input name. A batch handler, such as process or mapAccumBatch, takes it. An input with no new items has an empty slice.
Entry
type Entry<Inputs extends InputMap> = {
readonly [K in keyof Inputs]: {
readonly input: K;
readonly item: ItemOf<Inputs[K]>;
readonly index: number;
readonly offset?: string;
};
}[keyof Inputs];
An Entry is an item of a unit, tagged with the name of its input. For a local input, offset is the storage offset of that item. A read from entry.offset resumes after that entry. Entries read over HTTP omit the offset key. index is the position of the item in the whole unit, from 0. Check input to narrow item to the item type of that input. An entry handler, such as fold, emit, or mapAccum, takes one entry at a time.
Units
Unit
interface Unit<P = Record<string, string>> {
readonly projectionId: string;
readonly version: number;
readonly generation: number;
readonly params: P;
readonly ranges: Record<string, Range>;
readonly due: import("effect").Option.Option<number>;
readonly key: string;
}
A Unit identifies the unit that a handler runs. A handler gets the decoded params as params.
| Field | Meaning |
|---|---|
projectionId | The id of the projection |
version | The declared version |
generation | The declared generation |
params | The params that the declaration was bound with |
ranges | The range of each input that gave items, a close, or a removal. Other inputs have no entry |
due | Some(recordedTime) for a due process unit, otherwise None |
key | A string made from the id, the generation, the encoded params, ranges, and the due time when present |
A due unit without new input has empty slices and ranges. due is the recorded deadline, not the clock time at execution. Its key includes that time; units without a due time keep their existing key encoding.
A Commit that pins the unit, such as Commit.producer or Commit.atLeastOnce, reads the same ranges on a retry, so key stays the same. A Commit.transaction retry reads the inputs again, and a longer tail changes key. In an entry handler, ranges describes the whole unit, not the entry.
Range
interface Range {
readonly from: string;
readonly nextOffset: string;
readonly closed?: boolean;
readonly gone?: true;
}
A Range is the part of an input that a unit read: from after from up to nextOffset. closed and gone record a close or a removal of the input. The root entry point exports only the type.
Offsets
type Offsets<Inputs extends InputMap> = { readonly [K in keyof Inputs]: string };
Offsets holds the accepted offset of each input, by input name. The next pass reads after these offsets.
Snapshot
interface Snapshot<S, Inputs extends InputMap> {
readonly state: S;
readonly offsets: Offsets<Inputs>;
}
A Snapshot is a state value and the input offsets that it is at. Projection.read and Projection.current return one. state is undefined when the declaration has no state:.
Results
PassResult
interface PassResult<Inputs extends InputMap = InputMap> {
readonly status: "progress" | "caught-up" | "source-closed";
readonly units: number;
readonly items: number;
readonly offsets: Offsets<Inputs>;
}
Projection.pass returns a PassResult. units is 1 when the pass committed a unit, and 0 otherwise. items counts the input items of that unit. offsets holds the offsets after the pass.
| Status | Meaning |
|---|---|
progress | The pass committed a unit |
caught-up | The pass found no new input |
source-closed | Every input is closed and read, or the pass committed the unit of a removed input |
Progress
interface Progress<S, Inputs extends InputMap = InputMap> extends Snapshot<S, Inputs> {
readonly status: "caught-up" | "limit-reached" | "source-closed";
readonly wakeAt: import("effect").Option.Option<number>;
readonly units: number;
readonly items: number;
}
Progress is the result of a cycle of Projection.run, and an element of Projection.follow. state and offsets are the ones that the last pass of the cycle left. units counts the units that the cycle committed, and items counts their input items. wakeAt is the next recorded process wake in epoch milliseconds, or None. A cycle makes at most one due unit without input; return a time after unit.due for the next wake.
| Status | Meaning |
|---|---|
caught-up | The last pass found no new input |
limit-reached | The cycle committed limit units |
source-closed | Every input is closed and read, or an input is gone |
Options
RunOptions
interface RunOptions {
readonly limit?: number;
}
RunOptions holds the options of Projection.run. limit is the most units that a call commits. It has no default: with no limit, run continues until the inputs are caught up or closed. A limit that is not a positive safe integer fails the call with reason invalid-options.
FollowOptions
interface FollowOptions extends RunOptions {
readonly repairIntervalMs?: number;
readonly retry?: Schedule.Schedule<unknown>;
}
FollowOptions holds the options of Projection.follow. limit applies to each cycle.
| Field | Meaning |
|---|---|
repairIntervalMs | The longest wait for a change on an input, in milliseconds. The default is 30000 |
retry | The schedule for a retry after a retryable fault. The default is ProjectionRetry |
repairIntervalMs must be a positive safe integer.
AttachOptions
interface AttachOptions {
readonly triggers?: ReadonlyArray<Real.WakeOutput<{}>>;
readonly limit?: number;
readonly passRetry?: Schedule.Schedule<unknown>;
readonly durable?: boolean;
readonly batch?: number;
readonly concurrency?: number;
}
AttachOptions holds the options of Projection.attach and ProjectionWakeup.attach.
| Field | Meaning |
|---|---|
limit | The most units for each binding in a run. The default is 100 |
passRetry | Timing only. The runner calls the schedule once per retryable failed pass with no input. The default is ProjectionRetry |
durable | When true, the runner keeps a pending mark for each binding with work in the storage owner. The default is false for Projection.attach and true for ProjectionWakeup.attach |
triggers | Fixed-address wake classes whose rows request binding runs. Remote triggers require the process host. Default []; see Trigger streams |
batch | The most due marks that a pass reads. The default is 25 |
concurrency | The most bindings run at once. The default is 1; must be a positive safe integer |
An attach options object that still contains retry throws a RangeError naming passRetry.
See Binding concurrency for transaction, pool and handler rules.
With durable: false, pending work and timers reset on restart; a recorded timer then waits for its next input.
In both modes, a retryable failed binding uses the declaration's binding retry policy, defaulting to 1 s doubling to 1 hour. Handler errors and defects follow the same rule. A non-retryable projection or storage fault stops its binding until retry, schedule or restart. The passRetry schedule paces retryable failures of the runner's own marks, cursor or trigger operations.
In durable mode, each commit that writes an input also writes a pending mark for each binding that it touches, in the same transaction. A binding that catches up keeps a sleeping mark when its checkpoint records a wake; otherwise it deletes its mark, unless a write during the run marked it again. After a restart, an eviction, or a deploy, the runner reads the marks, runs due bindings and retains future timer deadlines. When a binding fails, the runner backs off or stops its mark and logs a warning, and the other bindings still run. A start makes stopped marks due once. By default a retryable failure starts a 1 s wait, doubling with consecutive failures up to 1 hour. The declaration's retry can change its timing and budget. The runner removes a mark whose params no longer decode, and logs a warning.
Durable mode needs a local memory or SQL storage Layer, which provides the projection marks. Over other storage, such as remote storage, acquiring the runner dies. The runner throws a RangeError when batch is not a positive safe integer.
Attachable
interface Attachable<B, HE = never, HR = never> {
readonly retry?:
| {
readonly schedule?: import("effect").Schedule.Schedule<unknown>;
readonly maxAttempts?: number;
onExhausted?(
this: void,
unit: import("@streamsy/projection").Unit<unknown>,
failure: import("@streamsy/projection").RunnerFailure,
): import("effect").Effect.Effect<void, HE, HR>;
}
| undefined;
readonly id: string;
readonly version: number;
readonly generation: number;
readonly params: StreamRoute.ParamCodecs;
readonly inputs: Inputs;
at(params: Readonly<Record<string, unknown>>): B;
}
// Inputs has no exported name. It is the record of input stream declarations and
// fixed addresses, by name. See /docs/reference/projection/constructors.
type Inputs = Readonly<Record<string, Input>>;
Attachable is the part of a declaration that attach reads. A Declaration from any constructor fits. at is a method, so a declaration with typed params fits too.
Declarations and state
Reference for the declaration and state types in @streamsy/projection. A declaration binds params into a projection, and a state definition sets the value that a projection keeps.
Commit
Reference for the Commit namespace and the Transaction service in @streamsy/projection. A Commit decides how a projection writes each unit and its checkpoint, and which guarantee it gives.