Wake streams
Reference for the wake stream members of the Streamsy namespace in @streamsy/core. A wake stream tells a client which source streams moved.
import { Streamsy, StreamAddress, type FollowFailure, type StreamId } from "@streamsy/core";
import type { StreamRoute, Streams } from "@streamsy/core/protocol";
import type { Context, Data, Schema, Stream } from "effect";A wake stream is a Durable State stream with one row for each source stream in a family. Each row holds the tail of its source after the last write, so a client follows one wake stream and reads only the sources that moved past its saved cursors. These members belong to the Streamsy namespace.
A wake declaration names no sources. Projection.wake from @streamsy/projection writes the rows: it links one source stream to one wake stream. For a host that runs Projection.wake, see Projections on Cloudflare.
Functions
Streamsy.wake
declare function wake<Self>(): <
const Name extends string,
const Template extends string,
C extends StreamRoute.ParamCodecs,
>(
name: Name,
template: Template,
options: { readonly params: C & StreamRoute.ExactTemplateParams<Template, C> },
) => Streamsy.StateClass<Self, Name, Template, C, Streamsy.WakeCollections>;
wake declares a wake stream class. It calls Streamsy.state with wakeCollections as the collections, so the class has the normal State service, including create, append, remove, read, follow, and collect.
import { Schema } from "effect";
import { Streamsy } from "@streamsy/core";
class UserWake extends Streamsy.wake<UserWake>()("App/UserWake", "users/:user/wake", {
params: { user: Schema.String },
}) {}The rules for name, template, and params are the same as for Streamsy.json. A wake stream on another storage owner must exist before a projection appends to it, so that owner creates the wake stream when it starts.
wakeActions compares each row offset with a saved cursor, so it assumes that the offsets of a source stream only grow. Do not create a stream again under the id of a removed or expired stream. The new stream starts at a lower offset, and a client with a saved cursor for the old stream skips its items.
Streamsy.followWake
declare function followWake<Self, P>(
declaration: Context.Key<Self, Streamsy.Api<P, Streamsy.WakeEvent>>,
params: P,
options: Streamsy.FollowWakeOptions,
): Stream.Stream<Streamsy.WakeBatch, FollowFailure, Self>;
followWake follows the wake stream params of declaration from options.from, and emits one WakeBatch for each batch that it reads. It calls wakeActions on each batch and carries an open snapshot from one batch to the next. It reads no source stream.
followWake reads options.cursors again for each batch, so you can update the map as you read the sources. Save each source cursor first and the wake cursor offset last. When the process stops between the two, the next run reads the same rows again, and wakeActions skips each source whose saved cursor already reaches the row offset.
import { Streamsy, type StreamId } from "@streamsy/core";
import { UserWake } from "./wake.ts";
const cursors = new Map<StreamId, string>();
const batches = Streamsy.followWake(UserWake, { user: "ann" }, { from: "-1", cursors });Streamsy.awaitWake
declare function awaitWake<Self, P>(
declaration: Context.Key<Self, Streamsy.Api<P, Streamsy.WakeEvent>>,
params: P,
target: { readonly stream: StreamId; readonly offset: string },
options?: Streamsy.AwaitWakeOptions,
): Effect.Effect<void, FollowFailure, Self>;
Wait for the source row to reach the requested offset under core compare. The helper catches up before checking, respecting resets, removals and snapshot controls across history pages. It uses the wake service over local or remote storage. AwaitWakeOptions is Omit<Streamsy.FollowWakeOptions, "cursors">: pass from to resume after a saved wake cursor, plus any Streams.FollowOptions except offset. Without from, each call reads the wake stream from the start. A saved cursor skips earlier rows, so the target must be observed after that cursor. Callers choose their own Effect.timeout; a closed wake stream without a matching row leaves the wait pending. See Accepted stream.
Streamsy.wakeActions
declare function wakeActions(
events: ReadonlyArray<Streamsy.WakeEvent>,
options: Streamsy.WakeActionsOptions,
): Streamsy.WakeActions;
wakeActions returns the actions for a batch of wake events. It reads no stream and changes none of its inputs. A client with its own HTTP reader decodes each page with WakeMessage and calls wakeActions directly. Pass the returned snapshot to the call for the next page.
wakeActions applies these rules, in event order:
- It collapses the rows of one stream in the batch to one item, at the position of the last row for that stream. The one exception is a delete and then an upsert, below.
- An upsert gives
Dirtywhen the rowoffsetsorts after the saved cursor of the stream, or when the row hasclosed. The saved cursor is the entry incursors, or-1when the map has none. - A delete gives
Removed. A delete and then an upsert of the same stream giveRemoved, thenDirtyfrom-1. - A
resetcontrol message drops the items before it in the batch, closes an open snapshot, and givesReset. Each later upsert in the batch givesDirtyfrom-1. - A
snapshot-startcontrol message opens a snapshot. Thesnapshot-endcontrol message that closes it gives the items of the rows before it, thenSnapshot.Snapshotholds each stream that an upsert named sincesnapshot-startand that no later delete removed.
A snapshot-end with no open snapshot gives no item.
Rows and messages
Streamsy.WakeRow
type WakeRow = {
readonly stream: StreamId;
readonly offset: Offset;
readonly closed?: true;
};
WakeRow is a row of a wake stream: the latest known state of a source stream. Streamsy.WakeRow is also the Schema of a row.
| Field | Meaning |
|---|---|
stream | The id of the source stream. It is also the State key of the row |
offset | A source position the writer vouches for: written in a source wake stream, accepted in an accepted stream |
closed | true when the source stream is closed. A row for an open stream leaves it out |
A row lags its source by one projection pass, because only the projection writes the row. A write to the source alone writes no row.
Streamsy.wakeCollections
declare const wakeCollections: {
readonly stream: { readonly schema: typeof Streamsy.WakeRow; readonly key: "stream" };
};
wakeCollections holds the only collection of every wake stream. Its State type is stream, and its key is the stream field of the row. A class from Streamsy.wake has wakeCollections as its collections.
Streamsy.WakeCollections
type WakeCollections = typeof Streamsy.wakeCollections;
WakeCollections is the type of wakeCollections.
Streamsy.WakeEvent
type WakeEvent = StreamAddress.CollectionsEvent<Streamsy.WakeCollections>;
WakeEvent is an item of a wake stream: a change to a WakeRow, or a State control message. See StreamAddress.CollectionsEvent.
Streamsy.WakeMessage
declare const WakeMessage: Schema.Codec<
Streamsy.WakeEvent,
StreamAddress.CollectionsEncodedEvent<Streamsy.WakeCollections>
>;
WakeMessage is the codec of a wake stream message: a stream change or a State control message. A client that reads raw JSON pages decodes each message with it, in place of a typed service.
import { Schema } from "effect";
import { Streamsy } from "@streamsy/core";
const decodePage = Schema.decodeUnknownEffect(Schema.Array(Streamsy.WakeMessage));Actions
Streamsy.WakeItem
type WakeItem = Data.TaggedEnum<{
Dirty: {
readonly stream: StreamId;
readonly from: string;
readonly offset: string;
readonly closed: boolean;
};
Removed: { readonly stream: StreamId };
Reset: {};
Snapshot: { readonly streams: ReadonlySet<StreamId> };
}>;
declare const WakeItem: Data.TaggedEnum.Constructor<Streamsy.WakeItem>;
WakeItem is an action for the consumer of a wake stream. The value Streamsy.WakeItem holds a constructor for each tag, and $is and $match.
| Tag | Action |
|---|---|
Dirty | Read stream from from up to at least offset, then save its cursor |
Removed | Drop the cursor and the local data of stream |
Reset | Drop every cursor. The items after this one start each stream from -1 |
Snapshot | A complete snapshot ended. Drop each stream that streams does not hold |
closed on Dirty is true when the row says that the source stream is closed.
Streamsy.WakeBatch
interface WakeBatch {
readonly items: ReadonlyArray<Streamsy.WakeItem>;
readonly offset: string;
readonly upToDate: boolean;
}
WakeBatch holds the actions of a batch of the wake stream. followWake emits one for each batch that it reads. offset is the nextOffset of that batch. Save it as the wake cursor after you handle every item. upToDate is true when the batch reached the tail of the wake stream.
Streamsy.FollowWakeOptions
interface FollowWakeOptions extends Omit<Streams.FollowOptions, "offset"> {
readonly from?: string;
readonly cursors: ReadonlyMap<StreamId, string>;
}
FollowWakeOptions holds the options of followWake. It takes the follow options of a stream, such as retry, with from in place of offset.
| Field | Meaning |
|---|---|
from | The saved wake cursor. The default is -1, which reads the wake stream from the start |
cursors | The saved source cursors, by stream id. followWake reads the map again for each batch |
Streamsy.WakeActionsOptions
interface WakeActionsOptions {
readonly cursors: ReadonlyMap<StreamId, string>;
readonly snapshot?: ReadonlySet<StreamId> | undefined;
}
WakeActionsOptions holds the options of wakeActions.
| Field | Meaning |
|---|---|
cursors | The saved source cursors, by stream id. A stream with no entry starts from -1 |
snapshot | The snapshot that the call for the previous page returned. Leave it out for the first page |
Streamsy.WakeActions
interface WakeActions {
readonly items: ReadonlyArray<Streamsy.WakeItem>;
readonly snapshot: ReadonlySet<StreamId> | undefined;
}
WakeActions is the result of wakeActions. items holds the actions of the batch, in order. snapshot holds the streams of a snapshot that is still open at the end of the batch, or undefined when no snapshot is open. Pass snapshot to the call for the next batch.