CheckpointStore
Reference for the checkpoint store in @streamsy/projection and @streamsy/projection/checkpoint. The store keeps the offsets and the state of each projection key in one compare-and-set record.
import {
CheckpointRecord,
CheckpointStore,
type CheckpointStoreApi,
type Identity,
type Loaded,
type ProjectionKey,
} from "@streamsy/projection";
import { fromStore, recordKey, type EncodedStore } from "@streamsy/projection/checkpoint";
import type { CommitBoundary, Streamsy } from "@streamsy/core";
import type { Effect, Layer, Option } from "effect";CheckpointStore keeps one record for each projection key: the input offsets, the encoded state, and a pinned unit. The built-in Commits need it. See Checkpoints and generations and Projection storage.
@streamsy/projection/checkpoint is the entry point for the author of a checkpoint Layer or a custom Commit. Each export has one section on this page:
| Export | @streamsy/projection | @streamsy/projection/checkpoint |
|---|---|---|
CheckpointStore | Yes | Yes |
CheckpointStoreApi | Yes | Yes |
CheckpointRecord | Yes | Yes |
Loaded | Yes | Yes |
ProjectionKey | Yes | Yes |
Identity | Yes | |
fromStore | Yes | |
recordKey | Yes | |
EncodedStore | Yes |
The store
CheckpointStore
class CheckpointStore extends Context.Service<CheckpointStore, CheckpointStoreApi>()(
"@streamsy/projection/CheckpointStore",
) {
static readonly layer: Layer.Layer<CheckpointStore, never, CommitBoundary | OwnerRecords>;
static readonly layerMemory: Layer.Layer<CheckpointStore>;
}
// OwnerRecords has no exported name in a public entry point. It is the service
// of the encoded records on the storage owner. Every storage host provides it.
interface OwnerRecords extends Context.ServiceClass.Shape<
"@streamsy/core/internal/OwnerRecords",
{
readonly read: (key: string) => Effect.Effect<Option.Option<string>, StorageFault>;
readonly write: (key: string, encoded: string) => Effect.Effect<void, StorageFault>;
readonly remove: (key: string) => Effect.Effect<void, StorageFault>;
}
> {}
CheckpointStore is the service key of the checkpoint store, and its service is a CheckpointStoreApi.
| Layer | Requires | Where the records live |
|---|---|---|
CheckpointStore.layer | The CommitBoundary and OwnerRecords of a storage host | On the storage owner, in its transaction |
CheckpointStore.layerMemory | Nothing | In the process, apart from any stream |
Every storage host provides CommitBoundary and OwnerRecords, including MemoryStorage.layer. With CheckpointStore.layer, the checkpoints and the streams on that owner commit in one transaction, and Commit.producer can create a missing output on that owner. A SQL host keeps the records in its streamsy_owner_records table.
CheckpointStore.layerMemory shares no owner with the streams, so a projection over it processes each unit at least once. Use it for rebuilt values, tests, and a remote host. Each build of the Layer starts with no records.
CheckpointStoreApi
interface CheckpointStoreApi {
readonly load: (key: ProjectionKey) => Effect.Effect<Loaded, ProjectionFault>;
readonly compareAndSet: (
key: ProjectionKey,
ifToken: string,
record: CheckpointRecord,
) => Effect.Effect<string, ProjectionFault>;
readonly remove: (key: ProjectionKey) => Effect.Effect<void, ProjectionFault>;
}
CheckpointStoreApi is the service of CheckpointStore. Each record has a token, which changes with each write.
loadreturns the record ofkeyand its token. When no record exists, the record isNoneand the token is"0". A record that does not decode fails with phaseloadand reasoninvalid-record.compareAndSetwritesrecordwhen the stored token isifToken, and returns the new token. When the stored token differs, it fails with phasecheckpointand reasontoken-conflict. The likely cause is a second writer on the same key.removedeletes the record ofkey. It ignores a key with no record.
A storage failure fails each call with reason storage-failure.
fromStore
declare function fromStore(store: EncodedStore): CheckpointStoreApi;
fromStore builds a CheckpointStoreApi over an EncodedStore. Both built-in Layers use it. Only @streamsy/projection/checkpoint exports it.
- It stores each record under
recordKey(key), as a JSON object with the record and a version number. The token is the version as a string. compareAndSetloads the record, compares the token, and writes the next version, all in onewithTransactioncall.
EncodedStore
interface EncodedStore {
readonly read: (key: string) => Effect.Effect<Option.Option<string>, ProjectionFault>;
readonly write: (key: string, value: string) => Effect.Effect<void, ProjectionFault>;
readonly remove: (key: string) => Effect.Effect<void, ProjectionFault>;
readonly withTransaction: <A, E, R>(
body: Effect.Effect<A, E, R>,
) => Effect.Effect<A, E | ProjectionFault, R>;
}
EncodedStore is a store of encoded strings by key, which fromStore turns into a checkpoint store. Only @streamsy/projection/checkpoint exports it.
read returns None for a key with no value. withTransaction must run body so that no other write to the same key happens between its read and its write. remove ignores a key with no value.
Keys and records
ProjectionKey
interface ProjectionKey {
readonly id: string;
readonly version: number;
readonly generation: number;
readonly params: Record<string, string>;
}
A ProjectionKey names a checkpoint record: the id, version, and generation of a declaration, and the params that it was bound with, each encoded with its codec. Projection.key returns the key of a bound projection.
recordKey
declare function recordKey(key: ProjectionKey): string;
recordKey returns the storage key of a record: a JSON array of "streamsy.projection.v1", the id, the version, the generation, and the params as JSON with sorted keys. Only @streamsy/projection/checkpoint exports it.
import { recordKey } from "@streamsy/projection/checkpoint";
const key = recordKey({
id: "order-totals",
version: 1,
generation: 2,
params: { workspaceId: "ws-42" },
});
// ["streamsy.projection.v1","order-totals","1","2","{\"workspaceId\":\"ws-42\"}"]A record key has no length limit. On MySQL, SQL storage stores a key over 512 code points under a hash, as SqlStorage.layer describes. An EncodedStore of your own receives the full key, and must fit it into the key limit of its own store.
Identity
interface Identity {
readonly id: string;
readonly version: number;
readonly generation: number;
readonly params: Record<string, string>;
readonly inputs: InputMap;
}
An Identity is what a runner needs to find and check the record of a bound projection: the fields of a ProjectionKey and the bound inputs. Every bound projection is an Identity. Only @streamsy/projection exports it.
Loaded
interface Loaded {
readonly record: Option.Option<CheckpointRecord>;
readonly token: string;
}
A Loaded is the result of load: the stored record, or None, and its token. The token is "0" when no record exists.
CheckpointRecord
interface CheckpointRecord {
readonly identity: { readonly inputs: { readonly [x: string]: string } };
readonly inputs: { readonly [x: string]: string };
readonly pending?: {
readonly ranges: {
readonly [x: string]: {
readonly from: string;
readonly nextOffset: string;
readonly count: number;
readonly upToDate?: boolean;
readonly closed?: boolean;
readonly gone?: true;
};
};
readonly due?: number;
readonly seqs?: { readonly [x: string]: number };
};
readonly wakeAt?: number;
readonly closed?: { readonly [x: string]: boolean };
readonly ended?: true;
readonly state?: string;
readonly adapters: {
readonly outputs?: {
readonly [x: string]: { readonly epoch: number; readonly nextSeq: number };
};
};
}
A CheckpointRecord is the stored progress of a projection key. CheckpointRecord is also the Schema.Struct of the record. Its encoded form is the same object, so Schema.fromJsonString(CheckpointRecord) gives its JSON codec.
| Field | Meaning |
|---|---|
identity | The stream id and content type of each input. A change stops the projection with reason identity-mismatch |
inputs | The accepted offset of each input. The next pass reads after it |
pending | A pinned unit: its optional recorded due time, the range and item count of each input, and the sequence of each output for Commit.producer |
wakeAt | Next process deadline in epoch milliseconds; absent when no timer is recorded |
closed | The inputs whose close the projection has handled |
ended | true when the final unit of a removed input committed. The next pass deletes the record |
state | The state as JSON |
adapters | The producer epoch and the next sequence of each output of Commit.producer |
A change to an input name, stream id, or content type fails the next load with phase load and reason identity-mismatch. A codec change with the same content type is not detected. Declare a new generation to start again.
Commit
Reference for the Commit namespace and the Transaction service in @streamsy/projection. A Commit decides how a projection writes each unit and its checkpoint, and which guarantee it gives.
Faults and retry
Reference for ProjectionFault and ProjectionRetry in @streamsy/projection. A projection fails with a ProjectionFault that names where the run stopped and why.