Skip to Content
Reference@streamsy/projection

CheckpointStore

Reference for the checkpoint store in @streamsy/projection and @streamsy/projection/checkpoint. The store keeps the offsets and the state of each projection key in one compare-and-set record.

import {
  CheckpointRecord,
  CheckpointStore,
  type CheckpointStoreApi,
  type Identity,
  type Loaded,
  type ProjectionKey,
} from "@streamsy/projection";
import { fromStore, recordKey, type EncodedStore } from "@streamsy/projection/checkpoint";
import type { CommitBoundary, Streamsy } from "@streamsy/core";
import type { Effect, Layer, Option } from "effect";

CheckpointStore keeps one record for each projection key: the input offsets, the encoded state, and a pinned unit. The built-in Commits need it. See Checkpoints and generations and Projection storage.

@streamsy/projection/checkpoint is the entry point for the author of a checkpoint Layer or a custom Commit. Each export has one section on this page:

Export@streamsy/projection@streamsy/projection/checkpoint
CheckpointStoreYesYes
CheckpointStoreApiYesYes
CheckpointRecordYesYes
LoadedYesYes
ProjectionKeyYesYes
IdentityYes
fromStoreYes
recordKeyYes
EncodedStoreYes

The store

CheckpointStore

class CheckpointStore extends Context.Service<CheckpointStore, CheckpointStoreApi>()(
  "@streamsy/projection/CheckpointStore",
) {
  static readonly layer: Layer.Layer<CheckpointStore, never, CommitBoundary | OwnerRecords>;
  static readonly layerMemory: Layer.Layer<CheckpointStore>;
}

// OwnerRecords has no exported name in a public entry point. It is the service
// of the encoded records on the storage owner. Every storage host provides it.
interface OwnerRecords extends Context.ServiceClass.Shape<
  "@streamsy/core/internal/OwnerRecords",
  {
    readonly read: (key: string) => Effect.Effect<Option.Option<string>, StorageFault>;
    readonly write: (key: string, encoded: string) => Effect.Effect<void, StorageFault>;
    readonly remove: (key: string) => Effect.Effect<void, StorageFault>;
  }
> {}

CheckpointStore is the service key of the checkpoint store, and its service is a CheckpointStoreApi.

LayerRequiresWhere the records live
CheckpointStore.layerThe CommitBoundary and OwnerRecords of a storage hostOn the storage owner, in its transaction
CheckpointStore.layerMemoryNothingIn the process, apart from any stream

Every storage host provides CommitBoundary and OwnerRecords, including MemoryStorage.layer. With CheckpointStore.layer, the checkpoints and the streams on that owner commit in one transaction, and Commit.producer can create a missing output on that owner. A SQL host keeps the records in its streamsy_owner_records table.

CheckpointStore.layerMemory shares no owner with the streams, so a projection over it processes each unit at least once. Use it for rebuilt values, tests, and a remote host. Each build of the Layer starts with no records.

CheckpointStoreApi

interface CheckpointStoreApi {
  readonly load: (key: ProjectionKey) => Effect.Effect<Loaded, ProjectionFault>;
  readonly compareAndSet: (
    key: ProjectionKey,
    ifToken: string,
    record: CheckpointRecord,
  ) => Effect.Effect<string, ProjectionFault>;
  readonly remove: (key: ProjectionKey) => Effect.Effect<void, ProjectionFault>;
}

CheckpointStoreApi is the service of CheckpointStore. Each record has a token, which changes with each write.

  • load returns the record of key and its token. When no record exists, the record is None and the token is "0". A record that does not decode fails with phase load and reason invalid-record.
  • compareAndSet writes record when the stored token is ifToken, and returns the new token. When the stored token differs, it fails with phase checkpoint and reason token-conflict. The likely cause is a second writer on the same key.
  • remove deletes the record of key. It ignores a key with no record.

A storage failure fails each call with reason storage-failure.

fromStore

declare function fromStore(store: EncodedStore): CheckpointStoreApi;

fromStore builds a CheckpointStoreApi over an EncodedStore. Both built-in Layers use it. Only @streamsy/projection/checkpoint exports it.

  • It stores each record under recordKey(key), as a JSON object with the record and a version number. The token is the version as a string.
  • compareAndSet loads the record, compares the token, and writes the next version, all in one withTransaction call.

EncodedStore

interface EncodedStore {
  readonly read: (key: string) => Effect.Effect<Option.Option<string>, ProjectionFault>;
  readonly write: (key: string, value: string) => Effect.Effect<void, ProjectionFault>;
  readonly remove: (key: string) => Effect.Effect<void, ProjectionFault>;
  readonly withTransaction: <A, E, R>(
    body: Effect.Effect<A, E, R>,
  ) => Effect.Effect<A, E | ProjectionFault, R>;
}

EncodedStore is a store of encoded strings by key, which fromStore turns into a checkpoint store. Only @streamsy/projection/checkpoint exports it.

read returns None for a key with no value. withTransaction must run body so that no other write to the same key happens between its read and its write. remove ignores a key with no value.

Keys and records

ProjectionKey

interface ProjectionKey {
  readonly id: string;
  readonly version: number;
  readonly generation: number;
  readonly params: Record<string, string>;
}

A ProjectionKey names a checkpoint record: the id, version, and generation of a declaration, and the params that it was bound with, each encoded with its codec. Projection.key returns the key of a bound projection.

recordKey

declare function recordKey(key: ProjectionKey): string;

recordKey returns the storage key of a record: a JSON array of "streamsy.projection.v1", the id, the version, the generation, and the params as JSON with sorted keys. Only @streamsy/projection/checkpoint exports it.

import { recordKey } from "@streamsy/projection/checkpoint";

const key = recordKey({
  id: "order-totals",
  version: 1,
  generation: 2,
  params: { workspaceId: "ws-42" },
});
// ["streamsy.projection.v1","order-totals","1","2","{\"workspaceId\":\"ws-42\"}"]

A record key has no length limit. On MySQL, SQL storage stores a key over 512 code points under a hash, as SqlStorage.layer describes. An EncodedStore of your own receives the full key, and must fit it into the key limit of its own store.

Identity

interface Identity {
  readonly id: string;
  readonly version: number;
  readonly generation: number;
  readonly params: Record<string, string>;
  readonly inputs: InputMap;
}

An Identity is what a runner needs to find and check the record of a bound projection: the fields of a ProjectionKey and the bound inputs. Every bound projection is an Identity. Only @streamsy/projection exports it.

Loaded

interface Loaded {
  readonly record: Option.Option<CheckpointRecord>;
  readonly token: string;
}

A Loaded is the result of load: the stored record, or None, and its token. The token is "0" when no record exists.

CheckpointRecord

interface CheckpointRecord {
  readonly identity: { readonly inputs: { readonly [x: string]: string } };
  readonly inputs: { readonly [x: string]: string };
  readonly pending?: {
    readonly ranges: {
      readonly [x: string]: {
        readonly from: string;
        readonly nextOffset: string;
        readonly count: number;
        readonly upToDate?: boolean;
        readonly closed?: boolean;
        readonly gone?: true;
      };
    };
    readonly due?: number;
    readonly seqs?: { readonly [x: string]: number };
  };
  readonly wakeAt?: number;
  readonly closed?: { readonly [x: string]: boolean };
  readonly ended?: true;
  readonly state?: string;
  readonly adapters: {
    readonly outputs?: {
      readonly [x: string]: { readonly epoch: number; readonly nextSeq: number };
    };
  };
}

A CheckpointRecord is the stored progress of a projection key. CheckpointRecord is also the Schema.Struct of the record. Its encoded form is the same object, so Schema.fromJsonString(CheckpointRecord) gives its JSON codec.

FieldMeaning
identityThe stream id and content type of each input. A change stops the projection with reason identity-mismatch
inputsThe accepted offset of each input. The next pass reads after it
pendingA pinned unit: its optional recorded due time, the range and item count of each input, and the sequence of each output for Commit.producer
wakeAtNext process deadline in epoch milliseconds; absent when no timer is recorded
closedThe inputs whose close the projection has handled
endedtrue when the final unit of a removed input committed. The next pass deletes the record
stateThe state as JSON
adaptersThe producer epoch and the next sequence of each output of Commit.producer

A change to an input name, stream id, or content type fails the next load with phase load and reason identity-mismatch. A codec change with the same content type is not detected. Declare a new generation to start again.

On this page