Skip to Content
Reference@streamsy/core

Storage contract

Reference for the storage contract in @streamsy/core. A storage adapter implements the Storage service, and the protocol runs on top of it.

import {
  CommitBoundary,
  Mutation,
  MutationApplied,
  MutationRejected,
  Mutations,
  Operation,
  OperationResult,
  OperationResults,
  Operations,
  ProducerPrecondition,
  Storage,
  StorageCapabilities,
  type CommitBoundaryApi,
  type StorageShape,
} from "@streamsy/core";
import type { Data, Effect, Option, PubSub, Scope, Stream } from "effect";
import type { SqlError } from "effect/sql";

Storage is the only service that a storage adapter implements. The protocol, HTTP, forks, expiry, and producer fencing run on top of it, so an application does not call it. For a worked adapter and its contract tests, see Write a storage adapter.

The stream records that these types carry, such as StreamRecord and StoredMessage, are on the Stream records page. ProducerPrecondition, Operation, Mutation, OperationResult, and MutationApplied each export a type and an Effect Schema with the same name.

Services

Storage

class Storage extends Context.Service<Storage, StorageShape>()("@streamsy/core/Storage") {}

Storage is the service key of a storage adapter, and its service is a StorageShape. The protocol Layer needs Storage, and every storage host Layer provides it.

StorageShape

interface StorageShape {
  readonly capabilities: StorageCapabilities;
  readonly record: (id: StreamId) => Effect.Effect<Option.Option<StreamRecord>, StorageFault>;
  readonly messages: (
    id: StreamId,
    window: MessageWindow,
  ) => Effect.Effect<ReadonlyArray<StoredMessage>, StorageFault>;
  readonly producer: (
    id: StreamId,
    producerId: ProducerId,
  ) => Effect.Effect<Option.Option<ProducerState>, StorageFault>;
  readonly mutate: (
    mutation: Mutation,
  ) => Effect.Effect<MutationApplied, MutationRejected | StorageFault>;
  readonly changes: (id: StreamId) => Stream.Stream<ChangeSnapshot, StorageFault>;
  readonly committed: Effect.Effect<
    PubSub.Subscription<ReadonlyArray<StreamId>>,
    never,
    Scope.Scope
  >;
  readonly nextExpiry: Effect.Effect<
    Option.Option<{ readonly at: number; readonly streamId: StreamId }>,
    StorageFault
  >;
}

StorageShape is the service of Storage. A read returns new values, so a caller cannot change the store through a returned object.

MemberDoes
capabilitiesSays what the adapter can do. See StorageCapabilities
record(id)Returns the record of the stream, or None when no stream has the id
messages(id, window)Returns the messages of the stream in window, in offset order
producer(id, producerId)Returns the last producer state that the stream accepted for producerId, or None
mutate(mutation)Applies every operation of mutation in one transaction, or none of them
changes(id)Emits a ChangeSnapshot of the stream when it changes. A snapshot with no change is valid
committedSubscribes to the stream ids of each later local commit
nextExpiryReturns the earliest expiry deadline, with its stream, or None when the adapter has no expiry index

mutate fails with MutationRejected when a precondition of an operation does not hold, and it then writes nothing. mutate dies when a mutation has two operations on the same stream. When capabilities.atomicScope is stream, it also dies for a mutation with more than one stream.

committed creates the subscription before it returns, so the subscription receives each commit after that point. Each item holds the ids that a commit created, appended to, or deleted, and it arrives after that commit. A rolled-back mutation publishes nothing. The subscription has no replay and shows only commits in this process, not commits by another process on the same database. The subscription ends when its scope closes.

StorageCapabilities

interface StorageCapabilities {
  readonly fork: "none" | "copy" | "chain";
  readonly atomicScope: "stream" | "store";
  readonly wake: "poll" | "push";
  readonly expiryIndex: "lazy" | "indexed";
}

StorageCapabilities holds the four features that an adapter declares. The value StorageCapabilities is the Schema of the type.

FieldValues
forknone refuses forks with NotSupported. copy copies the prefix of the source into the fork. chain shares the prefix
atomicScopestream limits a mutation to one stream. store allows one mutation across several streams
wakepush makes changes emit on each commit. poll makes changes check the record on an interval
expiryIndexindexed answers nextExpiry, so a host can sweep expired streams. lazy expires a stream when an operation reads it

CommitBoundary

class CommitBoundary extends Context.Service<CommitBoundary, CommitBoundaryApi>()(
  "@streamsy/core/CommitBoundary",
) {}

CommitBoundary is the service key of the transaction of the storage owner, and its service is a CommitBoundaryApi. Every Streamsy storage host Layer provides it, including MemoryStorage.layer. The projection checkpoint store needs it.

CommitBoundaryApi

interface CommitBoundaryApi {
  readonly withTransaction: <A, E, R>(
    body: Effect.Effect<A, E, R>,
  ) => Effect.Effect<A, E | SqlError.SqlError, R>;
}

CommitBoundaryApi is the service of CommitBoundary. Its only member, withTransaction, runs body in the transaction of the storage owner. The stream writes in body commit together, and on a SQL host the application SQL in body commits with them. When body fails, nothing in it commits.

  • A nested withTransaction joins the outer transaction and has no savepoint. To roll back, a failure must escape the outer body.
  • Keep body on the calling fiber. The memory host dies when another fiber writes in the transaction.
  • Do not wait in body for a wake that only its own commit publishes. The wake comes after the commit, so the wait never ends.

Only a SQL host fails with SqlError.

Mutations

Mutation

type Mutation = { readonly operations: readonly [Operation, ...Operation[]] };

Mutation is the argument of StorageShape.mutate: a non-empty, ordered list of operations that commit together. The adapter checks the preconditions of every operation before it writes. On the first failed check, it writes nothing and fails with MutationRejected, where index is the position of that operation.

Operation

type Operation =
  | {
      readonly _tag: "Create";
      readonly record: StreamRecord;
      readonly initialMessages: ReadonlyArray<StoredMessage>;
      readonly forkSource?: { readonly id: StreamId; readonly liveAtOffset: Offset };
    }
  | {
      readonly _tag: "Append";
      readonly streamId: StreamId;
      readonly messages: ReadonlyArray<StoredMessage>;
      readonly patch: RecordPatch;
      readonly expectedOffset?: Offset;
      readonly expectedClosed?: boolean;
      readonly producer?: ProducerPrecondition;
    }
  | {
      readonly _tag: "Delete";
      readonly streamId: StreamId;
      readonly reason: "delete" | "expiry";
      readonly expectedExpiresAtMs?: number;
    };

Operation is a step of a Mutation. Build one with Operations.

  • Create stores record and initialMessages. It fails with exists when the stream exists. With forkSource, it fails with fork-source-gone when the source is missing, deleted, or has a tail before liveAtOffset.
  • Append stores messages and applies patch to the record. It checks each precondition that it carries. expectedOffset must equal the tail, expectedClosed must equal closed, and producer.expected must equal the stored producer state.
  • Delete removes the stream. On a chain adapter, a stream with live forks becomes soft-deleted instead. With the reason expiry, it fails with expiry-mismatch unless expectedExpiresAtMs equals the stored deadline.

An Append or a Delete fails with not-found when no stream has the id, and with gone when the stream is soft-deleted.

ProducerPrecondition

type ProducerPrecondition = {
  readonly producerId: ProducerId;
  readonly expected: Option.Option<ProducerState>;
  readonly next: ProducerState;
};

ProducerPrecondition fences an Append on the state of a producer. The append fails with producer unless the stored state for producerId equals expected. None means that the stream has no state for producerId. When the append commits, the adapter stores next as the state of producerId.

OperationResult

type OperationResult =
  | { readonly _tag: "Created"; readonly record: StreamRecord }
  | { readonly _tag: "Appended"; readonly record: StreamRecord }
  | { readonly _tag: "Purged"; readonly record: StreamRecord }
  | { readonly _tag: "SoftDeleted"; readonly record: StreamRecord };

OperationResult is the result of an applied operation. record is the record after the operation. A Delete gives Purged when the adapter removed the stream, and SoftDeleted when the stream stays for its forks.

MutationApplied

type MutationApplied = {
  readonly _tag: "Applied";
  readonly results: readonly [OperationResult, ...OperationResult[]];
};

MutationApplied is the success value of mutate. results holds one result for each operation, in the order of the operations.

MutationRejected

interface MutationRejected {
  readonly _tag: "MutationRejected";
  readonly index: number;
  readonly reason:
    | "offset"
    | "closed"
    | "producer"
    | "exists"
    | "not-found"
    | "gone"
    | "fork-source-gone"
    | "expiry-mismatch";
  readonly record: Option.Option<StreamRecord>;
}

mutate fails with MutationRejected when a precondition does not hold. MutationRejected is a Schema.TaggedError, so it is an Error with a _tag that you can yield*. index is the position of the failed operation, and record is the stored record of its stream, or None when the stream does not exist.

ReasonWhen
offsetThe tail is not expectedOffset
closedclosed on the stream is not expectedClosed
producerThe stored producer state is not producer.expected
existsA Create found a stream with the id
not-foundAn Append or a Delete named a missing stream
goneAn Append or a Delete named a soft-deleted stream
fork-source-goneA Create named a fork source that it cannot use
expiry-mismatchAn expiry Delete found another deadline

A protocol append that meets MutationRejected reads the record again and plans its write again. It fails with StreamBusy after 64 attempts. A create that meets exists answers from the stored record.

Constructors

Operations

declare const Operations: Data.TaggedEnum.Constructor<Operation>;

Operations holds a constructor for each tag of Operation, and $is and $match. A constructor builds a plain object and does not validate it. The adapter decodes the values that it reads from its store.

import { Operations, StreamId } from "@streamsy/core";

const remove = Operations.Delete({ streamId: StreamId.make("drafts/main"), reason: "delete" });

OperationResults

declare const OperationResults: Data.TaggedEnum.Constructor<OperationResult>;

OperationResults holds a constructor for each tag of OperationResult, and $is and $match. It does not validate.

Mutations

declare const Mutations: Data.TaggedEnum.Constructor<MutationApplied>;

Mutations holds the Applied constructor of MutationApplied, and $is and $match. It does not validate.

On this page