Skip to Content
Reference@streamsy/core

MemoryStorage

Reference for the MemoryStorage namespace in @streamsy/core. It keeps streams in the process, for development and tests.

import { MemoryStorage, Streamsy, type CommitBoundary, type Storage } from "@streamsy/core";
import type { StreamsReader, StreamsWriter } from "@streamsy/core/protocol";
import type { Layer } from "effect";

MemoryStorage keeps streams in the memory of the process, and the data is gone when the scope of the Layer closes. Each mutation copies the whole store, so the time of an append grows with the size of the store. Use it for development and tests. See Memory.

MemoryStorage.layer

declare function layer<const Keys extends ReadonlyArray<Streamsy.Any>>(
  group: Streamsy.Group<Keys>,
  options?: MemoryStorage.Options,
): Layer.Layer<
  | Storage
  | CommitBoundary
  | OwnerRecords
  | CommitHooks
  | ProjectionMarks
  | Streamsy.Host<Streamsy.Group<Keys>>
>;

// OwnerRecords, CommitHooks, and ProjectionMarks have no exported name in a
// public entry point. See "Types with no exported name" at the end of this page.

layer returns a Layer that provides the stream services of group, over one memory store. The Layer also provides Storage, CommitBoundary, and the internal services OwnerRecords, CommitHooks, and ProjectionMarks. With them, the projection checkpoint store and the pending marks of a durable attached projection run on the same store. Each build of the Layer starts with an empty store.

import { ManagedRuntime } from "effect";
import { MemoryStorage } from "@streamsy/core";
import { shop } from "./streams.ts";

const runtime = ManagedRuntime.make(MemoryStorage.layer(shop));

MemoryStorage.layerProtocol

declare function layerProtocol(
  options?: MemoryStorage.Options,
): Layer.Layer<
  | CommitBoundary
  | CommitHooks
  | OwnerRecords
  | ProjectionMarks
  | Storage
  | StreamsReader
  | StreamsWriter
>;

// OwnerRecords, CommitHooks, and ProjectionMarks have no exported name in a
// public entry point. See "Types with no exported name" at the end of this page.

layerProtocol returns a Layer with the generic protocol reader and writer over a memory store, and no stream classes. StreamsReader and StreamsWriter read and write any stream id. Protocol tools and tests use layerProtocol, for example to write raw messages that a typed service does not write.

MemoryStorage.Options

interface Options {
  readonly constrained?: boolean;
  readonly pollIntervalMs?: number;
  readonly readLimit?: number;
  readonly longPollTimeoutMs?: number;
}

Options holds the options of layer and layerProtocol.

OptionMeaning
constrainedWhen true, the store has the smallest capabilities: copy forks, stream atomic scope, poll wakes, and a lazy expiry index
pollIntervalMsThe poll interval of changes in constrained mode. The default is 25 ms
readLimitThe largest number of messages in a catch-up page. The default is 1000
longPollTimeoutMsThe longest wait of a long-poll read. The default is 30 s

Without constrained, the store has chain forks, store atomic scope, push wakes, and an indexed expiry index. Use constrained: true in tests to check that a program does not depend on these capabilities.

Building the Layer dies with a RangeError when pollIntervalMs or longPollTimeoutMs is not a positive finite number, or when readLimit is not a positive safe integer.

Types with no exported name

The Layer signatures name three services from @streamsy/core/internal. They connect a storage host to @streamsy/projection, and no public entry point exports them.

interface OwnerRecords extends Context.ServiceClass.Shape<
  "@streamsy/core/internal/OwnerRecords",
  {
    readonly read: (key: string) => Effect.Effect<Option.Option<string>, StorageFault>;
    readonly write: (key: string, encoded: string) => Effect.Effect<void, StorageFault>;
    readonly remove: (key: string) => Effect.Effect<void, StorageFault>;
  }
> {}

interface CommitHooks extends Context.ServiceClass.Shape<
  "@streamsy/core/internal/CommitHooks",
  {
    readonly beforeCommit: (
      hook: (ids: ReadonlySet<StreamId>) => Effect.Effect<void, SqlError.SqlError>,
    ) => Effect.Effect<void, never, Scope.Scope>;
  }
> {}

interface ProjectionMarks extends Context.ServiceClass.Shape<
  "@streamsy/core/internal/ProjectionMarks",
  {
    readonly mark: (
      projection: string,
      params: ReadonlyArray<string>,
    ) => Effect.Effect<void, SqlError.SqlError>;
    readonly nextDue: (projection: string) => Effect.Effect<Option.Option<number>, StorageFault>;
    readonly summary: (
      projection: string,
      now: number,
    ) => Effect.Effect<ProjectionMarkSummary, StorageFault>;
    readonly due: (
      projection: string,
      now: number,
      limit: number,
    ) => Effect.Effect<ReadonlyArray<PendingMark>, StorageFault>;
    readonly read: (
      projection: string,
      params: ReadonlyArray<string>,
    ) => Effect.Effect<ReadonlyMap<string, PendingMark>, StorageFault>;
    readonly clear: (
      projection: string,
      params: string,
      seq: number,
    ) => Effect.Effect<void, StorageFault>;
    readonly sleep: (
      projection: string,
      params: string,
      seq: number,
      wakeAt: number,
    ) => Effect.Effect<void, StorageFault>;
    readonly remove: (projection: string, params: string) => Effect.Effect<void, StorageFault>;
    readonly fail: (
      projection: string,
      params: string,
      options: {
        readonly attempts: number;
        readonly failure: MarkFailure;
        readonly dueAt: Option.Option<number>;
      },
    ) => Effect.Effect<void, StorageFault>;
    readonly retry: (projection: string, params?: string) => Effect.Effect<number, StorageFault>;
    readonly resume: (projection: string) => Effect.Effect<number, StorageFault>;
    readonly failed: (
      projection: string,
      limit: number,
    ) => Effect.Effect<ReadonlyArray<PendingMark>, StorageFault>;
  }
> {}

// Disjoint counts of one projection's marks at the supplied time.
interface ProjectionMarkSummary {
  readonly fresh: number;
  readonly sleeping: number;
  readonly retryDue: number;
  readonly backingOff: number;
  readonly stopped: number;
  readonly nextDueAt: Option.Option<number>;
}

// One pending mark: a binding with input work, a retry, or a recorded wake.
interface PendingMark {
  readonly params: string;
  readonly seq: number;
  readonly attempts: number;
  readonly dueAt: number;
  readonly wakeAt: Option.Option<number>;
  readonly stopped: boolean;
  readonly failure: Option.Option<MarkFailure>;
}

interface MarkFailure {
  readonly at: number;
  readonly reason: Option.Option<string>;
  readonly message: string;
}

@streamsy/projection uses each service for one job:

  • OwnerRecords holds the encoded records of the projection checkpoint store
  • CommitHooks runs each hook that beforeCommit registers inside every later owned transaction that writes at least one stream id. The hook runs after the body and before the commit, and receives the ids that the transaction created, appended to, or deleted. Its writes commit or roll back with the transaction. The memory store turns a hook failure into a defect. Closing the scope of beforeCommit removes the hook
  • ProjectionMarks holds the pending marks of durable attached projections, keyed by projection id and params. The memory store keeps them with the streams, so they are gone when the scope of the Layer closes. See AttachOptions

ProjectionMarks.mark runs inside the commit hook and fails with SqlError; the other mark operations fail with StorageFault.

  • summary(projection, now) returns five disjoint counts: fresh, sleeping, retry due, backing off and stopped. Fresh rows have no wake and no failed attempts; sleeping rows have a wake and no failure or attempts. Its earliest due time includes sleeping marks and excludes stopped marks, as do nextDue and due.
  • mark raises the sequence, preserving attempts, stops and failure. A sleeping row becomes due now and clears its wake; other rows keep their due time, so writes never shorten backoff. fail(projection, params, { attempts, failure, dueAt }) preserves the sequence, inserts an absent mark, and records the failure with its message cut to 1,000 characters. Some(dueAt) backs off; None stops. Both clear the wake index.
  • clear deletes on a matching sequence; a moved sequence keeps the row but resets attempts, due time, wake, stop and failure.
  • sleep(projection, params, seq, wakeAt) resets failure state and keeps a matching row with dueAt and wakeAt both set to the deadline. A moved sequence becomes fresh and due now; an absent row is inserted asleep with sequence 1. Past wake times are stored unchanged.
  • retry(projection, params?) makes failed or stopped marks due now and clears their stop, preserving attempts, sequence and failure. It returns the number changed; a repeat before the marks run returns zero. Sleeping rows are left alone. It can select one binding by canonical params.
  • resume(projection) makes only stopped marks due now and clears their stop, preserving attempts, sequence and failure. It returns the number changed. Sleeping rows are left alone.
  • failed(projection, limit) lists at most limit marks with failed attempts, stopped first, then by due time and params. The old marks-level deferred count is summary.retryDue + summary.backingOff + summary.stopped.

memoryProjectionMarks() from @streamsy/core/internal creates a standalone private in-memory ProjectionMarksApi. It shares the mark operations above but has no stream owner or transaction boundary; each factory call creates an empty store.

On this page