MemoryStorage
Reference for the MemoryStorage namespace in @streamsy/core. It keeps streams in the process, for development and tests.
import { MemoryStorage, Streamsy, type CommitBoundary, type Storage } from "@streamsy/core";
import type { StreamsReader, StreamsWriter } from "@streamsy/core/protocol";
import type { Layer } from "effect";MemoryStorage keeps streams in the memory of the process, and the data is gone when the scope of the Layer closes. Each mutation copies the whole store, so the time of an append grows with the size of the store. Use it for development and tests. See Memory.
MemoryStorage.layer
declare function layer<const Keys extends ReadonlyArray<Streamsy.Any>>(
group: Streamsy.Group<Keys>,
options?: MemoryStorage.Options,
): Layer.Layer<
| Storage
| CommitBoundary
| OwnerRecords
| CommitHooks
| ProjectionMarks
| Streamsy.Host<Streamsy.Group<Keys>>
>;
// OwnerRecords, CommitHooks, and ProjectionMarks have no exported name in a
// public entry point. See "Types with no exported name" at the end of this page.
layer returns a Layer that provides the stream services of group, over one memory store. The Layer also provides Storage, CommitBoundary, and the internal services OwnerRecords, CommitHooks, and ProjectionMarks. With them, the projection checkpoint store and the pending marks of a durable attached projection run on the same store. Each build of the Layer starts with an empty store.
import { ManagedRuntime } from "effect";
import { MemoryStorage } from "@streamsy/core";
import { shop } from "./streams.ts";
const runtime = ManagedRuntime.make(MemoryStorage.layer(shop));MemoryStorage.layerProtocol
declare function layerProtocol(
options?: MemoryStorage.Options,
): Layer.Layer<
| CommitBoundary
| CommitHooks
| OwnerRecords
| ProjectionMarks
| Storage
| StreamsReader
| StreamsWriter
>;
// OwnerRecords, CommitHooks, and ProjectionMarks have no exported name in a
// public entry point. See "Types with no exported name" at the end of this page.
layerProtocol returns a Layer with the generic protocol reader and writer over a memory store, and no stream classes. StreamsReader and StreamsWriter read and write any stream id. Protocol tools and tests use layerProtocol, for example to write raw messages that a typed service does not write.
MemoryStorage.Options
interface Options {
readonly constrained?: boolean;
readonly pollIntervalMs?: number;
readonly readLimit?: number;
readonly longPollTimeoutMs?: number;
}
Options holds the options of layer and layerProtocol.
| Option | Meaning |
|---|---|
constrained | When true, the store has the smallest capabilities: copy forks, stream atomic scope, poll wakes, and a lazy expiry index |
pollIntervalMs | The poll interval of changes in constrained mode. The default is 25 ms |
readLimit | The largest number of messages in a catch-up page. The default is 1000 |
longPollTimeoutMs | The longest wait of a long-poll read. The default is 30 s |
Without constrained, the store has chain forks, store atomic scope, push wakes, and an indexed expiry index. Use constrained: true in tests to check that a program does not depend on these capabilities.
Building the Layer dies with a RangeError when pollIntervalMs or longPollTimeoutMs is not a positive finite number, or when readLimit is not a positive safe integer.
Types with no exported name
The Layer signatures name three services from @streamsy/core/internal. They connect a storage host to @streamsy/projection, and no public entry point exports them.
interface OwnerRecords extends Context.ServiceClass.Shape<
"@streamsy/core/internal/OwnerRecords",
{
readonly read: (key: string) => Effect.Effect<Option.Option<string>, StorageFault>;
readonly write: (key: string, encoded: string) => Effect.Effect<void, StorageFault>;
readonly remove: (key: string) => Effect.Effect<void, StorageFault>;
}
> {}
interface CommitHooks extends Context.ServiceClass.Shape<
"@streamsy/core/internal/CommitHooks",
{
readonly beforeCommit: (
hook: (ids: ReadonlySet<StreamId>) => Effect.Effect<void, SqlError.SqlError>,
) => Effect.Effect<void, never, Scope.Scope>;
}
> {}
interface ProjectionMarks extends Context.ServiceClass.Shape<
"@streamsy/core/internal/ProjectionMarks",
{
readonly mark: (
projection: string,
params: ReadonlyArray<string>,
) => Effect.Effect<void, SqlError.SqlError>;
readonly nextDue: (projection: string) => Effect.Effect<Option.Option<number>, StorageFault>;
readonly summary: (
projection: string,
now: number,
) => Effect.Effect<ProjectionMarkSummary, StorageFault>;
readonly due: (
projection: string,
now: number,
limit: number,
) => Effect.Effect<ReadonlyArray<PendingMark>, StorageFault>;
readonly read: (
projection: string,
params: ReadonlyArray<string>,
) => Effect.Effect<ReadonlyMap<string, PendingMark>, StorageFault>;
readonly clear: (
projection: string,
params: string,
seq: number,
) => Effect.Effect<void, StorageFault>;
readonly sleep: (
projection: string,
params: string,
seq: number,
wakeAt: number,
) => Effect.Effect<void, StorageFault>;
readonly remove: (projection: string, params: string) => Effect.Effect<void, StorageFault>;
readonly fail: (
projection: string,
params: string,
options: {
readonly attempts: number;
readonly failure: MarkFailure;
readonly dueAt: Option.Option<number>;
},
) => Effect.Effect<void, StorageFault>;
readonly retry: (projection: string, params?: string) => Effect.Effect<number, StorageFault>;
readonly resume: (projection: string) => Effect.Effect<number, StorageFault>;
readonly failed: (
projection: string,
limit: number,
) => Effect.Effect<ReadonlyArray<PendingMark>, StorageFault>;
}
> {}
// Disjoint counts of one projection's marks at the supplied time.
interface ProjectionMarkSummary {
readonly fresh: number;
readonly sleeping: number;
readonly retryDue: number;
readonly backingOff: number;
readonly stopped: number;
readonly nextDueAt: Option.Option<number>;
}
// One pending mark: a binding with input work, a retry, or a recorded wake.
interface PendingMark {
readonly params: string;
readonly seq: number;
readonly attempts: number;
readonly dueAt: number;
readonly wakeAt: Option.Option<number>;
readonly stopped: boolean;
readonly failure: Option.Option<MarkFailure>;
}
interface MarkFailure {
readonly at: number;
readonly reason: Option.Option<string>;
readonly message: string;
}
@streamsy/projection uses each service for one job:
OwnerRecordsholds the encoded records of the projection checkpoint storeCommitHooksruns each hook thatbeforeCommitregisters inside every later owned transaction that writes at least one stream id. The hook runs after the body and before the commit, and receives the ids that the transaction created, appended to, or deleted. Its writes commit or roll back with the transaction. The memory store turns a hook failure into a defect. Closing the scope ofbeforeCommitremoves the hookProjectionMarksholds the pending marks of durable attached projections, keyed by projection id and params. The memory store keeps them with the streams, so they are gone when the scope of the Layer closes. SeeAttachOptions
ProjectionMarks.mark runs inside the commit hook and fails with SqlError; the other mark operations fail with StorageFault.
summary(projection, now)returns five disjoint counts: fresh, sleeping, retry due, backing off and stopped. Fresh rows have no wake and no failed attempts; sleeping rows have a wake and no failure or attempts. Its earliest due time includes sleeping marks and excludes stopped marks, as donextDueanddue.markraises the sequence, preserving attempts, stops and failure. A sleeping row becomes due now and clears its wake; other rows keep their due time, so writes never shorten backoff.fail(projection, params, { attempts, failure, dueAt })preserves the sequence, inserts an absent mark, and records the failure with its message cut to 1,000 characters.Some(dueAt)backs off;Nonestops. Both clear the wake index.cleardeletes on a matching sequence; a moved sequence keeps the row but resets attempts, due time, wake, stop and failure.sleep(projection, params, seq, wakeAt)resets failure state and keeps a matching row withdueAtandwakeAtboth set to the deadline. A moved sequence becomes fresh and due now; an absent row is inserted asleep with sequence 1. Past wake times are stored unchanged.retry(projection, params?)makes failed or stopped marks due now and clears their stop, preserving attempts, sequence and failure. It returns the number changed; a repeat before the marks run returns zero. Sleeping rows are left alone. It can select one binding by canonical params.resume(projection)makes only stopped marks due now and clears their stop, preserving attempts, sequence and failure. It returns the number changed. Sleeping rows are left alone.failed(projection, limit)lists at mostlimitmarks with failed attempts, stopped first, then by due time and params. The old marks-leveldeferredcount issummary.retryDue + summary.backingOff + summary.stopped.
memoryProjectionMarks() from @streamsy/core/internal creates a standalone private in-memory ProjectionMarksApi. It shares the mark operations above but has no stream owner or transaction boundary; each factory call creates an empty store.