DurableObjectStorage
Reference for the DurableObjectStorage namespace in @streamsy/storage/durable-object. It keeps streams in the SQLite storage of a Cloudflare Durable Object.
import { DurableObjectStorage } from "@streamsy/storage/durable-object";
import type { CommitBoundary, Storage, StorageFault, Streamsy } from "@streamsy/core";
import type { StreamsReader, StreamsWriter } from "@streamsy/core/protocol";
import type * as Cloudflare from "@cloudflare/workers-types";
import type { Layer } from "effect";
import type { Reactivity } from "effect/reactivity";
import type { SqlClient } from "effect/sql";DurableObjectStorage keeps streams in the SQLite storage of one Durable Object. It builds the @effect/sql-sqlite-do client and runs SqlStorage over it. See Storage in the Cloudflare guide.
Layers
DurableObjectStorage.layer
declare function layer<const Keys extends ReadonlyArray<Streamsy.Any>>(
group: Streamsy.Group<Keys>,
options: DurableObjectStorage.Options,
): Layer.Layer<
| CommitBoundary
| CommitHooks
| OwnerRecords
| ProjectionMarks
| Reactivity.Reactivity
| SqlClient.SqlClient
| Storage
| Streamsy.Host<Streamsy.Group<Keys>>,
StorageFault
>;
// OwnerRecords, CommitHooks, and ProjectionMarks have no exported name in a
// public entry point. See "Types with no exported name" at the end of this page.
layer returns a Layer that provides the stream services of group over the SQLite storage of the object, and requires nothing. The Layer also provides Storage, CommitBoundary, the internal services OwnerRecords, CommitHooks, and ProjectionMarks, and the SqlClient with its Reactivity.
import { DurableObjectStorage } from "@streamsy/storage/durable-object";
import { shop } from "./streams.ts";
const storage = DurableObjectStorage.layer(shop, { client: { storage: ctx.storage } });- Pass the full
ctx.storagehandle asclient.storage. Each mutation runs in a transaction, so the client needs the transaction API of that handle - The long-poll timeout is
DEFAULT_LONG_POLL_TIMEOUT_MS, 25 s, unlesslongPollTimeoutMssets one - The migration, the store, and the option checks follow the rules of
SqlStorage.layer
In a Worker, pass a function that returns this Layer as the layer option of StreamsyObject.make from @streamsy/serve/cloudflare. The object builds it on first access. See Object lifecycle.
DurableObjectStorage.layerProtocol
declare function layerProtocol(
options: DurableObjectStorage.Options,
): Layer.Layer<
| CommitBoundary
| CommitHooks
| OwnerRecords
| ProjectionMarks
| Reactivity.Reactivity
| SqlClient.SqlClient
| Storage
| StreamsReader
| StreamsWriter,
StorageFault
>;
// OwnerRecords, CommitHooks, and ProjectionMarks have no exported name in a
// public entry point. See "Types with no exported name" at the end of this page.
layerProtocol returns a Layer with the generic protocol reader and writer over the SQLite storage of the object, and no stream classes. StreamsReader and StreamsWriter read and write any stream id. The options follow the rules of layer, including the 25 s long-poll default.
Options
DurableObjectStorage.Options
interface Options {
readonly client: {
readonly storage: Cloudflare.DurableObjectStorage;
readonly db?: Cloudflare.SqlStorage | undefined;
readonly spanAttributes?: Record<string, unknown> | undefined;
readonly transformResultNames?: ((str: string) => string) | undefined;
readonly transformQueryNames?: ((str: string) => string) | undefined;
};
readonly repairIntervalMs?: number;
readonly transactionRetryAttempts?: number;
readonly transactionRetryDelayMs?: number;
readonly readLimit?: number;
readonly longPollTimeoutMs?: number;
}
Options holds the options of layer and layerProtocol. client holds the options of SqliteClient.make in @effect/sql-sqlite-do, with storage required. The other fields are those of SqlStorage.Options.
Constants
DurableObjectStorage.DEFAULT_LONG_POLL_TIMEOUT_MS
declare const DEFAULT_LONG_POLL_TIMEOUT_MS: 25000;
DEFAULT_LONG_POLL_TIMEOUT_MS is the long-poll timeout of layer and layerProtocol when longPollTimeoutMs is not set: 25 000 ms. The other hosts use the 30 s default of Protocol.ProtocolOptions.
Types with no exported name
The Layer signatures name three services from @streamsy/core/internal. They connect a storage host to @streamsy/projection, and no public entry point exports them.
interface OwnerRecords extends Context.ServiceClass.Shape<
"@streamsy/core/internal/OwnerRecords",
{
readonly read: (key: string) => Effect.Effect<Option.Option<string>, StorageFault>;
readonly write: (key: string, encoded: string) => Effect.Effect<void, StorageFault>;
readonly remove: (key: string) => Effect.Effect<void, StorageFault>;
}
> {}
interface CommitHooks extends Context.ServiceClass.Shape<
"@streamsy/core/internal/CommitHooks",
{
readonly beforeCommit: (
hook: (ids: ReadonlySet<StreamId>) => Effect.Effect<void, SqlError.SqlError>,
) => Effect.Effect<void, never, Scope.Scope>;
}
> {}
interface ProjectionMarks extends Context.ServiceClass.Shape<
"@streamsy/core/internal/ProjectionMarks",
{
readonly mark: (
projection: string,
params: ReadonlyArray<string>,
) => Effect.Effect<void, SqlError.SqlError>;
readonly nextDue: (projection: string) => Effect.Effect<Option.Option<number>, StorageFault>;
readonly summary: (
projection: string,
now: number,
) => Effect.Effect<ProjectionMarkSummary, StorageFault>;
readonly due: (
projection: string,
now: number,
limit: number,
) => Effect.Effect<ReadonlyArray<PendingMark>, StorageFault>;
readonly read: (
projection: string,
params: ReadonlyArray<string>,
) => Effect.Effect<ReadonlyMap<string, PendingMark>, StorageFault>;
readonly clear: (
projection: string,
params: string,
seq: number,
) => Effect.Effect<void, StorageFault>;
readonly sleep: (
projection: string,
params: string,
seq: number,
wakeAt: number,
) => Effect.Effect<void, StorageFault>;
readonly remove: (projection: string, params: string) => Effect.Effect<void, StorageFault>;
readonly fail: (
projection: string,
params: string,
options: {
readonly attempts: number;
readonly failure: MarkFailure;
readonly dueAt: Option.Option<number>;
},
) => Effect.Effect<void, StorageFault>;
readonly retry: (projection: string, params?: string) => Effect.Effect<number, StorageFault>;
readonly resume: (projection: string) => Effect.Effect<number, StorageFault>;
readonly failed: (
projection: string,
limit: number,
) => Effect.Effect<ReadonlyArray<PendingMark>, StorageFault>;
}
> {}
// Disjoint counts of one projection's marks at the supplied time.
interface ProjectionMarkSummary {
readonly fresh: number;
readonly sleeping: number;
readonly retryDue: number;
readonly backingOff: number;
readonly stopped: number;
readonly nextDueAt: Option.Option<number>;
}
// One pending mark: a binding with input work, a retry, or a recorded wake.
interface PendingMark {
readonly params: string;
readonly seq: number;
readonly attempts: number;
readonly dueAt: number;
readonly wakeAt: Option.Option<number>;
readonly stopped: boolean;
readonly failure: Option.Option<MarkFailure>;
}
interface MarkFailure {
readonly at: number;
readonly reason: Option.Option<string>;
readonly message: string;
}
The rules for these services are those of SqlStorage.
ProjectionMarks.mark runs inside the commit hook and fails with SqlError; the other mark operations fail with StorageFault.
summary(projection, now)returns five disjoint counts: fresh, sleeping, retry due, backing off and stopped. Fresh rows have no wake and no failed attempts; sleeping rows have a wake and no failure or attempts. Its earliest due time includes sleeping marks and excludes stopped marks, as donextDueanddue.markraises the sequence, preserving attempts, stops and failure. A sleeping row becomes due now and clears its wake; other rows keep their due time, so writes never shorten backoff.fail(projection, params, { attempts, failure, dueAt })preserves the sequence, inserts an absent mark, and records the failure with its message cut to 1,000 characters.Some(dueAt)backs off;Nonestops. Both clear the wake index.cleardeletes on a matching sequence; a moved sequence keeps the row but resets attempts, due time, wake, stop and failure.sleep(projection, params, seq, wakeAt)resets failure state and keeps a matching row withdueAtandwakeAtboth set to the deadline. A moved sequence becomes fresh and due now; an absent row is inserted asleep with sequence 1. Past wake times are stored unchanged. The SQL store uses the existing(projection, due_at)index for both retries and sleeps.retry(projection, params?)makes failed or stopped marks due now and clears their stop, preserving attempts, sequence and failure. It returns the number changed; a repeat before the marks run returns zero. Sleeping rows are left alone. It can select one binding by canonical params.resume(projection)makes only stopped marks due now and clears their stop, preserving attempts, sequence and failure. It returns the number changed. Sleeping rows are left alone.failed(projection, limit)lists at mostlimitmarks with failed attempts, stopped first, then by due time and params. The old marks-leveldeferredcount issummary.retryDue + summary.backingOff + summary.stopped.