Skip to Content
Reference@streamsy/storage

BunStorage

Reference for the BunStorage namespace in @streamsy/storage/bun. It keeps streams in one SQLite file through bun:sqlite.

import { BunStorage } from "@streamsy/storage/bun";
import type { CommitBoundary, Storage, StorageFault, Streamsy } from "@streamsy/core";
import type { StreamsReader, StreamsWriter } from "@streamsy/core/protocol";
import type { SqliteClient } from "@effect/sql-sqlite-bun";
import type { Layer } from "effect";
import type { Reactivity } from "effect/reactivity";
import type { SqlClient } from "effect/sql";

BunStorage keeps streams in one SQLite file on Bun. It builds the bun:sqlite client and runs SqlStorage over it. See Use SQLite.

Layers

BunStorage.layer

declare function layer<const Keys extends ReadonlyArray<Streamsy.Any>>(
  group: Streamsy.Group<Keys>,
  options: BunStorage.Options,
): Layer.Layer<
  | CommitBoundary
  | CommitHooks
  | OwnerRecords
  | ProjectionMarks
  | Reactivity.Reactivity
  | SqlClient.SqlClient
  | Storage
  | Streamsy.Host<Streamsy.Group<Keys>>,
  StorageFault
>;

// OwnerRecords, CommitHooks, and ProjectionMarks have no exported name in a
// public entry point. See "Types with no exported name" at the end of this page.

layer returns a Layer that provides the stream services of group over one SQLite file, and requires nothing. The Layer also provides Storage, CommitBoundary, the internal services OwnerRecords, CommitHooks, and ProjectionMarks, and the SqlClient with its Reactivity. Your own SQL can use that client and commit with a stream mutation. See Commit with your own tables.

import { BunStorage } from "@streamsy/storage/bun";
import { shop } from "./streams.ts";

const storage = BunStorage.layer(shop, { client: { filename: "./streams.sqlite" } });

Before the client opens the file, the build prepares it in three steps. The build skips them when filename is ":memory:".

  1. When the open may write and create, which is the default, the build creates the parent folder. A readonly open, or create: false, creates no folder and no file.
  2. When the file exists, the build reads its format. It fails with operation migration.open when the file holds an older Streamsy format or a newer schema version, and does not change the file.
  3. Unless the open is readonly or sets disableWAL, the build sets WAL mode and a busy timeout of 0 on the file.

The client also has a busy timeout of 0 unless client.busyTimeout sets one. With a busy timeout of 0, SQLite does not wait for a locked file, so a busy file does not block the event loop. The store retries a busy transaction itself instead, as transactionRetryAttempts and transactionRetryDelayMs set. When SQLite fails while the file opens, the build fails with a StorageFault with operation migration.open. The fault is retryable when the file is busy or locked, and the preparation steps retry it with the same policy.

Several Bun processes can share one file. A commit in one process wakes the readers in another within repairIntervalMs. See Several processes.

The migration, the store, and the option checks follow the rules of SqlStorage.layer. The long-poll timeout is 30 s unless longPollTimeoutMs sets one.

BunStorage.layerProtocol

declare function layerProtocol(
  options: BunStorage.Options,
): Layer.Layer<
  | CommitBoundary
  | CommitHooks
  | OwnerRecords
  | ProjectionMarks
  | Reactivity.Reactivity
  | SqlClient.SqlClient
  | Storage
  | StreamsReader
  | StreamsWriter,
  StorageFault
>;

// OwnerRecords, CommitHooks, and ProjectionMarks have no exported name in a
// public entry point. See "Types with no exported name" at the end of this page.

layerProtocol returns a Layer with the generic protocol reader and writer over the SQLite file, and no stream classes. StreamsReader and StreamsWriter read and write any stream id. The file and the options follow the rules of layer.

Options

BunStorage.Options

interface Options {
  readonly client: SqliteClient.SqliteClientConfig;
  readonly repairIntervalMs?: number;
  readonly transactionRetryAttempts?: number;
  readonly transactionRetryDelayMs?: number;
  readonly readLimit?: number;
  readonly longPollTimeoutMs?: number;
}

Options holds the options of layer and layerProtocol. client is required. It holds the options of SqliteClient.make in @effect/sql-sqlite-bun, such as filename, readonly, create, disableWAL, and busyTimeout. The other fields are those of SqlStorage.Options.

Types with no exported name

The Layer signatures name three services from @streamsy/core/internal. They connect a storage host to @streamsy/projection, and no public entry point exports them.

interface OwnerRecords extends Context.ServiceClass.Shape<
  "@streamsy/core/internal/OwnerRecords",
  {
    readonly read: (key: string) => Effect.Effect<Option.Option<string>, StorageFault>;
    readonly write: (key: string, encoded: string) => Effect.Effect<void, StorageFault>;
    readonly remove: (key: string) => Effect.Effect<void, StorageFault>;
  }
> {}

interface CommitHooks extends Context.ServiceClass.Shape<
  "@streamsy/core/internal/CommitHooks",
  {
    readonly beforeCommit: (
      hook: (ids: ReadonlySet<StreamId>) => Effect.Effect<void, SqlError.SqlError>,
    ) => Effect.Effect<void, never, Scope.Scope>;
  }
> {}

interface ProjectionMarks extends Context.ServiceClass.Shape<
  "@streamsy/core/internal/ProjectionMarks",
  {
    readonly mark: (
      projection: string,
      params: ReadonlyArray<string>,
    ) => Effect.Effect<void, SqlError.SqlError>;
    readonly nextDue: (projection: string) => Effect.Effect<Option.Option<number>, StorageFault>;
    readonly summary: (
      projection: string,
      now: number,
    ) => Effect.Effect<ProjectionMarkSummary, StorageFault>;
    readonly due: (
      projection: string,
      now: number,
      limit: number,
    ) => Effect.Effect<ReadonlyArray<PendingMark>, StorageFault>;
    readonly read: (
      projection: string,
      params: ReadonlyArray<string>,
    ) => Effect.Effect<ReadonlyMap<string, PendingMark>, StorageFault>;
    readonly clear: (
      projection: string,
      params: string,
      seq: number,
    ) => Effect.Effect<void, StorageFault>;
    readonly sleep: (
      projection: string,
      params: string,
      seq: number,
      wakeAt: number,
    ) => Effect.Effect<void, StorageFault>;
    readonly remove: (projection: string, params: string) => Effect.Effect<void, StorageFault>;
    readonly fail: (
      projection: string,
      params: string,
      options: {
        readonly attempts: number;
        readonly failure: MarkFailure;
        readonly dueAt: Option.Option<number>;
      },
    ) => Effect.Effect<void, StorageFault>;
    readonly retry: (projection: string, params?: string) => Effect.Effect<number, StorageFault>;
    readonly resume: (projection: string) => Effect.Effect<number, StorageFault>;
    readonly failed: (
      projection: string,
      limit: number,
    ) => Effect.Effect<ReadonlyArray<PendingMark>, StorageFault>;
  }
> {}

// Disjoint counts of one projection's marks at the supplied time.
interface ProjectionMarkSummary {
  readonly fresh: number;
  readonly sleeping: number;
  readonly retryDue: number;
  readonly backingOff: number;
  readonly stopped: number;
  readonly nextDueAt: Option.Option<number>;
}

// One pending mark: a binding with input work, a retry, or a recorded wake.
interface PendingMark {
  readonly params: string;
  readonly seq: number;
  readonly attempts: number;
  readonly dueAt: number;
  readonly wakeAt: Option.Option<number>;
  readonly stopped: boolean;
  readonly failure: Option.Option<MarkFailure>;
}

interface MarkFailure {
  readonly at: number;
  readonly reason: Option.Option<string>;
  readonly message: string;
}

The rules for these services are those of SqlStorage.

ProjectionMarks.mark runs inside the commit hook and fails with SqlError; the other mark operations fail with StorageFault.

  • summary(projection, now) returns five disjoint counts: fresh, sleeping, retry due, backing off and stopped. Fresh rows have no wake and no failed attempts; sleeping rows have a wake and no failure or attempts. Its earliest due time includes sleeping marks and excludes stopped marks, as do nextDue and due.
  • mark raises the sequence, preserving attempts, stops and failure. A sleeping row becomes due now and clears its wake; other rows keep their due time, so writes never shorten backoff. fail(projection, params, { attempts, failure, dueAt }) preserves the sequence, inserts an absent mark, and records the failure with its message cut to 1,000 characters. Some(dueAt) backs off; None stops. Both clear the wake index.
  • clear deletes on a matching sequence; a moved sequence keeps the row but resets attempts, due time, wake, stop and failure.
  • sleep(projection, params, seq, wakeAt) resets failure state and keeps a matching row with dueAt and wakeAt both set to the deadline. A moved sequence becomes fresh and due now; an absent row is inserted asleep with sequence 1. Past wake times are stored unchanged. The SQL store uses the existing (projection, due_at) index for both retries and sleeps.
  • retry(projection, params?) makes failed or stopped marks due now and clears their stop, preserving attempts, sequence and failure. It returns the number changed; a repeat before the marks run returns zero. Sleeping rows are left alone. It can select one binding by canonical params.
  • resume(projection) makes only stopped marks due now and clears their stop, preserving attempts, sequence and failure. It returns the number changed. Sleeping rows are left alone.
  • failed(projection, limit) lists at most limit marks with failed attempts, stopped first, then by due time and params. The old marks-level deferred count is summary.retryDue + summary.backingOff + summary.stopped.

On this page