Skip to Content
Storage

Write a storage adapter

Implement the Storage service for a new backend, bind streams to it, and check it with the contract tests and the protocol conformance suite.

Storage is a single Effect service. An adapter that implements it and passes the contract tests is a backend for the stream services, the protocol routes, and the remote client.

The protocol, HTTP, forks, expiry, and producer fencing run above Storage. An adapter stores records and messages, applies mutations in order, and signals changes. To provide projection checkpoints and SQL transactions too, the adapter needs the services in Commit boundary.

The contract

import { Effect, Layer, Option, Stream } from "effect";
import { Storage, type StorageShape } from "@streamsy/core";

const adapter: StorageShape = {
  capabilities: { fork: "none", atomicScope: "stream", wake: "poll", expiryIndex: "lazy" },
  record: (id) => Effect.succeed(Option.none()),
  messages: (id, window) => Effect.succeed([]),
  producer: (id, producerId) => Effect.succeed(Option.none()),
  mutate: (mutation) => Effect.die("not implemented"),
  changes: (id) => Stream.empty,
  committed: Effect.die("not implemented"),
  nextExpiry: Effect.succeed(Option.none()),
};

export const layer = Layer.succeed(Storage, adapter);
MemberReturns
record(id)The stream record, or None
messages(id, window)Messages after after (exclusive), up to until (inclusive), with an optional limit
producer(id, producerId)The last accepted producer state
mutate(mutation)Applies an ordered set of operations in one transaction
changes(id)A level-triggered stream of change snapshots
committedA scoped PubSub subscription to stream-id arrays after successful local commits
nextExpiryThe earliest indexed deadline
capabilitiesWhat the adapter can do

Each read must return new values, so a caller cannot change the store through a returned object.

Bind streams

import { Layer } from "effect";
import { Protocol } from "@streamsy/core/protocol";
import { layer } from "./adapter.ts";
import { shop } from "./streams.ts";

export const shopStorage = Protocol.bind(Protocol.layer().pipe(Layer.provideMerge(layer)), shop);

shopStorage provides the stream services of shop, as MemoryStorage.layer(shop) does.

Protocol tier

Protocol.layer(options) from @streamsy/core/protocol is the protocol tier. It provides StreamsReader and StreamsWriter over Storage. These read and write any stream id with no codec and no stream class. Protocol.bind(graph, group) then gives one typed service for each stream in the group. Each shipped storage Layer is built this way, and each storage module also exports layerProtocol(options) for the generic pair alone. See Protocol.

Capabilities

CapabilityValuesEffect
forknone, copy, chainnone refuses forks with NotSupported. copy copies the prefix. chain shares it
atomicScopestream, storestore allows one mutation across several streams
wakepoll, pushpush signals on commit. poll uses the repair pass
expiryIndexlazy, indexedindexed answers nextExpiry, so a host can sweep. lazy expires on access

Mutations

A mutation is a non-empty ordered list of Create, Append, and Delete operations.

PreconditionOn
expectedOffsetAppend
expectedClosedAppend
producerAppend
expectedExpiresAtMsExpiry delete

Check every precondition before the mutation, in the same transaction. On the first failure, write nothing and answer MutationRejected { index, reason, record }.

ReasonWhen
offsetThe tail is not expectedOffset
closedThe stream is closed, or expectedClosed does not match
producerThe stored producer state is not the expected one
existsA create found an existing stream
not-foundAn append or delete named a missing stream
goneThe stream was removed
fork-source-goneA create named a removed fork source
expiry-mismatchAn expiry delete found a renewed deadline

The protocol maps each reason to a typed error. An I/O failure is StorageFault, with retryable.

Wakes

Each wake carries a ChangeSnapshot with present, currentOffset, closed, and softDeleted. A wake with no change is valid.

  • Register the subscriber before the first snapshot.
  • A push adapter keeps one pending wake for each subscriber.
  • Interrupting the subscriber, or closing the owner, releases the subscription.

committed publishes once after the outer transaction commits, with every stream id that the transaction wrote. A rollback publishes nothing. Hold one scoped PubSub and set committed to PubSub.subscribe(feed), so the subscription exists when the Effect returns. A caller that wants a Stream uses Stream.fromSubscription. Subscribers get no replay, and writes by other processes do not appear. Attached projections read committed. Readers read the coalesced changes(id).

Forks and expiry

A chain adapter stores the source id and fork offset, so a child read can join the source prefix. A removed source with live children is soft-deleted until the last child goes.

An indexed adapter stores each deadline. An expiry delete carries the deadline it saw. After expiry-mismatch, the next nextExpiry must move past that deadline.

Commit boundary

To offer CommitBoundary.withTransaction, an adapter needs one transaction owner, a way for mutate to join it, and wakes that publish after the outer commit. CommitBoundary comes from @streamsy/core.

Projections need more services from the same owner. Each comes from @streamsy/core/internal and joins a surrounding withTransaction:

ServiceUsed byDoes
OwnerRecordsCheckpointStore.layerReads, writes, and removes encoded strings by key
CommitHooksDurable attached projectionsRuns a hook at the end of each owned transaction, before it commits
ProjectionMarksProjection.attach with durable: true, ProjectionWakeup.attachKeeps the pending mark of each binding with work

An adapter that supports projections provides all four services from its storage Layer.

Contract tests

import { StorageContract } from "@streamsy/core/testing";
import { layer } from "./adapter.ts";

StorageContract.run({
  name: "my adapter",
  layer,
  expected: { fork: "none", atomicScope: "stream", wake: "poll", expiryIndex: "lazy" },
});

Run it under Bun with Vitest. The kit builds a new Layer for each case, gives a TestClock, and skips the cases that expected rules out. Set ownerRecords: true when the Layer also provides CommitBoundary, OwnerRecords, CommitHooks, and ProjectionMarks. The records cases then check that records, marks, and stream writes commit and roll back together.

Then serve Http.routes(Streamsy.group(Streamsy.Wildcard), { access: "full" }) over the adapter, and run the upstream server conformance suite against it. That suite checks the HTTP behavior that clients see.

What the tests establish

The contract tests check the Storage members against the capabilities that expected declares. With ownerRecords: true, they also check that owner records and projection marks share the transaction of stream writes. They do not check a SQL Transaction for Commit.transaction, the behavior of several processes on one database, or performance. Test those against your backend yourself.

Shipped adapters

CapabilityMemoryConstrained memoryBun SQLiteDurable Object SQLite
forkchaincopychainchain
atomicScopestorestreamstorestore
wakepushpollpush with repairpush with repair
expiryIndexindexedlazyindexedindexed, swept by the alarm

Projection storage

A custom checkpoint Layer can implement EncodedStore and call fromStore from @streamsy/projection/checkpoint. The record holds the encoded state and offsets. Commit.transaction also needs a SQL Transaction, which no storage capability provides. See Custom Commits and CheckpointStore.

On this page