Skip to Content
Streams

Declare streams

Declare a stream class with a service name, an id template and a codec, and collect stream classes in a group.

A stream is an ordered, append-only sequence of messages under one id. A stream class declares a set of streams with a service name, an id template, and a codec for the items. Each kind of stream has its own factory.

import { Schema } from "effect";
import { Streamsy } from "@streamsy/core";

export const Note = Schema.Struct({ id: Schema.String, text: Schema.String });

export class Notes extends Streamsy.json<Notes>()("Notes/Notes", "notes", {
  params: {},
  schema: Note,
}) {}
export class Blobs extends Streamsy.bytes<Blobs>()("Notes/Blobs", "blobs/:blobId", {
  params: { blobId: Schema.String },
  contentType: "image/png",
}) {}

The first argument is the service name. It is a global Effect service key, so two classes with the same name are the same service. Prefix each name with an application namespace, as in Notes/Notes. The second argument is the id template, which maps a set of params to the id of one stream. The third argument holds the params and the options of the factory.

Factories

FactoryItemsPage
Streamsy.bytesUint8ArrayByte streams
Streamsy.jsonOne schema value in each messageJSON streams
Streamsy.stateDurable State change events and controlsDurable State streams
Streamsy.wakeWakeRow change events and controlsWake streams
Streamsy.customThe items of the address that it returnsCustom streams

Byte streams

Streamsy.bytes<Self>()(name, template, { params, contentType? })

A byte stream is a plain Durable Stream, and its items are Uint8Array. contentType sets the content type that create, append, and fork send. The default is application/octet-stream.

append concatenates the items of one call into one request body, and the server stores that body as one message. The protocol does not frame byte streams. An HTTP read concatenates every message in the batch into one response body, so a remote reader gets one Uint8Array for each response and sees no append boundaries. If a reader needs record boundaries, encode them in the bytes, for example as newline-delimited records.

See Core Concepts in the Durable Streams docs.

JSON streams

Streamsy.json<Self>()(name, template, { params, schema })

A JSON stream holds one JSON value in each message. Its content type is always application/json. schema gives the item type. The typed API encodes each item with schema before an append, and an item that fails to encode fails the append with EncodeFault before anything is written. A read decodes each message with schema, and the first message that fails to decode fails the read with DecodeFault.

append sends the items as one JSON array, and the server stores each array element as a separate message. A read returns the same items with the same boundaries.

See JSON mode in the Durable Streams docs.

Durable State streams

Streamsy.state<Self>()(name, template, { params, collections })

A Durable State stream is a JSON stream of change events and control messages for keyed rows. Each key of collections is a collection type with a value schema and a key field. See Durable State streams.

Wake streams

Streamsy.wake<Self>()(name, template, { params })

A wake stream is a Durable State stream with one stream collection, and each row holds the tail of one source stream. See Wake streams.

Custom streams

Streamsy.custom<Self>()(name, { parse, address })

A custom stream is for an id grammar that a template cannot express. See Custom streams.

Templates and addresses

A template is a relative path. Each segment is a literal or a :name param. Every :name must be a key of params, and every key of params must appear in the template, or the declaration does not compile.

Each value in params is a codec that decodes from a string, so a param can have a type other than string. parse returns None when an id does not match the template or a param fails to decode.

import { Schema } from "effect";
import { Streamsy } from "@streamsy/core";
import { Note } from "./streams.ts";

export class Versions extends Streamsy.json<Versions>()(
  "Notes/Versions",
  "notes/:noteId/versions/:version",
  { params: { noteId: Schema.String, version: Schema.FiniteFromString }, schema: Note },
) {}

const address = Versions.at({ noteId: "n1", version: 3 }); // address.id === "notes/n1/versions/3"
const params = Versions.parse("notes/n1/versions/3"); // Option.some({ noteId: "n1", version: 3 })
const invalid = Versions.parse("notes/n1/versions/three"); // Option.none()
const owns = Versions.match("notes"); // false
MethodReturns
at(params)The StreamAddress of one stream
parse(id)The decoded params, or None
match(id)true when the stream class owns the id

A template with no params has one address, at({}). Projection inputs, DurableState.changes, and Serve routes take addresses.

Every id and template follows these rules:

  • / separates segments.
  • A segment cannot be empty, ., or ...
  • An id cannot contain %, ?, #, or \.
  • A template cannot start with / or contain *.

Groups

Streamsy.group(...streams) collects stream classes. Pass the group to a storage Layer, Http.routes, or RemoteStreams.layer.

import { Streamsy } from "@streamsy/core";
import { Blobs, Notes } from "./streams.ts";

export const notesApp = Streamsy.group(Notes, Blobs);

group throws a RangeError when two streams have the same name or the same template, or when two templates can match the same id. The overlap check treats a param as a match for any segment, so notes/:noteId and notes/latest overlap even when the codec of noteId rejects latest. The check skips custom streams. Streamsy.Wildcard is a custom stream that matches every id, and group throws unless it is last. See Custom streams in groups.

Over HTTP, Http.routes does not run the codec. See What the server checks.

Methods

To get the service of a stream class, yield* the class, as in yield* Notes. The service has these methods:

MethodDoesPage
create(params, options?)Creates the streamCreate and fork
fork(params, source, options?)Creates a child stream that starts with a prefix of sourceCreate and fork
remove(params)Deletes the streamCreate and fork
append(params, items, options?)Writes the items as one atomic appendWrite
decide(params, spec)Folds the stream, then appends with the tail it read as expectedOffsetWrite
read(params, options?)Streams the batches after an offset, up to the current tailRead
collect(params, options?)Reads to the current tail and returns the itemsRead
fold(params, initial, f, options?)Reads to the current tail and folds the items into a valueRead
follow(params, options?)Reads, then waits for new messages until the stream closesRead
session(params, { offset })Makes one raw long-poll readRead
head(params)Returns the tail and the configuration of the streamRead

Every method takes params, so a service reaches only the streams of its own class. fork takes source as params of the same class.

Offsets

Each message has an offset. An offset is an opaque string, and the offsets of one stream sort as strings in write order. ZERO_OFFSET is the offset before the first message. Do not compare offsets from two different streams.

Results and errors

create succeeds with Created or Exists. append succeeds with Appended or Duplicate. A protocol rejection, such as StreamNotFound or StreamClosed, fails the effect with a typed error that has a _tag and the stream id. The error type of each call lists only the rejections that its options allow. For example, append can fail with OffsetMismatch only when you pass expectedOffset.

To handle a rejection, catch it by its tag:

import { Effect } from "effect";
import { Notes } from "./streams.ts";

const tolerant = Effect.flatMap(Notes, (notes) =>
  notes.append({}, [{ id: "n2", text: "again" }]).pipe(
    Effect.catchTags({
      StreamNotFound: () => Effect.succeed(undefined),
      StreamClosed: (error) => Effect.logWarning(`closed at ${error.offset}`),
    }),
  ),
);

A call can also fail with a fault. A fault has a _tag and a message, and no stream id.

FaultCause
StorageFaultA local store failed
TransportFaultThe remote Layer could not send a request or read the response
EncodeFaultThe codec could not encode an item for an append
DecodeFaultThe codec could not decode a message on a read

Protocol services

@streamsy/core/protocol exports the StreamsReader and StreamsWriter services, the Streams functions that take a raw StreamAddress, and Protocol.layer. Storage adapters, protocol tests, and the HTTP code use them. Applications use stream classes.

On this page