Skip to Content
Streams

Read

Read stored batches after an offset, follow a stream for new messages, and fold a stream into a value.

A stream service reads the streams of its class. Each read starts after an offset and gives batches of items that the codec of the stream class decoded. For the full signatures, see Streamsy.Api and Streams.

interface Batch<A> {
  readonly items: ReadonlyArray<A>;
  readonly nextOffset: string;
  readonly upToDate: boolean;
  readonly closed: boolean;
}
FieldMeaning
itemsThe decoded items of the stored messages in the batch
nextOffsetThe offset of the last message in the batch, or the tail when the batch has no messages
upToDatetrue when nextOffset is the tail of the stream
closedtrue when the stream is closed and the batch reached the tail

Keep nextOffset to continue later. A read never includes the message at the offset that it starts from.

MethodEnds
read(params, { offset? })At the current tail
collect(params, { offset? })At the current tail
fold(params, initial, f, { offset? })At the current tail
follow(params, { offset?, retry? })When the stream closes
session(params, { offset, cursor? })After one long-poll wait
head(params)At once. It returns the tail, not messages

Start offset

The offset option sets where a read starts.

offsetReads from
Not set, or -1The start of the stream
nowThe current tail
An offsetThe message after this offset

Any other value fails the read with InvalidReadRequest. See Offsets for what an offset is.

Read

import { Effect, Stream } from "effect";
import { Notes } from "./streams.ts";

const sizes = Effect.flatMap(Notes, (notes) =>
  notes.read({}).pipe(
    Stream.map((batch) => batch.items.length),
    Stream.runCollect,
  ),
);

const everything = Effect.flatMap(Notes, (notes) => notes.collect({}));

read is an Effect Stream of batches. It does nothing until a sink runs it. It reads one page at a time and ends after the first batch whose upToDate is true. A stream with no messages gives one empty batch.

The storage Layer sets the size of a page with its readLimit option. The default is 1000 messages. Over the remote Layer, the server sets the page size. See Choose storage.

collect reads to the current tail and returns { items, offset }. offset is the tail that it read up to. Pass it as expectedOffset for a conditional write, or as offset to read on from there.

Every read restarts the TTL of a stream that has one. See Expiry.

Follow

import { Effect, Stream } from "effect";
import { Notes } from "./streams.ts";

const live = Effect.flatMap(Notes, (notes) =>
  notes.follow({}).pipe(
    Stream.flatMap((batch) => Stream.fromIterable(batch.items)),
    Stream.runForEach((note) => Effect.log(note.text)),
  ),
);

const consumer = Effect.gen(function* () {
  const fiber = yield* live.pipe(Effect.forkScoped);
  return fiber;
});

follow reads pages to the tail, as read does. It then makes one long-poll wait at a time, and sends the cursor of each wait in the next one. A wait returns every message after the offset, with no page limit.

follow gives no empty batch, except the last batch of a closed stream. The stream of batches ends after a batch whose closed is true. To stop earlier, interrupt the fiber, which also ends a parked wait, or use Stream.take to stop after a number of batches.

When a read fails with a retryable StorageFault or TransportFault, follow logs a warning and retries the read from the same offset, so it gives each item once. A TransportFault is retryable when the request failed or the server answered 429 or 5xx. A StorageFault is retryable when its retryable field is true. Any other failure ends the stream.

OptionEffect
offsetWhere the read starts. See Start offset
retryThe Schedule for retries. The default is exponential from 200 ms, jittered, with each wait capped at 30 s, and has no limit

read, collect, and fold do not retry. See Streams.FollowOptions.

Session

import { Effect } from "effect";
import type { ReadNextResult } from "@streamsy/core";
import { Notes } from "./streams.ts";

const first = Effect.flatMap(Notes, (notes) => notes.session({}, { offset: "-1" }));

const next = (previous: ReadNextResult) =>
  Effect.flatMap(Notes, (notes) =>
    notes.session({}, { offset: previous.nextOffset, cursor: previous.cursor }),
  );

session makes one long-poll wait, for an application with its own poll loop. offset is required. When messages follow offset, or the stream is closed, session returns at once. Otherwise it waits for an append, a close, or the long-poll timeout. The storage Layer sets the timeout with longPollTimeoutMs. The default is 30 s, and 25 s on Durable Object storage.

session returns a ReadNextResult with messages, nextOffset, upToDate, closed, cursor, and timedOut. The messages are raw bytes. session does not decode them and does not retry. timedOut is true when the wait ended with no new messages. Pass nextOffset and cursor to the next call.

Fold

import { Effect } from "effect";
import { Notes } from "./streams.ts";

const count = Effect.flatMap(Notes, (notes) =>
  notes.fold(
    {},
    () => 0,
    (total) => total + 1,
  ),
);

fold reads to the current tail and returns { state, offset }, as collect returns { items, offset }. The initial value is a function, so each fold starts from a new value and a mutable accumulator is safe. To fold only new items, pass the earlier offset and start from the earlier state.

To keep folding while the stream grows, fold over follow:

import { Effect, Stream } from "effect";
import { Notes } from "./streams.ts";

const live = Effect.flatMap(Notes, (notes) =>
  notes.follow({}).pipe(
    Stream.flatMap((batch) => Stream.fromIterable(batch.items)),
    Stream.runFold(
      () => 0,
      (total) => total + 1,
    ),
  ),
);

A fold over follow completes when the stream closes.

head(params) returns the tail and the configuration of the stream, and no messages. The result has contentType, nextOffset, closed, and the ttlSeconds or expiresAt that the stream was created with. head does not restart a TTL. See HeadResult.

Errors

MethodError type
read, collect, foldReadFailure
followFollowFailure
sessionFollowFailure without DecodeFault
headHeadFailure

Import these types from @streamsy/core. Each failure has a _tag.

_tagWhen
StreamNotFoundNo stream has the id, or the stream expired
StreamGoneThe stream was removed, and forks still use it
InvalidReadRequestoffset is not -1, now, or a valid offset. Not for head
NotSupportedfollow and session only, over the remote Layer, when the server refuses the long poll with a stream-not-supported header
DecodeFaultA message does not match the codec. Not for session or head
StorageFaultA local store failed
TransportFaultThe remote Layer could not send a request or read the response

A DecodeFault fails the whole batch and stops the read. The read does not skip the message, so the application selects the offset to continue from. Its message names the stream and the index of the message in the batch. See DecodeFault.

To read and fold the rows of a Durable State stream, see Read and fold rows.

Over HTTP

Every access rule allows GET and HEAD, so Http.routes serves reads for every stream in the group. See Expose the stream protocol.

RequestRead
GET, with or without offsetOne page, from the start when offset is not set
GET with offset and live=long-pollOne long-poll wait
GET with offset and live=sseA Server-Sent Events response
HEADThe tail and the configuration

offset takes the values in Start offset. A live mode needs offset.

A read answers with these headers:

HeaderValue
stream-next-offsetThe nextOffset of the batch. Send it as offset on the next request
stream-up-to-datetrue when the batch reached the tail. Always true on a long poll
stream-closedtrue when the stream is closed and the batch reached the tail
stream-cursorOn a long poll of an open stream only. Send it as cursor on the next long poll

A JSON stream answers with one JSON array of its messages. Another stream answers with the bytes of its messages, concatenated. The remote Layer splits a JSON array back into messages. See Byte streams.

Catch-up reads

A GET without live answers 200 with one page, also when the page is empty. The response has an etag, and a request whose If-None-Match equals it answers 304. cache-control is private, max-age=60, stale-while-revalidate=300, or public in place of private when cacheVisibility is "public". With offset=now, the response has an empty body ([] for JSON), stream-up-to-date: true, and cache-control: no-store.

Long poll

With live=long-poll, the server answers at once when messages follow offset or the stream is closed. Otherwise it waits for an append, a close, or longPollTimeoutMs. A wait that ends with messages answers 200 with every message after offset. A wait that ends with none answers 204 with cache-control: no-store.

The cursor is a number. The server sends a new one with each long poll, and it changes at least every 20 s. A client sends the last cursor back, so that a CDN can collapse the long polls of many readers into one request to the origin. A cursor that is not a non-negative whole number answers 400. See PROTOCOL.md.

follow and session over the remote Layer use long polls. The remote Layer does not use SSE.

SSE

With live=sse, the server answers with content-type: text/event-stream. It sends a data event for each batch with messages, then a control event with streamNextOffset. While the stream is open, the control event also has streamCursor, and upToDate: true once the reader reaches the tail. At the close, it has streamClosed: true, and the response ends.

A JSON stream sends each data event as a JSON array, and a text/* stream sends its text. Other streams send base64 and add the stream-sse-data-encoding: base64 response header.

A response ends after sseDeadlineMs, 60 s by default, scaled by a random factor from 0.9 to 1.1. It also ends when the stream is removed. It ends only between batches. To continue, the client reconnects with the last streamNextOffset.

Status codes

StatusWhen
200A read, a long poll with messages, an SSE response, or a HEAD
204A long poll that ended with no messages
304A catch-up read whose If-None-Match matches the etag
400offset or cursor is invalid, or a live mode has no offset
404No stream in the group matches the id, or no stream has the id
410The stream was removed, and forks still use it

A HEAD answers with content-type, stream-next-offset, and cache-control: no-store. It adds stream-closed, stream-ttl, and stream-expires-at when they apply.

On this page