Skip to Content
Streams

Write

Append items as one atomic write, close a stream, retry safely with a producer position, and make an append conditional on the tail offset.

A stream service appends to the streams of its class. Each append encodes its items with the codec of the stream class and writes them as one atomic append. For the full signatures, see Streamsy.Api and AppendCallOptions.

Append

import { Effect } from "effect";
import { Notes } from "./streams.ts";

const write = Effect.gen(function* () {
  const notes = yield* Notes;
  const result = yield* notes.append({}, [{ id: "n1", text: "hello" }]);
  return result.offset;
});

append(params, items, options?) encodes every item before it sends anything. An item that does not encode fails the append with EncodeFault, and nothing is written. The stream then stores all the items, or none of them.

ResultMeaning
AppendedThe append was written. offset is the new tail, and a read from it starts after these items
DuplicateOnly with producer. The stream already accepted this producer position and wrote nothing. See Producer retries

Both results have offset and closed. closed is true when the stream is closed after the call.

OptionEffect
closetrue closes the stream after the items. See Close
producerA producer position, so that a retry cannot write the items twice
expectedOffsetThe append runs only when the tail equals this offset. See Conditional writes
seqA writer sequence string. Each seq must sort after the last one, compared as strings

items must have at least one item, unless close is true. The compiler checks this. A JSON stream stores each item as a separate message, and a read returns the same items with the same boundaries. A byte stream concatenates the items of one append into one message, and an HTTP read concatenates the messages of a batch, so a byte read has no append boundaries. See Byte streams. An empty Uint8Array item makes the append die with a RangeError before anything is sent.

Each append restarts the TTL of a stream that has one. See Expiry.

To write a Durable State stream, see Write changes.

Close

import { Effect } from "effect";
import { Notes } from "./streams.ts";

const close = Effect.flatMap(Notes, (notes) => notes.append({}, [], { close: true }));

close: true with no items only closes the stream. With items, the append writes the items and closes the stream in the same atomic append. A close-only append on a stream that is already closed succeeds with Appended, so a close is safe to repeat.

After the close, an append with items fails with StreamClosed. The offset of the error is the tail of the closed stream. Reads still work. A batch that reaches the tail has closed: true, and follow ends after it. See Follow.

Producer retries

A producer position is { producerId, producerEpoch, producerSeq }. It names a writer and counts its appends. Producer.next returns the position with producerSeq increased by one.

import { Effect } from "effect";
import { Producer } from "@streamsy/core";
import { Notes } from "./streams.ts";

const importOne = (position: Producer.Position) =>
  Effect.gen(function* () {
    const notes = yield* Notes;
    const result = yield* notes.append({}, [{ id: "n3", text: "retried" }], {
      producer: position,
    });
    // Appended and Duplicate both acknowledge this position.
    return { result, next: Producer.next(position) };
  });

Advance the position only after Appended or Duplicate. On a retry, send the same items with the same position. The stream compares positions, not items, so a retry with other items also answers Duplicate and writes nothing.

The stream keeps the last epoch and the last producerSeq of each producerId. It checks each position against them:

PositionResult
A new producerId with producerSeq 0Appended
A new producerId with producerSeq above 0ProducerGap, with expectedSeq 0
The same epoch and the next producerSeqAppended
The same epoch and an accepted producerSeqDuplicate
The same epoch and a skipped producerSeqProducerGap
A higher epoch with producerSeq 0Appended. The higher epoch is now current
A lower epochStaleEpoch, with currentEpoch

A higher epoch with a producerSeq above 0 makes the append die with InvalidEpochSeq. The producer check runs before every other check, so a retry of an append that closed the stream still answers Duplicate.

A Duplicate result has the current tail as offset, and the last accepted producerEpoch and producerSeq. That producerSeq can be higher than the one you sent.

To hand a producerId to a new writer, start a higher epoch at producerSeq 0. Appends of the old writer then fail with StaleEpoch. Streamsy does not choose epochs. A writer that restarts and cannot build the same items again must store the items with their position before it appends. See Producer.next and PROTOCOL.md section 5.2.1.

Conditional writes

expectedOffset makes the append run only when the tail still equals it. ZERO_OFFSET means that the stream must be empty. On a miss, the append fails with OffsetMismatch, with expected and actual. It writes no items, does not close the stream, and does not change the producer state.

import { Data, Effect, Predicate } from "effect";
import { Orders, apply, initial } from "./orders.ts";

type Reservation = Data.TaggedEnum<{
  Rejected: {};
  Committed: { readonly offset: string };
}>;
const Reservation = Data.taggedEnum<Reservation>();

const reserve = (sku: string) =>
  Effect.gen(function* () {
    const orders = yield* Orders;
    const { state, offset } = yield* orders.fold({}, () => initial, apply);
    if (state.reserved.has(sku)) return Reservation.Rejected();
    const appended = yield* orders.append({}, [{ type: "reserved", sku }], {
      expectedOffset: offset,
    });
    return Reservation.Committed({ offset: appended.offset });
  }).pipe(Effect.retry({ while: Predicate.isTagged("OffsetMismatch"), times: 5 }));

fold returns the folded state and the tail that it read up to. That tail is the expectedOffset. When another writer appends first, the tail moves, the append fails with OffsetMismatch, and the retry folds again.

The error type of append follows its options. Only an append with expectedOffset can fail with OffsetMismatch, and only an append with producer can fail with StaleEpoch or ProducerGap.

expectedOffset is a Streamsy extension to the Durable Streams protocol. See Extensions.

Read, fold, append

decide(params, spec) runs the loop above in one call. It folds the stream, calls spec.decide with the state and the tail that it read, and appends with that tail as expectedOffset:

import { Effect } from "effect";
import { Orders, apply, initial } from "./orders.ts";

const reserve = (sku: string) =>
  Effect.gen(function* () {
    const orders = yield* Orders;
    return yield* orders.decide(
      {},
      {
        initial: () => initial,
        fold: apply,
        decide: (state) =>
          Effect.succeed(
            state.reserved.has(sku)
              ? { value: "rejected" as const, items: [] }
              : { value: "reserved" as const, items: [{ type: "reserved" as const, sku }] },
          ),
      },
    );
  });

The steps of decide:

  1. It folds the stream from the start with initial and fold.
  2. It calls spec.decide with the state and { offset }, the tail that it read. spec.decide must not write and must not change the state.
  3. When items is empty, it returns at once, with no append and no tail check.
  4. Otherwise it appends the items with expectedOffset set to that tail.
  5. When its own append fails with OffsetMismatch, it folds only the new items onto the same state and goes back to step 2.

spec.retry is the Schedule for step 5. The default is Schedule.recurs(3), so one call makes at most four appends. When the schedule ends, decide fails with the last OffsetMismatch. An OffsetMismatch that spec.decide itself fails with is not retried.

decide does not retry any other failure. A lost reply can hide an append that was written, so a caller that retries the whole call must make spec.decide safe against its own earlier write.

decide returns { value, offset, appended }. offset is the new tail after an append, or the tail that it read when there was none. appended is true when it wrote the items. Streams.decide(address, spec) is the same call on an address. See DecideSpec.

On a Durable State stream, also pass the offset to DurableState.changes. See Conditional writes on a State stream.

To project a stream into state and then append, see Read, decide, append.

Errors

append fails with AppendFailure<O>, where O is the type of its options. decide fails with ReadFailure, AppendFailure<{ expectedOffset: string }>, or an error of spec.decide. Import these types from @streamsy/core. Each failure has a _tag.

_tagWhen
StreamNotFoundNo stream has the id, or the stream expired
StreamGoneThe stream was removed, and forks still use it
StreamClosedThe stream is closed. offset is its tail
AppendConflictThe stream has another content type, or seq does not sort after the last one
StreamBusyThe storage lost 64 attempts in a row to other writers, or the server answered 503
NotSupportedOver the remote Layer only. Its capabilities do not allow the option, or the server refused it
OffsetMismatchWith expectedOffset only. The tail is not expectedOffset
StaleEpochWith producer only. The epoch is lower than the current epoch
ProducerGapWith producer only. The producerSeq skips a number
EncodeFaultAn item does not encode with the codec. Nothing was written
StorageFaultA local store failed
TransportFaultThe remote Layer could not send the request or read the response

The typed API sends the content type of the stream class, so AppendConflict for a content type means that something else created the stream, such as a raw HTTP client. See What the server checks.

These cases are caller bugs, and append dies with them in place of a typed failure:

  • InvalidAppendRequest, for an append with no items that does not close the stream.
  • InvalidEpochSeq, for a higher epoch that does not start at producerSeq 0.
  • A rejection that the options rule out, such as OffsetMismatch from an append without expectedOffset.
  • RangeError, for an empty Uint8Array item.

The StreamsWriter service from @streamsy/core/protocol keeps every rejection typed. See AppendError.

Over HTTP

POST on the stream path appends the body. Http.routes serves it for a stream whose access is "append" or "full". With "read", the server answers 405 with an allow header. See Expose the stream protocol.

HeaderValue
Content-TypeThe content type of the stream. Required when the body is not empty
Stream-Closedtrue closes the stream. With an empty body, the request only closes it
Producer-Id, Producer-Epoch, Producer-SeqThe producer position. Send all three, with whole numbers for epoch and seq
Stream-Expected-OffsetexpectedOffset
Stream-Seqseq

A JSON stream takes a JSON array and stores each element as one message, or takes one JSON value and stores it as one message. Another stream stores the body as one message. The body can be at most maxMessageSize, 1 MiB by default.

A successful append answers with stream-next-offset, the new tail. It adds stream-closed: true when the stream is closed, and producer-epoch and producer-seq when the request had a producer position.

StatusWhen
200Appended, for a request with a producer position and a body
204Appended for any other request, or Duplicate
400A header is invalid, the body is empty without Stream-Closed: true, Content-Type is missing, the JSON does not parse or is [], or a new epoch does not start at 0
403StaleEpoch. producer-epoch holds the current epoch
404No stream in the group matches the id, or no stream has the id
409StreamClosed, OffsetMismatch, ProducerGap, or AppendConflict
410The stream was removed, and forks still use it
413The body is larger than maxMessageSize
503StreamBusy. Try again later

A 409 for StreamClosed has stream-closed: true and the tail in stream-next-offset. A 409 for OffsetMismatch has the tail in stream-next-offset. A 409 for ProducerGap has producer-expected-seq and producer-received-seq.

The remote Layer sends expectedOffset and producer only when its capabilities allow them. Without the capability, the append fails with NotSupported before any request, with feature set to "expected-offset" or "producer". See Connect to a remote server.

On this page