Write
Append items as one atomic write, close a stream, retry safely with a producer position, and make an append conditional on the tail offset.
A stream service appends to the streams of its class. Each append encodes its items with the codec of the stream class and writes them as one atomic append. For the full signatures, see Streamsy.Api and AppendCallOptions.
Append
import { Effect } from "effect";
import { Notes } from "./streams.ts";
const write = Effect.gen(function* () {
const notes = yield* Notes;
const result = yield* notes.append({}, [{ id: "n1", text: "hello" }]);
return result.offset;
});append(params, items, options?) encodes every item before it sends anything. An item that does not encode fails the append with EncodeFault, and nothing is written. The stream then stores all the items, or none of them.
| Result | Meaning |
|---|---|
Appended | The append was written. offset is the new tail, and a read from it starts after these items |
Duplicate | Only with producer. The stream already accepted this producer position and wrote nothing. See Producer retries |
Both results have offset and closed. closed is true when the stream is closed after the call.
| Option | Effect |
|---|---|
close | true closes the stream after the items. See Close |
producer | A producer position, so that a retry cannot write the items twice |
expectedOffset | The append runs only when the tail equals this offset. See Conditional writes |
seq | A writer sequence string. Each seq must sort after the last one, compared as strings |
items must have at least one item, unless close is true. The compiler checks this. A JSON stream stores each item as a separate message, and a read returns the same items with the same boundaries. A byte stream concatenates the items of one append into one message, and an HTTP read concatenates the messages of a batch, so a byte read has no append boundaries. See Byte streams. An empty Uint8Array item makes the append die with a RangeError before anything is sent.
Each append restarts the TTL of a stream that has one. See Expiry.
To write a Durable State stream, see Write changes.
Close
import { Effect } from "effect";
import { Notes } from "./streams.ts";
const close = Effect.flatMap(Notes, (notes) => notes.append({}, [], { close: true }));close: true with no items only closes the stream. With items, the append writes the items and closes the stream in the same atomic append. A close-only append on a stream that is already closed succeeds with Appended, so a close is safe to repeat.
After the close, an append with items fails with StreamClosed. The offset of the error is the tail of the closed stream. Reads still work. A batch that reaches the tail has closed: true, and follow ends after it. See Follow.
Producer retries
A producer position is { producerId, producerEpoch, producerSeq }. It names a writer and counts its appends. Producer.next returns the position with producerSeq increased by one.
import { Effect } from "effect";
import { Producer } from "@streamsy/core";
import { Notes } from "./streams.ts";
const importOne = (position: Producer.Position) =>
Effect.gen(function* () {
const notes = yield* Notes;
const result = yield* notes.append({}, [{ id: "n3", text: "retried" }], {
producer: position,
});
// Appended and Duplicate both acknowledge this position.
return { result, next: Producer.next(position) };
});Advance the position only after Appended or Duplicate. On a retry, send the same items with the same position. The stream compares positions, not items, so a retry with other items also answers Duplicate and writes nothing.
The stream keeps the last epoch and the last producerSeq of each producerId. It checks each position against them:
| Position | Result |
|---|---|
A new producerId with producerSeq 0 | Appended |
A new producerId with producerSeq above 0 | ProducerGap, with expectedSeq 0 |
The same epoch and the next producerSeq | Appended |
The same epoch and an accepted producerSeq | Duplicate |
The same epoch and a skipped producerSeq | ProducerGap |
A higher epoch with producerSeq 0 | Appended. The higher epoch is now current |
| A lower epoch | StaleEpoch, with currentEpoch |
A higher epoch with a producerSeq above 0 makes the append die with InvalidEpochSeq. The producer check runs before every other check, so a retry of an append that closed the stream still answers Duplicate.
A Duplicate result has the current tail as offset, and the last accepted producerEpoch and producerSeq. That producerSeq can be higher than the one you sent.
To hand a producerId to a new writer, start a higher epoch at producerSeq 0. Appends of the old writer then fail with StaleEpoch. Streamsy does not choose epochs. A writer that restarts and cannot build the same items again must store the items with their position before it appends. See Producer.next and PROTOCOL.md section 5.2.1.
Conditional writes
expectedOffset makes the append run only when the tail still equals it. ZERO_OFFSET means that the stream must be empty. On a miss, the append fails with OffsetMismatch, with expected and actual. It writes no items, does not close the stream, and does not change the producer state.
import { Data, Effect, Predicate } from "effect";
import { Orders, apply, initial } from "./orders.ts";
type Reservation = Data.TaggedEnum<{
Rejected: {};
Committed: { readonly offset: string };
}>;
const Reservation = Data.taggedEnum<Reservation>();
const reserve = (sku: string) =>
Effect.gen(function* () {
const orders = yield* Orders;
const { state, offset } = yield* orders.fold({}, () => initial, apply);
if (state.reserved.has(sku)) return Reservation.Rejected();
const appended = yield* orders.append({}, [{ type: "reserved", sku }], {
expectedOffset: offset,
});
return Reservation.Committed({ offset: appended.offset });
}).pipe(Effect.retry({ while: Predicate.isTagged("OffsetMismatch"), times: 5 }));fold returns the folded state and the tail that it read up to. That tail is the expectedOffset. When another writer appends first, the tail moves, the append fails with OffsetMismatch, and the retry folds again.
The error type of append follows its options. Only an append with expectedOffset can fail with OffsetMismatch, and only an append with producer can fail with StaleEpoch or ProducerGap.
expectedOffset is a Streamsy extension to the Durable Streams protocol. See Extensions.
Read, fold, append
decide(params, spec) runs the loop above in one call. It folds the stream, calls spec.decide with the state and the tail that it read, and appends with that tail as expectedOffset:
import { Effect } from "effect";
import { Orders, apply, initial } from "./orders.ts";
const reserve = (sku: string) =>
Effect.gen(function* () {
const orders = yield* Orders;
return yield* orders.decide(
{},
{
initial: () => initial,
fold: apply,
decide: (state) =>
Effect.succeed(
state.reserved.has(sku)
? { value: "rejected" as const, items: [] }
: { value: "reserved" as const, items: [{ type: "reserved" as const, sku }] },
),
},
);
});The steps of decide:
- It folds the stream from the start with
initialandfold. - It calls
spec.decidewith the state and{ offset }, the tail that it read.spec.decidemust not write and must not change the state. - When
itemsis empty, it returns at once, with no append and no tail check. - Otherwise it appends the items with
expectedOffsetset to that tail. - When its own append fails with
OffsetMismatch, it folds only the new items onto the same state and goes back to step 2.
spec.retry is the Schedule for step 5. The default is Schedule.recurs(3), so one call makes at most four appends. When the schedule ends, decide fails with the last OffsetMismatch. An OffsetMismatch that spec.decide itself fails with is not retried.
decide does not retry any other failure. A lost reply can hide an append that was written, so a caller that retries the whole call must make spec.decide safe against its own earlier write.
decide returns { value, offset, appended }. offset is the new tail after an append, or the tail that it read when there was none. appended is true when it wrote the items. Streams.decide(address, spec) is the same call on an address. See DecideSpec.
On a Durable State stream, also pass the offset to DurableState.changes. See Conditional writes on a State stream.
To project a stream into state and then append, see Read, decide, append.
Errors
append fails with AppendFailure<O>, where O is the type of its options. decide fails with ReadFailure, AppendFailure<{ expectedOffset: string }>, or an error of spec.decide. Import these types from @streamsy/core. Each failure has a _tag.
_tag | When |
|---|---|
StreamNotFound | No stream has the id, or the stream expired |
StreamGone | The stream was removed, and forks still use it |
StreamClosed | The stream is closed. offset is its tail |
AppendConflict | The stream has another content type, or seq does not sort after the last one |
StreamBusy | The storage lost 64 attempts in a row to other writers, or the server answered 503 |
NotSupported | Over the remote Layer only. Its capabilities do not allow the option, or the server refused it |
OffsetMismatch | With expectedOffset only. The tail is not expectedOffset |
StaleEpoch | With producer only. The epoch is lower than the current epoch |
ProducerGap | With producer only. The producerSeq skips a number |
EncodeFault | An item does not encode with the codec. Nothing was written |
StorageFault | A local store failed |
TransportFault | The remote Layer could not send the request or read the response |
The typed API sends the content type of the stream class, so AppendConflict for a content type means that something else created the stream, such as a raw HTTP client. See What the server checks.
These cases are caller bugs, and append dies with them in place of a typed failure:
InvalidAppendRequest, for an append with no items that does not close the stream.InvalidEpochSeq, for a higher epoch that does not start atproducerSeq0.- A rejection that the options rule out, such as
OffsetMismatchfrom an append withoutexpectedOffset. RangeError, for an emptyUint8Arrayitem.
The StreamsWriter service from @streamsy/core/protocol keeps every rejection typed. See AppendError.
Over HTTP
POST on the stream path appends the body. Http.routes serves it for a stream whose access is "append" or "full". With "read", the server answers 405 with an allow header. See Expose the stream protocol.
| Header | Value |
|---|---|
Content-Type | The content type of the stream. Required when the body is not empty |
Stream-Closed | true closes the stream. With an empty body, the request only closes it |
Producer-Id, Producer-Epoch, Producer-Seq | The producer position. Send all three, with whole numbers for epoch and seq |
Stream-Expected-Offset | expectedOffset |
Stream-Seq | seq |
A JSON stream takes a JSON array and stores each element as one message, or takes one JSON value and stores it as one message. Another stream stores the body as one message. The body can be at most maxMessageSize, 1 MiB by default.
A successful append answers with stream-next-offset, the new tail. It adds stream-closed: true when the stream is closed, and producer-epoch and producer-seq when the request had a producer position.
| Status | When |
|---|---|
| 200 | Appended, for a request with a producer position and a body |
| 204 | Appended for any other request, or Duplicate |
| 400 | A header is invalid, the body is empty without Stream-Closed: true, Content-Type is missing, the JSON does not parse or is [], or a new epoch does not start at 0 |
| 403 | StaleEpoch. producer-epoch holds the current epoch |
| 404 | No stream in the group matches the id, or no stream has the id |
| 409 | StreamClosed, OffsetMismatch, ProducerGap, or AppendConflict |
| 410 | The stream was removed, and forks still use it |
| 413 | The body is larger than maxMessageSize |
| 503 | StreamBusy. Try again later |
A 409 for StreamClosed has stream-closed: true and the tail in stream-next-offset. A 409 for OffsetMismatch has the tail in stream-next-offset. A 409 for ProducerGap has producer-expected-seq and producer-received-seq.
The remote Layer sends expectedOffset and producer only when its capabilities allow them. Without the capability, the append fails with NotSupported before any request, with feature set to "expected-offset" or "producer". See Connect to a remote server.