Commit
Reference for the Commit namespace and the Transaction service in @streamsy/projection. A Commit decides how a projection writes each unit and its checkpoint, and which guarantee it gives.
import {
Commit,
HandlerFailed,
Transaction,
type Accumulated,
type CheckpointRecord,
type CheckpointStore,
type Emitted,
type Folded,
type Loaded,
type OutputBatch,
type ProjectionFault,
type ProjectionKey,
type Unit,
} from "@streamsy/projection";
import type { CommitBoundary, StreamAddress } from "@streamsy/core";
import type { Cause, Data, Effect, Layer, Pipeable } from "effect";
import type { SqlClient } from "effect/sql";A Commit takes the unit that a runner read, runs the handler, and writes the checkpoint. Its result type decides which constructors it fits, and its guarantee says how often the effects of a unit happen. See Commits and Custom Commits.
| Commit | Result | Guarantee | Needs |
|---|---|---|---|
Commit.checkpoint | Folded | Exactly once for the saved value | CheckpointStore |
Commit.transaction | Any | Exactly once in the SQL transaction | CheckpointStore, Transaction |
Commit.producer | Emitted, Accumulated | Effectively once for the outputs | CheckpointStore |
Commit.atLeastOnce | void or WakeAt | At least once | CheckpointStore |
Commits
Commit.checkpoint
declare const checkpoint: Commit.Commit<Folded<unknown>, never, CheckpointStore>;
checkpoint runs the handler, then saves the encoded state and the input offsets in one compare-and-set write. It fits fold, and it is the default Commit of fold. When the write fails, the next pass runs the steps again, so keep the steps free of external effects.
Commit.transaction
declare const transaction: Commit.Commit<
| void
| WakeAt
| Folded<unknown>
| Emitted<Readonly<Record<string, never>>>
| Accumulated<unknown, Readonly<Record<string, never>>>,
SqlClient.SqlClient,
CheckpointStore | Transaction
>;
transaction runs the handler, the output appends, and the checkpoint write (including the returned wake request) in one SQL transaction of the storage owner. It fits every constructor. The handler gets the SqlClient of the transaction, so its SQL commits or rolls back with the checkpoint.
- Every declared output must be on the storage owner of the transaction. When one is not, the unit fails with phase
loadand reasonunsupported-composition. - A stream append in the handler to a stream on another owner fails with phase
processand reasonunsupported-composition. - When the checkpoint changed after the runner read it, the unit fails with phase
checkpointand reasontoken-conflict. - An output append failure has phase
process: storage or transport faults usestorage-failure, a busy output usesoutput-busy, and other rejections or encoding failures useinvalid-output. - Only
transaction, alone or wrapped bywithCompensation, acceptsprepare. It runspreparebefore it opens the transaction. - A retry reads the inputs again, so a longer tail can change
unit.key. An external call in the handler happens at least once.
The Transaction service opens the transaction. A memory host has no SqlClient, so transaction needs a SQL host.
Commit.producer
declare const producer: Commit.Commit<
Accumulated<unknown, Readonly<Record<string, never>>> | Emitted<Readonly<Record<string, never>>>,
never,
CheckpointStore
>;
producer appends the items of each output with a producer position, so a retry does not write a unit twice. It fits emit, mapAccum, and mapAccumBatch. The handler must give the same items for the same unit.
- It runs the handler.
- When an output has items, it saves the input ranges and the next sequence of each such output in the checkpoint. This write is the pin.
- It appends the items of each output with its own producer position. A retry sends the same positions, and the server drops the duplicates.
- It saves the state and the new offsets.
| Producer field | Value |
|---|---|
producerId | id/v<version>, then / and the canonical params JSON when there are params |
producerEpoch | The generation |
producerSeq | The pinned sequence of that output |
Storage refuses a producer id over 255 code points. When the producerId above is longer, producer appends under id/v<version>/# and the SHA-256 hex of the full id instead. When the projection id itself is long, producer cuts the part before # by code point, so the whole id is 255 code points. A retry computes the same id, and a producerId of at most 255 code points is used as it is.
producercreates a missing output on the storage owner of the checkpoint. An output on another owner must exist before the first append.- A closed output fails the unit with phase
pinand reasoninvalid-output. - When the stored epoch of an output is not the generation, or a newer generation has appended to the output, the unit fails with reason
stale-epoch. - When a pinned unit runs again and an output that was pinned has no items, or an output that was not pinned has items, the unit fails with phase
processand reasoninvalid-output. - The appends to several outputs do not commit together.
Commit.atLeastOnce
declare const atLeastOnce: Commit.Commit<void | WakeAt, never, CheckpointStore>;
atLeastOnce saves the input ranges and recorded due time of the unit, runs the handler, and then saves the new offsets and returned wake request. It fits process. When the handler or the last write fails, the next pass runs the same unit again, with the same unit.key. A due unit's key includes its recorded deadline, so different deadlines have different keys. Use unit.key as the idempotency key of an external call. Set the declaration's binding retry policy for a retry budget and onExhausted, and return Projection.wakeAt for a deferred retry or a snooze. See the effects guide for the complete pattern.
Commit.make
declare function make<Result, Provides = never, Requires = never>(
options: Omit<Commit.Commit<Result, Provides, Requires>, typeof Commit.TypeId | "pipe">,
): Commit.Commit<Result, Provides, Requires>;
Custom Commits do not support timer handlers in this release. A typed WakeAt result does not fit a custom Commit<void>; use Commit.atLeastOnce or Commit.transaction, including compensation wrappers, for timers.
make returns a custom Commit from its name, guarantee, load, apply, and optional remove. make infers the result and the required services from apply and load. When the Commit provides handler services, write the Result, Provides, and Requires type arguments.
A custom Commit must follow these rules:
- Never store
nextwhenrunfails. - Make the write conditional on
loaded.token. When the token moved, fail with phasecheckpointand reasontoken-conflict. - Return the token
"0"fromloadwhen no record exists. - Return the stored
statefromload, so thatProjection.readreads it. - Declare the guarantee of the actual write. When
runand the write are not atomic, declareat-least-once.
A custom Commit reads the input range again on a retry, so a retry can see a longer tail. It does not accept prepare, and it must append the outputs of an Emitted or Accumulated result itself.
Commit.withCompensation
declare const withCompensation: {
<Result, R>(
f: (
result: Result,
cause: Cause.Cause<ProjectionFault | HandlerFailed>,
unit: Unit,
) => Effect.Effect<void, never, R>,
): <P, Q>(self: Commit.Commit<Result, P, Q>) => Commit.Commit<Result, P, Q | R>;
<Result, P, Q, R>(
self: Commit.Commit<Result, P, Q>,
f: (
result: Result,
cause: Cause.Cause<ProjectionFault | HandlerFailed>,
unit: Unit,
) => Effect.Effect<void, never, R>,
): Commit.Commit<Result, P, Q | R>;
};
withCompensation wraps a Commit. When the handler succeeded and the Commit then failed, the wrapped Commit calls f with the handler result, the failure, and the unit, and then fails with the same failure. Call it directly or through .pipe. The wrapped Commit keeps the pin and the prepare support of self.
f cannot fail. Compensation does not make an external effect atomic.
The Commit type
The names in these signatures are members of the Commit namespace.
Commit.Commit
interface Commit<in Result, in Provides = never, out Requires = never> extends Pipeable.Pipeable {
readonly [TypeId]: Variance<Result, Provides, Requires>;
readonly name: string;
readonly guarantee: "exactly-once" | "effectively-once" | "at-least-once";
readonly load: (key: ProjectionKey) => Effect.Effect<Loaded, ProjectionFault, Requires>;
readonly apply: (
pending: Pending<Result, Provides>,
) => Effect.Effect<void, ProjectionFault | HandlerFailed, Requires>;
readonly remove?: (key: ProjectionKey) => Effect.Effect<void, ProjectionFault, Requires>;
}
A Commit writes the units of a projection. Result is the handler result that it takes, Provides holds the services that it gives the handler, and Requires holds the services that it needs.
| Field | Meaning |
|---|---|
name | A name for messages |
guarantee | How often the effects of a unit happen |
load | The function that loads the checkpoint record of a key, and its token |
apply | The function that runs the handler of a unit and writes the result and the checkpoint |
remove | The optional function that deletes the checkpoint record of a key. Projection.forget calls it |
A Commit with a result type that does not fit a constructor fails at the commit: field of the definition.
Commit.Pending
interface Pending<Result, Provides = never> {
readonly key: ProjectionKey;
readonly unit: Unit;
readonly loaded: Loaded;
readonly next: CheckpointRecord;
readonly run: Effect.Effect<Result, HandlerFailed, Provides>;
}
apply takes a Pending: a unit that is ready to commit.
| Field | Meaning |
|---|---|
key | The checkpoint key |
unit | The unit, with the encoded params |
loaded | The record and the token that the runner loaded before it read the unit |
next | The record to save after the unit: the new offsets, and the state from before the unit |
run | The Effect that runs the handler. It fails with HandlerFailed, and the runner unwraps the handler error from it |
To save the state, put the encoded field of a Folded or Accumulated result into next as state.
Commit.Variance
interface Variance<in Result, in Provides, out Requires> {
readonly result: (_: Result) => void;
readonly provides: (_: Provides) => void;
readonly requires: () => Requires;
}
Variance holds the type parameters of a Commit under TypeId. It has no run-time use.
Commit.TypeId
declare const TypeId: "~@streamsy/projection/Commit";
TypeId is the key of the Variance field of every Commit. make adds it, so make does not take it.
Results
The root entry point exports these types. The Commit namespace exports the same types under the same names, and also the constructors of Folded, Emitted, and Accumulated.
HandlerFailed
interface HandlerFailed {
readonly _tag: "HandlerFailed";
readonly error: unknown;
}
HandlerFailed wraps a handler failure inside a Commit. error holds the error of the handler. The runner unwraps it, so pass, run, and follow fail with the error of the handler itself. HandlerFailed is an Error with a _tag that you can yield*. Commit.HandlerFailed is the same class.
Folded
interface Folded<S> {
readonly _tag: "Folded";
readonly value: S;
readonly encoded: string;
}
declare const Folded: Data.TaggedEnum.ConstructorFrom<Folded<unknown>, "_tag">;
Folded is the result of a fold handler. value is the state after the unit, and encoded is the state as JSON. The constructor Commit.Folded builds one.
Emitted
interface Emitted<O> {
readonly _tag: "Emitted";
readonly outputs: OutputBatch<O>;
}
declare const Emitted: Data.TaggedEnum.ConstructorFrom<
Emitted<Readonly<Record<string, never>>>,
"_tag"
>;
Emitted is the result of an emit handler. outputs holds the items of the unit for each output. The constructor Commit.Emitted builds one.
Accumulated
interface Accumulated<S, O> {
readonly _tag: "Accumulated";
readonly value: S;
readonly encoded: string;
readonly outputs: OutputBatch<O>;
}
declare const Accumulated: Data.TaggedEnum.ConstructorFrom<
Accumulated<unknown, Readonly<Record<string, never>>>,
"_tag"
>;
Accumulated is the result of a mapAccum or mapAccumBatch handler: the state after the unit, as in Folded, and the output items, as in Emitted. The constructor Commit.Accumulated builds one.
OutputBatch
type OutputBatch<O> = {
readonly [K in keyof O]: {
readonly address: StreamAddress.StreamAddress<unknown>;
readonly items: ReadonlyArray<unknown>;
};
};
An OutputBatch holds the items of a unit for each output: the address after at bound the params, and the items in entry order. The items of a State output are changes and State control messages. An output with no items has an empty array.
Services
Transaction
class Transaction extends Context.Service<
Transaction,
{
readonly owner: object;
readonly sql: SqlClient.SqlClient;
readonly withTransaction: <A, E, R>(
body: Effect.Effect<A, E, R>,
) => Effect.Effect<A, E | ProjectionFault, R>;
}
>()("@streamsy/projection/Transaction") {
static readonly layer: Layer.Layer<Transaction, never, CommitBoundary | SqlClient.SqlClient>;
}
Transaction is the service of the SQL transaction that Commit.transaction uses. owner is the storage owner of the transaction, sql is the client that the handler gets, and withTransaction runs an Effect in the transaction.
Transaction.layer builds the service from the CommitBoundary and the SqlClient of a SQL storage host. A failed transaction fails with phase checkpoint and reason storage-failure. A memory host has no SqlClient, so Transaction.layer does not fit it. See Checkpoints and generations.
Units and progress
Reference for the unit, batch, and result types in @streamsy/projection. A unit is one slice of each input that a projection reads, handles, and commits together.
CheckpointStore
Reference for the checkpoint store in @streamsy/projection and @streamsy/projection/checkpoint. The store keeps the offsets and the state of each projection key in one compare-and-set record.