Skip to Content
Reference@streamsy/projection

Commit

Reference for the Commit namespace and the Transaction service in @streamsy/projection. A Commit decides how a projection writes each unit and its checkpoint, and which guarantee it gives.

import {
  Commit,
  HandlerFailed,
  Transaction,
  type Accumulated,
  type CheckpointRecord,
  type CheckpointStore,
  type Emitted,
  type Folded,
  type Loaded,
  type OutputBatch,
  type ProjectionFault,
  type ProjectionKey,
  type Unit,
} from "@streamsy/projection";
import type { CommitBoundary, StreamAddress } from "@streamsy/core";
import type { Cause, Data, Effect, Layer, Pipeable } from "effect";
import type { SqlClient } from "effect/sql";

A Commit takes the unit that a runner read, runs the handler, and writes the checkpoint. Its result type decides which constructors it fits, and its guarantee says how often the effects of a unit happen. See Commits and Custom Commits.

CommitResultGuaranteeNeeds
Commit.checkpointFoldedExactly once for the saved valueCheckpointStore
Commit.transactionAnyExactly once in the SQL transactionCheckpointStore, Transaction
Commit.producerEmitted, AccumulatedEffectively once for the outputsCheckpointStore
Commit.atLeastOncevoid or WakeAtAt least onceCheckpointStore

Commits

Commit.checkpoint

declare const checkpoint: Commit.Commit<Folded<unknown>, never, CheckpointStore>;

checkpoint runs the handler, then saves the encoded state and the input offsets in one compare-and-set write. It fits fold, and it is the default Commit of fold. When the write fails, the next pass runs the steps again, so keep the steps free of external effects.

Commit.transaction

declare const transaction: Commit.Commit<
  | void
  | WakeAt
  | Folded<unknown>
  | Emitted<Readonly<Record<string, never>>>
  | Accumulated<unknown, Readonly<Record<string, never>>>,
  SqlClient.SqlClient,
  CheckpointStore | Transaction
>;

transaction runs the handler, the output appends, and the checkpoint write (including the returned wake request) in one SQL transaction of the storage owner. It fits every constructor. The handler gets the SqlClient of the transaction, so its SQL commits or rolls back with the checkpoint.

  • Every declared output must be on the storage owner of the transaction. When one is not, the unit fails with phase load and reason unsupported-composition.
  • A stream append in the handler to a stream on another owner fails with phase process and reason unsupported-composition.
  • When the checkpoint changed after the runner read it, the unit fails with phase checkpoint and reason token-conflict.
  • An output append failure has phase process: storage or transport faults use storage-failure, a busy output uses output-busy, and other rejections or encoding failures use invalid-output.
  • Only transaction, alone or wrapped by withCompensation, accepts prepare. It runs prepare before it opens the transaction.
  • A retry reads the inputs again, so a longer tail can change unit.key. An external call in the handler happens at least once.

The Transaction service opens the transaction. A memory host has no SqlClient, so transaction needs a SQL host.

Commit.producer

declare const producer: Commit.Commit<
  Accumulated<unknown, Readonly<Record<string, never>>> | Emitted<Readonly<Record<string, never>>>,
  never,
  CheckpointStore
>;

producer appends the items of each output with a producer position, so a retry does not write a unit twice. It fits emit, mapAccum, and mapAccumBatch. The handler must give the same items for the same unit.

  1. It runs the handler.
  2. When an output has items, it saves the input ranges and the next sequence of each such output in the checkpoint. This write is the pin.
  3. It appends the items of each output with its own producer position. A retry sends the same positions, and the server drops the duplicates.
  4. It saves the state and the new offsets.
Producer fieldValue
producerIdid/v<version>, then / and the canonical params JSON when there are params
producerEpochThe generation
producerSeqThe pinned sequence of that output

Storage refuses a producer id over 255 code points. When the producerId above is longer, producer appends under id/v<version>/# and the SHA-256 hex of the full id instead. When the projection id itself is long, producer cuts the part before # by code point, so the whole id is 255 code points. A retry computes the same id, and a producerId of at most 255 code points is used as it is.

  • producer creates a missing output on the storage owner of the checkpoint. An output on another owner must exist before the first append.
  • A closed output fails the unit with phase pin and reason invalid-output.
  • When the stored epoch of an output is not the generation, or a newer generation has appended to the output, the unit fails with reason stale-epoch.
  • When a pinned unit runs again and an output that was pinned has no items, or an output that was not pinned has items, the unit fails with phase process and reason invalid-output.
  • The appends to several outputs do not commit together.

Commit.atLeastOnce

declare const atLeastOnce: Commit.Commit<void | WakeAt, never, CheckpointStore>;

atLeastOnce saves the input ranges and recorded due time of the unit, runs the handler, and then saves the new offsets and returned wake request. It fits process. When the handler or the last write fails, the next pass runs the same unit again, with the same unit.key. A due unit's key includes its recorded deadline, so different deadlines have different keys. Use unit.key as the idempotency key of an external call. Set the declaration's binding retry policy for a retry budget and onExhausted, and return Projection.wakeAt for a deferred retry or a snooze. See the effects guide for the complete pattern.

Commit.make

declare function make<Result, Provides = never, Requires = never>(
  options: Omit<Commit.Commit<Result, Provides, Requires>, typeof Commit.TypeId | "pipe">,
): Commit.Commit<Result, Provides, Requires>;

Custom Commits do not support timer handlers in this release. A typed WakeAt result does not fit a custom Commit<void>; use Commit.atLeastOnce or Commit.transaction, including compensation wrappers, for timers.

make returns a custom Commit from its name, guarantee, load, apply, and optional remove. make infers the result and the required services from apply and load. When the Commit provides handler services, write the Result, Provides, and Requires type arguments.

A custom Commit must follow these rules:

  1. Never store next when run fails.
  2. Make the write conditional on loaded.token. When the token moved, fail with phase checkpoint and reason token-conflict.
  3. Return the token "0" from load when no record exists.
  4. Return the stored state from load, so that Projection.read reads it.
  5. Declare the guarantee of the actual write. When run and the write are not atomic, declare at-least-once.

A custom Commit reads the input range again on a retry, so a retry can see a longer tail. It does not accept prepare, and it must append the outputs of an Emitted or Accumulated result itself.

Commit.withCompensation

declare const withCompensation: {
  <Result, R>(
    f: (
      result: Result,
      cause: Cause.Cause<ProjectionFault | HandlerFailed>,
      unit: Unit,
    ) => Effect.Effect<void, never, R>,
  ): <P, Q>(self: Commit.Commit<Result, P, Q>) => Commit.Commit<Result, P, Q | R>;
  <Result, P, Q, R>(
    self: Commit.Commit<Result, P, Q>,
    f: (
      result: Result,
      cause: Cause.Cause<ProjectionFault | HandlerFailed>,
      unit: Unit,
    ) => Effect.Effect<void, never, R>,
  ): Commit.Commit<Result, P, Q | R>;
};

withCompensation wraps a Commit. When the handler succeeded and the Commit then failed, the wrapped Commit calls f with the handler result, the failure, and the unit, and then fails with the same failure. Call it directly or through .pipe. The wrapped Commit keeps the pin and the prepare support of self.

f cannot fail. Compensation does not make an external effect atomic.

The Commit type

The names in these signatures are members of the Commit namespace.

Commit.Commit

interface Commit<in Result, in Provides = never, out Requires = never> extends Pipeable.Pipeable {
  readonly [TypeId]: Variance<Result, Provides, Requires>;
  readonly name: string;
  readonly guarantee: "exactly-once" | "effectively-once" | "at-least-once";
  readonly load: (key: ProjectionKey) => Effect.Effect<Loaded, ProjectionFault, Requires>;
  readonly apply: (
    pending: Pending<Result, Provides>,
  ) => Effect.Effect<void, ProjectionFault | HandlerFailed, Requires>;
  readonly remove?: (key: ProjectionKey) => Effect.Effect<void, ProjectionFault, Requires>;
}

A Commit writes the units of a projection. Result is the handler result that it takes, Provides holds the services that it gives the handler, and Requires holds the services that it needs.

FieldMeaning
nameA name for messages
guaranteeHow often the effects of a unit happen
loadThe function that loads the checkpoint record of a key, and its token
applyThe function that runs the handler of a unit and writes the result and the checkpoint
removeThe optional function that deletes the checkpoint record of a key. Projection.forget calls it

A Commit with a result type that does not fit a constructor fails at the commit: field of the definition.

Commit.Pending

interface Pending<Result, Provides = never> {
  readonly key: ProjectionKey;
  readonly unit: Unit;
  readonly loaded: Loaded;
  readonly next: CheckpointRecord;
  readonly run: Effect.Effect<Result, HandlerFailed, Provides>;
}

apply takes a Pending: a unit that is ready to commit.

FieldMeaning
keyThe checkpoint key
unitThe unit, with the encoded params
loadedThe record and the token that the runner loaded before it read the unit
nextThe record to save after the unit: the new offsets, and the state from before the unit
runThe Effect that runs the handler. It fails with HandlerFailed, and the runner unwraps the handler error from it

To save the state, put the encoded field of a Folded or Accumulated result into next as state.

Commit.Variance

interface Variance<in Result, in Provides, out Requires> {
  readonly result: (_: Result) => void;
  readonly provides: (_: Provides) => void;
  readonly requires: () => Requires;
}

Variance holds the type parameters of a Commit under TypeId. It has no run-time use.

Commit.TypeId

declare const TypeId: "~@streamsy/projection/Commit";

TypeId is the key of the Variance field of every Commit. make adds it, so make does not take it.

Results

The root entry point exports these types. The Commit namespace exports the same types under the same names, and also the constructors of Folded, Emitted, and Accumulated.

HandlerFailed

interface HandlerFailed {
  readonly _tag: "HandlerFailed";
  readonly error: unknown;
}

HandlerFailed wraps a handler failure inside a Commit. error holds the error of the handler. The runner unwraps it, so pass, run, and follow fail with the error of the handler itself. HandlerFailed is an Error with a _tag that you can yield*. Commit.HandlerFailed is the same class.

Folded

interface Folded<S> {
  readonly _tag: "Folded";
  readonly value: S;
  readonly encoded: string;
}

declare const Folded: Data.TaggedEnum.ConstructorFrom<Folded<unknown>, "_tag">;

Folded is the result of a fold handler. value is the state after the unit, and encoded is the state as JSON. The constructor Commit.Folded builds one.

Emitted

interface Emitted<O> {
  readonly _tag: "Emitted";
  readonly outputs: OutputBatch<O>;
}

declare const Emitted: Data.TaggedEnum.ConstructorFrom<
  Emitted<Readonly<Record<string, never>>>,
  "_tag"
>;

Emitted is the result of an emit handler. outputs holds the items of the unit for each output. The constructor Commit.Emitted builds one.

Accumulated

interface Accumulated<S, O> {
  readonly _tag: "Accumulated";
  readonly value: S;
  readonly encoded: string;
  readonly outputs: OutputBatch<O>;
}

declare const Accumulated: Data.TaggedEnum.ConstructorFrom<
  Accumulated<unknown, Readonly<Record<string, never>>>,
  "_tag"
>;

Accumulated is the result of a mapAccum or mapAccumBatch handler: the state after the unit, as in Folded, and the output items, as in Emitted. The constructor Commit.Accumulated builds one.

OutputBatch

type OutputBatch<O> = {
  readonly [K in keyof O]: {
    readonly address: StreamAddress.StreamAddress<unknown>;
    readonly items: ReadonlyArray<unknown>;
  };
};

An OutputBatch holds the items of a unit for each output: the address after at bound the params, and the items in entry order. The items of a State output are changes and State control messages. An output with no items has an empty array.

Services

Transaction

class Transaction extends Context.Service<
  Transaction,
  {
    readonly owner: object;
    readonly sql: SqlClient.SqlClient;
    readonly withTransaction: <A, E, R>(
      body: Effect.Effect<A, E, R>,
    ) => Effect.Effect<A, E | ProjectionFault, R>;
  }
>()("@streamsy/projection/Transaction") {
  static readonly layer: Layer.Layer<Transaction, never, CommitBoundary | SqlClient.SqlClient>;
}

Transaction is the service of the SQL transaction that Commit.transaction uses. owner is the storage owner of the transaction, sql is the client that the handler gets, and withTransaction runs an Effect in the transaction.

Transaction.layer builds the service from the CommitBoundary and the SqlClient of a SQL storage host. A failed transaction fails with phase checkpoint and reason storage-failure. A memory host has no SqlClient, so Transaction.layer does not fit it. See Checkpoints and generations.

On this page