Tables and effects
Write each unit of input to your own SQL tables in the checkpoint transaction, or run external effects at least once.
Projection.process runs an Effect once for each unit. Use it to write rows to your own SQL tables, or to call another system. For the full signature, see Projection.process.
Process
process(batch, unit, context) gets the whole unit and returns an Effect of void or a WakeAt request. batch has one Slice for each input, with its items in stream order. context is an object:
context.preparedis the result ofprepare, orundefinedwhen the declaration has none. See Prepare.context.attemptis the binding run attempt, starting at 1 and increasing after failed attached runs. Directrunandfollowalways supply 1.context.outputsholds the bound addresses of the declared outputs. It exists only when the declaration listsoutputs. See Declared outputs.
Destructure the fields that the handler uses, as in process: (batch, unit, { prepared }) => .... A handler that uses none can take (batch, unit).
To handle one entry at a time, wrap an entry handler in Projection.each. It runs the handler for every entry in order, one after another: all entries of the first declared input, then those of the next. It discards the results.
process has no default Commit. Choose one by what the handler writes:
| Commit | Use it for | Guarantee |
|---|---|---|
Commit.transaction | SQL rows on the same storage as the checkpoint | The rows and the checkpoint commit together |
Commit.atLeastOnce | Calls to other systems | The handler can run more than once for a unit |
Commit.checkpoint and Commit.producer do not fit process, and the compiler rejects them. See Commits.
Timers
Return Projection.wakeAt(time) from a process handler to replace the binding's
one recorded timer. time is epoch milliseconds or an Effect DateTime. It must
convert to a non-negative safe integer; fractional, negative or unsafe times throw
RangeError. Returning nothing clears the timer. Ordinary return values are
discarded as before.
Timers require Commit.atLeastOnce or Commit.transaction, including
Commit.withCompensation around either. The compiler rejects a typed wake request
paired with a custom Commit<void>. Custom timer Commits are not supported in this
release. The context has no timer member.
import { Effect, Option } from "effect";
import { Streamsy } from "@streamsy/core";
import { Commit, Projection } from "@streamsy/projection";
import { Schema } from "effect";
class Deadlines extends Streamsy.json<Deadlines>()("Timers/Deadlines", "deadlines", {
params: {},
schema: Schema.Finite,
}) {}
const deadlines = Projection.process({
id: "deadlines",
params: {},
inputs: { deadlines: Deadlines },
commit: Commit.atLeastOnce,
process: (batch, unit) =>
Effect.gen(function* () {
if (Option.isSome(unit.due)) {
yield* Effect.log(`Deadline ${unit.due.value} is due`);
}
const next = batch.deadlines.items.at(-1);
return next === undefined ? undefined : Projection.wakeAt(next);
}),
});A due unit has unit.due: Some(recordedTime). With no new input, every slice is
empty and unit.ranges is {}. If input arrives after the deadline, one unit
contains both the items and unit.due. Input before the deadline runs normally,
with unit.due: None, and its result replaces or clears the timer. A deadline can
still run after every input has closed. A due unit contributes no accepted row.
Use the recorded unit.due to decide what is due. Reading Clock is useful for a
late policy, but must not change which effects a pinned retry repeats. The unit
key includes the recorded due time, and Commit.atLeastOnce pins that time with
the ranges. Commit.transaction saves the next timer with the handler's SQL and
checkpoint; a failed write rolls them all back.
Projection.run, pass and follow make a due unit only when they run at or after
the time. They do not wait for the deadline. follow can discover it on an input
change or repair cycle. A cycle commits at most one due unit without input, even
when the handler returns a time already passed. Return a time after unit.due
for the next wake. Projection.attach and ProjectionWakeup.attach wake at the recorded deadline.
Projection.attach wakes a sleeping binding at its recorded deadline. Its process
wait re-reads the wall clock at least every 30 seconds while a deadline exists,
so after host suspend a passed wake is discovered within 30 seconds of resume.
ProjectionWakeup.attach exposes that deadline to the Durable Object alarm host,
which arms its alarm for the wake time. Wakes can be late; handlers use unit.due
to apply their late policy. A wake during backoff waits for backoff to end. A
stopped binding keeps its checkpoint timer and fires late after retry or restart.
With durable: false, pending work and timers live in private memory: restart or
eviction loses a timer's wake until the binding's next input. Use durable marks
on persistent storage for restart recovery.
Return a time after unit.due for the next wake. Repeatedly returning a passed
time leaves the mark due immediately and can run the handler at pass speed.
SQL tables
With Commit.transaction, the handler runs inside the SQL transaction that saves the checkpoint. The Commit provides the transaction's SqlClient, so the rows and the new offsets commit together or roll back together.
import { Effect } from "effect";
import { SqlClient } from "effect/sql";
import { Commit, Projection } from "@streamsy/projection";
import { Orders } from "./streams.ts";
const rows = Projection.process({
id: "order-rows",
params: {},
inputs: { orders: Orders },
commit: Commit.transaction,
process: Projection.each(({ item }) =>
Effect.gen(function* () {
const sql = yield* SqlClient.SqlClient;
yield* sql`INSERT INTO order_rows (order_id, amount) VALUES (${item.orderId}, ${item.amount})`;
}),
),
});The application creates and migrates its tables. Streamsy does not create them.
To write rows and also keep a value, use a fold step under Commit.transaction. The step can return an Effect that uses the SqlClient:
import { SqlClient } from "effect/sql";
const orderTotals = Projection.fold({
id: "order-totals",
commit: Commit.transaction,
params: {},
inputs: { input: Orders },
state: { schema: Schema.Finite, initial: 0 },
fold: (sum, { item }) =>
Effect.gen(function* () {
const sql = yield* SqlClient.SqlClient;
yield* sql`INSERT INTO order_totals (order_id, total)
VALUES (${item.orderId}, ${item.amount})
ON CONFLICT (order_id) DO UPDATE SET total = excluded.total`;
return sum + item.amount;
}),
});Commit.transaction needs a SQL host with the CheckpointStore and Transaction services. A memory host has no Transaction. See Layers.
Before the handler runs, the Commit loads the record again inside the transaction. When another writer changed the record after the unit was read, the unit fails with a ProjectionFault with phase checkpoint and reason token-conflict, and nothing commits. A failed handler rolls back its rows.
An append that the handler makes to a stream on the same storage commits with the rows. An append to a stream on other storage fails with phase process and reason unsupported-composition. To write declared output streams in the same transaction, see Transaction outputs.
Declared outputs
Under Commit.transaction, a process declaration can list outputs, as emit does. Each output is a stream declaration or a fixed address, and each declaration must take a subset of params. The handler gets the bound address of each output in context.outputs, and it appends to them itself.
import { Array, Effect } from "effect";
import { Streams, StreamsWriter } from "@streamsy/core/protocol";
import { Commit, Projection } from "@streamsy/projection";
import { Orders, OrderTotals } from "./streams.ts";
const copyTotals = Projection.process({
id: "copy-totals",
params: {},
inputs: { orders: Orders },
outputs: { totals: OrderTotals },
commit: Commit.transaction,
process: (batch, _unit, { outputs }) =>
Effect.gen(function* () {
const rows = Array.map(batch.orders.items, (order) => ({
orderId: order.orderId,
total: order.amount,
}));
if (!Array.isReadonlyArrayNonEmpty(rows)) return;
const { wire } = yield* outputs.totals.stream;
yield* Streams.append(outputs.totals, rows).pipe(Effect.provideService(StreamsWriter, wire));
}),
});- Before
prepareand the handler run, the Commit checks that every output shares the storage of theTransactionservice. An output on other storage fails the unit with phaseloadand reasonunsupported-composition, and the handler does not run. - The appends commit with the checkpoint, or roll back with it.
- The Commit does not create the outputs. Create each output first, or create it in the handler.
preparegets the same addresses in itscontext.outputs.Commit.atLeastOncedoes not provideSqlClient, so the compiler rejectsoutputswith it.
FoldByKey.process builds a handler of this shape, which folds rows of a State stream.
Prepare
prepare(batch, unit) runs before the transaction opens. Use it for work that must not hold the transaction open, such as a slow read from another system. Its result reaches process as context.prepared. When the declaration lists outputs, prepare also gets { outputs } as its third argument.
prepareneedsCommit.transaction, orCommit.transactionwrapped withCommit.withCompensation. With any other Commit,Projection.processthrows aRangeErrorwith the messageprepare requires Commit.transaction.prepareruns at least once for each unit. A retry runs it again. It must not make effects that a second run would repeat.- When
preparefails, the run fails with its error and the transaction does not open. - The runner's caller provides the services that
prepareneeds.
prepare and process see the same unit.key within an attempt. A transaction retry reads the inputs again, and the key changes when new input extends a range. Do not use unit.key as an idempotency key that must hold across retries of Commit.transaction.
External effects
Commit.atLeastOnce saves the input ranges of the unit in the checkpoint record, then runs the handler, then saves the new offsets. When a pass stops between the two saves, the next pass reads the same ranges again and runs the handler again with the same unit.key. Pass unit.key to the other system as an idempotency key, so that it can drop the repeated call.
Commit.atLeastOnce needs only the CheckpointStore service, so it works on a memory host.
When the handler fails, the pass fails, the offsets stay, and the next pass runs the unit again. Commit.atLeastOnce keeps no delivery queue and has no retry policy of its own. Use unit.key as the external call's idempotency key, set the declaration's binding retry policy for a retry budget and onExhausted, and return Projection.wakeAt for a deferred retry or a snooze. See the effects guide for the complete pattern.