Skip to Content
Projections

Operate State tables

Rebuild a State table, fix a keyed fold, and follow the rules that Streamsy does not check for tables and folds.

This page lists the operator steps for State tables and keyed folds, and the rules that keep their rows right. Streamsy checks none of these rules at run time. A mistake gives wrong rows with no error, as a mistake with checkpoints does.

Operator steps

CaseSteps
Rows are lost or suspectStateTable.rebuild(table)
Host start, or a new table over a stream with historyProjection.run(table) before the folds attach. See Start order
A fold has a bug, or a fold was renamedThe fold fix
You want an empty streamStateTable.reset(table), then Projection.forget for every binding of every projection that writes the stream
A checkpoint was moved by hand to skip a bad recordNothing more. The fold skips the record, as intended

rebuild and the start run read only the State stream. A fold fix reads the sources again.

Fix a fold

A fold fix folds every source again from zero into empty rows. It uses the generation of the folds and a reset of the stream. A new generation does not fence the old runners of a Commit.transaction projection, so you stop them first.

  1. Stop the old runners of every projection that writes the stream.
  2. Run StateTable.reset(table). It appends a reset, which clears every row of the stream.
  3. Deploy every projection that writes the stream with a raised generation.
  4. Run each source address once. A source with no pending mark or new source write or trigger request stays unfolded until you run or schedule its binding.

A reset clears the whole stream, so the fix folds again every projection that writes it, not only the one with the bug. Readers see the rows empty and then fill again. Choose a time when that is acceptable.

Streamsy has no list of the source addresses of a projection. Keep your own list, for example from the stream ids in storage or from the checkpoint records of the old generation. After step 4, compare the number of checkpoint records of the new generation with your list.

Remote writers under Commit.producer must stop before the reset too. A reader that does not handle reset must stop while you run the fix.

Wrong orders

The order is the operator's job. A wrong order does not fail. In the Streamsy tests, two sources fold 10 and 20 into one counter, so the right answer is 30:

OrderEffectCounter in the test
The steps aboveThe rows equal those of a fresh fold30
A new generation runs before the resetIt folds from zero over the old rows. Counters double60
An old runner keeps running after the resetIt folds onto empty rows from its old offset. Rows lose history4 in place of 34, after one more record of 4

Fold identity

The id, version and generation of a fold are its identity. A fold with a new identity has no checkpoint, so it folds every source from zero over the rows that the old identity left.

A rename is a fold fix. In the Streamsy tests, a fold renamed over kept rows gave a counter of 60 in place of 30. To rename a fold, or to split one source into two, follow Fix a fold with the new ids in step 3. Pin the ids of your folds in a test, so that an accidental rename fails the build.

Rules that Streamsy does not check

RuleWhat breaks itEffect of the mistake
Treat id, version and generation as the identity of a fold. A rename is a fold fixA refactor renames a fold or splits a sourceThe renamed fold folds history over its old rows. Counters double
Do not forget a fold binding and keep its rowsProjection.forget with no resetThe same doubling. See the forget steps
Append a reset, or a snapshot that omits a fold's rows, only as part of a fold fixAn operator or another writer appends oneThe fold continues on empty rows and loses history
Stop the old runners before a fold fixA deploy that overlaps two processesOld fold output lands on the rows of the fix
One writer per collectionTwo projections, or a projection and an outside writer, write one collectionRows from the two writers interleave
One writer per projection key, the table binding includedA table under both attach and followThe slower writer fails with reason token-conflict

A checkpoint that an operator moves by hand, to skip a bad record, needs no step. The fold skips that record, which is the intent.

Faults and recovery

The table binding, sync and rebuild report storage, history and message failures as ProjectionFault. A storage-failure is retryable unless its underlying stream fault is non-retryable. Missing history (history-unavailable), rejected reads (invalid-source) and invalid stored State messages (invalid-record) are non-retryable. An attached binding backs off alone on a retryable failure and stops on a non-retryable fault. You can also fix the cause and call StateTable.rebuild. See Faults.

A fault in the table also fails every fold that writes the stream with the same fault, because each fold calls sync before it reads rows.

Recovery and health checks need to account for failed marks:

  • Fixing the cause does not resume the binding. Rebuilding the table, a source commit or a trigger row neither shortens backoff nor ends a stop. A retryable failure waits by default from 1 s doubling to 1 hour under the declaration's binding retry policy. A stopped binding waits for registry.retry, registry.schedule or a runner restart. After fixing the cause, request the binding when it must run sooner.
  • Count every failed mark. In either mode, inspect marks.retryDue, marks.backingOff and marks.stopped, and use registry.failed for the causes. A Durable Object's Wakeups.status reports their sum as the ProjectionWakeup deferred count, including stopped bindings. Also inspect the runner's failure and triggers for pass and trigger-stream failures.

SQLite only

The tables need the SQLite family. StateTable.migrate fails on Postgres and MySQL with reason unsupported-composition.

On this page