Operate State tables
Rebuild a State table, fix a keyed fold, and follow the rules that Streamsy does not check for tables and folds.
This page lists the operator steps for State tables and keyed folds, and the rules that keep their rows right. Streamsy checks none of these rules at run time. A mistake gives wrong rows with no error, as a mistake with checkpoints does.
Operator steps
| Case | Steps |
|---|---|
| Rows are lost or suspect | StateTable.rebuild(table) |
| Host start, or a new table over a stream with history | Projection.run(table) before the folds attach. See Start order |
| A fold has a bug, or a fold was renamed | The fold fix |
| You want an empty stream | StateTable.reset(table), then Projection.forget for every binding of every projection that writes the stream |
| A checkpoint was moved by hand to skip a bad record | Nothing more. The fold skips the record, as intended |
rebuild and the start run read only the State stream. A fold fix reads the sources again.
Fix a fold
A fold fix folds every source again from zero into empty rows. It uses the generation of the folds and a reset of the stream. A new generation does not fence the old runners of a Commit.transaction projection, so you stop them first.
- Stop the old runners of every projection that writes the stream.
- Run
StateTable.reset(table). It appends areset, which clears every row of the stream. - Deploy every projection that writes the stream with a raised
generation. - Run each source address once. A source with no pending mark or new source write or trigger request stays unfolded until you run or schedule its binding.
A reset clears the whole stream, so the fix folds again every projection that writes it, not only the one with the bug. Readers see the rows empty and then fill again. Choose a time when that is acceptable.
Streamsy has no list of the source addresses of a projection. Keep your own list, for example from the stream ids in storage or from the checkpoint records of the old generation. After step 4, compare the number of checkpoint records of the new generation with your list.
Remote writers under Commit.producer must stop before the reset too. A reader that does not handle reset must stop while you run the fix.
Wrong orders
The order is the operator's job. A wrong order does not fail. In the Streamsy tests, two sources fold 10 and 20 into one counter, so the right answer is 30:
| Order | Effect | Counter in the test |
|---|---|---|
| The steps above | The rows equal those of a fresh fold | 30 |
A new generation runs before the reset | It folds from zero over the old rows. Counters double | 60 |
An old runner keeps running after the reset | It folds onto empty rows from its old offset. Rows lose history | 4 in place of 34, after one more record of 4 |
Fold identity
The id, version and generation of a fold are its identity. A fold with a new identity has no checkpoint, so it folds every source from zero over the rows that the old identity left.
A rename is a fold fix. In the Streamsy tests, a fold renamed over kept rows gave a counter of 60 in place of 30. To rename a fold, or to split one source into two, follow Fix a fold with the new ids in step 3. Pin the ids of your folds in a test, so that an accidental rename fails the build.
Rules that Streamsy does not check
| Rule | What breaks it | Effect of the mistake |
|---|---|---|
Treat id, version and generation as the identity of a fold. A rename is a fold fix | A refactor renames a fold or splits a source | The renamed fold folds history over its old rows. Counters double |
Do not forget a fold binding and keep its rows | Projection.forget with no reset | The same doubling. See the forget steps |
Append a reset, or a snapshot that omits a fold's rows, only as part of a fold fix | An operator or another writer appends one | The fold continues on empty rows and loses history |
| Stop the old runners before a fold fix | A deploy that overlaps two processes | Old fold output lands on the rows of the fix |
| One writer per collection | Two projections, or a projection and an outside writer, write one collection | Rows from the two writers interleave |
| One writer per projection key, the table binding included | A table under both attach and follow | The slower writer fails with reason token-conflict |
A checkpoint that an operator moves by hand, to skip a bad record, needs no step. The fold skips that record, which is the intent.
Faults and recovery
The table binding, sync and rebuild report storage, history and message failures as ProjectionFault. A storage-failure is retryable unless its underlying stream fault is non-retryable. Missing history (history-unavailable), rejected reads (invalid-source) and invalid stored State messages (invalid-record) are non-retryable. An attached binding backs off alone on a retryable failure and stops on a non-retryable fault. You can also fix the cause and call StateTable.rebuild. See Faults.
A fault in the table also fails every fold that writes the stream with the same fault, because each fold calls sync before it reads rows.
Recovery and health checks need to account for failed marks:
- Fixing the cause does not resume the binding. Rebuilding the table, a source commit or a trigger row neither shortens backoff nor ends a stop. A retryable failure waits by default from 1 s doubling to 1 hour under the declaration's binding retry policy. A stopped binding waits for
registry.retry,registry.scheduleor a runner restart. After fixing the cause, request the binding when it must run sooner. - Count every failed mark. In either mode, inspect
marks.retryDue,marks.backingOffandmarks.stopped, and useregistry.failedfor the causes. A Durable Object'sWakeups.statusreports their sum as theProjectionWakeupdeferredcount, including stopped bindings. Also inspect the runner'sfailureandtriggersfor pass and trigger-stream failures.
SQLite only
The tables need the SQLite family. StateTable.migrate fails on Postgres and MySQL with reason unsupported-composition.