Engine Database Schema¶
Vayu uses SQLite via sqlite_orm. The schema is defined in engine/src/db/database_impl.hpp and the
struct definitions live in engine/include/vayu/types.hpp. sync_schema() adds new columns
automatically on startup - no migration scripts are needed for additive changes.
Database's own code is nine translation units under engine/src/db/ (issue #1614), one per table
family, over that shared internal header: database.cpp (construction, locking, transactions),
db_collections.cpp, db_requests.cpp, db_environments.cpp, db_runs.cpp (runs and every
run-scoped artifact: metric ticks, monitor samples, results, the webhook inbox), db_specs.cpp,
db_credentials.cpp, db_config.cpp and db_maintenance.cpp (schema migration, corruption
recovery, init()'s startup repair passes, workspace backup). database.hpp's section comments
name which file defines each member.
Breaking changes: a destructive schema change is now a versioned migration, not a wipe.
PRAGMA user_versionis the marker:Database::Database'smigrate_before_sync(issue #1514) reads it beforesync_schema ()ever runs, and a version behind the engine's own gets whatever data-preserving pass that version's cut-over needs, run against the raw file with its ownsqlite3connection - ahead ofsync_schema (), because that call is what wouldDROP COLUMNa retiring column before a later pass could read it. A version newer than the engine's is refused outright, naming both versions, rather than opened and possibly written to with settings this build does not understand. The script-to-elements migration below is the first of these; the same mechanism serves the next destructive change, not a new one invented per issue. An additive change - a new column, or existing data reshaped into a new one alongside the old - still gets a startup repair pass instead, the shape the header-strip pass uses.
Startup validation, backup and recovery¶
Database::Database (engine/src/db/database.cpp) reads PRAGMA user_version before anything
else - a version newer than this engine's throws immediately and the constructor fails outright
(see the script-to-elements migration); that is
not corruption, and none of the recovery below applies to it. Past that gate it probes the file -
opens it and runs sync_schema() before anything else uses the database - and sorts the answer
three ways (issue #2420):
- Valid: the probe finished.
- Corrupt: a file with no SQLite header (checked without opening it, see below), or a probe
that failed with
SQLITE_CORRUPT(11) orSQLITE_NOTADB(26). Only this answer reaches the recovery branch. - Unavailable: any other failure -
SQLITE_BUSYorSQLITE_LOCKED(another connection holds the file),SQLITE_FULL,SQLITE_READONLY,SQLITE_IOERR,SQLITE_CANTOPEN, a filesystem error, a migration step that failed. None of these says anything about the file, so none of them may quarantine it.
Every connection the constructor opens - the migration's, the probe's, the one it keeps - waits
BUSY_TIMEOUT_MS (10 s) for another connection's lock before its first statement, the same
wait init() applies from dbBusyTimeout afterwards, so a lock that is let go within that
window costs a slower start rather than a refusal. What happens next:
| Outcome | What happens to the files | What is recorded |
|---|---|---|
A newer user_version than this engine understands |
Nothing - the constructor throws before the file is touched, and the daemon does not start | (the exception, naming both versions) |
An older user_version |
Before anything raises the version, a copy is saved to backups/vayu-<stamp>-pre-v<N>.db (the copy before a migration). If it cannot be written, nothing is touched and the daemon does not start |
(the exception, naming the path and why the copy failed) |
| Cannot be examined (locked, disk full, read-only, I/O error) | Nothing is touched, the daemon does not start, the reason is named | (the exception, naming the path, the SQLite code and message, and what to do) |
| Opens cleanly | Once the engine is listening, <db>.bak is replaced by a VACUUM INTO snapshot of it, validated by the same probe, flushed to disk and renamed into place - so the backup is only ever taken from a database that validated, and is never a partial copy. A clean shutdown after a session that wrote anything takes the same snapshot again (#2421), so the backup holds that session's work |
Nothing |
Corrupt, <db>.bak passes the same validation |
The corrupt file set is quarantined and the backup copied back over it | restored_from_backup |
Corrupt, <db>.bak is corrupt too |
The backup is left untouched as evidence, the corrupt file set is quarantined, and a fresh empty database is created | backup_also_corrupt |
Corrupt, <db>.bak cannot be examined |
Nothing is touched and the daemon does not start, naming the backup: whether it would restore is unknown | (the exception, naming the backup) |
| Corrupt, there is no backup | The corrupt file set is quarantined and a fresh empty database is created | started_fresh_quarantined |
| Corrupt, and the quarantine rename fails too | <db>, <db>-wal and <db>-shm are deleted so the daemon can still start |
deleted_corrupt |
The refusal reaches the daemon's main, which prints it on stderr and exits 1 - the same path the
newer-schema refusal takes. A snapshot the backup refresh cannot examine is discarded with a reason
that says so (the snapshot could not be examined: ...), and the previous <db>.bak stands, as it
does for one that did not validate.
The shutdown refresh (Database::refresh_recovery_backup_on_shutdown) runs after the run drain,
while the engine still holds its lock. It is skipped when sqlite3_total_changes64 on the shared
connection has not moved since init() (a write through any other connection is not counted),
checkpoints the WAL (TRUNCATE, a reader another process holds tolerated), and is bounded by the
smaller of SHUTDOWN_BACKUP_DEADLINE_MS (2 s) and what is left of the app's 5 s shutdown window:
the copy is interrupted at the deadline and one that finishes after it is discarded unvalidated.
Validation, the flush and the rename that follow an in-time copy run to completion. Every way it
stops leaves the previous <db>.bak and logs a db warning saying so.
Starting fresh is deliberate - the alternative is a daemon that fails to start on every launch forever - but it is total: collections, requests, environments, saved examples, spec documents and run history are all gone, and the app comes up looking like a fresh install.
<db>.bakis not a user backup. It is rewritten once the engine listens and again at a clean shutdown, so it exists to give a corrupt file something to restore from, not to let anyone go back to last week. Work written after the last refresh (a session that ended in a crash or a power cut rather than a clean shutdown) is not in it - and the row above where it is the only copy left is the one that ends in an empty workspace. The snapshot a user owns isPOST /workspace/backup/vayu-cli backup, described in architecture.md.The copy taken before a rewrite is in
backups/, not beside the database.<db>.bakis refreshed, once the engine listens, from a file the migration or repair has already rewritten, so it cannot be the way back. The copy before a migration is.
The copy before a migration (issue #2426)¶
Before migrate_before_sync raises user_version - on the stamp-only path as
well as before the transaction of a step that rewrites rows - the constructor's
call writes a VACUUM INTO snapshot of the file to
backups/vayu-<stamp>-pre-v<N>.db beside it, where N is the version the file
had. It is the copy an engine at version N opens, and the only one: the
<db>.bak refresh that follows replaces the backup from the migrated file.
- No copy, no migration. When the snapshot cannot be written (the folder
cannot be created, the disk is full) the constructor throws
Vayu could not migrate the database at <path> to schema version <V>: no copy of it could be saved first (...), the daemon does not start, and the file is left byte for byte as it was. A file that holds no table yet is not copied. - The same retention as a workspace backup. It is pruned with
maxBackupsRetained(default 5,0= unlimited), counted together with the snapshotsPOST /workspace/backupwrites, the setting read from the file before anyDatabaseexists. It is the newest file when it is written, so the start that writes it never prunes it; five later backups do. Exempting it from the count was weighed and declined: a full copy of every stored credential that nothing ever removes is what this replaced. - Only the constructor's own call takes it.
probe_databaseruns the same migration on a candidate it is about to judge (a.bakrecovery may restore, the refresh's own snapshot), which is not the workspace a person goes back to, and takes none. - The header-strip pass takes one too, labelled
pre-repair(below).
Engines before this wrote <db>.pre-migration.bak and <db>.pre-upgrade.bak
(with their -wal and -shm) beside the database and never removed them. They
are removed on a later start than the one that migrated: one that validated,
found the file already at SCHEMA_VERSION before writing anything, and has
refreshed <db>.bak. A file modified since that start began is left alone.
The quarantine (issue #984)¶
A database that is corrupt is moved, not deleted: <db> becomes
<db>.corrupt-<epoch-ms> with its -wal and -shm sidecars renamed alongside.
SQLite's own tooling can usually pull most rows back out of a damaged file, and
only while the file exists:
Two rules bound it:
- At most two sets are kept. These are full-size copies written unattended into the user's data directory; the newest two survive each recovery and older ones are pruned. Two, not one, because the set before the current one is what says this has happened before.
- The backup is validated before it is trusted. It gets the same open plus
sync_schema()probe the main file gets - which also migrates a backup taken by an older build before it is committed to. A corrupt backup is left exactly where it is rather than written over the original, which is whatbackup_also_corruptrecords; one that cannot be examined stops the start before the original is moved.
A quarantined set is the whole file as it was, so when the file was in fact healthy, putting it back by hand undoes the recovery: see architecture.md.
One implementation detail worth knowing before changing the order of that code:
the validation probe cannot be the first thing to touch the file. SQLite
deletes the -wal and -shm when it opens a file whose header is not a
database's, and a -wal holds committed transactions the main file does not, so
the sidecars would be gone before the quarantine could move them. The
constructor reads the 16-byte SQLite header itself first and only opens a file
that could plausibly be a database.
The recovery marker (issue #922)¶
The record of that cannot live in the database, because the database is the
thing that just went away, so it is a small JSON file beside it, <db>.recovery,
written by engine/src/db/recovery.cpp:
{
"outcome": "started_fresh_quarantined",
"at": 1755870000000,
"quarantinedPath": "/home/someone/.local/share/vayu/vayu.db.corrupt-1755870000000"
}
at is epoch milliseconds, as every other engine timestamp is.
quarantinedPath is present only when a set was moved aside - absent for
deleted_corrupt and for any marker written before #984 - so a reader tests for
it rather than deriving it from outcome. Two rules the readers depend on:
- A clean start writes no marker. The file's presence alone is what
distinguishes a wiped database from a genuine first run, which produces an
empty database too. There is deliberately no
okoutcome. - A marker is never cleared by the engine. The engine can be restarted, or
crash, before any client has polled it; clearing it on the next clean start
would lose exactly the case the file exists for. Showing a notice about it
once is the reader's job - the app remembers the
atit has acknowledged (app/src/stores/recovery-notice-store.ts).
Database reads the marker once at construction and exposes it as
Database::recovery(), so the polled GET /health costs no file access. That
endpoint reports it as a recovery node, absent on a clean start - see
api-reference.md - and the app renders it as a
dismissible banner naming the database path and, when there is one, the
quarantined file, how to copy it back to undo the recovery and the .recover
command that reads it.
Reclaiming freed pages¶
Deleting rows - a prune, a trash purge, the metrics drop - moves their pages to SQLite's
freelist, where the engine reuses them and the filesystem never sees them again, so a workspace
that once held weeks of heavy runs keeps that high-water mark forever. Database::init() ends
with a guarded VACUUM that gives them back (issue #990).
Guarded, because a VACUUM rewrites the whole database: it runs only when both thresholds are
crossed - at least 25% of the file's pages are free and they hold at least 10 MiB
(VACUUM_MIN_FREELIST_PERCENT / VACUUM_MIN_RECLAIMABLE_BYTES, compile-time, deliberately not
configurable). The fraction alone would rewrite a nearly-empty workspace on every start; the byte
floor alone would rewrite four gigabytes to win back eleven megabytes. Neither threshold crossed
means the pass reads three PRAGMAs and returns.
Three things worth knowing about where it sits:
- Last in
init(), after the run prune and the trash purge, so it sees the pages every sweep freed rather than only the first one's. - On the storage's own held connection (
open_forever), under the DB mutex, never on a second one. The truncating checkpoint below needs the file unmapped, because Windows refuses to truncate a file any connection has mapped (mmap_size) - with a mapping the rewrite committed and the file stayed its old size. No connection maps it (Connection settings). (POST /workspace/backup'sVACUUM INTOruns on its own separate connection, since it writes a different file, and so does the run-row reader, which maps nothing and opens only afterinit.) Nothing contends for it:initruns before the HTTP listener starts, and the instance lock has already refused a second engine. - The
VACUUMis followed by awal_checkpoint(TRUNCATE). Under WAL the rewritten image lands in the-walfile and the database is not resized until a checkpoint copies it back, so without it the pass would free every page it set out to and leave the file exactly as large as it found it.
The outcome is logged at info level (Reclaimed N KB of freed database pages), because the cost
is paid on the startup path - db.init() returns before /health answers, and the app's readiness
poll budgets 45 seconds against it before handing the engine to the renderer's own health poll. A
rewrite slow enough to matter is one this line explains.
Best-effort like the sweeps before it: a failure is a warning, never a daemon that will not start.
auto_vacuum was rejected as the alternative - switching an existing database's mode requires a
full VACUUM anyway, and it changes page bookkeeping for every write thereafter.
The header-strip pass (issues #1487, #1491)¶
Database::strip_stored_managed_headers(), called from init(), disables the X-Vayu-Version,
X-Request-ID and User-Agent rows a pre-#1229 renderer wrote into a saved request's headers
column on the app's own behalf - default headers are applied per transfer now and never stored.
Like the passes above it runs before /health can answer, but it is scoped so a workspace with
nothing left to touch pays close to nothing for it:
- A SQL
LIKEoverheadersfirst, on undeleted rows only. Only rows whose header text could contain one of the three names, and whosedeleted_atisNULL, are loaded into aRequestand JSON-parsed;LIKEis ASCII case-insensitive by default, the same fold the exact predicate applies to a matched key. A workspace already handled costs one scan of the column, not a materialisation of every request's body and script. A request in the trash is left byte-identical - see below. - A row is matched by value shape, not by name or header family alone (issue #1491): a
X-Vayu-Versiononly when its value is a plainMAJOR.MINOR.PATCHversion, anX-Request-IDonly when its value is the exact lowercase RFC 4122 v4 UUIDgenerateUUID()produced, and aUser-Agentonly when its value isVayu/followed by that same plain version - never a name or family match broad enough to catch a value the user typed on purpose. - A match is disabled and marked, never deleted (issue #1491): the row's
enabledis set tofalse(so it is never sent - the wire is exactly as clean as deleting it would have been) and itssourceis set to"legacy-default", the app-readable markerKeyValueEntry.sourcealready carries for an app-written row (issue #1481). The row survives for a user to find in the editor and re-enable, in case a match turns out to be wrong. - One transaction for the rewrite, not one implicit commit per candidate row - the same shape
seed_default_configuses. - A one-time marker. Once a pass finds nothing left to change, it records a
managedHeadersStrippedconfig entry (advanced, no everyday user story) and every later start skips the scan outright rather than re-running it to confirm nothing changed. #1492 will give this kind of migration bookkeeping a proper home; until then a config entry is where this file's other internal-only flags already live. - A
backups/vayu-<stamp>-pre-repair.dbsnapshot is written immediately before the first row this pass ever rewrites, under the same retention as the copy before a migration. When it cannot be written the pass rewrites nothing and leaves its marker unset, so the next start tries again. - Trash is caught up on restore, not on the startup scan. A request already in the trash when
this pass runs is skipped, so it stays byte-identical for as long as it sits there; restoring it
(
restore_request_locked/restore_collection_locked) runs the same disable-and-mark step on its way back, which is a no-op if the pass already handled it before the request was deleted.
The empty-parent pass (issue #3451)¶
Database::clear_empty_parent_ids(), called from init(), rewrites every collection stored with
parent_id = '' to NULL, trashed rows included. Until #3451, POST/PUT /collections stored a
parentId: "" as written: a second spelling of the tree root that the change feed and
get_requests_in_subtree read as root while the append scan, the root normalize of
POST /reorder and a restore did not, so such a row tied with the first root, and a delete and
restore of it came back with reparentedToRoot: true, as if its parent were gone. The write
paths now store "" as NULL (read_parent_id_field, see
Accepted field shapes); this pass is the rows written
before.
- No done marker and no schema bump. Both statements, the count and the
UPDATE, are served byidx_collections_parent_id, so a workspace with nothing to rewrite pays one index probe per start, not the scan the header-strip pass marks itself done to avoid. Running every start also catches a''an older engine writes after a downgrade. An older engine readsNULLas the root, as it always did, so the rewrite is nothing it must be stopped from reading past, which is what a bump is for (Schema versions). - One statement, so one commit.
updated_atis left alone: the row meant the root all along. - A
backups/vayu-<stamp>-pre-repair.dbsnapshot is written immediately before the first rewrite, as the header-strip pass does. When it cannot be written the pass rewrites nothing, logs a warning, and the next start tries again.
The script-to-elements migration (issue #1514)¶
Issue #1513 landed elements additively, beside the two script columns, with a startup repair pass
(fold_scripts_into_elements) that gave a 0.26-shaped row an elements entry without touching
pre_request_script / post_request_script - nothing ran an element yet, so the columns had to
keep driving execution. Issue #1514 lands the pipeline that runs one, which is what makes the two
columns dead data and the cut a real migration rather than another additive pass. The repair pass,
its scriptsFoldedIntoElements marker and its .pre-elements-fold.bak snapshot are gone; PRAGMA
user_version is the marker now.
migrate_before_sync (path, busy_timeout) (engine/src/db/db_maintenance.cpp, declared in database_impl.hpp
so Database::Database's constructor - engine/src/db/database.cpp - can call it before any
sync_schema (), including the constructor's own validation probe) opens path with a raw
sqlite3 connection, independent of sqlite_orm:
- Read
PRAGMA user_version. Newer than this engine'sSCHEMA_VERSION(currently10, see Schema versions) throwsstd::runtime_errornaming both versions - not inside the constructor's probe/recovery try-catch, so the exception reaches the daemon's own startup failure path, naming both versions, rather than surfacing from the probe as a generic refusal to examine the file. Equal toSCHEMA_VERSIONis a fast no-op. Anything older is copied first (the copy before a migration).0(every database written before this issue, folded by #1513's pass or not) proceeds to fold.1and2have no script column left, so they skip the fold and are only stamped. - Fold, if either table still has a script column. For each of
requestsandcollectionsthat does (eitherpre_request_scriptorpost_request_scriptalone is enough - a table needs folding for), in one transaction, per table: read its actual column set fresh (PRAGMA table_info),ALTER TABLE ... ADD COLUMN elements TEXT NOT NULL DEFAULT '[]'when the column is not there yet - a genuinely pre-cutover database (older than #1513) never ran the additive pass that would have added it, so this migration is the first thing that does - and select''for whichever script column the table does not carry, rather than aSELECTthat names a column absent from this row's table and fails to prepare. Then, per row: parseelements, append a{"id": "el_...", "kind": "script.pre" | "script.post", "enabled": true, "config": {"script": "..."}}for each non-blank script column not already represented inelements(the check that makes a database #1513's additive pass already folded safe to re-run through this migration without a duplicate), and write the row back. A database neither table needs folding for (a fresh install, or one some other path already brought to this shape) skips the transaction. A fold failure rolls the transaction back and throws, naming the SQLite message that caused it. A refusedBEGIN IMMEDIATE(another writer holds the lock), a refused stamp and a failedCOMMITare treated the same way: each throws that message through the sameexec_or_throwhelper (a failedCOMMITis rolled back first), and the stamp-only path throws on a failed stamp too. The file is left as it was, scripts still in their columns anduser_versionunmoved, so the next start runs the migration again. The constructor's first call is unguarded and the probe's call classifies the throw as unavailable, never corrupt, so neither path quarantines the file. - Stamp
user_version = SCHEMA_VERSION. Both writes (the no-fold path and the fold's commit) go through one helper,stamp_schema_version, so a bump of the constant is a bump of every stamp. Only after this returns doessync_schema ()see a mapping with nopre_request_script/post_request_scriptcolumns andALTER TABLE ... DROP COLUMNthem - the ordering this migration exists to guarantee, perengine/CLAUDE.md's "Removing a column" rule.
GET /elements/kinds's script.pre / script.post entries gained apply in the same issue, so a
row this migration folds is not just storage-compatible but immediately runnable.
The file-trust restatement (schema version 4)¶
Version 4 stated an explicit unresolved on every stored file reference (absent written as
false: the editor was the one writer that left the key out) and removed the allowed folders the
root rule refuses. The restatement is superseded by
the file-grants migration, which now runs in its
place for every older database and reads an absent key the way version 4 did when the database
predates it. The folder removal still runs, in the same step:
- Removes the allowed folders the root rule refuses. A
file_rootsrow whose pathrefused_root_reasonrejects - a filesystem or drive root, the home folder, or a folder containing it - is deleted and logged at warning level with the reason, so a user can see why Settings > Files no longer lists it.FileAccessPolicy::from_databasetherefore reads only rows the route would accept, and carries no second check.
The file-grants migration (schema version 6)¶
From version 6 the engine derives a file reference's trust from the allowed folders and the
picked files (file_grants) and never reads it from a body (#2356). Stored bodies
still carry the old unresolved key, so the same migrate_before_sync transaction, for any
database below version 6, after the fold:
- Grants every file a person chose. A file reference stored as chosen -
unresolved: false, or, on a database below version4, no key at all - whosesrcis an absolute path holding no{{and still resolves to a regular file becomes onefile_grantsrow, its canonical path, deduplicated (vayu::http::grantable_file_path). A path that is gone, a templated one,true, a non-boolean, and (from version4on) an absent key grant nothing. A chosen concrete path that is gone (deleted, or on a drive or share not mounted at upgrade time) is the one of those a person loses, so each is logged at warning level once the step has run, with itstable, rowidandpath, followed by one warning carrying thecount; this happens on the stamp-only path too, where a pre-version-4body without the key leaves nothing to rewrite. The step runs once, so such a file reads as not chosen even after the drive returns, until it is picked again. - Removes the key from every stored body. The two places a body
FilePlansends is stored:requests.body(trash included) and the top-levelbodyofruns.config_snapshot(the payload a history entry is re-opened from), a binaryfileand everytype: "file"form part. Only values whose text contains"file"are parsed; one that is not JSON is left byte-identical and logged at warning level, never dropped.request_examples.bodyis a response's text andinbox_requests.bodythe bytes a listener received: neither is ever parsed as a request body, so neither is visited (both are stored and read back byte-exact, a zero byte included, seerequest_examplesandinbox_requests).collectionsand the other tables store no request body. - Removes the allowed folders the root rule refuses, as version
4did.
file_grants is created by this step (CREATE TABLE IF NOT EXISTS, FILE_GRANTS_DDL in
db_maintenance.cpp) because it runs before sync_schema (); the statement is exactly what the
mapping generates, or sync_schema () would rebuild the table empty. All of it is planned before
anything is written, so a database with nothing to change is only stamped; either way the
copy before a migration is written first. A second run finds nothing to change
and grants nothing twice. A database already at 6 does not run it on its way to a later
version (plan_file_trust_migration plans nothing from 6 on), so a root rule that has since
moved never removes a folder allowed under the rule it was written with. The cost: a body a
non-editor caller managed to store as unresolved: false before the upgrade is granted once,
like a pick.
The sibling-order migration (schema version 7)¶
From version 7 a collection's sub-folders and its requests share one order space (#2535):
siblings sort by (order, kind, createdAt, id) with a sub-folder before a request on a tie, so
an imported Postman collection can keep a request above or between its folders. Before it, each
kind was numbered from 0 and displayed as two blocks, folders first; read with the merged
comparator, those numbers would interleave (f0 r0 f1 r1). So the same migrate_before_sync
transaction, for any database below version 7, after the file-grants step:
- Moves every request past its collection's sub-folders. For each collection with at least
one sub-folder, every request whose
collection_idit is gains one more than the highestorderamong the collections whoseparent_idit is - trashed folders included, so a folder restored from the trash cannot land on a request - and, when the collection's lowest requestorderis negative, that much again, so every request sorts strictly after every folder. Trashed requests move with the rest. Relative order within each kind is unchanged, so the merged comparator lists each parent exactly as the two blocks did. A collection with no sub-folder, trashed or live, is untouched.
plan_sibling_order_shift (db_maintenance.cpp) reads one shift per collection before anything
is written, so a database with no request beside a sub-folder is only stamped; either way the
copy before a migration is written first. A plan read SQLite refuses throws
rather than reading as "nothing to shift", and a refused shift rolls the whole transaction back
with user_version unmoved. The step is not idempotent on its own - run twice, it would shift
twice - so the version stamp, written in the same transaction, is its only done-marker.
Schema versions¶
SCHEMA_VERSION lives in engine/include/vayu/db/database.hpp. It is bumped by the commit that
changes make_vayu_storage's mapping - an added column included - because sync_schema () runs
without preserve: an older engine opening a database with a column it does not map would rebuild
that table without it. It is also bumped by a migration step that must run exactly once and that an
older engine must not read past (versions 4, 6 and 7); the version is that step's done-marker. A bump is one-way. Once a newer engine has opened a
workspace, every engine that carries the version gate refuses it at startup (the refusal above), so
the release that carries a bump says so in its notes. The way back is the
copy before a migration: install the older release, then
restore backups/vayu-<stamp>-pre-v<N>.db as architecture.md
describes; anything written since the update is not in it. The gate first shipped in 0.27.0: 0.26.0
and earlier have none, and opening a newer workspace with one rebuilds requests and collections
empty (their pre_request_script / post_request_script columns are NOT NULL with no default, which
sqlite_orm cannot add to an existing table) and then copies that emptied file over <db>.bak, with no
error. A shipped binary cannot be changed, so nothing in the engine can stop that. install.sh
refuses to install a version below 0.27.0 over an existing db/vayu.db unless given
--allow-data-loss. The Windows installer and winget cannot gate an install: a check added to a
newer installer does not exist in the older installer being run, and winget runs the vendor
installer of whichever version is pinned. On Windows the only protection is not installing an older
version over a newer workspace; restore a Workspace backup first if you must.
| Version | Change | Migration step |
|---|---|---|
0 |
Every database before #1514 (never stamped) | - |
1 |
Scripts folded into elements; pre_request_script / post_request_script dropped (#1514) |
the fold above |
2 |
request_examples.postman_response added (the Postman saved response an example was imported from); requests.disable_cookies, disabled_system_headers, disable_url_encoding and postman_protocol_behavior added (#1765) |
none: sync_schema () adds the nullable and defaulted columns, the migration only stamps |
3 |
file_roots added - the folders request-body files may be read from. Also the fence for the binary body's file reference: an engine at 2 would send such a body bodiless |
none: sync_schema () creates the table, the migration only stamps |
4 |
No mapping change. Every stored body's file references state unresolved, and the allowed folders that are a filesystem root, the home folder or a folder containing it are removed. An engine at 3 read an absent key as chosen |
the file-trust restatement |
5 |
oauth_tokens.raw_response dropped (#1781): the provider response, refresh and id tokens included, was stored and never read; runs.origin and runs.origin_client added (#1817) |
none: sync_schema () drops the one column at the constructor's probe and adds the defaulted and nullable two, the migration only stamps |
6 |
file_grants added - the files a person picked (#2356). Every stored file reference loses unresolved; the ones stored as chosen become grants. An engine at 5 would read the missing key as untrusted and refuse every picked file |
the file-grants migration |
7 |
No mapping change. A collection's sub-folders and requests share one order space (#2535); every request moves past its collection's sub-folders so the display is unchanged. An engine at 6 would number the two kinds apart again on its next create or reorder |
the sibling-order migration |
8 |
collections.spec_file and data_file added (#2765): the files a collection's Spec and Data tabs are bound to, so a backup carries them |
none: sync_schema () adds both defaulted columns, the migration only stamps |
9 |
requests.timeout_ms added (#2937): a request's own timeout, NULL for the defaultTimeout setting; environments.deleted_at added (#2970): the environment trash; request_examples.deleted_at added (#2971): the example trash; client_certificates.deleted_at added (#2973): the certificate trash; idx_requests_deleted_at added (#3006): a partial index on the stamped requests |
none: sync_schema () adds the nullable columns and creates the index, the migration only stamps |
10 |
spec_declined_operations added (#3031): the operations a person declined by deleting the request for them, so a sync does not re-create them. An engine at 9 does not know the table, so it would re-create every declined operation on its next sync |
none: sync_schema () creates the table, the migration only stamps |
The #1765 columns joined version 2 rather than bumping to 3 because no
released build has ever stamped 2: the version shipped only on the unreleased
branch that carried both changes. sync_schema () runs on every start whatever
the stamp, so a development database an earlier build of that branch already
stamped 2 still gets the four columns. Merge gate: before this merges, run
git grep "SCHEMA_VERSION\s*=" $(git describe --tags --abbrev=0); if the last
tag already shows 2 (the postman_response change was released on its own),
the #1765 columns need their own bump to 3 and a row of their own here.
The #1817 runs columns joined version 5 the same way: 0.40.0, the last
release, stamps 4, so no released build has stamped 5. Merge gate: the
same git grep against the last tag must still show 4; if a release shipped
5 without these columns, they need their own bump to 6. The release notes of
the release that carries version 5 say so under Changed (one-way: every
earlier engine refuses the workspace), naming both #1781 and #1817.
Version 6 is a bump of its own: 0.41.0 released 5, and every release since
stamps it. Merge gate: the git grep against the last tag must show 5; a
column that joins 6 later must check the tag the same way. The release that
carries 6 says so under Changed, naming #2356.
Version 7 is a bump of its own even though no release stamps 6: joining an
unreleased version works for what sync_schema () redoes on every start, and
the sibling-order shift is a one-time rewrite of stored rows. A development
database an unreleased build already stamped 6 must still run it, and would
skip it at 6. The release that carries 7 says so under Changed, naming
2535.¶
Version 8 is a bump of its own: 0.45.0 released 7. Merge gate: the
git grep against the last tag must show 7; a column that joins 8 later
must check the tag the same way. The release that carries 8 says so under
Changed, naming #2765.
Version 9 is a bump of its own even though no release stamps 8: an engine
built between #2765 and #2937 stamps 8 too, and its sync_schema () would drop
timeout_ms, and every request's timeout with it, from a workspace a newer build
had opened; at 9 it refuses the workspace instead. Merge gate: the
git grep against the last tag must show 7 or 8; a column that joins 9
later must check the tag the same way. The release that carries 9 says so
under Changed, naming #2937.
The #2970 environments.deleted_at column joined version 9 the same way the
1817 columns joined 5: 0.45.0, the last release, stamps 7, so no released¶
build has stamped 9, and sync_schema () adds the nullable column on every
start whatever the stamp. Merge gate: the git grep against the last tag
must still show 7 or 8; if a release shipped 9 without this column, it
needs its own bump to 10 and a row of its own here. The release that carries
9 names #2970 beside #2937 under Changed.
The #2971 request_examples.deleted_at column joined version 9 by the same
argument and under the same gate: the last release still stamps 7, and
sync_schema () adds the nullable column on every start whatever the stamp. If
a release shipped 9 without it, it needs its own bump and its own row here.
The release that carries 9 names #2971 beside #2970 and #2937 under Changed.
The #2973 client_certificates.deleted_at column joined version 9 by the same
argument and under the same gate: the last release still stamps 7, and
sync_schema () adds the nullable column on every start whatever the stamp. If
a release shipped 9 without it, it needs its own bump and its own row here.
The release that carries 9 names #2973 beside #2971, #2970 and #2937 under
Changed.
The #3006 index idx_requests_deleted_at joined version 9 under the same gate:
any change to make_vayu_storage's mapping bumps the version, an index
included, and sync_schema () creates it on every start whatever the stamp, so
a database an earlier build of this branch stamped 9 still gets it. If a
release shipped 9 without it, it needs its own bump and its own row here. The release that carries 9 names #3006 beside #2971, #2970 and
2937 under Changed.¶
Version 10 is a bump of its own: 0.46.0 released 9. Merge gate: the
git grep against the last tag must show 9; a change that joins 10 later
must check the tag the same way. The release that carries 10 says so under
Changed, naming #3031.
Connection settings¶
PRAGMAs are per-connection state, so the engine applies them when the storage opens its one
long-lived connection (Database::Impl in engine/src/db/database_impl.hpp; the values are in
vayu::core::constants::database). The config_entries rows named below are re-applied by
init once they can be read.
| PRAGMA | Value | Why |
|---|---|---|
journal_mode |
WAL |
Readers (the run-row reader, a report poll) do not wait on a writer |
synchronous |
dbSynchronous, default 1 (NORMAL) |
See dbSynchronous |
busy_timeout |
dbBusyTimeout, default 10 s |
How long a statement waits on another connection's lock |
cache_size |
dbCacheSize, default 64 MB |
|
temp_store |
2 (memory) |
|
mmap_size |
0 (off) |
An I/O error inside a mapped region is a signal (SIGBUS on POSIX), not SQLITE_IOERR, so a failing disk or a truncated file would kill the engine with no log line. Mapping 256 MB cut SQLite's read time by 5-55%, which is 1-6 ms per 1,000 result rows (about 50 ms at the default 10,000 stored errors), and Windows cannot truncate a mapped file, so the engine stays unmapped (#2433). Measured on Windows with a release SQLite and a warm OS cache, over a file of 2 KB result rows and 8 KB bodies, cache 64 MB, medians of 7 interleaved runs, mapped against unmapped, first pass then second pass in one connection. 185 MB file (fits the map): scan of 15,000 results rows 118 against 202 ms, then 67 against 146 ms; 7,500 random body reads 273 against 339 ms, then 252 against 341 ms. 560 MB file (the map covers part): scan of 50,000 rows 485 against 532 ms, then 336 against 544 ms; 7,500 body reads 326 against 344 ms, then 335 against 375 ms. Only Windows was measured |
wal_autocheckpoint |
1000 pages |
Passive: it copies frames back but never shrinks the -wal file |
journal_size_limit |
64 MiB |
Once a checkpoint has emptied the log and the next write restarts it, the -wal file is cut back to this size. Without it one large transaction (a run prune, a big import) leaves a -wal of its own size beside the database for the rest of the session; only a clean shutdown (which checkpoints and deletes the -wal) or the startup VACUUM gives it back (#2433) |
The run-row reader (see runs) opens query_only with mmap_size = 0 and sets none of the
write-side settings.
Tables¶
ID format, and who owns it. Every row ID in the database is generated by the
engine, with vayu::utils::generate_id("col_") (see engine/src/utils/id.cpp) -
<prefix>_ + a random UUIDv4 (lowercase 8-4-4-4-12 hex, e.g.
col_3f2b1c9a-...). No client can supply one: POST /collections, /requests
and /environments reject a body carrying id with a 400, and
POST /import/apply rejects a per-item id the same way, taking opaque
tempIds and returning the map of the real IDs it generated (issues #96, #97).
So the format is a property of the data rather than a convention clients are
trusted to follow. The engine formerly built IDs from a millisecond timestamp, so
two rows created in the same millisecond collided and - because persistence
upserts (storage.replace()) - silently merged; the random UUID removes that.
globals.id is the one exception: a fixed literal "globals".
collections¶
Stores folder/group hierarchy for requests.
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | col_ + UUID |
parent_id |
TEXT | NULL for root collections; never '' (the empty-parent pass) |
name |
TEXT | |
description |
TEXT | Default "" |
variables |
TEXT | JSON: Record<string, VariableValue> |
auth |
TEXT | JSON: RequestAuth (POST/PUT /collections refuse inherit, #3445; a row stored before that is read as no auth of its own) |
elements |
TEXT | JSON array of elements (issue #1513, run by #1514); default "[]" |
data_schema |
TEXT | JSON: the declared data contract; default "{}" |
openapi |
TEXT | JSON: the bound spec document; default "{}" |
spec_file |
TEXT | JSON: the spec file the Spec tab was picked from (issue #2765); default "{}" |
data_file |
TEXT | JSON: the data file a run reads its rows from (issue #2765); default "{}" |
order |
INTEGER | Position among the parent's children, its sub-folders and requests in one space (#2535, Ordering); default 0 |
created_at |
INTEGER | Unix ms |
updated_at |
INTEGER | Unix ms |
deleted_at |
INTEGER | Unix ms; NULL while the collection is live (issue #988) |
elements - the ordered, typed behaviours attached to this collection (issue #1512): extractors,
assertions, timers, controllers, scripts and metrics, each {"id", "kind", "enabled", "name"?,
"config"} and validated against the registry GET /elements/kinds serves
(vayu::core::Registry::validate, engine/include/vayu/core/elements.hpp). [] - the default,
and what an explicit null on PUT resets to - means no elements. The only script source since
issue #1514's cut-over: a pre-cutover collection's scripts are folded in here as script.pre /
script.post entries by migrate_before_sync, and
preRequestScript / postRequestScript are refused (400, naming elements) on every route that
used to accept them. POST /compose resolves a request's whole chain of these (its own
collection's, root to leaf, then the request's) into one list, minus anything a inherit.disable
entry names.
data_schema - which columns this collection's data files are expected to
carry, so {{data.column}} and pm.iterationData can be checked before a run
(issue #599):
{} - the default, and what an explicit null on PUT resets to - means the
collection declares no contract. columns must be an array of unique, non-empty
strings (at most 1024 of them, each at most 256 characters); declaredAt is
epoch ms and fileName is a display label, both optional. Validation lives in
apply_collection_fields, so POST, PUT and POST /import/apply all enforce
it. NOT NULL with a default_value, which is what lets sync_schema
ALTER TABLE ADD COLUMN it onto an existing collections table - pre-existing
rows backfill to {} (the same migration shape as config_entries.keywords).
The schema is stored; the rows never are. A data file's contents are
user data of unknown sensitivity and are persisted nowhere - not here, not in a
run snapshot, which records the row count (dataRowCount) plus the file's name
and content hash (dataFileName, dataFileHash, issue #2051) and whether its
numbers were read as numbers (dataFileCastNumbers, issue #2540), never its
path or rows. The file's location is data_file, below. See
Data-driven runs.
openapi - the OpenAPI document this collection is bound to (issue #637):
{} - the default, and what an explicit null on PUT resets to - means the
collection is bound to nothing, so unbinding needs no verb of its own. A
non-empty object is a binding and must carry a specId that resolves to a row in
spec_documents; specHash records which version of the
document the collection was last synced to, and syncedAt when. Shape validation
lives in apply_collection_fields so the write paths that accept a binding enforce it, and the
resolvability check sits in the route cores beside reject_missing_collection -
POST /import/apply binds specs it is about to write in the same transaction.
The version half is the engine's (issue #709). A client sends the specId;
whichever write path receives it fills in specHash from the document that id
names and syncedAt from the moment of the write - the same division
spec_documents.hash itself draws. A specHash or syncedAt a client sends on
POST /collections or POST /import/apply is dropped first (issue #3348), so
the pair always comes from the store; a stale one arises only when the
document under a binding is replaced, and a run then reports hash_mismatch. This matters because both contract
readers - coverage and response-schema validation - require the binding's hash
to equal the stored document's before they engage: until #709 the import path
stored {specId} alone, and every run of an imported collection therefore
reported no contract at all. Database::stamp_hashless_spec_bindings() repairs
the rows written before that rule, on every startup, stamping the document's
fetched_at (when the import that made the binding stored it, not the moment of
the restart) and skipping any binding whose document is gone.
The edge is stored here; the document is not. Several collections may
bind the same spec, so nothing here owns it: no cascade reaches spec_documents,
and DELETE /specs/:id is refused while any collection names the row. NOT NULL
with a default_value, the same migration shape as data_schema above.
spec_file and data_file - the files this collection's Spec and Data tabs are bound to (issue #2765), stored on the row so a workspace backup carries them:
{"path":"/home/me/specs/pets.yaml","fileName":"pets.yaml"}
{"path":"/home/me/data/users.csv","fileName":"users.csv","hash":"<hex sha256>","castNumbers":true}
{} - the default, and what an explicit null on PUT resets to - means no
file. A non-empty binding must carry path and fileName as non-empty strings
and nothing beyond its members: spec_file holds those two, data_file adds an
optional hash (the SHA-256 of the file's bytes as 64 lowercase hex
characters, the digest a run records as dataFileHash) and an optional boolean
castNumbers (how the app reads the file's numbers, issue #2540). Validation
lives in apply_collection_fields, so POST and PUT enforce it; POST
/import/apply refuses either field on a collection item, because an import
describes a document from elsewhere.
The path's form is not judged. It is true of the machine that picked the file,
so a Windows path restored onto Linux is stored and read back as written, and
the app shows it as a file it cannot find. A stored path is not a grant:
whether a request body may send a file is still the
file_roots and file_grants decision, and
nothing reads these columns to send one. The engine never opens either file.
Neither column leaves the machine through an export: POST /export/postman and
POST /specs/export copy their own fields off the row and name no binding.
Deleting the collection keeps both on the trashed row, a restore brings them
back unchanged and a purge removes them with the row. NOT NULL with a
default_value, the same migration shape as data_schema above.
auth is a JSON discriminated union: {"mode":"none"} | {"mode":"bearer","token":"..."} |
{"mode":"basic","username":"...","password":"..."} | {"mode":"apikey","key":"...","value":"...","in":"header"|"query"} |
{"mode":"oauth2","config":{…}} (see requests.auth and oauth_tokens).
Collections are always auth sources - they never store {"mode":"inherit"}. They may store
{"mode":"noauth"}, the inheritance terminator meaning "descendants inherit no credentials"
(distinct from none, which means nothing is set at that level); POST /compose resolves both
during the inherit walk (request_composer.cpp), so neither reaches the engine's parse_auth -
which would treat them as no auth anyway.
Cap. (issue #1485) variables, auth, data_schema, openapi,
spec_file and data_file are each
refused at write time with a 413 naming the field, its size and the cap, when
the serialized value is over json::MAX_FIELD_SIZE (10 MiB,
engine/include/vayu/core/constants.hpp) - through the same shared
apply_json_field the requests columns are capped by. A file
binding's path over file_binding::MAX_PATH_BYTES (96 KiB) or fileName over
MAX_FILE_NAME_BYTES (1 KiB) is a 413 naming the member the same way; both sit
above the longest path and file name any of the three platforms can open. Unlike
requests, no collection column is ever substituted on read: GET
/collections carries the same whole value GET /collections/:id does, so
there is no truncatedFields here.
Cascade delete: deleting a collection performs BFS to collect all descendant IDs and stamps
deleted_at on them and their requests in one transaction (Database::delete_collection(),
issue #988). The hard cascade is the purge (DELETE /trash/:id, or retention): BFS over the
stamped descendants only, re-parenting a live collection it meets to the root (issue #1883),
then all their requests before the collections deepest-first, wrapped in a single transaction so a
crash mid-cascade cannot leave a half-deleted subtree. See Database::purge_collection_locked().
parent_id forms a tree, but SQLite enforces no such constraint, so the BFS carries a visited
set and is cycle-safe: a self-parent (parent_id == id) or an A -> B -> A loop terminates
instead of growing the work list forever under the global DB mutex (which would hang every
endpoint, including /health). POST /collections rejects the writes that would create such a
cycle (see api-reference.md - POST /collections), so cycles only arise from
data written before that validation existed; the visited set is what lets deletion recover from it.
requests¶
Stores individual HTTP request definitions.
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | req_ + UUID |
collection_id |
TEXT | FK → collections.id (not enforced by SQLite FK) |
name |
TEXT | |
description |
TEXT | Default "" |
method |
TEXT | GET / POST / PUT / PATCH / DELETE / etc. |
method_source |
TEXT | Which app setting wrote method; NULL when none |
url |
TEXT | |
params |
TEXT | JSON array of KeyValueEntry[] |
headers |
TEXT | JSON array of KeyValueEntry[] |
body |
TEXT | JSON discriminated union (see below) |
body_type |
TEXT | Denormalized mirror of body.mode; kept for queries |
auth |
TEXT | JSON discriminated union (see below) |
elements |
TEXT | JSON array of elements (issue #1513, run by #1514); default "[]" |
order |
INTEGER | Position among the collection's children, its sub-folders and requests in one space (#2535, Ordering); default 0 |
follow_redirects |
INTEGER | Boolean; default 1 (follow) |
max_redirects |
INTEGER | Hops allowed while following; default 10 |
timeout_ms |
INTEGER | The request's own whole-transfer timeout in ms, 1-86400000; NULL for the defaultTimeout setting (issue #2937) |
http_version |
TEXT | 'auto' | 'http1.1' | 'http2'; default 'auto' |
verify_ssl |
INTEGER | Boolean; verify the TLS certificate; default 1 |
stream |
INTEGER | Boolean; consume the response as SSE; default 0 |
disable_cookies |
INTEGER | Boolean; keep the cookie jar out of this request's transfer; default 0 (issue #1765) |
disabled_system_headers |
TEXT | JSON array of lowercased header names the engine does not add on its own; default "[]" (issue #1765) |
disable_url_encoding |
INTEGER | Boolean; write path values and an api-key query parameter as typed; default 0 (issue #1765) |
postman_protocol_behavior |
TEXT | The imported Postman item's protocolProfileBehavior, JSON text in source member order; NULL when none (issue #1765) |
spec_operation |
TEXT | JSON: which spec operation this is; NULL when none |
mock_response_mode |
TEXT | Which saved example a mock server answers with; default 'first' (issue #481 phase 3) |
mock_example_id |
TEXT | The example id mock_response_mode == 'fixed' names; NULL when none. Cleared when that example is deleted through its route, and not set again by its restore (issue #3422) |
created_at |
INTEGER | Unix ms |
updated_at |
INTEGER | Unix ms |
deleted_at |
INTEGER | Unix ms; NULL while the request is live (issue #988) |
params / headers - stored as a JSON array of objects:
Disabled rows ("enabled":false) are preserved in storage and filtered at HTTP-execution time only.
Duplicate keys are allowed. A query row may carry "valueless": true for a
bare key with no = (issue #1772). The enabled query rows of params follow
url: a write that states url and no params reconciles them with the URL's
query, and rewrites the column only when that changes a row (issue #3551, see
POST /requests); no startup pass touches
rows stored before it. A pre-#1229 renderer also wrote its own X-Vayu-Version,
X-Request-ID and User-Agent rows here; the header-strip pass
disables them once, at startup, leaving each row in place with source: "legacy-default".
An optional source key ("body-mode" | "stream" | "legacy-default") marks a row
an app setting or that repair pass wrote rather than the user - the auto Content-Type
a body-mode change adds, the Accept the Event stream toggle adds, or a row the
header-strip pass disabled - so it can tell its own row apart from a hand-typed one
across a reload; retyping the row's key or value clears it (issues #1481, #1491).
A Postman import also keeps two keys nothing sends, for the Postman export to
write back: a header row's type other than "text" ("default") and a
query row's boolean equals.
elements - same shape, and the same script-to-elements cut-over, as
collections.elements above; see that entry.
body - discriminated union:
{"mode":"none"}
{"mode":"json"|"text"|"graphql"|"jsonrpc"|"xml","content":"..."}
{"mode":"form-data"|"x-www-form-urlencoded","fields":[{"key":"...","value":"...","enabled":true}]}
{"mode":"binary","file":{"src":"/abs/a.bin","fileName":"a.bin","contentType":"image/png"}}
form-data file part carries "type":"file" and, in place of value, the
same file members a binary body's file does: src (an absolute path, stored
as written) and fileName and contentType (both optional, omitted when
empty). No trust is stored: whether src may be sent is decided by the
file_roots and file_grants at each send, and
the unresolved a read answers is derived then (#2356). Every write drops the
key an older client sends (apply_request_fields), and the
schema version 6 migration
removed it from every row written before. A text part may also carry
"type":"text". No file's bytes are ever stored; a binary body needs a file
object (schema version 3).
A json or text body a Postman import produced from a raw body also
carries rawLanguage: the options.raw.language the document declared when it
named no Vayu mode ("javascript", "html", "yaml"; the body is text), or
"" when it declared none (the body was sniffed). Nothing sends it, though the
import wrote the Content-Type row html and javascript need as an ordinary
body-mode header row. The Postman exporter writes it back while importing the
body again would still give its stored mode, so an unlabelled body exports
without options as Postman wrote it.
auth - discriminated union (same shape as collection auth, plus inherit):
{"mode":"none"}
{"mode":"inherit"} // resolved at execution time
{"mode":"bearer","token":"..."}
{"mode":"basic","username":"...","password":"..."}
{"mode":"apikey","key":"...","value":"...","in":"header"|"query"}
{"mode":"oauth2","config":{ /* OAuth2Config */ }}
A Postman import may add postman to any of these: the document's own auth
block ({"type":"oauth2","oauth2":[{"key","value","type"}]}), kept when the
Postman exporter could not otherwise write it back as it was. Nothing sends
it; the exporter writes it verbatim while it still maps to the stored auth,
and every secret-blanking export blanks its credential attributes, the PKCE
code_verifier and the credentials among an oauth2 block's
tokenRequestParams / authRequestParams / refreshRequestParams rows
included. Every write of an auth column drops it once it no longer maps to
the auth being written (without_stale_postman_source), so an edited
credential does not stay behind in it.
The oauth2 config holds the grant type, endpoints, client id/secret,
placement options, etc. Three optional lists hold extra request parameters
(issue #2536): tokenParams, authorizationParams and refreshParams, each
an array of {key, value, enabled, sendAs?} rows. sendAs is body, url or
header and, when absent, body; an authorizationParams row stores none,
because the authorization request carries its parameters in the authorize
URL's query. A config with no extra parameters omits the lists. What the
engine sends from them, and how they enter the token cache key, is in
api-reference.md. A Postman
import fills them from the block's tokenRequestParams, authRequestParams
and refreshRequestParams
(postman.md).
Secret fields (clientSecret, password) are stored
in plaintext here, same as bearer/basic credentials - the v1 posture. The
resolved access tokens live separately in oauth_tokens.
Cap. (issue #1485) params, headers, body and auth are each refused
at write time - 400 if the value is not an object/array of the shape above,
413 naming the field, its size and the cap - when the serialized value is
over json::MAX_FIELD_SIZE (10 MiB, engine/include/vayu/core/constants.hpp),
the same limit GET /requests (the list route) has always read a column
against. A row written before this cap existed can still be oversized: GET
/requests/:id answers with it whole, however large, but GET
/requests?collectionId= substitutes the field's documented default and adds
truncatedFields (an array of the substituted field names, e.g. ["body"],
omitted when nothing was substituted) to that row, so a caller reading the
list can tell a genuinely empty field from one it cannot see the whole of.
Nothing recomputes truncatedFields at write time or stores it - it is
derived fresh on every list read from the same size check.
follow_redirects / max_redirects / http_version / stream - the request's
execution options, surfaced in the request builder's Settings tab and
serialized as followRedirects / maxRedirects / httpVersion / stream. The
first three mirror the executable vayu::Request fields of the same name, so the
saved options are what POST /execute and POST /runs apply - http_version
governs both Send and load test alike; there is no separate per-run protocol
control.
stream has no vayu::Request mirror, deliberately (issue #574). It is not
a transfer option the executor applies but a choice of which delivery path
runs, read off the POST /execute payload by read_stream_flag; the column is
where the app's Event stream toggle persists so the choice survives a tab switch
and a bulk import carries it. payload_from_stored does not add it to a by-id
composition, so a client that pipes POST /compose into POST /execute - MCP's
run_collection_smoke, among others - still gets a buffered send rather than a
202 it never asked to parse. That stays true now that MCP can stream
(issue #575): run_request sets stream from its own argument on every call
rather than inheriting it from a row, and the app's Code panel reads the flag off
the request row rather than out of the composed payload, for the same reason.
All four columns are NOT NULL with a DEFAULT, which is what lets
sync_schema() add them to an existing, non-empty requests table - a
NOT NULL column with no default cannot be added by ALTER TABLE ADD COLUMN.
Rows written before the columns existed backfill to 1 / 10 / 'auto' / 0,
i.e. the behaviour they already had (a row predating http_version could only
ever have run HTTP/1.1, since nghttp2 was not yet linked). max_redirects is
clamped to 0..100 on write.
timeout_ms - the request's own timeout (issue #2937, schema version 9),
serialized as timeout and edited in the same Settings tab. Unlike the four
above it is nullable with no DEFAULT, on the spec_operation precedent, and
sync_schema() adds it the same way: NULL means "use the defaultTimeout
setting", which is what every row written before the column existed backfills
to. A stored number would freeze the setting at the value it had on the day the
row was saved, so the default is never written into the row. A value outside
1-86400000, or not a number, is refused on write with the 400 POST
/execute answers (REQUEST_TIMEOUT_FIELD), never clamped. payload_from_stored
emits timeout only when the column holds one, so a by-id composition of a
request that states none leaves the setting to /execute and /runs.
disable_cookies / disabled_system_headers / disable_url_encoding /
postman_protocol_behavior - Postman's per-request protocolProfileBehavior
settings (issue #1765), serialized as disableCookies /
disabledSystemHeaders / disableUrlEncoding / postmanProtocolBehavior and
surfaced beside the redirect options in the Settings tab. The three typed
ones mirror vayu::Request fields and ride every composed payload:
disable_cookieskeeps the cookie jar out of the transfer, both halves - no jar cookie is attached and noSet-Cookieis stored - on the buffered and streaming design sends and a scenario load run's per-virtual-user session. A pre-request script'spm.cookies.jar()write is still persisted, applied directly because no transfer carries it.disabled_system_headersis unioned into the per-sendsuppressed_default_headers, souser-agent,accept-encodingand the correlation id are refused as a per-send opt-out refuses them, and two more names gain meaning:acceptremoves libcurl's implicitAccept: */*andcontent-typeremoves the Content-Type a body implies (multipart keeps its boundary header) and aheadersrow marked"source": "body-mode", the body's own Content-Type row (composition lists those names asbodyModeHeaders).host,content-length,connection,cache-controlandpostman-tokenare stored and exported but change nothing on the wire.disable_url_encodingwrites a path variable's value (at composition and when a bind or the residual pass settles it) and an api-key or OAuth 2.0 query parameter as typed, and it turns off the Postman encoding of the whole URL (wire_url, #2465): with encoding on a raw space is sent as%20and"as%22, with the toggle on the space is refused, by a message that names the space and the toggle. libcurl's dot-segment normalisation is unchanged.
postman_protocol_behavior is nothing the engine sends: POST /export/postman
starts from it and overwrites only the keys a typed column now disagrees with,
so explicit defaults, keys Vayu does not apply and the source's member order
survive a round trip; GET answers it as this text, so a copy written back
from it (the app's Duplicate) keeps the same bytes. The three typed columns are NOT NULL with a DEFAULT
and the carrier is nullable, for the reason the paragraph above gives.
spec_operation - which operation of the collection's bound
openapi document this request is (issue #637):
method and path are required and operationId is optional, because an
OpenAPI operation may declare none. path is the templated path from the
document, never a concrete URL - it is the identity a re-fetched spec is diffed
against and the key coverage counts by, and a substituted URL would match nothing
in the document it came from, so a path that does not start with / is a
400. Two requests may name the same operation.
Nullable rather than NOT NULL with a "{}" default, unlike the four columns
above: a request that answers to no operation declares nothing, and NULL is
that - {} would be a second spelling of the same absence for every reader to
handle. A nullable column is ALTER TABLE ADD COLUMN-friendly without a default.
Both request serializers emit it as specOperation, null when the column is,
and apply_request_fields applies it, so POST, PUT and POST /import/apply
all carry it.
method_source - which app setting last wrote method, still unclaimed by
the user (issue #1505). The only writer today is the GraphQL body mode: entering
it sets method to POST on a fresh GET and stamps method_source: "graphql";
leaving it reverts method to GET only while the marker is still there, and
any other write to method clears it, the same "still ours" rule
switchAutoHeader's KeyValueEntry.source (issue #1481) applies to a header
row. Nullable on the spec_operation precedent directly above, for the same
reason: NULL is the only spelling of "no marker" and the column is
ALTER TABLE ADD COLUMN-friendly without a default. apply_request_fields
validates a present value against the one string the app writes ("graphql")
and rejects anything else with a 400, so a marker this app does not recognize
can never park itself here and revert a method the user actually chose.
http_version stores Request::http_version (the requested protocol, an
enum member spelled as text) - a different value and a different value space
from Response::http_version, the negotiated protocol string
("HTTP/1.1" / "HTTP/2" / "") that lands in a design run's trace_data
(see results) and the live /execute response. Conflating the
two would show a user a protocol they asked for but were not actually granted.
On create or on an explicit null reset, this column seeds from the
defaultHttpVersion config_entries row ('auto' unless
changed) - a write-time default only, never consulted at execution. Changing
the global afterward does not alter a request already saved.
request_examples¶
Stores saved example responses for a request (issue #481) - what an importer
found next to it (Postman's item.response[], an OpenAPI operation's
responses) and, once the mock server lands, what that server answers with.
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | exa_ + UUID |
request_id |
TEXT | FK → requests.id (not enforced by SQLite FK) |
name |
TEXT | Required on write; no default |
status |
INTEGER | HTTP status, 200-599; default 200 |
headers |
TEXT | JSON array of KeyValueEntry[] |
body |
TEXT | Response body, verbatim; capped (see below) |
content_type |
TEXT | Denormalized from headers; "" when unstated |
order |
INTEGER | Sort order within the request; default 0 |
origin |
TEXT | import | user; NOT NULL, default import |
body_truncated |
INTEGER | body stops short of the captured response; NOT NULL, default 0 |
suppressed |
INTEGER | A tombstone: an imported example the user deleted; NOT NULL, default 0 |
spec_example_key |
TEXT | The examples map key this was imported from; NULL when there is none |
postman_response |
TEXT | The Postman saved response this example has - from an import or from a save in the app - as JSON text; NULL when it has none (schema version 2) |
created_at |
INTEGER | Unix ms |
updated_at |
INTEGER | Unix ms |
deleted_at |
INTEGER | Nullable, Unix ms. NULL while live; the delete's instant once a user example is in the trash (#2971) |
body is stored and read back byte-exact, a zero byte included (#3112). sqlite_orm reads a
std::string column through strlen, so the mapping uses db::BodyBytes
(declared in types.hpp, adapted in database_impl.hpp), a std::string whose extractor takes the length from
sqlite3_column_bytes. The column is still TEXT, so no schema version moved and a row written
before the fix reads whole: the write always bound its full length.
headers is the same KeyValueEntry[] shape a request's headers use, not the
JSON object result_bodies and inbox_requests
store. A stored example is re-served rather than only displayed, so repeated
names (Set-Cookie) and the author's ordering both have to survive - an object
keeps neither.
order is what makes "the first example" a stable answer. A bulk import
writes every example of one request in the same millisecond, so created_at
ties for all of them and an id tiebreak would return the author's list
shuffled; GET /requests/:id/examples orders by order, then created_at,
then id. A create that states no order appends after the request's current
examples, exactly as a new request appends within its collection.
origin records who wrote the row (issue #588). It carries default_value
rather than being a fresh table's column, so sync_schema ALTERs it onto an
existing request_examples and every pre-existing row backfills to import -
which is what all of them are, since import was the only writer until the app
could save a live response as an example. It exists because the OpenAPI spec
sync replaces the examples a document produced and must leave the ones a person
saved alone, and nothing else about a row says which it is - POST /specs/sync
(issue #655) is the reader: applying a change to a request deletes that
request's import rows and writes the document's in their place. The rows are
the engine's own since issue #869 - the payload says only whether to refresh a
request's imported examples, and what is written is what the document being
stored documents - so a sync cannot manufacture a row it would then refuse to
replace, nor one for a response no document describes. The write paths validate it against those two
values and 400 on anything else; no read path in the app displays it.
body_truncated (issue #659) records that body is only the first slice of
the response it was captured from - the trace's maxTraceBodyBytes cap. Added
the same ALTER-friendly way as origin: NOT NULL with a default_value, so
sync_schema() adds it and every pre-existing row backfills to false, which is
what they all are (an importer copies a whole documented body; only the app's
save-as-example ever had a partial one). It is stored rather than inferred
because a short body is a legitimate body - nothing about the row says otherwise
- and a mock server serves the bytes verbatim, so an undisclosed partial example
is answered as though it were complete. The Examples panel is the reader,
painting a "Partial body" chip; before the column, the fact lived only in the
example's name, which the save dialog invites the user to edit.
suppressed (issue #722) marks a row as a tombstone: an origin="import"
example the user deleted. Added the same ALTER-friendly way as the two columns
above, and false is right for every pre-existing row - before it, a delete
removed the row. It exists because deleting an imported example otherwise
recorded nothing, while the sync refresh above rewrites every imported row of a
request it applies any change to, so the next rename-only sync re-created what
the user had removed. The row is kept so the refresh can skip that response
status (the identity a document's example keeps across a reworded
description, which its name does not), and body, headers,
content_type and postman_response are cleared when the flag goes on, since nothing serves a
tombstone. get_request_example, get_request_examples and
count_request_examples all filter suppressed rows out - so the list route, a
mock server and an export behave exactly as though the delete had removed it -
and get_suppressed_request_examples is the one read that sees them, for the
one caller that must. A user row is never tombstoned: nothing re-creates a
saved response, so there is no intent for a sync to keep, and its delete goes
to the trash instead (deleted_at, below).
deleted_at (#2971) is the trash stamp of a user example, on the
environments.deleted_at shape: one row, no cohort, NULL
while live. DELETE /requests/:id/examples/:exampleId stamps it on a user
row and tombstones an import row, so a row is never both suppressed and
stamped. Every read filters it exactly as it filters suppressed
(get_request_example, get_request_examples, count_request_examples, and
get_suppressed_request_examples too), so a trashed example is gone to the
list route, a mock server, both exports and the spec sync's refresh, and does
not count against the cap. The trash lists it as a root while its request is
live (parentId is the request); a restore clears the stamp alone and is
refused while the request is trashed or gone, or already holds
MAX_PER_REQUEST live examples. The create path's id-collision check reads
request_example_id_taken, which sees every row, so a fresh id can never
overwrite a hidden one.
spec_example_key (issue #1457) records which entry of a 3.x response's
named examples map the import took this example's payload from - the first
one, unwrapped, since firstNamedExample never read past it. Nullable rather
than NOT NULL with a default, on the requests.spec_operation precedent: a row
that predates the column, or one taken from a single example or sampled off a
schema, names no entry, and NULL is the only spelling of that - there is
nothing to backfill it from. It exists because a value comparison alone cannot
tell an edited example from a new one: once its body has changed, it no longer
equals the entry it came from, so exporting it back would add it beside that
entry rather than replace it. The bound export reads the key to find its way
back to the same entry when the document still declares it, and falls back to
adding a new one when it does not. Not a display field: no app surface reads it.
postman_response (schema version 2) is the Postman saved response this
example has - from an import or from a save in the app - as compact JSON
text in Postman's member order: an imported one is the item.response[] entry
the import took it from, in the source's own order, and one saved from a live
response in the app (#1763) is built by the engine
(core::postman_saved_response_text) from the request as written when it was
sent and the server's reason phrase. Every member is kept verbatim except
name and body, which are null placeholders holding only their position -
the columns are the values, so there is one copy of each. It exists because
the rest of a saved response has no Vayu column: the request it was recorded
against (originalRequest, which differs per example and from the request's
current state), the status text the server sent, the editor's preview
language and type, cookies, the response time, and the header rows as written
(name fields, numeric values). The Postman export is the one reader
(core/postman_export.cpp): it writes originalRequest,
responseTime and unknown members back as stored, and the status text,
header[] and preview members only while status and headers still say
what was imported, and cookie while the enabled Set-Cookie rows do,
regenerating them otherwise - so an edit made in Vayu is
never contradicted by a stale copy. An import or a save in the app writes
it: POST /import/apply and POST /requests/:id/examples accept a string that
parses as a JSON object (capped at
request_example::MAX_POSTMAN_RESPONSE_BYTES), POST /requests/:id/examples
builds it from a savedFrom instead (left NULL, with a warning, when the
result would be over that cap), and PUT accepts only null, which clears it. The list route does not return
it.
Cascade. Examples are owned by their request. DELETE /requests/:id and
the delete_collection cascade stamp the requests and leave their examples
untouched (issue #988): every route reaching an example checks its request
first, so they are unreachable while it is in the trash and come back with it.
The purge of a request or a collection (purge_request_locked,
purge_collection_locked) removes each request's examples before the request,
by request_id and whatever their stamp, and so does the spec sync's hard
delete. Every read here is by request_id or by example id, so a row left
behind would be unreachable rather than merely stale. A tombstone and a trashed
example are rows like any other and go with the same purges.
Caps. A body over request_example::MAX_BODY_BYTES (1 MiB) is a 400, never
a truncation - a half-body served as if whole is worse than a refused write - and
a request holds at most MAX_PER_REQUEST (100) examples. Both constants live in
engine/include/vayu/core/constants.hpp.
spec_documents¶
Stores OpenAPI documents, bound to collections by
collections.openapi (issue #637).
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | spec_ + UUID |
content |
TEXT | The document, verbatim; capped (see below) |
source_url |
TEXT | Where it was fetched from; NULL when pasted/uploaded |
fetched_at |
INTEGER | Unix ms |
hash |
TEXT | Hex sha256(content), computed engine-side |
operations |
TEXT | JSON array: the declared-operation index, "" = none |
response_schemas |
TEXT | JSON object: the response schema index, "" = none |
content is stored as text rather than a parsed model, because every feature
stacked behind this needs the document: the Spec tab renders it, sync
re-fetches and diffs against it, response validation resolves $refs through it,
and export writes a new one beside it. A parse here would be re-done by each of
them anyway and would lose whatever the author wrote that the parser did not
model.
hash is the content+hash pair body_blobs uses, and is
never taken from the caller - POST /specs and POST /import/apply both
compute it through spec_content_hash, and a body carrying hash or fetchedAt
is a 400. A scenario run of a bound collection stamps specId + specHash
into its runs.config_snapshot under scenario.openapi, and that stamp
only means anything because both sides of a later comparison were computed by the
same code on the same bytes.
Ownership, and why there is no cascade. A spec is bound by collections
rather than owned by one: several may bind the same row, and unbinding one must
leave it there for the others. So no cascade deletes a document -
DELETE /specs/:id is refused with a 409 naming the binder while any
collection still names it. There is likewise no PUT /specs/:id: a document that
changed is a different document, and rewriting one in place would invalidate the
hash every run of every bound collection was stamped with.
Lifetime, and the sweep that enforces it (issue #718). Having no owner left
these rows with no way to die: every POST /specs/sync mints a new document and
moves the binding off the old one, unbinding leaves it, re-binding mints another,
and deleting a bound collection takes its requests and not the document - while
DELETE /specs/:id needs an id and there is no list route to get an unreachable
one from. Weekly syncs of a 12 MB document therefore stranded ~600 MB a year that
nothing could read or reclaim. The rule now, and the only one:
A document lives while a collection binds it, or while a retained run names it in
runs.config_snapshotunderscenario.openapi.specId.
Database::sweep_orphaned_spec_documents() applies it and returns how many rows
went. Run-referenced documents live as long as the runs do, deliberately: a
scenario run stamps the specId it was planned against and its report describes
that contract, so the source has to outlive the binding by exactly the retention
window - which is why the sweep is the tail of prune_runs_configured(), the
pass that releases those references, and thereby runs on startup and at the end
of every run. It also runs after spec_sync_apply (the accretion source) and
after delete_collection (the last binder going away). Two rules keep it honest:
a document written within SPEC_DOCUMENT_SWEEP_GRACE_MS (10 min) is always
spared, because binding stores the document before the write
that names it (POST /specs/bind, POST /collections) and a document in that window is a bind in flight, not an orphan;
and the pass never throws or reads content, so no caller can fail over
housekeeping and no 10 MiB blob is loaded to decide its fate.
Cap. A document over the live maxSpecDocumentBytes
config_entries value (default 10 MiB, aligned with
json::MAX_FIELD_SIZE) is a 400 naming the size and the cap, on both write
paths - never a truncation, and never the engine's general body cap refusing the
whole request without naming the document
(Request bodies).
operations is what the document declares - a JSON array of
{operationId?, method, path, responses[]} in document order - read by
contract coverage (issue #629).
Derived, never supplied (issue #853): the engine reads content as it stores
it (core/openapi_document.cpp, JSON first and YAML second) and writes this
column from that parse, the way it computes hash rather than taking one; a
write carrying an operations field is a 400. One reader answers what a
document declares, which is what keeps this column agreeing with the
spec_operation stamped on each request. NOT NULL with a default_value, the
same migration shape request_examples.origin uses, so sync_schema() ALTERs it
onto an existing table and every pre-existing row backfills to "".
"" means no index, which is not the same as a document declaring nothing: a
run of a collection bound to such a document reports no coverage block at all,
and GET /specs/:id reads the column back as null rather than [] so the two
stay distinguishable. A row can hold "" because it predates the column, or
because the document declares no operation at all - a stored file that is not a
contract. Capped at 2000 rows on the write, which is refused with the count rather
than silently truncated.
response_schemas is what the document declares each response looks like
(issue #628) - a JSON object of {refRoots?, operations: [{operationId?, method,
path, responses: [{status, contentType, schema?}]}]}, read when a response comes
back with a body to check. Every status an operation declares is listed (#3028);
one with no schema to check against is a status-only row (contentType: "", no
schema), so the check and coverage agree on the statuses and a default or
4XX never judges a body under a status the document names itself. An x-
extension key under responses is not a status and is listed in neither index
(#3084).
Derived, never supplied (issue #860), off the same read as operations: a write carrying a responseSchemas field is a
400, and one read for both is what keeps them agreeing about which operation
declares which status.
A column of its own rather than a field on operations because the two are read
at different moments: the operation index is parsed when a run's plan resolves
and held for the run's whole life, while schemas are touched only per response -
and they are orders of magnitude larger.
refRoots holds the document's components.schemas / definitions /
x-vayu-bundled subtrees once, and each schema keeps its $refs as written;
validation merges the two into one root document. Inlining instead would copy a
shared Error schema into every operation naming it, and a recursive schema - a
tree node whose child is itself - has no finite expansion at all. Schemas are
translated out of OpenAPI's dialect as they are derived (3.0's nullable, a
union with null on the type or, beside allOf / anyOf / oneOf / $ref /
not / enum with no type, anyOf: [<schema>, {"type":"null"}]; its
draft-04 boolean exclusiveMinimum, and the OpenAPI-only keywords that constrain
no body): the column holds JSON Schema and nothing else, because that is what the
validator reads.
"" means no index on the same terms as operations: a response of such a
document reports checked: false with the reason no_index, never a body that
passed, and GET /specs/:id reads it back as null. The serialized index is
held to the same maxSpecDocumentBytes cap as the document, refused on the write
with the count and the cap rather than truncated.
The table itself was new when it landed, so sync_schema() created it outright;
operations and response_schemas are its migrated columns.
spec_declined_operations¶
The operations a person declined by deleting the request for them (schema
version 10, #3031), or by moving it out of the contract (#3446), so that a sync
of the document does not re-create them.
POST /specs/diff lists such an operation
under declined instead of added, and a policy: "safe" sync never creates
it.
| Column | Type | Notes |
|---|---|---|
id |
INTEGER PK | Autoincrement; never leaves the engine |
collection_id |
TEXT | The bound collection the decision was made under |
operation_id |
TEXT | The operation's operationId, "" when it declares none |
method |
TEXT | The identity's method, as requests.spec_operation records it |
path |
TEXT | The identity's path template |
request_id |
TEXT | The request whose delete, or move out of the contract, wrote the row |
UNIQUE (collection_id, method, path, operation_id): one row per operation
under one binding, and a second request's delete for the same operation takes
the row over (request_id moves to it). operation_id is "" rather than NULL
for that reason - SQLite treats two NULLs as distinct, so a NULL would let one
endpoint hold two rows. The constraint's own index serves every read, which is
by collection_id.
A row of its own rather than a flag on the request, because the decision has to outlive the request row: a purge removes it. Who writes and forgets a row, each in the transaction of the write named:
- Written by a person's delete:
DELETE /requests/:idof a request whosespec_operationnames a whole identity (method and path), andDELETE /collections/:idfor each such request its own delete stamps. The collection is the nearest ancestor that binds a document (core::nearest_spec_binding, resolved by the route and passed in, soDatabasereads no binding itself). Deleting the bound collection itself writes nothing. A sync's own (hard) deletes write nothing. - Written too by a person's move out of the contract (#3446): a
PUT /requests/:idwith acollectionId, aPOST /reorderbatch (every stamped request under a moved collection included), or aPUT /collections/:idwith aparentId(every stamped request in the moved subtree, #3470), that leaves a stamped request where the nearest binding is not the same contract (declined_by_moveinroutes.hpp: another document, none, or the same document bound by a collection outside the source's contract walk). The collection is the source's nearest binding, read before the move, and the same write clears the request'sspec_operation(Database::save_request/apply_reorder, orrebind_collectionwhen the samePUTalso unbinds, given the decision by the route). - Forgotten when the request comes back from the trash (by
request_id, alone or with its collection's cohort) while it still records an operation (forget_declined_by_locked's guard): a request a move unstamped claims nothing back, so its move's row stands; when aPOST /specs/synccreates a request for the same operation (core::declines_taken_back, the diff's own identity rule run against the document being stored); whenPUT /collections/:idunbinds the collection (after the rows a move in the same body wrote, so one written under the collection itself goes too), orPOST /specs/bindbinds it to a document with anotherspecHash; and when the bound collection itself is purged. - Kept when the deleted request is purged, by hand or by
trashRetentionDays: emptying the trash is not changing one's mind.
The identity is read the way the diff reads a request's
(core::parse_spec_identity), and the diff follows a row into a re-fetched
document by the rule it follows a request with (operationId first, method and
path shape second), so a declined operation the document renamed is still the
declined one. A live request claiming the operation outranks the row.
environments¶
Stores named variable sets.
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | env_ + UUID |
name |
TEXT | |
description |
TEXT | Default "" |
variables |
TEXT | JSON: Record<string, VariableValue> |
is_active |
INTEGER | Boolean; 0 or 1. At most one row is 1 - see below |
created_at |
INTEGER | Unix ms |
updated_at |
INTEGER | Unix ms |
deleted_at |
INTEGER | Nullable, Unix ms. NULL while live; the delete's instant once it is in the trash (#2970) |
is_active marks the environment clients resolve against by default, and at
most one row carries it. The invariant is enforced in the DB layer, not the
routes: every write path that can store an active environment calls
Database::deactivate_other_environments_locked first, inside the same
transaction, so activating one environment deactivates the previous one
atomically - no reader can observe two actives, and none can observe an
intermediate zero. Three paths reach this table (POST /environments, PUT
/environments/:id, and POST /import/apply); putting the rule in the handlers
would mean repeating it in each, which is how this column previously ended up
honoured on create but not on update.
A trashed row is never active. DELETE /environments/:id stamps
deleted_at and clears is_active in one write, and a restore clears the stamp
alone, so the row comes back inactive. Every read (get_environments,
get_environment) filters deleted_at IS NULL, and so does
deactivate_other_environments_locked: the rule above is about live rows, and a
stamped one already holds 0. An environment owns no rows, so unlike
collections its stamp is no cohort key; it is always its own
trash root, and a purge (DELETE /trash/:id, or retention) removes the row.
Writing isActive: true is the switch - there is no separate endpoint and
no companion request to clear the old one. Clearing entirely is spelled as
writing isActive: false to the environment that holds the flag, since there is
no "no environment" row to write true to.
Selecting is still a client action: the engine never applies an active
environment to a request, which must always name its own environmentId. What
changed is where the choice lives. Storing it here rather than in client-local
state is what makes it survive a restart and a reinstall, and what lets two
clients on the same database agree - the app mirrors it into
session-store.ts for synchronous reads and reconciles on launch
(useActiveEnvironmentRestore), treating the engine's value as the truth.
Cap. (issue #1485) variables is refused at write time with a 413 naming
the field, its size and the cap, when the serialized value is over
json::MAX_FIELD_SIZE (10 MiB, engine/include/vayu/core/constants.hpp) -
the same apply_json_field guard the requests and
collections columns share.
client_certificates¶
The registry that maps a host to the client certificate Vayu presents to it (issue #707). Read into the transport policy at the point of use - once per run on the load and collection paths - and applied by the shared applier every outbound transfer goes through. See api-reference.md for the routes and the matching rule.
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | cert_ + UUID |
host |
TEXT | Hostname, lower-cased, no scheme/port/path, or *.example.com for every subdomain of it (issue #803). IPv6 without brackets |
port |
INTEGER | NULL = answers for the host on every port |
cert_path |
TEXT | Path to the certificate file |
key_path |
TEXT | Path to the private key file; "" for a p12 entry, which carries its own |
cert_format |
TEXT | pem or p12 (issue #833). NOT NULL, default pem - what a row written before this column existed is |
passphrase |
TEXT | The key's passphrase, "" when it has none. Plaintext - see below |
created_at |
INTEGER | Unix ms |
updated_at |
INTEGER | Unix ms |
deleted_at |
INTEGER | Nullable, Unix ms. NULL while live; the delete's instant once it is in the trash (#2973) |
The private key never enters this database. cert_path and key_path are
paths, and the engine opens the files at send time - the strongest storage
decision available without a keystore, and the reason a registry entry survives
being copied between machines only if the files do.
cert_format is stored rather than sniffed per send. The applier runs on
every handle of every load run, so reading the file each time would pay for the
same answer thousands of times - and a format nothing recorded is one the
Settings card cannot print back for the user to correct. It is written once, at
the moment the entry is registered, from the body or from the file's own first
bytes; client_cert_rejection refuses a row whose bytes contradict it.
The passphrase is stored in plaintext, and that is disclosed rather than
hidden. It is the same treatment every other credential in this file already
gets - request auth, and oauth_tokens.access_token / refresh_token - and
libsodium, though linked, encrypts nothing here. OS-keychain storage is
explicitly out of scope for the transport epic (#704, decision 6); it would be
its own change, touching every stored credential rather than this one column.
The wire is narrower than the file: reads never echo the passphrase, answering
hasPassphrase instead.
A trashed row is never presented. DELETE /client-certificates/:id stamps
deleted_at, and both reads (get_client_certificates,
get_client_certificate) filter deleted_at IS NULL. The first is what
resolve_transport_policy reads, so the stamp takes the entry off every send at
once. An entry owns no rows, so, as with environments, its
stamp is no cohort key: it is always its own trash root, a restore clears the
stamp alone, and a purge (DELETE /trash/:id, or retention) removes the row.
The passphrase stays on a trashed row, in plaintext, until that purge - a
restore gives the entry back exactly as it was, and nothing short of the purge
removes the value from the file.
port is nullable rather than a sentinel because "every port" is the absence
of a port, the same distinction result_bodies.stream_events draws. At most one
row may claim a given host + port pair - the routes answer 409 on the
second, and a wildcard pattern is a host like any other, so the rule covers it
unchanged. Several rows may still match one transfer once patterns overlap
(#803), and they are ranked rather than tie-broken: closest host, then port. A
tie would need the same host and port twice, which is the pair the 409
forbids. The rule counts live rows only (client_certificate_for_target, the
one definition the routes and the trash share): a trashed entry frees its
target, and restoring it while a live row has taken that target is the
restore's 409.
file_roots¶
The folders a request-body file may be read from without being picked file by
file (schema version 3). A file reference (see requests.body) is
sent when its canonical path lies under one of these or is a
file_grants row; both are read once per design send and once
per run. See api-reference.md
for the routes.
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | froot_ + UUID |
path |
TEXT UNIQUE | Canonical absolute folder path, no trailing separator |
created_at |
INTEGER | Unix ms |
path is canonical when written (symlinks resolved, . and .. folded), so
two spellings of one folder are one row, and the 409 the route answers for a
second is backed by the constraint. A filesystem or drive root and the home
folder itself are never stored: the route refuses them, and the
schema version 4 migration
removed any stored before.
file_grants¶
The files a person picked, each of which may be sent from wherever it lies (schema version 6,
2356): the per-file half of the trust rule beside file_roots, read with it once¶
per design send and once per run (FileAccessPolicy::from_database).
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | fgrant_ + UUID |
path |
TEXT UNIQUE | Canonical absolute path of a regular file |
created_at |
INTEGER | Unix ms |
path is the file's canonical form when it was picked (grantable_file_path: every symlink
resolved, never a {{ template), and a send compares the canonical form of what it is about to
read with it exactly, never resolving the stored path again: a link re-pointed after the pick
resolves to another file and is refused. The
schema version 6 migration wrote the first rows,
one per file a stored body marked chosen. See
api-reference.md for the routes; like file_roots,
this table is written only by a caller holding the engine's grant secret.
globals¶
Singleton table; always has exactly one row with id = "globals".
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | Always "globals" |
variables |
TEXT | JSON: Record<string, VariableValue> |
updated_at |
INTEGER | Unix ms |
runs¶
Stores design-mode and load-test run records. Defined in database_impl.hpp (make_table("runs", …));
struct is db::Run in engine/include/vayu/types.hpp.
| Column | Type | Notes |
|---|---|---|
id |
TEXT PK | run_ + UUID |
request_id |
TEXT | FK → requests.id (optional; set in design mode) |
environment_id |
TEXT | FK → environments.id (optional) |
type |
TEXT | "design", "load" or "scenario" (a collection run) |
status |
TEXT | "pending" / "running" / "completed" / "failed" / "stopped" |
config_snapshot |
TEXT | JSON: the run's request as composed, credentials withheld (see below) |
start_time |
INTEGER | Unix ms |
end_time |
INTEGER | Unix ms; 0 = no end recorded (readers guard on > 0); the API emits it only once terminal (below) |
summary |
TEXT | JSON: whole-run results, written once at terminal status ("" = not written) |
baseline |
INTEGER | 1 when the run is pinned as a baseline. NOT NULL DEFAULT 0 |
has_warnings |
INTEGER | 1 when summary.warnings is non-empty. NOT NULL DEFAULT 0 |
origin |
TEXT | Who started the run: "app" / "mcp" / "other". NOT NULL DEFAULT 'other' |
origin_client |
TEXT | The MCP client's name, mcp runs only (nullable) |
status moves through Database::transition_run_status (#3340), which takes
an edge only when the row's current status is one the caller names: there is no
wildcard, so a run that is completed, failed or stopped stays that way unless a
caller names that exact status as the one being left. The read, the decision and the
write hold one DB mutex acquisition. A busy database is retried; a terminal write is
retried until it lands, because a lost final status leaves the run running, which
retention never prunes. A graceful shutdown waits for that worker, and a daemon killed
before the write lands leaves the row to startup reconciliation (below). A refused move
is logged (run id, the statuses named, the one found) and answered, never thrown.
A load run's phase (setup, sending, draining, summarising, settled; #3341)
is in-memory state of the active run (core/run_phase.hpp) and is not stored: status
is the only column that records where a run is, and the phase adds no column and no
schema version. What a Stop meant is visible only in the terminal status it left
(stopped when it decided the outcome; a Stop in the natural-end drain leaves the
run's own completed or failed).
Every status write after the insert goes through it, each naming its real edge. A load
or collection run is inserted pending and a design send running. A worker moves
pending to running as it starts and running to its terminal status as it ends; a
run that fails before its worker starts, or whose move to running threw, fails from
pending. POST /runs/:runId/stop on a run no worker holds (a buffered design send, or a
run whose worker has not registered yet) moves pending or running to stopped. A
Stop in that window therefore stands: the worker's move to running is refused and it
ends without sending, and a buffered send's result is stored under a row that stays
stopped. A worker that registered and moved to running between the route's look-up
and the move is signalled as well. The one move out of a terminal status is completed
to failed, when a design send throws after its result was recorded
(guard_recorded_send): its caller got no response, so the row must not say it
completed.
end_time moves with the status edge transition_run_status takes. Both inserts seed it to start_time up
front (seed_run_times, http/routes/execution.cpp). A move to running clears it to
0, so a live report measures up to now rather than to the seed; a design send,
inserted running, keeps the seed until its terminal move. update_run_end_time stamps
it when a load run's load ends, before its cleanup and summary, and when a collection
run's teardown is done, before its summary; the terminal move that follows keeps any
value later than start_time; otherwise (the seed, or 0) the terminal move stamps now. A
move to any other status leaves it alone.
A run killed by a daemon crash never reaches a terminal status:
reconcile_orphaned_runs marks it failed (with a partial summary for a load run, see
summary below) and keeps the end_time it recorded, or gives it start_time when it
recorded none (0, a run that died while running). The restart time would report a
duration spanning however long the daemon was down. This is safe without an owner column
because the engine holds the instance lock before the database opens, so no live engine
owns the rows it meets (issue #2422). db::Run::end_time defaults to 0 as the
backstop for a future insert site that forgets to seed - 0 is the "no end recorded"
sentinel, and readers (GET /runs/:runId/report, the app's dashboard) guard on > 0.
The stored seed is never what an API client reads for a live run: GET /runs, GET
/runs/:runId, the report's metadata.endTime and the baseline row emit endTime: null
while status is pending or running and the column once it is terminal, through
vayu::json::serialize_run_end_time (#2953). A run a crash left behind is failed after
reconciliation, so it reports the end it kept, or its start_time.
baseline is the pin set by
PUT /runs/:runId/baseline: the
known-good run later runs of the same request (or, for a load run, the same collection) are compared against. Several rows
may carry it at once - the engine records the pin and leaves the choice of which
baseline applies to a given run to the client. NOT NULL with a default_value,
which is what lets sync_schema ALTER TABLE ADD COLUMN it onto an existing
runs table (the same pattern as summary and requests.follow_redirects);
rows written before the column read as unpinned. Retention reads it - see
below.
has_warnings (issue #1527) is derived, never written directly: every
Database::update_run_summary call parses the JSON it is about to store into
summary and sets this column to whether that JSON's warnings array (issue
1503) is non-empty, in the same row write. The two columns are stamped¶
together and can never disagree. It exists apart from summary so
GET /runs's paginated list can read it straight
off the row it already fetched for every page - no extra query, and no risk
of the compact-summary cache (RunSummaryCache, keyed on the immutable
config_snapshot) ever caching a completion-time fact. The same cache's
second table holds a terminal load or collection run's list-row
resultSummary (#2130, #2954), built from summary and only once status is terminal: every writer stores
summary before the terminal status, so from then on the column is final.
origin / origin_client (#1817, schema version 5) record who started
the run, as the starting client asserted it on POST /execute or POST /runs
(read_run_origin, http/routes/execution.cpp; the rules are in
POST /execute). Client-asserted metadata, not
identity: any caller can claim any kind and nothing authorises on it.
origin_client is set only for mcp, trimmed and capped at 128 characters,
and stored as the client sent it. The default_value on origin is what lets
sync_schema ALTER TABLE ADD COLUMN it onto an existing table, and it is the
honest answer for every row stored before: nobody recorded one. No data
migration. The client's origin object is kept out of config_snapshot
(sanitize_config_snapshot), so these two columns are its one record.
GET /runs?origin=<kind> filters on origin
(run_filter_where, db/db_runs.cpp) with no index: History's list is already
bounded by retention and ordered by idx_runs_start_time. The column sits past
config_snapshot in the stored row, so reading it walks the snapshot's overflow
pages; like every filter there, the term is a CASE that reads its column only
when the filter is set (#2548).
summary holds the aggregates GET /runs/:runId/report used to rebuild by scanning every
metric row of the run: totals, the cumulative latency percentiles, the status-code distribution,
bytes, and the script-validation tallies. It is written once - by run_manager.cpp when the run
reaches completed/stopped, and best-effort (setup_overhead and cleanup_overhead left at
0, with a wall-clock test_duration) when it fails. setup_overhead is the seconds from the
load run's worker taking the run up to load start, its script.setup pass included;
cleanup_overhead is the seconds from the end of load to its background threads stopped (issue
2894). A summary with no cleanup_overhead was stored before that issue, and its¶
setup_overhead holds the cleanup figure: the report reads it as cleanupOverhead and leaves
setupOverhead out. "" means no terminal status has written one: the run is still
pending/running, or it failed on a path that writes none. The report then stands on the run's
sampled results alone, and a failed load run says so with summary.summaryMissing. NOT NULL
with a "" default, so sync_schema() can
ALTER TABLE ADD COLUMN it onto an existing table (same pattern as requests.follow_redirects).
A run the engine died under (issue #2895) is given a partial summary at the next start, in
the same transaction reconcile_orphaned_runs marks it failed in: a load run whose summary
is "" and that wrote at least one metric_ticks row is given the last row's
cumulative figures (one ORDER BY id DESC LIMIT 1 read per such run, get_last_metric_tick),
under this column's own keys, plus "partial": true:
{
"partial": true, "total_requests": 40, "test_duration": 4.0, "rps": 10.0, "cancelled": 1,
"send_rate": 11.0, "throughput": 10.0, "dropped_requests": 4,
"skipped_requests": 3, "stalled_ms": 300.0,
"bytes_sent": 4000, "bytes_received": 40000, "status_codes": { "200": 38, "503": 2 }
}
rps is total_requests / test_duration, not the tick's trailing current_rps. Nothing a tick
holds for its last second only (current_*, window_*, the latency percentiles) is copied, and
nothing it never measures (latency, peak_concurrency, avg_in_flight, queue_wait_avg,
error_types, sampling, the sending window, connection_replays) is written, so the report
reads each absent key as it reads an older engine's summary. The report flags the run with
summary.partial and leaves its latency block out (#2938), derived on read: a run already
reconciled by an earlier version reads without latency with nothing rewritten. A summary the run already stored is never replaced, a run that wrote no tick
keeps "", design and collection runs are given nothing, and has_warnings stays 0.
{
"total_requests": 100, "cancelled": 0, "rps": 50.0, "send_rate": 51.0, "throughput": 49.5,
"test_duration": 2.0, "setup_overhead": 0.25, "cleanup_overhead": 0.03,
"requests_sent": 102, "requests_expected": 100, "sending_duration": 2.0,
"completed_while_sending": 96,
"peak_concurrency": 8, "avg_in_flight": 4.2, "dropped_requests": 2,
"skipped_requests": 0, "stalled_ms": 0.0, "queue_wait_avg": 1.5,
"bytes_sent": 1024, "bytes_received": 8192,
"http_version_downgraded": 0, "connection_replays": 1,
"status_codes": { "200": 90, "500": 7, "0": 3 },
"error_types": { "timeout": 2, "connection_failed": 1 },
"latency": { "count": 97, "min": 1.0, "max": 90.0, "avg": 12.5, "p50": 10.0,
"p75": 15.0, "p90": 20.0, "p95": 25.0, "p99": 30.0, "p999": 35.0 },
"phases": {
"dns": { "p50": 0.1, "p95": 0.2, "p99": 1.4, "max": 12.0, "count": 100, "mean": 0.15 },
"connect": { "p50": 0.3, "p95": 0.9, "p99": 8.2, "max": 40.1, "count": 100, "mean": 0.42 },
"tls": { "p50": 0.0, "p95": 0.0, "p99": 22.0, "max": 61.0, "count": 100, "mean": 0.31 },
"firstByte": { "p50": 2.9, "p95": 4.0, "p99": 6.1, "max": 30.0, "count": 100, "mean": 3.1 },
"download": { "p50": 0.1, "p95": 0.3, "p99": 0.9, "max": 4.2, "count": 100, "mean": 0.14 }
},
"slow_requests": { "count": 6, "thresholdMs": 25 },
"tests": { "sampled": 10, "passed": 9, "failed": 1 },
"assertions": {
"exact": { "passed": 40, "failed": 0 }, "sampled": { "passed": 9, "failed": 1 },
"errored": 0, "teardownFailed": 0
},
"thresholds": {
"checks": [
{ "metric": "latencyP99Ms", "limit": 50, "actual": 30.0, "passed": true, "evaluated": true }
],
"passed": 1, "failed": 0
},
"schemaValidation": {
"sampled": 40, "checked": 36, "valid": 30, "failed": 6, "unevaluated": 0,
"uncheckedReasons": { "body_not_json": 4 },
"failures": [ { "step": "get pet", "status": 200, "path": "/id", "message": "..." } ],
"failuresTotal": 6
}
}
phases carries the whole-run distribution of each network phase, drained from the collector's
five per-phase HdrHistograms - which every successful completion feeds, unlike the sampled
results rows. Each phase's mean (issue #2782) is what the report's timingBreakdown.avg*Ms
read; a summary stored before it has no mean, and its report averages the sampled results
rows' traces instead, as every report did before. Keyed by wire name (firstByte,
not ttfb), and the only section here written in camelCase, because the report passes it through
verbatim rather than translating it. Omitted when the run recorded no distribution - the
phaseHistograms setting (or the run's own phase_histograms) off, or nothing successful
completed - which keeps the report's timingBreakdown.phases absent rather than showing five
zeroed rows; a zeroed TLS row would claim the handshake was free, which is a different statement
from "this run did not measure it". A zero inside a present section is meaningful and kept: a
run over reused connections genuinely has a TLS p50 of 0.
cancelled counts the requests a user stop cancelled before they answered (issue #2136). They
are in neither total_requests nor status_codes, which is why the count is stored: nothing
else here can recover it. Always written, 0 included; a summary stored before the issue has no
key and the report reads it as 0.
drainCut ({"cancelled": 15}, issue #3538) is written only when a Stop cut the drain of a run
that ended on its own and cancelled something (RunContext::drain_cut): its cancelled repeats
the top-level cancelled, because every request such a Stop cancels is one the drain was waiting
on. Absent on every other run, a stopped one included, never 0. The report serves it as
summary.drainCut and the list row as resultSummary.drainCut, and a verdict reader treats the
run as not judged unless a budget already failed it, since the cut requests are the slowest of the
run and in no other figure.
connection_replays (issue #2799) counts the requests libcurl re-sent on a new connection after
the server dropped or refused a reused one before answering. Each replayed request is still one
entry in total_requests and status_codes, by the outcome of its last attempt, so the target
received total_requests + connection_replays requests. Written by the completed and the failed
run alike, 0 included; a non-zero count also adds a connection_replayed entry to warnings.
A summary stored before the issue has no key, and the report then leaves
summary.connectionReplays out rather than reading it as 0.
avg_in_flight (issue #2821) is the mean number of requests in flight over the run: the
collector's in-flight integral over test_duration (MetricsCollector::average_in_flight, issue
3567), the time every request spent between its submission to the event loop and its completion,¶
a Stop's cancellation included, so timer waits between requests and requests refused before the
event loop count for nothing. A summary stored before issue #3567 holds the old figure under the
same key, the latency sum over test_duration, which left cancelled requests out; no schema
version marks the change. Written by the completed and the failed run alike, 0 included; the report's summary.avgInFlight reads it. A summary stored before the
issue has no key, and the report leaves avgInFlight out rather than reading it as 0. A
collection (design-mode) run's summary does not carry it.
error_types (issue #2400) is every error the run recorded, by the type name its stored
error traces carry (timeout, connection_failed, ...), counted by the collector as it
happened; a type with no error is left out. It is exact where the error records in results
stop at maxStoredErrors, and the report's errors.types reads it while errors.withDetails
counts the stored records. Written by the completed and the failed run alike, {} included. A
summary stored before the issue has no key, and the report then counts the stored records by
type as it always did. A design-mode collection run's summary does not write it.
slow_requests (issue #2399) is every successful completion at or past the run's
slow_threshold_ms, counted by the collector as it happened, and the threshold it was counted
against. It is exact where the slow traces in results are a sample capped at max_slow_results;
the report shows both, as slowRequests.count and slowRequests.tracesStored. Written by the
completed and the failed run alike, 0 included, and omitted when the threshold was 0,
which disables slow capture. A summary stored before the issue has no key either, and the report
then counts the stored slow traces as it always did.
requests_sent, requests_expected and sending_duration (issue #2392) are what the run
submitted, what its mode set out to submit (targetRps x duration for a rate-limited run, the
iteration count for an iterations one, 0 for an open-ended mode; a scenario iterations
run stores what it sent when it reached its own end, because its iterations x steps figure is
only an upper bound once an iteration is abandoned, issue #2914), and the seconds it
submitted in: load start to the moment it stopped submitting, before the end-of-run drain that
test_duration includes. An iterations run that reached its own end stores its whole
test_duration there instead (see the report's
two clocks). send_rate is requests_sent over
sending_duration; rps and throughput stay completed requests over test_duration, and so
does every threshold that reads them. A failed run stores its wall-clock test_duration in
sending_duration too. Always written, zeros included: a summary with no sending_duration key
predates the issue, and the report keeps that run's old rate-fidelity formula.
skipped_requests and stalled_ms (issue #2911) are the requests a constant_rps pacer owed
beyond its 100 ms catch-up window and skipped instead of bursting, and the schedule time they
cover in milliseconds (skipped_requests ÷ targetRps). They are counted apart from
dropped_requests, which is the maxInFlight cap's: for a run that reaches its deadline
requests_sent + skipped_requests + dropped_requests is requests_expected. Always written,
zeros included, by the completed and the failed run alike, and 0 on every mode but
constant_rps; a summary stored before the issue has neither key and the report reads both as
0. requests_sent / requests_expected (the report's rateControl.achievement) is not adjusted
for them: the skips are what separate a stalled generator from a slow target. The metric ticks
carry the same two cumulative keys.
completed_while_sending (issue #2882) is total_requests as it stood when sending_duration
ended, before the drain, so it is never above total_requests; the report's
summary.completedWhileSending reads it. Where sending_duration is the whole test_duration (an
iterations run that reached its own end, a failed run) it is total_requests. Always written,
0 included; a summary stored before the issue has no key, and the report leaves
completedWhileSending out rather than reading it as 0.
tests is omitted when deferred script validation did not run, which is what keeps the
report's testValidation section absent rather than reporting zero tests. Beside its counts it
carries console ([{level, text, count?}], the first 10 distinct lines the replayed scripts
logged, each cut to 500 bytes, count written only on a line logged more than once) and
consoleDropped (occurrences a full list turned away), each written only when non-empty and
non-zero (#3127). A scenario load run writes them on each step's own
scenario.steps[].tests (the same object, built by the same test_totals_payload) and never on
the run's tests. A summary stored before them has neither.
assertions (issue #3537) is every failed check a load run's verdict reads, written by both
load run kinds in one shape: exact and sampled (each {passed, failed}) are the two
populations a maxAssertionFailureRatePct check judges, exact every assert.* outcome,
required extract.* miss and inline pm.test call counted on each completion and sampled the
deferred replay that tests also counts. errored is the element executions that ended in
error (a script that threw, an element that could not run): the summed error of a
single-request run's elements rows, or of a scenario load run's scenario.steps[].elements
rows, never scenario.errored, which also counts a step that got no response. A deferred
script's throw is not in it, since the replay counts it in tests.failed. teardownFailed is
the lifecycle.teardown rows whose outcome is failed or error; a failed setup fails the
run instead. Absent when the run made no assertion and nothing errored, and every member is
written once the block is, zeros included. The failed run's summary never writes it. A summary
stored before the issue has none; the report's assertions, its scenario.assertions and the
list row's resultSummary read it through read_run_assertions (http/routes/runs.cpp), the
one place such a summary is derived from: a scenario load run's scenario.assertions as
exact, and a single-request run's assert.* and script.* rows of elements as exact
and its tests as sampled, which is what each read before. Nothing derives errored or
teardownFailed for it.
schemaValidation
follows the rule tests does (issue #682): omitted when the deferred pass over the run's sampled
responses walked nothing, so an unbound collection's report carries no block rather than one
saying its contract held. Written in camelCase because the report passes it through verbatim,
the phases rule. Its sampled count is stored rather than derived because it is the
denominator that says these tallies describe the run's bounded reservoir and not the run -
unlike coverage beside it, which is exact. thresholds follows
the same rule and for the same reason: absent when the run declared no
budgets, so the report's
thresholdValidation section is left out rather than claiming a run passed nothing. Its metric
keys are the wire names the payload declared, carried through unchanged; the report derives
verdict from failed rather than storing it, so the two cannot contradict. Each check's
evaluated follows the same absent-vs-zero rule one level down (issue #1484): a latency percentile
with no completed requests writes evaluated: false and omits actual rather than storing the
default 0, which a reader could not tell apart from a genuine 0ms measurement; such a check counts
toward failed. An evaluated maxAssertionFailureRatePct check also stores
assertionsExact and assertionsSampled (each {passed, failed}) and, when the sampled one is
non-empty, sampleRate (issue #2892), so a report read after a restart still names the two
populations its actual was the worse of; a check stored before the issue has none of them and
nothing backfills them. The writer is
vayu::core::build_run_summary_payload and the reader is apply_run_summary
(http/routes/runs.cpp); runs_route_test.cpp round-trips the pair, so the key names cannot
drift apart silently.
Do not confuse this results summary with the summary key on a GET /runs list row - that
one is a derived view of config_snapshot
(url/method/mode/duration/concurrency/comment/requestName) and is
never stored. It is built once per run id and held in a bounded in-memory cache
for as long as the process lives (issue #1150), which is sound only because
config_snapshot is write-once: create_run sets it, and every later write to a
run row - status, end time, summary, baseline, the startup reconcile - is a
read-modify-write that puts the same string back. A change that edits a stored
snapshot in place breaks that, and would have to invalidate the cache in
engine/include/vayu/http/run_summary_cache.hpp.
The list row's resultSummary on a terminal load run (#2130) is read from
this results summary (rps, latency.p95, the status_codes error rate, the
thresholds verdict, the assertions block as assertions, with its two
populations summed as assertionsPassed / assertionsFailed (#2820, #3458,
3537), drainCut as it is stored (#3538), and for a capacity run the¶
capacity section's stop_reason, max_healthy_concurrency and
knee_concurrency (#2930); build_load_result_summary in
http/routes/runs.cpp) and, on a terminal collection run, from scenario.passed
/ failed / skipped / errored / stopped and the same thresholds verdict
(build_scenario_result_summary, #2954), held in the same cache, which is
sound for the same reason: each update_run_summary writer stores the summary before the
terminal status, and nothing rewrites it after. A change that writes the
summary after the terminal status, or rewrites it later, breaks that too.
A scenario run's summary is a different shape, written by
vayu::core::build_scenario_summary_payload (core/scenario_runner.cpp): the
four keys apply_run_summary reads plus everything sequence-shaped under
scenario, which the report serves as its own section.
{
"total_requests": 5, "status_codes": { "0": 1, "200": 4 },
"http_version_downgraded": 0, "test_duration": 1.8, "rps": 2.8,
"latency": {
"count": 5, "min": 12.0, "max": 240.0, "avg": 71.6,
"p50": 40.0, "p75": 60.0, "p90": 240.0, "p95": 240.0, "p99": 240.0, "p999": 240.0
},
"scenario": {
"iterations": 3, "iterations_completed": 3, "steps_executed": 6,
"passed": 4, "failed": 0, "skipped": 1, "errored": 1, "stopped": 0,
"steps_stored": 6, "steps_dropped": 0, "steps_not_run": 0,
"stop_on_error": false
}
}
total_requests is the number of requests sent and status_codes their
distribution, in the load summary's shape (issue #2822): a step the plan skipped
is in scenario.steps_executed and in neither, while a request that got no
response is in both, under "0". Both are here precisely because the report
would otherwise read the rows that survived maxScenarioStoredSteps, call their
count the run's size, and count a skipped step's status-0 row as a failure.
status_codes is written even when empty: {} is a run in which no step sent,
and a summary with no status_codes key (a run stored before the issue, whose
total_requests is the step count) reports its split from the stored rows.
http_version_downgraded (#3262) is how many of those requests asked for
HTTP/2 and negotiated something older, the load summary's key and unit, always
written (a 0 is a run that counted and saw none, a missing key a run stored
before the issue). rps is requests sent over test_duration. steps_dropped is what
that cap thinned - always successes or stopped steps, never a failure.
stopped (#3397) counts the steps a Stop or an engine shutdown cut short; with
passed, failed, skipped and errored it sums to steps_executed, and a
stopped step is in neither total_requests nor status_codes, as a load run's
stop-cancelled request is not. A summary stored before the issue has no
stopped (read as 0), and the report and the list row then leave it out.
steps_not_run is the
plan positions errors cost (issue #2526): for every iteration an errored step
ended, the positions after it the iteration had not attempted; an iteration
that setNextRequest(null), a jump or a stop ended adds nothing.
stop_on_error is the run's stopOnError as it ran. The report serves both as
scenario.stepsNotRun / scenario.stopOnError, and only when stored: a summary
written before the issue, and a scenario load run's, carries neither. The step
row that ended an iteration short of the plan's end carries the same
per-iteration count as stepsNotRun on its trace_data, absent on every other
row, and a step row whose extractors found nothing carries extractorMisses
(#3053) on the same absent-at-0 terms.
A scenario load run's scenario object (build_scenario_load_summary in
core/scenario_load.cpp, then collect_summary_inputs in core/run_manager.cpp)
adds assertions, {passed, failed} (issue #2820): the exact assertion
population exact_assertion_totals reads, every assert.* outcome, required
extract.* miss and inline pm.test call, never the deferred sampled replay
under tests. Absent, never zeros, for a run that made none; a sequential
collection run never writes it and nothing backfills a run stored before it. It is
the same figure as the top-level assertions block's exact, which the report
serves as scenario.assertions (absent at zero, as stored); the list row reads
the block.
latency (issue #2389) is the run's latency over the steps that sent a transfer (a step the
plan skipped or one refused before the wire has no duration), exact nearest-rank over the full
sample rather than a histogram. It is computed once and is the same sample the run's thresholds
judge, so latency.p99 equals a latencyP99Ms check's actual. A run in which no step sent
stores {"count": 0} alone, and the report omits its latency block on that marker; a summary
with no latency key (a run stored before the issue) reports from the stored rows. A load run's
latency carries count too (#3540): a load run that timed no request stores zeroed percentiles
with count: 0, its report omits the latency block on that marker, and its history row omits
p95Ms. A load summary stored before count existed has none, so its stored percentiles stand.
config_snapshot redaction - the snapshot is the run payload as composed,
with its credentials withheld. Composition has already written every
{{variable}}'s value into the payload, so before persistence
sanitize_config_snapshot (utils/json.cpp) withholds three things (issue
1803):¶
- the top-level
authobject is reduced to just{"mode": "..."}- an allowlist, so no current or future auth field (clientSecret,password, tokens) leaks into a stored run; - the value of every
headersentry named in the shared secret set (Authorization,Proxy-Authorization,Cookie,Set-Cookie,X-Api-Key, ...;vayu::utils::is_secret_field_name, case-insensitive) or by the request's ownapikeyauth becomes"<redacted>", the name kept - in either shapeheadersarrives in, an object's member or each{name, value}line'svalue(issue #2466), and a repeated line every time; - every occurrence of a value of a variable marked
secret- raw, strictly percent-encoded (RFC 3986), as composition writes a query value, as it writes a URL's path, fragment and user info (a space is%20while/ @ & : + 'stay raw there, soa b/cin a path isa%20b/c; a user name and a password are spelled alike except at the user's own colon), JSON-escaped (a JSON body is stored as text, sopa"ss\wordispa\"ss\\wordinside it) and XML-escaped (character data, and each attribute delimiter) - insideurl,params,headers,body.contentandbody.fieldsbecomes<redacted>. The values come from the scopes the payload names: globals,environmentId, and the collection chain ofrequestId's request (a scenario run: of its collection and of every folder a step sits in, #3270). A value under four characters is left alone, because it recurs in ordinary text and masking it would shred the snapshot.
results.trace_data still records what was sent, credentials included, by
design (architecture). Rows stored before #1803 stay
as they were written: nothing rewrites a stored snapshot, and the schema
version did not change. A run exported as a file is held to a stricter rule than
the snapshot: POST /export/run-report
copies an allowlist of the snapshot (header and parameter names, the body's mode
and size, the URL with its credentials blanked) and none of its values.
config_snapshot's body is capped at maxTraceBodyBytes (issue #1486), the
same limit and the same sibling-key shape the results trace bodies
use - sanitize_config_snapshot truncates body.content in place and records
bodyTruncated: true / bodyBytes: <original length> on the body object when
it cuts. Without this cap a single oversized send (a large upload, a big JSON
fixture) left a permanently bloated config_snapshot behind: the one-time
summary build the cache above pays per run id, and the q filter's LIKE /
collectionId's json_valid / json_extract (run_filter_where,
db/db_runs.cpp, evaluated only when that filter is set, #2548) all scan the
stored string's full length on every call that uses them. The by-id route
(GET /runs/:id) returns whatever was stored, cap included - it never
re-derives or re-truncates.
config_snapshot for a scenario run - a run started from a scenario block
(see POST /runs) stores a step manifest, never
the resolved plan. The manifest replaces the block as sent, rather than
sitting beside it (scenario_snapshot, http/routes/execution.cpp).
{
"source": "collection", "collectionId": "col_123", "recursive": false,
"iterations": 3, "dataRowCount": 2,
"dataFileName": "users.csv", "dataFileHash": "<sha256 hex>",
"dataFileCastNumbers": true,
"steps": [
{ "index": 0, "requestId": "req_1", "name": "Log in",
"url": "https://{{host}}/login", "method": "POST" }
]
}
url is the stored, uncomposed one. The resolved plan is credential-grade -
it carries resolved Authorization headers, and an apikey auth with
in: "query" puts a live key in the composed URL - and the redaction above
reads one composed request, not a plan's steps or the auth built for them. The plan lives in
memory for the run's life and nowhere else. Inline data rows are not
snapshotted either: they are user data of unknown sensitivity, and the manifest
records only their count plus the file's name and content hash (dataFileName
and dataFileHash, each absent when the run named no file; issue #2051), never
its path, and dataFileCastNumbers: true when the client read the file's
numbers as numbers (issue #2540). That key is written only when true: absent is
the one spelling of "read as text", which is also how every run recorded before
it reads, so it needed no migration or schema version. A single-request load
run's snapshot carries the same keys at its top level (load_data_snapshot),
the flag written from the validated payload rather than the body's own copy. The writer is vayu::core::build_scenario_manifest
(core/scenario_plan.cpp). Only the scenario block is replaced: the run's
top-level options (environmentId, failOnSchemaError, stopOnError,
thresholds and the rest) stay beside it as sent, which is what keeps two runs
that differ only in one of them from reading as the same run.
config_snapshot's defaultHeaders (issue #1488) - a load or collection
run's snapshot also carries the run's resolved default-header decision at
start: {"userAgent": "Vayu/0.26.0", "requestId": false, "acceptEncoding":
true, "expectContinue": false} (expectContinue, #2516, is
DefaultHeaderPolicy::expect_continue). This is a deliberate, narrow exception to
Default request headers's "nothing
here is stored" rule - that rule is about not writing an applied default back
into a saved request, the way the renderer used to freeze X-Request-ID
into one before issue #1229. Here nothing is applied to anything: the value is
read back only to compare one run's report against another's, most often a
pinned baseline against a newer run after loadNegotiateCompression's default
changed underneath both. Written once at POST /runs alongside the rest of
the snapshot (with_default_headers_snapshot, http/routes/execution.cpp);
absent on a design run and on any run recorded before this issue.
Retention - runs are append-only in normal use (every design-mode click adds a runs row,
every load run its metric_ticks/results), so
Database::prune_runs(max_runs, max_age_days, after_selection, max_design_runs) trims the
history. The count caps are per kind (issue #2429): load and scenario runs are counted
under maxRunsRetained, design Sends under maxDesignRunsRetained, and neither kind
evicts the other, so an afternoon of Sends cannot push last week's load runs out. A run is a
victim when it falls beyond its kind's most-recent N (ordered by start_time) or its
start_time is older than runRetentionDays days, one age for every kind; each knob is
disabled by 0. Runs still running/pending are never pruned and never count
toward either cap, and neither are runs whose baseline is set: a pin that retention can expire is
not a pin, and pins counting toward a cap would let a handful of them evict the recent history
the cap exists to keep. Deletion goes through the run cascade (runs + their metric_ticks +
their monitor_samples + their results), batched inside transactions that release the DB mutex between batches so a large
backlog cannot stall /health, SSE, or the runs poll. Because victims are chosen under an
earlier acquisition, each batch re-reads its rows under its own lock and spares one that has
since been pinned, gone back in flight or been deleted (issue #1906); that batch loop is
remove_unexempt_runs. "Clear run history" (DELETE /runs,
Database::clear_runs, issue #2429) is its second caller: every stored id is a candidate, and the
same per-row exemption under the batch lock keeps baselines and in-flight runs. The cascade itself lives in one function
(remove_run_cascade_locked), which both delete_run and remove_unexempt_runs call, so a new
child table is wired into every delete at once. prune_runs_configured() reads the three
knobs (config, data_retention, defaults 200 / 100 / 30) and runs at startup (Database::init) and
after a run reaches a terminal status (design mode's record_design_result, and the load-run
completion/failure paths in run_manager.cpp). What a prune frees is disk the file keeps until
Reclaiming freed pages gives it back.
A run row read without the DB mutex (#2679). Database::get_run_snapshot reads one row
through a second connection to the file, read-only (query_only), unmapped (mmap_size = 0) and
guarded by its own small mutex, never the DB mutex. WAL lets it read the last commit while the
main connection writes, so it waits on neither a results chunk nor the automatic WAL checkpoint
inside that chunk's commit, which holds the DB mutex for 21-35 ms about every 4,500 rows, and it
does not see a transaction still in flight. It is opened on its first call, after init's
migration, reclamation and checkpoint have run, and closed before the main connection, which
stays the last to close and so the one that checkpoints and deletes the -wal. get_run stays
the read under the DB mutex, the one a read-then-write composes with.
oauth_tokens¶
Cached OAuth 2.0 access/refresh tokens, keyed by config identity. Written by the
token client (engine/src/http/oauth_client.cpp); struct is db::OAuthToken.
Auto-created by sync_schema().
| Column | Type | Notes |
|---|---|---|
cache_key |
TEXT PK | accessTokenUrl \x1f clientId \x1f credentialsId \x1f username?, then one \x1f <list>:<key>=<value>(\x1e ...) segment per extra parameter list with a sendable row, a credential-named row's value left out (recipe) - byte-identical to the app's computeOAuth2CacheKey (omits scope/audience/resource) |
access_token |
TEXT | Bearer token (plaintext at rest) |
token_type |
TEXT | Defaults to "Bearer" when the provider omits it |
refresh_token |
TEXT | "" when none |
scope |
TEXT | Granted scope, if returned |
expires_in |
INTEGER | Seconds; 0 = non-expiring |
created_at |
INTEGER | Unix ms |
Expiry is now > created_at + expires_in*1000 − skew, with skew = min(45s, expires_in/2),
so a token living less than 90s is still valid for the first half of its life. On refresh the
refresh_token rotates when the provider issues a new one; a rejected refresh
token (invalid_grant or invalid_client) clears the row, only while the row
still holds that refresh token
(Database::delete_oauth_token_if_refresh_matches, one lock acquisition, #3344),
and falls back to a fresh grant. Any other refresh failure, an unavailable
token endpoint included, keeps the row (#3360).
A load run does refresh mid-run: a run whose auth resolves to a
header-placed, expiring oauth2 token gets a watchdog thread (a collection load
run one per distinct cache key its steps send, #3199) that re-acquires
oauth2RefreshLeadMs (default 60s) before expiry, writes the new row here and
republishes the header onto every later transfer. That lead, the floor between
two renewals (oauth2RefreshMinIntervalMs), the retry backoff after a refused
renewal (oauth2RefreshRetryMs doubling to oauth2RefreshRetryMaxMs) and how
often the watchdog wakes to notice the run ended (oauth2RefreshPollIntervalMs,
which bounds how long a finished run waits to join it) are all settings under
Services, read once when a run arms its watchdog, so a change
applies to the next run started. It stays out of the way
for the shapes it cannot renew - tokenPlacement: "query" (the credential is in the
URL every transfer copies), autoRefreshToken: false, an authorization_code
grant with no refresh token, a non-expiring token, a token that lost to a
user-supplied Authorization header, and a collection step whose auth waited
on its scripts. Each refresh, and any failure, is reported in the run's auth
section; a failed refresh never fails the run. See plan_oauth2_refresh and
plan_auth_refresh (engine/src/http/auth_resolver.cpp), and
plan_scenario_auth_refresh and run_auth_refresh
(engine/src/core/auth_refresh.cpp).
Tokens are plaintext at rest (v1 posture), in a database file only its owner can read
(0600 in a 0700 directory on POSIX); the row is cleared via DELETE /oauth2/token. The
provider's raw response (refresh and id tokens included) is not stored: the column was
dropped at schema version 5.
metric_ticks¶
The time series for a load test: one wide row per persisted tick (~1/s), written by the
metrics producer thread. Struct is db::MetricTick. Auto-created by sync_schema().
| Column | Type | Notes |
|---|---|---|
id |
INTEGER PK | Autoincrement |
run_id |
TEXT | FK → runs.id |
timestamp |
INTEGER | Unix ms - the tick's single wall-clock sample |
payload |
TEXT | JSON: the complete tick object (below) |
payload is one data[] entry of GET /runs/:runId/metrics - the app's snake_case
LoadTestMetrics shape, built once at write time by
vayu::core::build_metric_tick_payload instead of being reassembled per request:
{
"timestamp": 1730000001000, "elapsed_seconds": 1.0,
"requests_completed": 120, "requests_failed": 2,
"http_errors": 0, "no_response": 2, "cancelled": 0,
"current_rps": 118.4, "current_send_rate": 120.2, "window_avg_in_flight": 1.1,
"window_avg_latency_ms": 8.9, "window_avg_queue_wait_ms": 0.2,
"current_concurrency": 10, "send_rate": 120.0,
"throughput": 118.4, "backpressure": 3, "error_rate": 1.66,
"dropped_requests": 0, "skipped_requests": 0, "stalled_ms": 0.0,
"bytes_sent": 4096, "bytes_received": 65536,
"status_codes": { "200": 118, "0": 2 },
"status_seconds": [ { "second": 0, "span": 1, "counts": [118, 0, 0, 0, 2] } ],
"latency_p50_ms": 8.1, "latency_p95_ms": 20.4, "latency_p99_ms": 31.9
}
requests_failed, http_errors and no_response count errors by the one definition
(What counts as an error); the breakdown pair is
absent from a tick an engine older than issue #2124 stored, whose requests_failed counted
no-response failures only. cancelled is the requests a user stop has cancelled so far, which
none of the three counts (issue #2136); absent from a tick stored before that issue.
current_rps, current_send_rate, window_avg_latency_ms and window_avg_queue_wait_ms read
the trailing window of the row's tick (vayu::core::TrailingWindow, issue #2393): at least the
last second, divided by its actual span, from the differences of the run's cumulative counters
across it. They are the same values the live frame of that tick carries, so a chart drawn live
and one drawn from these rows agree. The two means are null for a window in which nothing
completed. The three keys besides current_rps are absent from a tick stored before that issue,
whose current_rps was the completions since the previous row over the time since it. No schema
version marks the change: they are keys inside payload, which sync_schema() never reads.
window_avg_in_flight (issue #2821) is the same window's mean requests in flight: the time
requests spent in flight during the window over its span (issue #3567), so a request open across
the window counts whether or not anything completed. Always a number, 0 for a window with
nothing in flight. A tick stored before issue #3567 holds the latency its window's completions
carried over its span under the same key, which read 0 while a request longer than the window
was open.
current_concurrency and backpressure beside it are instantaneous gauges (curl transfers
active at the tick, and RunContext::in_flight), which a virtual user in a timer wait does not
hold, so a run whose users wait in step aliases them between 0 and the user count. Absent from a
tick stored before the issue; like the keys above, no schema version marks it.
status_seconds (issue #2866) is the seconds the row's gate closed, each with the responses that
completed in it by class (vayu::core::StatusSecondCounter): [{second, span, counts}],
second counted from load start, the clock elapsed_seconds counts from since #2881,
span 1, and counts [2xx, 3xx, 4xx, 5xx, error], where error is code 0 (no response)
and any code outside 200-599, the classes the app's status chart draws. A second is closed
250 ms after it ends, so a row usually holds one second, two when its gate's ~1 s cadence crosses two
second ends, and [] when none closed since the last gate (always before the first second closes); across a run's rows the
seconds run on from 0, and a second nothing completed in is an entry of zeros. The live gate
frame of the same tick carries the same entries as statusSeconds. The second still open when the
run ends is stored nowhere, like its percentiles, and the seconds that close after the last row ride
the final live frame only, so a run opened from History ends a point or two before the live chart did. Absent from a tick stored before the issue, whose status
chart keeps diffing status_codes between rows; like the keys above, no schema version marks it.
status_codes itself is unchanged, still the cumulative map.
Two consequences of the row being the tick:
- Pagination is tick-aligned.
GET /runs/:runId/metricspages rows, and a row is a whole tick, so a page boundary can no longer hand back a half-populated bucket (which row-paginating the retired EAVmetricstable did every ~277 ticks). elapsed_secondsis written with the row, so it keeps counting across page boundaries instead of restarting at 0 on each page.
elapsed_seconds is the seconds since load start, the instant the report's test_duration
is measured from, after the request is built and any script.setup has run (issue #2395). It is
the live stream's origin: a row's elapsed_seconds is the elapsedSeconds of the gate frame of
the same tick (issue #2881). The first row reads between 0 and about 1.0: the 1 Hz gate
counts from the metrics thread's start, so after a setup script the first stored gate can land
anywhere up to one interval after load start. No row is stored before load start: a gate that
closes while setup is still running sends its live frame and writes nothing. The cumulative rates in the same row, send_rate and throughput, are
requests sent and completed since load start, divided by the same elapsed_seconds. Rows an
engine older than issue #2881 stored count from the run's first stored row instead, which reads
0 and sits about one second behind the live clock; nothing rewrites them, and such a run may
also hold rows written during setup. Ticks an engine older than issue #2395 stored divided their
rates by the time since the run was registered, so a run with a slow setup understated both in
its early rows.
Latency percentiles in a tick (latency_p50_ms, latency_p95_ms, latency_p99_ms) cover the
row's second: every 100 ms window the hdr_interval_recorder handed back since the previous
row, merged as histograms before the percentile is read (issue #2385). The row stores no sample
count; a second with no completion stores none of the three keys (issue #2386), absent rather
than null because the run export types them as numbers. Rows an engine older than issue #2386
stored keep 0 for such a second: nothing rewrites them, so a reader of an older run still meets
a 0 that means "no completion", not "answered in 0 ms". The whole-run cumulative percentiles live in
runs.summary, never here. The partial second between the last row and the end of the
run is stored nowhere: it reaches only the final live frame. Rows an engine older than issue #2385
stored hold the last 100 ms window of their second instead, about a tenth of its completions.
monitor_samples¶
The server vitals a run scraped from a target endpoint it was configured to
watch (monitor on POST /runs): one row per successful scrape, written by the
run's own scrape thread. Struct is db::MonitorSample. Auto-created by
sync_schema().
| Column | Type | Notes |
|---|---|---|
id |
INTEGER PK | Autoincrement |
run_id |
TEXT | FK → runs.id |
timestamp |
INTEGER | Unix ms - when the engine scraped |
payload |
TEXT | JSON: the sample object (below) |
payload is one data[] entry of GET /runs/:runId/monitor, and the
data: of a live monitor SSE frame - one shape for both, built by
vayu::core::build_monitor_sample_payload:
{
"timestamp": 1730000001000,
"series": { "node_cpu_seconds_total": 3.75, "process_resident_memory_bytes": 1048576 }
}
Its own table rather than wider metric_ticks rows. That payload's key set
is the GET /runs/:runId/metrics contract, pinned by stats_route_test.cpp;
and a scrape lands on the user's own cadence (250-60000ms), which does not line
up row for row with the 1/s tick. A scrape that read nothing writes no row at
all - a stored sample with no readings would draw a line through a hole in the
data - so gaps are counted in the run summary instead, under
monitor.failures (no 2xx answer) or monitor.noMatchScrapes (a 2xx answer
carrying none of the requested names).
Deleted with the run by the same cascade (remove_run_cascade_locked).
The EAV metrics table this replaced is gone. It stored one row per
(run_id, name, timestamp) sample, ~20 rows per second of a run, and was kept read-only
after the switch so runs recorded by an older engine still rendered. Retention deletes those
runs within runRetentionDays, so the read path outlived its data; it was removed in issue #177,
along with the MetricName enum and db::Metric. sync_schema() only syncs the tables the
storage still declares - it never drops one that was removed from it - so Database::init()
issues an explicit DROP TABLE IF EXISTS metrics, and an upgraded database sheds the table and
its rows on first start. The freed pages return to SQLite's freelist for reuse; the file itself
shrinks only when the guarded reclamation at the end of Database::init() decides they are worth
a rewrite - see Reclaiming freed pages.
results¶
Individual request outcomes - all errors plus sampled successes (sampling is configurable in
MetricsCollector). Struct is db::Result.
| Column | Type | Notes |
|---|---|---|
id |
INTEGER PK | Autoincrement |
run_id |
TEXT | FK → runs.id |
timestamp |
INTEGER | Unix ms |
status_code |
INTEGER | HTTP status, or 0 for transport errors (so totals reconcile) |
status_text |
TEXT | Wire reason phrase or canonical IANA text |
latency_ms |
REAL | Perceived latency (completion − submitted_at), not wire time. On a load run's error row, the time until the transfer gave up (#2412); 0 for a request with no duration, which a real 0 reading cannot be told apart from (see Latency population) |
error |
TEXT | Error message for failures; empty on success |
trace_data |
TEXT | JSON (headers/body/timing breakdown) - design mode + errors + slow samples |
A load run's captured response headers and bodies are not here - they live in
result_bodies / body_blobs. That split is load-bearing:
Database::get_results loads every row for a run with no limit and
calculate_detailed_report JSON-parses each trace_data on every report fetch of a finished
run. At ~200 bytes per trace that is free; with bodies inline it would be megabytes read and
parsed per fetch, to compute aggregates that never look at a body. The report of a load run that
is not terminal yet reads none of these rows (#2659): it is polled while the run is in flight, and
takes its counts from the run's in-memory collector until the status turns terminal.
Written in chunks. A load run's results, with their result_bodies rows and
the body_blobs those introduce, are written by Database::add_results_batch in
chunks of RESULTS_WRITE_CHUNK_ROWS (250) results, one transaction each, staying off the DB
mutex for 1 ms between chunks (#2548): the mutex is not fair, and a writer that re-locks at once
wins it back from the reader it just woke. A reader that lands during the flush waits for one
chunk (about 7 ms in a Release build, 21-35 ms for the one whose commit runs the automatic WAL
checkpoint) rather than the whole write, and one that reads this table may see the run's results
part-written. A report poll waits for neither: its one read, the run row, takes no DB mutex
(runs, #2679). A body row always commits with its result; a failure part-way keeps the
chunks already committed. Blob dedup spans the whole flush, not one chunk.
trace_data timing keys are all in ms and carry the Ms suffix: totalMs, wireMs,
queueWaitMs, dnsMs, connectMs, tlsMs, firstByteMs, downloadMs. totalMs is perceived
latency; wireMs is libcurl's CURLINFO_TOTAL_TIME less its wait for a connection slot (#2394);
queueWaitMs = totalMs − wireMs is time spent queued before the wire, inside the generator or in
libcurl (see the split). A buffered design send adds a ninth, engineMs (#2132):
Vayu's own cost for that exchange, the handler's elapsed time less wireMs, script time and an OAuth 2.0 token fetch, taken
before the write (see One timing
convention). No other writer stores it.
The writers store different subsets, at different nesting, so read the one you need rather
than assuming all eight are there and flat (engineMs aside, above):
| Writer | What lands in trace_data |
|---|---|
Load run, success sample (load_strategy.cpp) |
timing only, flat, all eight keys. Written for a completion the 1-in-success_sample_rate sampler selects (only while save_timing_breakdown is on), or one that crossed slow_threshold_ms (which also adds isSlow / thresholdMs, and is stored whether or not the breakdown toggle is on). The two have separate retention budgets - max_success_results and max_slow_results - and an outlier never consumes a sampling slot. A scenario load run started with scenario.data adds dataRowIndex, the row that iteration bound - the only per-result record of it, since a scenario load run stores no per-step rows. |
Load run, error (load_strategy.cpp) |
an error envelope (error_type, message, request_number) with the eight keys nested under timing, present whenever totalMs > 0, plus dataRowIndex for a scenario load run given scenario.data. A data_binding_failed error is written for a request that was never sent at all - a {{data.column}} naming a column its row does not carry - so it has no timing. |
Design mode (record_design_result in execution.cpp) |
all eight keys flat, unconditionally - the same set the live /execute response carries, so a restored response shows exactly what the live one did (a skipped phase is stored as 0) - plus flat engineMs for a buffered send, the one value the live timing.engineMs returned; absent for a streaming send, an auth-failure row and any row stored before #2132. Written on every single request, alongside a nested request object plus error_type / error_message when the exchange failed and response when it did not. The two can coexist (#3423): every driver reports a failure at status 0 (mark_failed, the stream's settle_end_reason), and one that came after a header block arrived (an undecodable body, a timeout mid-body, a redirect limit, a proxy's refusal) keeps its whole response node beside the error, as the live answer shows it, so response is not a success flag. A failure that received no header block (a refused connection, an unresolved host, a failed handshake) has no response node; a proxy's answer counts as one, so headers a proxy sent on its tunnel are kept too. The response node carries headers (the folded, lower-cased map), headerLines (#2460: the lossless {name, value} list in wire order with Set-Cookie per line, written only when non-empty - absent on a load sample and on any row stored before the field, so a reader falls back to headers), redirects (#2461: the followed 3xx hops as {status, statusText, headerLines, url, schemeDowngrade} - url the request that hop answered with its credential query values blanked, schemeDowngrade true when it went over https and the next hop over http (#2949) - written only when non-empty - absent when nothing was followed and on any row stored before the field; a hop stored before #2949 lacks url and schemeDowngrade), body, httpVersion - the negotiated protocol, "" when nothing was negotiated, same convention as the live /execute response (see POST /execute) - and httpVersionDowngraded, true when the request asked for HTTP/2 and got something older; a row written before either field existed simply has no such key, so restore-response.ts must default both. Rows written by older engines omitted zero-valued phases and all of totalMs/wireMs/queueWaitMs, so readers must default missing keys (perceived total also lives in the latency_ms column). |
Load run, script validation (store_validation_failures in run_manager.cpp) |
not a captured request: the one synthetic row the post-run replay of the un-inlined script.post elements leaves when an assertion failed (status_code 0, error Script validation failures; none is written for a run that failed nothing). trace_data is {failures, failureCounts, failuresDistinct, totalFailed, totalPassed}. failures holds each distinct failure message once, in first-seen order, at most 10 (MAX_FAILURE_MESSAGES); a scenario step's name prefixes it (step name: assertion: message) and is part of its identity. failureCounts is parallel to failures: how many times each failed across the sampled responses. failuresDistinct is every distinct message the replay saw, kept or not, so failuresDistinct - len(failures) were turned away. totalFailed / totalPassed count assertions, each occurrence once, so the counts sum to totalFailed unless messages were turned away. failureCounts and failuresDistinct are absent on a row stored before #3127, which also repeated a message once per sample; a reader falls back to a count of 1 each. The number of responses replayed is the report's testValidation.samplesTested, not stored here. trace.failures is the only field that tells the row from a captured sample. |
| Scenario run, one row per step execution (core/scenario_runner.cpp) | the design-mode writer's trace exactly - it is build_result_trace - plus five keys naming the step: iteration (0-based), stepIndex, stepName, requestId and outcome (passed / failed / skipped / errored / stopped, the last a step a Stop or shutdown cut short, #3397), and a sixth, dataRowIndex, present only for a run given scenario.data - the row that iteration bound, which is the only record of which row a wrapped pass re-used. The rows themselves are never stored; the snapshot keeps the row count (dataRowCount) plus the file's name and content hash (dataFileName, dataFileHash), never its path or rows. A skipped row - a pre-request script called pm.execution.skipRequest() - carries the request node and no response, because nothing was sent; restore-response.ts already answers null for that shape rather than building a hollow 0-byte response. Bodies are capped the same way. The row count is bounded by maxScenarioStoredSteps (config, data_retention, default 5000; 0 = unlimited), biased so that every step that failed, errored or was skipped is kept and successes, with stopped steps, fill the remainder - what was thinned is reported in runs.summary, never silently. |
A streaming design run (POST /execute with "stream": true, issue #573)
adds one node the others never carry: events, holding items (the first
sseMaxStoredEvents events, each with event / data / optional sourceId /
receivedAt, and dataTruncated + dataBytes on one that hit the per-event
cap), totalEvents (every event received, whatever was stored), eventsTruncated
(true when the two differ) and endReason. cap_trace_bodies walks the
request/response body nodes and does not reach this one, so its cap is applied
when the node is built rather than when the trace is stored - which is why
eventsTruncated is computed from the true total rather than derived from
whatever the cap happened to be.
The request node also carries rawRequest - the message the transfer actually put on the
wire, the same string the live POST /execute response returns
(issue #348). It is what a restored raw-request view reads, because it is the only copy that
carries what libcurl added on our own behalf: the cookie jar's
Cookie line, Accept, the real Content-Length. Values in it are not redacted, matching
the live field's contract - it lands in the same node that already stores the resolved
Authorization header among request.headers (see Security for why
this column and runs.config_snapshot answer differently). The key is absent on a step that
sent nothing (a pm.execution.skipRequest()) and on every row written before #348, so a reader
falls back to rebuilding the view from method/url/headers/body -
restore-response.ts's sentSide is that reader.
The request node carries a second header map, sentHeaders - the record of what the
transfer issued, the same map the live POST /execute response
returns as requestHeaders (issue #664). It is headers minus the entries the transfer
suppresses (a form-data Content-Type, which libcurl writes itself with the boundary) and a
blank Content-Type or Cookie row, which is not sent (any other blank row is sent empty and
recorded as "", issue #3220), plus what the engine derives at send time: the body-implied
Content-Type and the default headers this send
added - the User-Agent, a negotiated Accept-Encoding, a correlation id where one is switched
on. None of those is written into the request row itself (issue #1229): they are applied per send,
so a stored request cannot carry a stale one. Both maps are stored because both are read, and they
answer different questions - sentHeaders is what the response pane's sent-headers disclosure
means, while headers is the request as composed, which is what a pre-request script saw and
what design-run-seed.ts reseeds a request tab from. Both are one value per name: a name the
request carries on more than one line (issue #2466) is stored with its values joined by ", "
(a Cookie's by "; "), under the first line's spelling. Beside them, sentHeaderLines is
the sent record unfolded - one {name, value} entry per line in wire order, the live response's
requestHeaderLines - written whenever sentHeaders is. Values are not redacted, same contract
as rawRequest beside it. Both sent keys are absent on a step that sent nothing, sentHeaders
on every row written before #664 and sentHeaderLines on every row written before #2466, so a
reader falls back to sentHeaders and then to headers. The load-run writers store no sent
record at all - the load driver passes nullptr for it, to keep an allocation off the hot path -
so a sampled capture's replay reads the composed map either way.
The request node also names what auth wrote (#1835): authQueryParam (string), the raw
unencoded name of the query parameter apply_auth appended to url for an API key with
in: "query" or an OAuth 2.0 token placed in the query, and authHeaders (array of strings),
the header names an API-key block filled. They exist because the URL and rawRequest cannot say
which query parameter is a credential, so a reader that masks one (the MCP server) has nothing to
go on without them. Each is absent, never empty, when auth wrote none of that kind and on every
row written before #1835; the stored url and rawRequest stay raw beside them. The live
POST /execute answer carries the same two keys at its top level (#1845), written by the same
function. That function also writes authNotice (string, #2474) when the request's auth mode
is one Vayu stores but does not execute (digest, aws, ntlm, hawk, oauth1, edgegrid,
jwt): the request went out without authentication and the notice says so, naming the mode. It
says so too for a bearer token or API key name that is empty, which writes nothing (#3203).
It is also written when an OAuth 2.0 token placed in the header yielded to the request's own
Authorization header and was not fetched (#3202). Absent when empty, on the same rule.
The design-mode request.body and response.body are capped at maxTraceBodyBytes
(config, data_retention, default 5 MiB) before storage, so downloading one 50 MB response does
not live in SQLite forever. request.rawRequest ends with that same body and is capped to the
same limit, body half only - its header block is never cut, being the reason the field is
stored at all. When a body is cut, its node gains two keys:
Key on request / response |
Type | Meaning |
|---|---|---|
bodyTruncated |
bool | Present and true only when the stored body is a prefix, not the whole body |
bodyBytes |
int | The original body length in bytes; the stored body is at most the first maxTraceBodyBytes of it (see below for where the cut lands) |
response carries bodyBytes cut or not (#3423), written by build_result_trace before the
cap, which keeps it; a request carries it only when cut. On every response it is a byte count,
which a reader cannot recount from body (a JSON string's length is not its UTF-8 size): the
length the engine held, which for a bodyCapped response is the prefix the read cap kept, and
for a streaming run the bytes the server delivered (#3238), which can exceed what body holds
(none for text/event-stream, at most maxDesignResponseBodyBytes otherwise). A row stored
before #3423 carries it on response only when cut or streamed.
A response.body that is not text (not valid UTF-8 over its whole length, or carrying a NUL) is
stored as its base64, and only then does response carry bodyEncoding: "base64" (#2458):
the same decision and spelling the live POST /execute answer makes for bodyRaw, including
that a body the read cap cut (bodyCapped) is judged, and stored as text, without an incomplete
last UTF-8 character. Absent means
the stored body is the text itself, which every row written before the key holds. Such a body is
capped by its decoded bytes, cut at a multiple of three so the stored slice still decodes, and
its bodyBytes counts decoded bytes.
A text body's cut is on a character boundary (#3423): an incomplete last UTF-8 character the
byte count would leave is dropped (without_incomplete_utf8_tail), so the stored body can be up
to three bytes shorter than the cap, and a reopened body never ends in a U+FFFD the server did
not send. rawRequest's body half is cut the same way. A header value off the wire can still
carry a byte that is not UTF-8, so record_design_result dumps the trace with
error_handler_t::replace, turning it into U+FFFD rather than throwing. The cap is applied by
vayu::json::cap_trace_bodies (utils/json.cpp) to the trace build_result_trace
(http/request_exchange.cpp) produces. It applies to the design-mode writer and
to the scenario runner, which shares that trace builder - the load-run writers
store timing/error envelopes, not bodies.
That design-mode subset is what rebuilds the request builder's response pane (Timing tab included)
after a restart - see app/src/modules/request-builder/utils/restore-response.ts, which surfaces
bodyTruncated/bodyBytes as a "body truncated for storage" notice, and restores a failed
exchange that kept its response node as the failed exchange with the server's headers beside its
error.
A design run has exactly one results row. GET /runs/:runId serves it (as result)
alongside the run itself, in addition to GET /runs/:runId/report's results array - the
same row, read by two routes for two different callers.
result_bodies¶
The response captured for one sampled load-run result, one-to-one with a results row.
Struct is db::ResultBody. Read only by GET /runs/:runId/samples;
nothing on the report path touches it.
| Column | Type | Notes |
|---|---|---|
result_id |
INTEGER PK | The results.id this exchange belongs to (not autoincrement) |
run_id |
TEXT | FK → runs.id; what the run cascade and the endpoint filter on |
headers |
TEXT | JSON object of the response headers, as received |
blob_id |
INTEGER | FK → body_blobs.id, or 0 when no body was stored |
body_bytes |
INTEGER | Size of the body as received, before any truncation |
truncated |
INTEGER | 1 when the stored bytes are a prefix (maxSampleBodyBytes) |
binary |
INTEGER | 1 when the body was stored as a descriptor rather than as text |
content_type |
TEXT | The response's Content-Type, "" when it sent none |
stream_events |
INTEGER | Events the transfer delivered when it was a stream; NULL otherwise, and NULL on every row written before 0.17.2 |
Which completions get a row. Not a uniform sample - a uniform slice of a 30M-request run is a thousand identical 200s. Three buckets, all decided before anything is copied:
| Bucket | Bound |
|---|---|
| Every error | maxStoredErrors (the error store's own cap) |
| Slow outliers | max_slow_results, the existing slow-request reservoir |
The first EXEMPLARS_PER_STATUS (3) of each distinct status code |
max_exemplar_results (64), and unlike its neighbours not a reservoir - an exemplar that gets displaced is not an exemplar |
The buckets overlap, and the overlap resolves toward the other store: an outlier that is also one of its status code's first three stays charged to the slow budget, and a sampled completion stays charged to the sampling budget. The exemplar store holds only what no other budget wanted. Claiming an exemplar is what decides that a body is captured, separately from which budget pays - so a completion that is both sampled and an exemplar is stored as a sample and still keeps its body.
stream_events is a count, not a list. The stored body of a streaming
sample already is the text/event-stream bytes, so
GET /runs/:runId/samples parses the
events back out of it with the same SseParser the live path feeds rather than
storing a second copy that could disagree with the first. What a reader cannot
recover from those bytes is how many events the transfer actually delivered -
the body may be a prefix - so that one number is stored beside them. NULL means
"this was not a stream", which is why the column is nullable rather than
defaulted to 0.
A uniformly sampled success (success_sample_rate) is deliberately body-less.
Budgets. maxSampleBodyBytes (config, data_retention, default 32 KiB) caps a single body;
maxSampleBytes (default 2 MiB) is the whole-run budget. Once the run budget is spent, samples
keep their headers and metadata and lose only their bodies - the row then has blob_id = 0 with
body_bytes > 0, and runs.summary's sampling.sample_bodies_dropped counts them, so the UI
can say the set is incomplete rather than presenting a biased subset as the whole story. Both
defaults are far below design mode's maxTraceBodyBytes (5 MiB): a design run stores one
exchange the user asked for, a load run stores tens nobody asked for individually.
Binary bodies. Since issue #1229 the engine negotiates compression by default and libcurl
decodes what it negotiated, so a gzip response is stored decoded - but a request carrying its own
Accept-Encoding, or a run with loadNegotiateCompression off against a server that compresses
anyway, still yields compressed bytes; images and protobuf arrive that way regardless.
Those are stored as a descriptor (binary = 1, blob_id = 0, with body_bytes and
content_type), never as text - error_handler_t::replace would keep dump() from throwing and
hand the reader a mojibake that reads like a real response. The rule is
vayu::core::looks_binary (core/sample_capture.cpp): a content type that is not text-shaped,
or a bounded prefix that is not valid UTF-8 / contains a NUL.
No redaction. Captured data is stored verbatim, consistently with design-mode traces, which
already store request headers as sent. A response Set-Cookie is captured along with everything
else. The mitigation is the run's own marker - sampling.response_bodies_captured in
runs.summary - which the Samples tab reads to warn, plus the run cascade below, which makes
maxRunsRetained (the load and collection run cap) the expiry for anything credential-shaped a
capture picked up.
Per-run request. There is deliberately no per-sample request copy: a load run's request is
constant across iterations and already lives in runs.config_snapshot, and the event-loop path
never populates Response::request_headers at all (only the synchronous client.cpp does).
body_blobs¶
One row per distinct captured body within a run - the dedup table. Struct is db::BodyBlob.
| Column | Type | Notes |
|---|---|---|
id |
INTEGER PK | Autoincrement; result_bodies.blob_id points here |
run_id |
TEXT | FK → runs.id; scopes dedup to one run |
hash |
TEXT | Lowercase hex SHA-256 of content (vayu::core::body_digest) |
content |
TEXT | The stored bytes, already truncated to maxSampleBodyBytes |
content is read back whole. Capture classes a body as text from its first 1024 bytes alone,
so a body with a zero byte further in is stored here as text, with all its bytes, and reads back
with all of them through the same db::BodyBytes mapping inbox_requests.body uses (#3159). The
column is still TEXT and the write always bound the full length, so rows stored before the
mapping read whole too. GET /runs/:id/samples lists such a body as base64
(bodyEncoding).
Load-test responses are overwhelmingly identical, so 1000 samples of one 2 KiB body store 2 KiB, not 2 MB. The digest is taken over the stored (already truncated) bytes: two bodies that differ only past the truncation point are byte-identical as stored, and storing them twice would be storing the same row twice.
Dedup is scoped per run rather than globally so that deleting a run deletes its blobs with no
cross-run refcount to maintain. Both tables are removed by remove_run_cascade_locked - bodies
before the results they hang off, so a delete interrupted between the two leaves results without
bodies rather than body rows pointing at nothing.
Both tables are new in 0.15.0. sync_schema() creates new tables outright, so there is no
migration: an existing database picks them up on the next startup and older runs simply have no
rows in them.
inbox_requests¶
What a webhook inbox listener captured. Struct is
db::InboxRequest.
| Column | Type | Notes |
|---|---|---|
id |
INTEGER PK | Autoincrement; also the SSE event id on GET /inbox/:id/live |
inbox_id |
TEXT | The inbox that recorded it - not an FK; inboxes are in memory |
received_at |
INTEGER | Unix ms |
method |
TEXT | Any verb cpp-httplib routes (GET/HEAD/POST/PUT/PATCH/DELETE/OPTIONS) |
path |
TEXT | Raw path as received (percent-escapes kept), no query |
query |
TEXT | Raw query string, without the ? |
headers |
TEXT | JSON object; a repeated name is joined with , |
body |
TEXT | Stored bytes, truncated to the inboxMaxBodyBytes setting (default 64 KiB) |
body_bytes |
INTEGER | Size as received, which exceeds length(body) when truncated |
body_truncated |
INTEGER | 1 when body is a prefix |
remote_addr |
TEXT | Peer address the request arrived from |
body is read back whole. A capture holding a zero byte (abc\0def) is stored with all its
bytes and lists with all of them - as base64, since NUL is never text - through the same
db::BodyBytes mapping request_examples.body uses (#3112).
Bounded as it is written. add_inbox_request inserts and trims in one transaction, keeping the
newest inboxMaxCaptures rows per inbox (default 500, settable 1-10000) and deleting the oldest by
id - insertion order, which for an append-only table is arrival order and, unlike received_at,
cannot tie. A capture the caller was told had landed can therefore never be missing its insert, nor
an untrimmed table its bound. The retention and body limits are resolved by read_inbox_limits
once, when the inbox starts, so every row belonging to one inbox was truncated and retained by the
same rule - see api-reference.md for the settings.
Cleared at startup. An inbox lives only as long as the engine process that opened it, so after
a restart every row here belongs to an inbox nothing can list. Database::init calls
clear_inbox_requests_all for the same reason it reconciles orphaned runs: the previous process's
leftovers are dealt with before anything can read them. Rows are stored at all - rather than kept
on the heap - so a long-lived listener's capture list is paged off disk instead of growing a
running daemon's memory.
New in 0.16.0. sync_schema() creates new tables outright, so there is no migration.
config_entries¶
Engine configuration registry - each tunable setting with UI metadata. Read by GET /config,
written by POST /config. Struct is db::ConfigEntry.
| Column | Type | Notes |
|---|---|---|
key |
TEXT PK | e.g. workers, dbCacheSize, liveTickIntervalMs |
value |
TEXT | Current value (parsed per type) |
type |
TEXT | "integer" / "string" / "boolean" / "number" / "enum" |
label |
TEXT | Display label |
description |
TEXT | Help text |
category |
TEXT | Which sidebar row it renders under; one of general_engine, network_performance, services, observability, data_retention, limits, scripting_sandbox |
default_value |
TEXT | Default as string |
min_value |
TEXT | Optional minimum (numbers) |
max_value |
TEXT | Optional maximum (numbers) |
options |
TEXT | JSON array of {value, label}; "enum" entries only |
updated_at |
INTEGER | Unix ms |
requires_restart |
INTEGER | Boolean; the running engine keeps the old value until restarted |
advanced |
INTEGER | Boolean; an internal, rendered collapsed under "Advanced" |
keywords |
TEXT | JSON array of extra search terms; "[]" when the entry declares none |
unit |
TEXT | What a numeric value measures (ms, sec, days, bytes); NULL for a count |
depends_on |
TEXT | Key of a boolean entry in the same category this one means nothing without; NULL when the entry stands on its own |
A row can outlive the catalogue that wrote it (issue #1492). seed_default_config
only touches two lists on every start - the keys it upserts and a fixed retired-key
list it deletes - so a row for a key neither list names (a workspace a newer engine
opened, then reopened by this one) is left untouched: still on disk, still readable by
get_config_entry, but not one this build's seed_default_config declared. GET
/config and a successful POST /config's echoed entries both filter to
Database::is_known_config_key, the set seed_default_config populated on this
start, rather than to "every row in the table" - so neither ever describes a setting
this build has no label, no validator and no bounds for. POST /config refuses a
write to such a key the same way it refuses a key with no row at all: "Unknown config
key", invalid_config. is_known_config_key answers from that populated set, never
from whether a row exists - a row existing is exactly the case this guards.
category is a closed set, and it is the app that closes it: the renderer
draws one sidebar row per category it declares
(app/src/modules/settings/engine-categories.ts) and drops an entry whose
category it does not know, so a value outside the list above is a setting with
no screen. seed_default_config is the only writer, and two guards pin the two
lists against each other from opposite sides:
ConfigRouteTest.EverySeededEntrySitsInADeclaredCategory reads the seeded rows
against a set copied into the C++ test, and engine-categories.test.ts reads
the seven engine/src/db/config_seeds/*.cpp files (#1611) against the
renderer's registry - so a category added to the seed and to the C++ set but
forgotten in the app still fails. Reseeding rewrites an existing row's metadata
while keeping its value, which is how a retired category
(database_performance, folded into general_engine in #586) carries an
upgraded database across with nothing to migrate by hand.
depends_on (issue #1610) names a relation, never a value: correlationIdHeader
carries depends_on: "correlationIdEnabled", since the header means nothing
until the id is switched on. ConfigSeeder::validate_dependencies, called once
every category's seed function has run, refuses a depends_on naming a key
nothing seeded, a key in a different category, or a key that is not itself
"boolean" - the same refusal shape as a key seeded twice. GET /config
serializes it as dependsOn, omitted when absent; POST /config never reads
it back, because it describes the catalogue, not a stored value. The settings
screen (SettingsMain.tsx) nests the dependent's card immediately beneath its
parent's, indented and disabled until the parent reads true.
The seed itself is one file per category (issue #1611):
Database::seed_default_config (engine/src/db/db_config.cpp) is the ordered
list of seven calls, one per engine/src/db/config_seeds/{general, network,
services, observability, data_retention, limits, scripting}.cpp - each file
holding exactly the ConfigEntry literals for the category its name says, over
the shared ConfigSeeder and helpers in config_seeds/seed.hpp. Adding a
setting is editing the one category file it belongs to; adding a category is
one new file and one call.
A stored value is the user's, except where an entry opts in to follow a
changed default (#2419). A seed call written seed (entry, adopt_new_default)
moves a row whose value still equals the default its previous build stored to
the new default, once, and records that default so the next start finds nothing
to do; a value the user moved off the old default is kept, and so is every entry
that does not opt in. Opt-in is per entry because a default can differ with no
build having changed it: workers seeds from the machine's core count, and a
workspace file moved to another machine must keep its row. Only dbSynchronous
opts in.
requires_restart / advanced / keywords are NOT NULL with a default_value
(false, false and "[]"), so
sync_schema adds them to an existing table with ALTER TABLE ADD COLUMN
rather than a copy-and-recreate. They are metadata, not user data: every startup
re-seeds each row's metadata (values are preserved), so the backfill only holds
between the ALTER and that upsert. requires_restart replaced a
"(Requires Restart)" suffix in label that the app and the MCP tool surface
both parsed out of the prose - one stale label misinformed both at once, and a
test now asserts no label carries the substring.
keywords is JSON-in-TEXT like options, but never null: an entry with no
terms stores "[]", so GET /config always sends an array and no client
branches on absent-vs-empty. It holds what a user types that the entry's key,
label and description never say ("ram" for dbCacheSize), which is why a
seeded keyword may not repeat a word those three fields already carry - the
search matches them first and ranks them above keywords, so a duplicate only
lifts the entry over better matches. config_route_test.cpp scans the seeded
catalogue for both rules.
unit is nullable, like min_value / max_value: an entry that measures
nothing declares none, and NULL is that - a nullable column is ALTER-friendly
without a default_value. It is what lets the app put the unit on the input
where a unit belongs, and "bytes" additionally selects the human-readable
size formatting. The app used to decide that last part from a hardcoded list of
three keys, which is the "one branch defines, another re-derives" shape: an
entry seeded here was invisible to it until someone edited a TypeScript array,
and two of the three keys had already been retired out from under it.
config_route_test.cpp guards that every key ending Ms / Bytes / Days
declares the matching unit, that no non-numeric entry declares one, and that no
description restates a unit the input already carries.
options is nullable and populated only for type: "enum" entries. It is
JSON-in-TEXT, the same convention as every other structured column in this
schema (variables, auth, config_snapshot, ...), never a delimited string.
Labels travel with values so the Settings UI never holds its own
value-to-label map that could drift from the engine's. min_value / max_value
were considered and rejected as a place to carry the option list - they are
engine-side validation only and unread by the app, whereas options is part of
the client contract: the renderer cannot draw the dropdown without it.
Three entries are seeded as enum today (via ConfigSeeder in
engine/src/db/config_seeds/).
Two of them derive their options from the C++ enumeration that gives the
values meaning, rather than from a literal list, so the picker cannot offer a
value the engine rejects or omit one it accepts.
defaultHttpVersion derives from the same HttpVersion enumeration that
validates requests.http_version - see requests above:
[
{"value": "auto", "label": "Auto"},
{"value": "http1.1", "label": "HTTP/1.x"},
{"value": "http2", "label": "HTTP/2"}
]
value is this instance's current global, read fresh (not cached) on
every request create; changing it applies to the next request created, never
retroactively.
proxyMode derives from all_proxy_modes()
(engine/include/vayu/http/transport_policy.hpp), the same enumeration
resolve_transport_policy parses the stored value back into:
[
{"value": "environment", "label": "From environment"},
{"value": "system", "label": "From system"},
{"value": "manual", "label": "Manual"},
{"value": "off", "label": "None"}
]
string entries, proxyUrl, proxySystemUrl and
proxyBypass, and the four are read together as one policy at the point of use
- see Proxy settings for the values, the
cross-field rule POST /config enforces over proxyMode + proxyUrl, and why
proxyUrl holds credentials in plaintext exactly as oauth_tokens does.
proxySystemUrl is the one config row the app writes rather than the user:
the Electron main process resolves the operating system's proxy through
Chromium and stores the answer there for system mode to read. It is a visible
row rather than a hidden channel, because a proxy a user cannot see is the
failure this whole area exists to end.
customCaCertificates is the one text entry - a multi-line string, which
is a rendering distinction rather than a value-space one: the app draws a
textarea for it because what it holds is pasted PEM whose line breaks are the
format. The certificates themselves are public, so nothing here is a
credential; the engine materializes them as ca-bundle.pem beside this
database (see TLS trust settings), which
is derived state a delete only costs a rewrite.
dbSynchronous is the exception: SQLite's durability levels ("0" Off,
"1" Normal, "2" Full) are its enumeration rather than ours, so that one
list is literal. After a power cut or an operating-system crash, Off can
leave the whole workspace file corrupt; Normal, the default, cannot corrupt
it but may lose the last few seconds of writes (in WAL it syncs only at a
checkpoint, so a commit costs no fsync); Full keeps every committed write at
the cost of a sync per commit. A workspace still holding the old default (Off,
before #2419) moves to Normal once at the first start of the build that changed
it; one that had picked Normal or Full keeps its choice.
Indexes¶
Declared alongside the tables in make_vayu_storage() (engine/src/db/database_impl.hpp). sqlite_orm
requires index arguments to precede the table arguments. sync_schema() creates them on startup
for fresh and pre-existing databases, so adding an index is additive and needs no migration step;
it still counts as a mapping change for the schema version.
| Index | Column | Query paths that rely on it |
|---|---|---|
idx_metric_ticks_run_id |
metric_ticks.run_id |
get_metric_ticks_paginated / count_metric_ticks (every GET /runs/:id/metrics), get_metric_ticks_since (the legacy SSE poll), and the remove_all in the run cascade |
idx_monitor_samples_run_id |
monitor_samples.run_id |
get_monitor_samples_paginated / count_monitor_samples (every GET /runs/:id/monitor) and the remove_all in the run cascade |
idx_results_run_id |
results.run_id |
get_results and the remove_all in delete_run |
idx_result_bodies_run_id |
result_bodies.run_id |
get_result_bodies_paginated / count_result_bodies (every GET /runs/:id/samples) and the remove_all in the run cascade |
idx_body_blobs_run_id |
body_blobs.run_id |
The remove_all in the run cascade |
idx_requests_collection_id |
requests.collection_id |
get_requests_in_collection (every sidebar load) and cascade delete |
idx_collections_parent_id |
collections.parent_id |
The cascade-delete BFS in Database::delete_collection, which does one lookup per node in the subtree |
idx_runs_start_time |
runs.start_time |
get_all_runs and get_runs_paginated, which sort start_time DESC on every GET /runs, and get_runs_before, which seeks it to a before cursor (#2978). The id tiebreak needs no second column: the plan sorts only the rows sharing one start_time, which the LIMIT bounds |
idx_inbox_requests_inbox_id |
inbox_requests.inbox_id |
Every inbox read - the capture page, the live poll, the per-insert retention trim and DELETE /inbox/:id/requests |
idx_runs_request_id |
runs.request_id |
GET /runs?requestId= and useLastDesignRunQuery's single-run lookup (get_runs_paginated with a request_id filter) |
idx_request_examples_request_id |
request_examples.request_id |
Every example read - GET /requests/:id/examples, the create path's append scan, and the remove_all in both the request and collection cascades |
idx_requests_deleted_at |
requests.deleted_at |
Database::get_trash's request listing (GET /trash), which otherwise scans every request row and walks the overflow pages of a large body to reach the stamp (#3006). Partial: WHERE deleted_at IS NOT NULL, so it holds only the trashed rows and a live write never touches it; a query reaches it only by filtering deleted_at IS NOT NULL, which is what the listing does |
metric_ticks and results are the unbounded-growth tables - a load run writes one tick row per
second (the retired EAV metrics table cost roughly 20 rows for the same second) - so without run_id
indexes a lookup slows down with every run ever recorded, not just the current one. collections.parent_id is a nullable column; sqlite_orm indexes it
without special handling.
Guarded by DatabaseTest.CreatesIndexesOnFreshDatabase and
DatabaseTest.RecreatesIndexesOnExistingDatabase in engine/tests/db_test.cpp, which read
sqlite_master directly rather than trusting what sqlite_orm reports about itself.
VariableValue shape¶
Used in collections.variables, environments.variables, and globals.variables:
{
"value": "https://api.example.com",
"enabled": true,
"secret": false,
"type": "string",
"createdAt": 1784967810149
}
secret is a UI masking hint only - values are not encrypted at rest. type is a UI/script
conversion hint, one of "string" (default), "number", "boolean", "json" - it controls
how scripts read the variable via pm.*.get(...). A Postman import stores a variable's declared
string / number / boolean type here, and on a collection or folder also an optional
description - read by nothing but the Postman exporter, which writes it back; the app's
variables editor does not show it.
createdAt (ms epoch) is the app's row-ordering key: the variables editor lists a scope
oldest-first. It is optional - a row written before the field existed, or stripped by an
engine older than the fix for issue #135, simply has none, and the app sorts an absent value as
older than everything. Neither side may backfill it on an existing variable: stamping a legacy
row at save time is what made it leapfrog the row the user had just added. Only the three places
that genuinely create a variable stamp it - the app when the user types a new row, the engine's
pm.*.set() when a script introduces a key that did not exist, and the Postman importer
(to_var_record, #3252), which stamps each row of a collection's, folder's, environment's or
globals' variable array with one clock reading per import plus the row's position, so the
file's order survives the sorted blob. A re-import into the globals scope through
POST /import keeps a replaced global's stored createdAt, or its absence
(merge_globals_locked).
The engine round-trips the whole shape through vayu::json::parse_variables /
serialize_variables (engine/src/utils/json.cpp) when POST /execute persists script-set
variables. A field added here must be added to both, or a design run erases it from disk;
engine/tests/script_variables_test.cpp pins the round trip field by field. POST /execute
also skips the write entirely for a scope no script changed, so sending a request no longer
touches a collection's / environment's updated_at.