Request Storage Design¶
Overview¶
This document explains how requests are stored and executed in Vayu, particularly regarding variable resolution.
Architecture¶
1. Request Definitions (Database)¶
Location: requests table
Format: Templates with variables
Requests are stored WITH variables (e.g., {{baseUrl}}/api/users) in the database. This allows:
- ✅ Reusability across different environments
- ✅ Easy updates to request templates
- ✅ Environment-specific variable values
Example:
{
"id": "req_123",
"name": "Get Users",
"method": "GET",
"url": "{{baseUrl}}/api/users",
"headers": {
"Authorization": "Bearer {{token}}"
}
}
The url / params invariant¶
Enabled query parameters live inside url. params[] mirrors them for the
editor, disabled entries included.
url is the wire truth: every execution path - design Send, collection scenario
run, load run - sends its query verbatim, and no send path reads the query
rows of params[]. The Params table maintains the invariant on the app's side
by rewriting url on every edit, keeping disabled rows in params[] only.
The engine maintains it on every write (issue #3551): a POST or PUT
/requests that states url and no params has its query rows reconciled with
the URL by the same rule the URL bar uses (below), so a request written from a
URL alone (an MCP or API caller) stores one row per pair, and the exports and
the spec diff, which read params[], see its query. Rows that already agree
are left byte for byte; a write that states params stores them as sent (see
engine/api-reference.md).
A path row ("in": "path", issue #1764) is the one row the engine reads:
url keeps the :name segment verbatim and composition writes the row's value
into it at send time (see
engine/api-reference.md). A path row
never joins the query.
The URL bar keeps the same invariant the other way: typing into it merges
the parsed query into params[] rather than replacing the list, so a disabled
row (invisible in the query by design) survives, and a key removed from the URL
is removed from params[] too, rather than left behind as a row the URL no
longer carries (mergeParamsFromUrl, issue #1482). Enabled rows take the
URL's pair order, so the table's row-ordered rebuild writes back the order that
was typed, and a disabled row keeps its slot after the enabled row that
preceded it (issue #3426).
Query rows hold the query as the URL spells it (issue #1771), which is
Postman's own row model: ?q=a%20b is the row a%20b, never decoded, in
either encoding mode. Decoding is not reversible - %2B would come back as a
+ a server reads as a space, and %2541 as %41 - and because the query
encoding never encodes %, a raw row written back is the pair it came from.
A row is written into url with Postman's query rule (postman-url-encoder's
QUERY_ENCODE_SET plus unparseSingle's additions): space, ", #, ',
<, >, controls and non-ASCII (per UTF-8 byte) become %XX, & does in
both key and value, and = only in a key. Everything else, +, |, % and
%XX included, goes out as typed, and a whole {{variable}} token is kept.
disableUrlEncoding writes rows as typed. The app's
encodeQueryComponent (modules/request-builder/utils/query-encoding.ts) and
the engine's encode_query_component are pinned to one another by
engine/tests/fixtures/query-encoding-conformance.json. A row with an empty
value is written as key=, and one with valueless: true (Postman's
"value": null, a URL's ?flag) as a bare key (issue #1772). A pair a
script's query.add or addQueryParams writes is a row and takes the same
rule. The value a stored {{variable}} token resolves to at send time is URL text instead, as
Postman re-parses the substituted URL: its /, &, =, ? and # are
structure (issue #1773), and the rest is encoded by Postman's set for the
path, the query or the fragment it lands in, the scheme and authority taking
it as it stands (issue #2465). The stored URL itself stays as typed; at send
time the whole URL is written as Postman writes it (each component by its own
set, a backslash before the query as /, the host lowercased), so /a b"
goes out as /a%20b%22 while an escape such as %2F is kept, and with
disableUrlEncoding nothing is.
A Params table edit rewrites only the pairs it touches. Each enabled row
takes the first unused pair of the current url with the same key and value
as written, and keeps that pair's bytes; then a row carried over unchanged from
before the edit (same id, key and value) may take a pair that percent-decodes
to it, which is the case an older version's decoded row needs. An edited or new
row is never matched decoded - + typed over %2B is written + - and a row
with no pair is encoded. The query ends at the fragment, which is put back
after the rebuilt query. So a pair written by another rule - a row an
older version stored decoded (+05:00 for %2B05%3A00), an Insomnia, JMeter
or OpenAPI import's encodeURIComponent join, a Postman key= - survives an
edit of another row. Nothing re-encodes a stored url: it is sent as stored,
apart from the send-time encoding above, and an import writes it once.
Path variables are the one kind of row the engine reads (issue #1764). A
row with "in": "path" names a :name segment the URL keeps verbatim (key
without the colon), and never reaches the query: the query builders skip it,
and at compose time the engine puts the value of the last enabled row with
that key into its segment, {{variable}}-resolved and then percent-encoded
with Postman's path encode set (space, ", <, >, `, #, ?, {,
}, controls and non-ASCII; a / in a value makes more segments, as it does
in Postman); an empty value or no enabled row leaves :name literal. An inline
POST /compose always carries the editor's path rows as request.params
([] when there are none), since the editor may be ahead of the saved row and
an absent key falls back to the stored rows. A row keeps every member it
arrived with (type, description from a Postman import) through edits. The URL bar keeps the rows in step with
the segments, but only on an edit that changes them: a query-only edit leaves
every path row alone (a declared row no segment uses included), and a segment
renamed on its own keeps its row's value (syncPathRows in
modules/request-builder/utils/path-variables.ts). A :name with no stored
row is not given one until its value is edited: the Params tab shows an empty
row for it (displayPathRows), so opening a request does not change it, and
an import stores only the rows the source declares.
A writer that stores the query only in params[] (stating params with no
query in url) therefore stores a request that sends nothing of it; the
write-time reconcile runs only for a write that leaves params out. That was issue #590: every importer split the query
out of the URL, so an imported request dropped its query on every send until the
user happened to edit the Params table once. Imports now restore the invariant at
parse time (parseImport, see
app/import-collections/README.md).
An empty body keeps its mode¶
A content-based body (json, text, graphql, jsonrpc, xml) is stored as
{ "mode": "<mode>", "content": "<string>" }, and an empty content is still a
body of that mode - clearing the editor to retype it is not the same action as
picking None. The save path tests bodyMode alone rather than falling back
to mode: "none" when the content is an empty (falsy) string, which used to
collapse a cleared JSON/text/XML/GraphQL/JSON-RPC body to none on save (issue
1490) while the mode selector kept showing the old mode until the next reload.¶
A file body stores a path, never bytes¶
A binary body is stored as { "mode": "binary", "file": { "src", "fileName"?,
"contentType"? } }, and a form-data file part carries the same members on
its row; a text part may carry contentType alone, the type it declares for
itself (#2539). src is the absolute path as written; the file itself is never
stored, so a request moves between machines as a name plus a path to relink.
No trust is stored either (#2356): a path is sent when it lies under an allowed
folder (file_roots, Settings > Files) or is a file a person picked
(file_grants), decided by the engine at each send. Every read of a stored
body answers a derived unresolved - true for a path this machine would
refuse - for the editor's "Not chosen on this machine" line; the key a client
writes is dropped, and the schema-version-6 migration granted every file the
editor had stored as chosen before removing the key from every row.
A send's history record keeps request.bodyFile {fileName, size, sha256} for a
binary body - never the bytes or the path; see
the engine API reference.
2. Request Execution¶
Process:
1. User clicks "Send" in the UI
2. Frontend resolves variables using the selected environment
3. Resolved request is sent to /execute endpoint
4. Backend executes the HTTP request
5. Backend stores both the resolved request AND response in execution history
Variable Resolution Happens: - ✅ In frontend before execution (for immediate feedback) - ✅ In pre-request scripts (variables can be modified) - ✅ Results are stored in execution history
3. Execution History (Results)¶
Location: results table → trace_data field
Format: JSON containing both request and response
The trace_data stores the RESOLVED request that was actually sent:
{
"request": {
"method": "GET",
"url": "https://api.example.com/api/users",
"headers": {
"Authorization": "Bearer abc123token"
},
"body": "..."
},
"response": {
"headers": {...},
"body": "..."
},
"dnsMs": 10,
"connectMs": 50,
...
}
4. Saved Example Responses (Database)¶
Location: request_examples table
Format: One stored response per row, owned by a request
Separate from execution history, and deliberately so: a results row records a
response that happened and is pruned with its run, while an example is a
response the request documents and lives as long as the request does. Examples
are what an importer found next to a request - Postman's saved responses, an
OpenAPI operation's documented ones - which every parser used to drop, because
there was nowhere to keep them (issue #481).
{
"id": "exa_123",
"requestId": "req_123",
"name": "200 - A user",
"status": 200,
"headers": [{ "key": "Content-Type", "value": "application/json", "enabled": true }],
"body": "{\"id\":1}",
"contentType": "application/json",
"order": 0
}
Unlike a request definition, an example is stored verbatim: no {{variable}}
resolution happens on the way in or out, because it records what a server
answered rather than what a client should send. order is part of the contract -
a mock server serves the first example of a matched request.
Reached through GET /requests/:id/examples. Written by import (nested on the
request item of POST /import/apply, so the whole tree lands in one transaction)
or by Save as example on a response. The request builder's Examples tab
lists them and can remove one; an example is never edited in place.
Deleting the request - or the collection above it - takes its examples with it,
in the sense the next section describes: they stay on the row while the request
is in the trash, unreachable because every read of them checks the owner first,
and a purge removes them in the same transaction as the request.
5. The deletion lifecycle (issue #988)¶
Deleting a collection or a request does not remove it. DELETE
/collections/:id and DELETE /requests/:id stamp deleted_at on the row - and,
for a collection, on its whole subtree in one transaction - and every read
surface filters stamped rows out. To the sidebar, an export, an MCP tool or a
scenario plan, the row is gone; only GET /trash can still see it.
A row leaves that state one of three ways:
| What happens | |
|---|---|
POST /trash/:id/restore |
The stamp is cleared and the row is back exactly as it was - order, parent_id and every field untouched. |
DELETE /trash/:id |
The old hard cascade: the subtree, its requests and their examples are removed for good. |
| Retention | The same purge, run at startup for anything deleted more than trashRetentionDays ago (default 30; 0 keeps the trash forever). |
Two rules make a restore mean something precise:
- The cohort. One delete stamps its subtree with one timestamp, and a restore clears exactly the rows carrying the timestamp of the row it was given. A request the user deleted separately before its collection keeps its own stamp, so restoring the collection leaves it in the trash - as a trash root of its own, now that its collection is live again.
- Re-parenting. A restored collection whose parent is gone or itself deleted
comes back at the tree root. A request cannot do this -
collection_idis required - so restoring one whose collection is in the trash is refused with a409that names the request and its collection, and the trash root to restore first: the listed entry holding the collection, which may be an ancestor whose delete took it (issue #3449).
A purge is deliberately not limited to the cohort: it takes every stamped row
in the subtree, whichever delete stamped it, because a request left under a
removed collection would be reachable by no read and restorable by nothing. It
takes stamped rows only (issue #1883): a live collection found under a purged
one was never listed in the trash, so it is re-parented to the tree root - the
restore's rule - and the walk stops there, leaving anything below it alone. A
collection's trash entry counts both: the cohort a restore puts back
(collections, requests) and what this walk would destroy
(purgeCollections, purgeRequests, issue #3449).
The write side keeps that state from arising. POST /collections and
PUT /collections/:id refuse a parentId naming a collection in the trash with
a 409 (Collection '<id>' is in the trash - restore it first). A parent id the
store has never seen is still accepted, as before, because an import names
parents it has not written yet.
One thing a soft delete deliberately does not release: a stamped collection
still binds its OpenAPI document, so the orphan sweep leaves that document alone
until the collection is purged, and DELETE /specs/:id refuses it with a 409
naming the binder as in the Trash (#3446). Reclaiming it earlier would restore a
binding that points at nothing.
A collection's spec-file and data-file bindings (collections.spec_file /
data_file, issue #2765: the path and name of the file its Spec and Data tabs
read) are columns of the row, so they need no step of their own: a delete
leaves them on the stamped row, a restore brings them back unchanged, and a
purge removes them with it. A stored path grants nothing and is never exported
or imported; it is a location on the machine that picked it.
One delete path stays hard, by decision (issue #1046): the requests a
POST /specs/sync removes because the re-fetched document no longer declares
their operation. A sync is a reconciliation to a document rather than a person
removing a request, and it is the one delete whose removals are shown before
they happen - POST /specs/diff reports every one, the app renders them as
ticks the user can untick, and policy: "safe" declines deletions altogether.
Stamping them instead would fill the Trash with operations a document dropped,
where restoring one puts back a request the current document cannot explain. A
caller that wants those rows recoverable leaves them out of the sync payload and
deletes them with DELETE /requests/:id, which is soft like every other delete
a person makes.
Deleting the request for an operation is a decision that lasts (#3031),
as deleting an imported example has been since #722. A person's delete of a
request whose spec_operation names an operation of the document a collection
above it binds - one request, or every request a deleted tag folder stamps -
records that operation in spec_declined_operations under the nearest bound
collection, in the delete's own transaction. A row of its own rather than a
flag on the request, because the decision outlives the row: purging the
request, or retention purging it, keeps the record. POST /specs/diff lists
the operation as declined instead of added, and policy: "safe" never
re-creates it, so a sync of an unchanged document cannot bring the request
back. The decision is taken back by restoring the request from the trash, by a
sync that creates the operation on purpose (the user ticking it), and by
unbinding the collection or binding it to different bytes; purging the bound
collection takes its records with it. Deleting the bound collection itself
records nothing, and a sync's own hard deletes record nothing either: the
document dropped those operations, nobody declined them.
Moving the request out of the contract is the same decision (#3446). A move
through POST /reorder, PUT /requests/:id or a parentId on
PUT /collections/:id (#3470) that leaves a stamped request
outside the contract it answered to - under a collection whose nearest binding
names another document, none, or the same document bound by a collection
outside the source's contract walk (a sibling root) - records the operation
under the binding it left and clears the request's spec_operation, in the
move's own transaction; a moved collection does it for every stamped request
beneath it. The route compares the two bindings (declined_by_move, over
spec_contract_subtree_ids; for a batch or a moved collection, the binding
after and the walk are read off the tree the write stores, SpecTreeMove) and
passes the decision down, so Database
still reads no binding. A move within one contract keeps the stamp, a move
between the root and a sub-folder bound to the same document included. Because the moved request
no longer records the operation, restoring it from the trash after a later
delete forgets nothing: a restore takes back only a record its request can
claim again.
An environment goes to the same trash (#2970), by the simplest form of the
same mechanism. DELETE /environments/:id stamps deleted_at on its one row
and every read (GET /environments, composition, a run's variable scopes, the
Postman export) filters the stamp out, so an environmentId naming a trashed
environment resolves as one naming nothing. It owns no rows, so there is no
cohort to keep and no subtree to walk; it hangs under no collection, so it is
always a trash root and its restore can never be refused or re-parented. The one
field a delete rewrites is is_active: a trashed environment gives up the
active slot in the same write, so the at-most-one-active rule never has to
reason about a row nobody can see, and a restore brings it back inactive rather
than taking the slot from whatever was activated since. A purge, explicit or by
retention, removes the row.
A saved example a person made goes there too (#2971), one level below a
request. DELETE /requests/:id/examples/:exampleId on an origin: "user"
example stamps request_examples.deleted_at on its one row, and every example
read (the list, a mock server, both exports, the spec sync's refresh and the
per-request cap) filters it out. An imported example keeps the tombstone it has
had since #722 instead: a spec sync reads those to leave a removed response
alone, and a trash entry would let a restore bring back what the document's
next sync is told to skip. An example hangs under its request the way a request
hangs under its collection, and takes the request's rule rather than the
collection's: it is a trash root only while its request is live (under a
trashed request it is part of what that request's entry stands for), and a
restore is a 409 while the request is in the trash or gone, since an example
has no root of its own to come back to. A restore is also refused when the
request already holds its 100 live examples: the delete freed a slot a later
save may have taken. Purging a request or collection removes its examples
whatever their stamp, so a trashed example never outlives its request.
A client certificate goes there as well (#2973), in the environment's shape:
DELETE /client-certificates/:id stamps client_certificates.deleted_at on its
one row, it hangs under nothing, and it is always a trash root, named by its
host or host:port. Both registry reads filter the stamp out, and the
transport policy every send resolves is built from them, so a trashed
certificate is presented to no host by any send or run that starts after the delete. Its restore can be
refused for one reason: the registry holds one entry per host and port, so a
live entry that has taken the target since the delete makes the restore a 409
naming it. The row keeps its plaintext passphrase until a purge, explicit or by
retention, removes it.
6. Response Viewer¶
The Response Viewer shows the RESOLVED request in the "Raw Request" tab: - Shows exactly what was sent over the wire - Includes resolved variable values - Complete HTTP request string with headers and body
Benefits of This Design¶
- Template Reusability: Keep request definitions clean and reusable
- Environment Flexibility: Same request works across dev/staging/prod
- Full Audit Trail: Execution history shows exactly what was sent
- Debugging: See resolved values in response viewer
- Historical Accuracy: Can review past executions with actual values used
Data Flow¶
┌─────────────────────────────────────────────────────────────┐
│ Request Definition (Database) │
│ { url: "{{baseUrl}}/users", headers: {"Auth": "{{token}}"} }│
└───────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Variable Resolution (Frontend + Scripts) │
│ baseUrl → https://api.example.com │
│ token → abc123 │
└───────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Execute HTTP Request (Backend) │
│ GET https://api.example.com/users │
│ Headers: { "Auth": "abc123" } │
└───────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Store in Execution History (trace_data) │
│ { │
│ "request": { resolved values }, │
│ "response": { ... } │
│ } │
└─────────────────────────────────────────────────────────────┘
Implementation Details¶
Backend Changes¶
File: engine/src/http/routes/execution.cpp
- Stores resolved request in
trace_data.request - Includes method, URL, headers, and body
- Works for both successful and failed requests
Frontend Display¶
File: app/src/components/request-builder/components/ResponseViewer/index.tsx
- "Raw Request" tab shows the complete HTTP request
- "Headers" tab separates request headers (blue) and response headers (green)
- Request headers show the actual resolved values
Future Enhancements¶
Potential improvements: 1. Variable Diff View: Show which variables were used and their values 2. Request History Comparison: Compare requests across different runs 3. Export with Context: Export including variable values used