The Correct Answer Was a Terrible Bargain
A coding agent can find a path that returns the requested answer and still make a terrible product decision. Asked to show the current count, it can load the history, count the rows, and return the correct number. The successful test says nothing about what that answer will cost when the request repeats every thirty seconds for the next three years.
The expensive bargain usually appears after release, when more history and more users multiply work that once looked harmless. A path that finishes today can become millions of unnecessary scans without ever becoming logically wrong.
Two questions change the assignment. Can routine work that already touches the data maintain a cheap version of the answer? What is the smallest request and the smallest payload that can complete the user’s task? Those questions treat recurrence and data movement as part of correctness rather than cleanup for a later performance project.
Our monitoring dashboard exposed the cost. The page asked for a few current counts and the latest activity time for each collection, but its storage path loaded and decoded the full metrics history. Once the store reached roughly 6.3 million rows, one refresh could pull about 500 MB into memory and end in a 504 timeout.
The browser displayed the delay, so the screen took the blame. Drawing the page was not the first problem; a tiny answer had inherited the cost of the entire past.
The screen showed where the bargain became visible, not where it was made.
Follow the Wait Backward
Performance work becomes useful when the team can name what occupies the user’s wait. The dashboard request crossed several stages. The browser asked the local service for data. The service queried the metrics store, assembled a response, encoded it as JSON, moved the bytes over the local connection, and handed the result back for rendering.
Any stage could have been the limit. Expensive math might dominate one path, while another reads too much from storage or repeats work in the pipeline. A response may carry unnecessary data. The trigger can also run a heavy task at the moment the user needs an answer. Looking at the screen alone could not distinguish the causes.
The first repair changed the question asked of storage. The dashboard needed the number of metric rows for each estate and the latest timestamp. It did not need every historical row and its tags. Aggregate queries allowed the database to return those answers directly.
Indexes then gave the database a short route to the relevant rows. Instead of opening millions of records and performing the filtering in application memory, the query could seek to the estate and metric it needed.
The change removed the largest block of unnecessary work. The first repair also exposed a more precise question. The investigation was not finished.
The Next Limit Was Waiting Behind It
One endpoint still took about 9.4 seconds. The code had stopped reading the entire table, but it was fetching all historical rows for several named metrics and then choosing the latest value in memory.
A busy estate made the flaw visible. One source held 2.76 million rows. A single probe could take about 0.45 seconds. The remaining metrics repeated that cost before the request could continue.
The next change asked for the latest row by metric and estate. A composite index matched that question, so each probe could go directly to the newest value instead of walking the estate’s history.
This sequence matters because it is how bottlenecks behave. The full-table scan dominated the first measurement. Once it disappeared, the historical query became a meaningful part of the wait. Removing one limit did not prove that the rest of the path was fast. It made the next limit possible to see.
Premature tuning often misses that progression. A team improves a visible loop by 30 percent while a database scan still owns 90 percent of the response. The improvement is real and the user feels almost none of it. The largest avoidable block of work deserves attention first.
Pay Once While the Data Is Already Moving
An aggregate query can make a repeated answer much cheaper, but it still reconstructs the answer when someone asks. If the write path already visits every new measurement, the product can sometimes maintain a small count, newest timestamp, or summary as part of that routine work. A bounded addition to each write can replace a read whose cost grows with the entire history.
That trade is attractive when the answer is requested constantly and the underlying history grows without a natural limit. One hour spent refining the write path can remove thousands of times more recurring work downstream because every later reader receives the prepared fact instead of rebuilding it.
Incremental state creates its own obligation. The summary needs tests, a repair path, and a way to prove it still agrees with the underlying records. The strategy is not to cache everything; it is to pay a small, controlled cost where the data is already moving when that cost eliminates a large and repeated reconstruction later.
We Were Shipping the Shape of Our Code
After the queries improved, the response itself became the next question. The graph endpoint represented each node and connection as a separate JSON object. Every object repeated field names. Every connection repeated two long identifiers. Some fields were never read by the browser but still crossed the boundary on every refresh.
The format was convenient because it resembled the structures used inside the program. Repetition made it expensive on the network response.
The repair began by removing fields the dashboard did not use. It then changed the wire format so repeated values appeared once. Node properties moved into parallel arrays. Connections referred to the position of each node instead of repeating long identifiers. Classification codes used a small dictionary and an index. Coordinates could be packed at the precision the visualization required.
The response kept the meaning the browser needed while dropping much of the repeated representation. Tests set a payload ceiling under 5 MB for a graph with about 50,000 nodes and 70,000 connections.
The work did not make the underlying graph smaller. The new response stopped shipping the shape of the server’s objects as if that shape were part of the user’s requirement.
The distinction appears in many systems. An API returns every column because the database row already contains them. A report downloads every event because the summary is computed in the browser. An AI workflow sends the full conversation again because extracting the relevant state would require another step. Convenient internal representations can become external bottlenecks when the system grows.
Removing Work Beats Performing It Faster
The dashboard repair used several optimizations, but the central strategy was subtraction. The database stopped returning rows the request did not need. The application stopped decoding history only to discard it. The response stopped repeating names and identifiers. Later hardening also capped the number of estates and rows one read could fan across, which prevented a poisoned local stats store from turning a normal request into unbounded work.
Only after the work was bounded did tuning the remaining path make sense.
This is a useful discipline for AI systems because new capability can hide new waste. A model can summarize a large result, so a pipeline sends the large result every time. An agent can retry a slow operation, so the trigger runs it repeatedly. Faster inference reduces one wait and exposes storage, serialization, or tool coordination as the next limit.
The answer is not always a more powerful model or a faster algorithm. Sometimes the system should stop asking the question that creates the work.
Start With the User’s Wait
Good performance measurements begin with the action a person is trying to complete. For the dashboard, the meaningful interval began when the user asked for the current view and ended when the view became usable. That interval provided a path to trace backward. Database time, response assembly, encoding, transfer, parsing, and rendering could each be measured inside it.
Other products have different waits: a command-line user cares how long it takes before the next prompt is ready. A programmer working with an agent cares whether a tool result arrives while the reasoning remains relevant. A person opening an application cares about the moment the first useful action becomes available, not the moment every background task finishes.
The system can perform substantial work without making the user wait if the triggers align with the rhythm of use. Indexing can follow capture. Maintenance can run while the person is reading or writing. A summary can be prepared before the next session asks for it.
The useful goal is a system that stays busy while the user is busy and stands ready when the user wants to act.
That goal prevents a benchmark from becoming the product. A component can become impressively fast without changing the wait that the user experiences.
The Investigation Is a Loop
The practical method we used in MOOTx01 is simple enough to remember. Begin with the visible wait and trace the request through every layer it crosses. Measure the amount of work as well as the time. A fast query that returns 500 MB is still carrying a future problem.
Identify the stage that dominates the current wait. Ask whether the work is required before asking how to make it faster. Remove unnecessary work, bound what remains, and measure the whole path again.
The second measurement matters because the bottleneck will move. Storage may give way to serialization. A smaller response may reveal rendering work. A faster pipeline may expose a trigger that runs too often.
Optimization is therefore not one clever change. It is a sequence of grounded questions whose answers change the next question.
AI can accelerate that loop. An agent can search traces, compare code paths, inspect query plans, and repeat benchmarks across two implementations. The same pass can hold more of the pipeline in context than a person can inspect at once. The result is a shorter path from a visible symptom to the next useful measurement.
The human still chooses which wait matters. Experience recognizes a metric that looks impressive but has no effect on the task. Perspective asks whose time the system is consuming. Imagination considers how today’s acceptable path behaves after another year of data.
The dashboard did not need faster paint. It needed the rest of the system to stop making the screen wait for work the user never asked to see.
Find that work before you tune it.
Off-Axis Labs: All the science, fewer casualties.
Source Notes
MOOTx01 maintainers,
packages/libs/ObserverSink/Sources/ObserverSink/StatsStore.swift, schema history and dashboard query notes.MOOTx01 maintainers,
apps/moot-mgr/Sources/MootManager/MootManager.swift.MOOTx01 maintainers,
apps/moot-mgr/Sources/MootManager/APIPayloads.swift.MOOTx01 Git history:
ed8b517efor aggregate queries after the 6.3-million-row scan,d0426293for latest-value indexed queries after the 9.4-second endpoint,ce5562daandf449f2a1for payload reduction,150bd2a9for dictionary coding, andb0bbd749for bounded read fanout.
How I Write With AI
These articles are written with AI, but the AI does not decide what I believe. I bring the experience, evidence, conclusions, and responsibility for every word. The AI helps me structure and edit that material according to rules I developed over 35 years of business writing.
A few of those rules: begin with a situation the reader recognizes; establish the consequence and useful question within three paragraphs; teach through a real causal story; explain technical ideas in ordinary language; acknowledge the strongest fair complication; cite the evidence; and remove jargon, marketing language, fake certainty, and synthetic rhythm. Chicago supplies the style. Kate L. Turabian supplies the intellectual discipline. Thank you, Kate.
In short, I have taught the AI to write the way I write. That matters because I respect your time. We are working on large intellectual systems with knowledge worth sharing, but the work has traditionally accumulated faster than I could explain it well. AI can now carry much of the structural and editorial load without reducing the ideas to slogans.
If you wonder whether the voice is really mine, try the Listen function. The articles sound like me because the experience is mine, the rules are mine, the judgment is mine, and—most importantly—the message is mine.