Closing: What to do this quarter

Install the collector. The plugin in Chapter 8, or your own hooks that write the same schema. Start keeping traces this week, before the oracle exists. Unlabeled traces are still an asset, and the collector is the part with no dependencies.
Pick one repository and write the first hidden tests. Twenty tasks, each with a test that fails on the base commit for the right reason. Bug reports with reproductions are the easiest source.
Run the oracle in a container with no network. On the same machine at first. Move it to a runner the agents cannot log into before you train on anything.
Measure the flip rate. Dispatch the twenty tasks. Count the flips. That number, times your sessions per day, is your data rate, and Chapter 12 says what it buys.
Keep the failures. They are half of the preference set.
Set aside the held-out tasks now. Before the first training run, not after. The leak cannot be undone.
Train the first adapter at 500 flips. On all layers, with replay, with the shuffled-label control beside it. Evaluate on the held-out set. Expect it to be worse than the rented model and better than the base model. That is the first point on a curve the team now owns.

Cite this

Anderson, M. (2026). What to do this quarter. In Building a continuous delivery system for fine-tuned open-source, open-weight models trained on your organization's traces (Closing). macanderson.com. https://macanderson.com/research/continuous-delivery-of-fine-tuned-open-weight-models/what-to-do-this-quarter

BibTeX

@incollection{anderson2026continuousdeliveryof,
  author    = {Anderson, Mac},
  title     = {What to do this quarter},
  booktitle = {Building a continuous delivery system for fine-tuned open-source, open-weight models trained on your organization's traces},
  year      = {2026},
  url       = {https://macanderson.com/research/continuous-delivery-of-fine-tuned-open-weight-models/what-to-do-this-quarter}
}

Cite this

Updates by email