Image: web-assets.dd-static.net · rights & removal

Teaching a 9B model to investigate production alerts

Reporting by Datadog Security LabsRead the original at datadoghq.com

Executive Summary

A methodology was developed to combine fast, rules-based change attribution with deeper agentic investigation by fine-tuning a smaller language model. The process leveraged an adaptation of the data flywheel blueprint: generating investigation traces using a larger teacher model, selecting successful traces based on external labels derived from Bits Investigation results, and fine-tuning a smaller student model. This method demonstrated that a specialized model, Qwen3.5-9B, could achieve 87% of the large model's recall at 5% of the inference cost. Furthermore, fine-tuning changed the model’s investigative behavior, shifting it from broad searching to focused evidence collection by reducing search calls and gathering more specific data points per investigation. The work suggests that agentic investigations can be made practical for high-volume production alerts if costly frontier models are replaced by smaller, specialized models trained on real-world investigation traces.

Facts Only

* A deployment, feature flag, or configuration change can trigger an alert requiring investigation.
* Rules-based approaches identify potentially relevant changes quickly but have limited accuracy on complex incidents.
* Agentic investigations using frontier models are too expensive at scale for every alert.
* Qwen3.5-9B was fine-tuned on traces from investigations generated by GLM-5.3.
* The fine-tuned model achieved 87% of GLM-5.3’s recall with only 5% of the investigation cost ($0.003 vs $0.06).
* A single 40 GB A100 GPU can support approximately 100,000 investigations per week at the fine-tuned rate.
* The change attribution process involved using Bits Investigation to derive proxy labels for changes, which were used during fine-tuning.
* The training data involved generating 100 teacher traces from internal incidents and selecting examples based on matching proxy labels.
* The process involved supervised fine-tuning using 16-bit LoRA on the assistant responses in selected traces.
* Customer incident results for the fine-tuned model reached 0.62 Recall@5, compared to 0.52 for the base GLM-5.3.

Full Take

The core insight demonstrated is that specialization via a data flywheel—generating high-quality, labeled investigation traces from powerful teacher models and iteratively training smaller student models—can effectively bridge the gap between expensive frontier reasoning and practical, scalable agentic applications. The process shifts the burden of complex reasoning from runtime inference to offline training, creating an asset (the fine-tuned model) that can be deployed cheaply at scale. This moves the field away from treating frontier models as the sole engine for every operational task toward a tiered approach where specialized models handle high-volume attribution while retaining frontier models for novel or highly ambiguous scenarios. The introduction of reinforcement learning based on production feedback, using Bits Investigation outcomes as an automated reward signal, suggests a pathway to self-improving systems that don't rely solely on explicit human annotation, introducing a mechanism for continuous, scalable quality improvement outside traditional labeling loops. The real implication is that the cost barrier to operationalizing deep reasoning can be significantly lowered by structuring investigative tasks into repeatable, feedback-rich sequences.

From the original · Datadog Security Labs

Scott Kramer Staff Engineer Junaid Ahmed Vice President, Engineering A deployment, feature flag, or configuration change may trigger an alert, but identifying which recent change most likely contributed to the alert can require a time-consuming investigation across multiple services and systems.

Read the full story at datadoghq.com

Sentinel — provisional

Raw score, 0 to 1 (uncalibrated)0.10

No strong signs of machine writing were found in the source article. Provisional estimate, not a finding that a person wrote it.

The model's own note (unverified)

This text reads like a detailed engineering whitepaper or blog post detailing a machine learning methodology applied to observability data, showing high specificity and internal methodological depth.

This looks only at the wording of the original source article, not at this page's AI-written sections. A small local AI model made this estimate. It has not been checked against known human and machine texts, so treat it as provisional. It cannot show who wrote an article.

What the model noticed in the source article

Moderate sentence length variance; use of technical jargon interspersed with explanatory flow.

High logical progression tracing a specific research/engineering process (flywheel) through model fine-tuning and empirical results.

Detailed description of methodology, dataset creation, and quantitative results, suggesting internal technical reporting.

Specific, complex claims regarding model performance (Recall@5 scores), cost estimates ($0.003 per investigation), and specific internal workflows seem grounded in a technical context.

Signs the model read as against machine writing

Use of specific, self-referential engineering terminology (LoRA, teacher/student models, Recall@5, A100 GPU costs) indicative of an internal technical publication.

The narrative structure follows a classic R&D exposition: Problem -> Existing Solutions -> Proposed Novel Approach (Flywheel) -> Methodology -> Results -> Future Directions.