Beyond the Dashboard: Mastering Autonomous AI Routing with End-to-End Observability

In the race to integrate Generative AI into enterprise workflows, the economic imperative is clear: don’t use a Ferrari to drive to the mailbox. This "tiering" philosophy—routing simple queries to lightweight, high-speed models and reserving frontier-class models for complex reasoning—is the gold standard for sustainable AI infrastructure. However, moving this pattern from a whiteboard concept to a production-grade system introduces a new, critical operational risk: the autonomous decision-making layer.

When a router autonomously selects which model handles a request, the traditional operational question of "is the service up?" becomes dangerously insufficient. The new, harder question is: "Is the router making optimal decisions, and would we know the moment it started making bad ones?" Many AI stacks currently ship with a generic metrics suite—request counters and latency histograms—that fails to answer this fundamental query. This article explores how to architect a transparent, self-correcting routing system using New Relic and a local Ollama-based stack to ensure AI reliability.

The High Cost of the "Invisible" Router

The risks of unmonitored AI routing are not theoretical. We have already entered an era where "black box" deployments lead to tangible, often expensive, failures.

Chronology of Failure

• Model Drift (2023): Stanford and UC Berkeley researchers observed GPT-4’s accuracy on prime number identification plummet from 84% to 51% over three months. Without continuous evaluation, this degradation would remain invisible until a customer reported a failure.
• Legal Liability (2024): Air Canada was held responsible for misinformation provided by its chatbot, with the Civil Resolution Tribunal rejecting the defense that the AI was "responsible for its own actions." This established a legal precedent: there is no "AI excuse" for bad output.
• Data Leakage (2023): Samsung banned internal use of generative tools after engineers inadvertently pasted proprietary source code into prompts. This serves as a stark reminder that if your observability stack logs full prompt content without a robust data governance layer, your monitoring tool becomes a security liability.

These incidents demonstrate that the absence of a "human-in-the-loop" oversight layer is not just a technical oversight—it is a business vulnerability. As the EU AI Act begins to mandate automatic logging and post-market monitoring for high-risk systems, the question for engineering teams is no longer whether to instrument these systems, but how far behind the curve they will be if they wait.

The Architecture: A Three-Tier Local Router

To demonstrate a robust solution, we built a reference implementation using a FastAPI gateway, LiteLLM, and Ollama—entirely hosted on local infrastructure.

The Routing Logic

The gateway assigns a "complexity score" (0–10) to every incoming prompt based on six deterministic heuristics:

1. Prompt length
2. Code content detection
3. Multi-step logic identifiers (e.g., "step by step," "algorithm")
4. Explicit depth requests (e.g., "explain in detail")
5. Math density
6. Multiple question detection

This scoring maps to three tiers:

• 0–3 (Tier-1): Gemma3:270m (Fast, low-cost)
• 4–7 (Tier-2): Phi3 (Balanced performance)
• 8–10 (Tier-3): Qwen2.5:7b (Complex reasoning)

By utilizing regex-based scoring, the routing overhead is effectively zero, providing an immediate, high-performance decision layer that precedes the model inference itself.

Instrumenting the Unknown: The Observability Gap

While New Relic’s AI Monitoring (AIM) is optimized for major providers like OpenAI and Bedrock, our custom local stack presented a unique challenge. However, by leveraging LiteLLM’s native New Relic callback, we were able to bridge this gap.

Automated Instrumentation

By running the LiteLLM proxy under the New Relic Python agent, every call—regardless of the underlying Ollama model—is wrapped as a LlmChatCompletionSummary event. This allows the system to appear in the AI Monitoring UI, indistinguishable from hosted API calls. Furthermore, distributed tracing ensures that a single request’s ID persists from the initial gateway call through the proxy and down to the final Ollama inference, providing a continuous "waterfall" view of the entire transaction.

Filling the Schema Gaps

Standard AI monitoring tools do not inherently understand why a router chose a specific model. To address this, we implemented two custom event types:

1. LlmRoutingDecision: Captures the complexity score, the specific heuristic triggers, and the final tier assignment.
2. LlmResponseEvaluation: Tracks fast heuristics like empty/refusal detection and latency-per-tier budgets.

Note on Data Visualization: When building dashboards for these metrics, avoid the common pitfall of FACETing on continuous floats, which can silently truncate data. Use histogram(eval_score, 1, 10) to ensure numerical integrity.

The Reality of Local Inference

Operating locally means facing the harsh realities of hardware constraints. Our initial tests showed cold-start latencies of up to 126 seconds for the Qwen 7B model. We mitigated this through operational tuning:

• Warm-up requests: Sending a dummy prompt upon container initialization.
• Memory persistence: Setting OLLAMA_KEEP_ALIVE=60m to ensure models remain resident in memory.
• Honest Budgeting: Rather than artificially inflating latency budgets, we calibrated our alerts to the measured reality of the hardware, ensuring that the observability system reports true performance rather than idealized metrics.

Implications: Closing the Loop with Autopilot

The most sophisticated observability setup is useless if it remains a passive dashboard. We wanted to move beyond mere "alerting" to "autonomous remediation."

The Circuit Breaker Pattern

We configured a CRITICAL alert condition: if two or more LlmResponseEvaluation events report a quality failure within five minutes, the system triggers a Workflow Automation sequence.

1. Analysis: The system executes an Autopilot investigation against the gateway’s APM entity to identify the root cause.
2. Human-in-the-Loop: The analysis is posted to Slack. The workflow waits for a human reaction.
3. Corrective Action: Upon approval, a webhook triggers a routing-override on the gateway, forcing all traffic to the fast tier-1 model.

This mechanism avoids the "Excessive Agency" risk defined by OWASP LLM08. By requiring a human to approve the shift to the circuit-breaker state, we retain the speed of automation while keeping the responsibility of production changes under human governance.

Conclusion: Engineering for Transparency

The build demonstrated that the surface area of AI observability is much larger than just tracking token costs. It involves understanding the decision-making process of the router, the reliability of the model output, and the ability to intervene when things go wrong.

By treating AI routing as a first-class citizen of your observability stack, you move from "hoping" your AI works to "knowing" exactly how it performs. For engineering teams, the lesson is clear: do not wait for a regulatory mandate or a high-profile failure to implement visibility. The most effective infrastructure is the one that tells you exactly how it’s failing—before your customers do.

About the Author:
Arnav Bal has over nine years of experience in technical presales and cloud assurance. He specializes in building cloud-native testing strategies and integrating AI/ML into complex enterprise environments. His work focuses on bridging the gap between cutting-edge AI implementation and robust, enterprise-grade cloud operations.

Disclaimer: The views expressed in this article are those of the author and do not necessarily reflect the official policy of New Relic. Solutions offered are environment-specific and provided for educational purposes.