Running AI in production: what I took to the Datadog Secure panel

The fundamentals have not changed. They matter more.

The fundamentals of good development, DevOps, DevSecOps and SRE do not change. They become more important. The biggest change is code generation.

I built a company in the first wave of DevOps. The principles have not changed. The speed has. DevOps 1.0 removed the wall between development and operations. This is DevOps 2.0. It adds AI agents to the team. Code is cheap and fast, so there is more code and more change than there has ever been, and every one of those changes still has to be built, tested, released and run by something you trust.

The business value comes from the same place it always did: flow, feedback and automation. Apply them to the agents too. In our own team, AI does the routine SRE work, so our engineers spend their time on the work that needs judgement. I have built about twenty internal apps with Claude against our business systems. That work used to take months. Now it takes days.

We are SREs. We run production systems for our customers, and Managed Runtime Assurance means we prove each day that those systems are safe to run. Everything below comes from doing that.

Our pipeline is the worked example

Most of my answers came back to how we build our own software, because it is the clearest example I have of the fundamentals holding up when AI writes the code. AI does most of the work. People make the decisions. In order:

1. Claude writes the code. People own the intent and the decisions.
2. Static code analysis runs on every pull request. It stops the merge when it finds a real risk.
3. AI fixes the findings. Each finding comes with a pull request that contains the fix.
4. Codex reviews every pull request. One AI model checks the work of a different model.
5. Feature flags control every release. We release to a small group first, and we can stop a feature immediately.
6. DORA metrics measure the delivery. Deployment frequency, lead time, change failure rate and time to restore. They show that speed and safety increase together.
7. Datadog monitors production. It records every deployment, every trace and every security signal.

In plain terms: AI writes the code. AI checks the code. AI fixes the code. People approve. Datadog shows the proof.

Before an AI service is ready for the business

This was one of the questions I led on. The answer is unglamorous. That is the point. The foundations do not change for AI. An AI service needs the same pipeline, the same ownership, the same security controls and the same SRE practices as every other service you run.

We put observability in place before go-live, not after. If you cannot see the model calls, the data and the cost, the service is not ready. We see AI-built apps go live without it. That is the first gap we close. Start-ups bring us apps they built in Lovable or Replit. They work in the demo, but they have no monitoring and no owner, so our first job is to add both.

The reason observability comes first is that an AI service depends on many parts you do not own. Every model, data source and provider is a new failure point and a new security boundary. So we trace each request from end to end: the input, the data, the prompt, the model version, the output and the cost. The same trace finds faults and finds attacks. Take an app that calls a model provider, a vector database and a payment API. When the model provider slows down, the trace shows it straight away. Without the trace, the team blames its own code.

A bad AI result is an incident

Not a quirk. Not a prompt to tweak. It is an incident, and it gets the same incident process as anything else. The first question is whether the bad result is an attack, because prompt injection can look exactly like a quality problem.

Then each bad result becomes an evaluation test in the pipeline, so the same fault cannot ship again. A typical case is a chatbot that starts giving wrong answers after a release. The trace shows the retrieval step sent old data to the model. The model was right. The change was the cause.

Evidence comes from the pipeline, not from a slide

Asked what would give us confidence that an AI service is ready for critical operations, I said two things have to exist first: an owner and observability. Without an owner and without data, you have no evidence, only an opinion.

With them in place, the evidence comes from the pipeline and the tests: Code Security results, LLM Observability evaluations, SLOs and deployment tracking. In our own pipeline, every pull request shows the static analysis result, the Codex review and the tests. After release, Datadog shows the deployment and the SLO. That is our evidence trail. The evidence is a by-product of running properly.

Two habits hold it together. Put one set of tags on all data: environment, service and version. Give each service one owner in the catalogue. Then an AI fault can be tied to the release, the infrastructure and the team behind it.

Give every team the same data and the same SLOs, and show security findings in the pull request, where developers work. Teams disagree when they look at different numbers.

The AI security risks I worry about most

Secrets, prompt injection and too many permissions. These are the risks people get wrong most often.

• Scan every pull request for code faults, vulnerable packages and secrets. An AI coding agent puts an API key in a configuration file; secret scanning on the pull request stops the merge before the key ever reaches the repository.
• Give every agent its own identity, the minimum permissions it needs and short-lived credentials. An agent borrowing a person's access leaves you unable to say who did what.
• Watch the runtime, not only the build. Some attacks only exist once the service is talking to real users and real data.

Privacy sits alongside these. Trust depends on what you send to the model and what you keep, so remove personal data before it reaches the model, and from logs and traces too, and know where your provider keeps your data. There is more on logs as a data-protection surface on our blog.

Fast and safe are not opposites

The last question I led on was how to keep security and monitoring strong while still releasing quickly. Speed and security are not opposites. The mistake is one big gate at the end. Put the controls in the pipeline instead, which is all "shift left" ever meant, and then use ordinary good SRE practice: SLOs, error budgets and feature flags.

Our pipeline is the example. We release a new feature behind a flag to a small group first. If the error rate breaks the SLO, we turn off the flag in seconds. We do not need a full rollback.

Where people stay in the loop

AI is good at repeatable, high-volume work with clear inputs and outputs. It cannot be accountable. So we give AI small jobs we can audit, and our SREs use it for health checks, triage and patching. People make the decisions you cannot reverse, and our SREs approve every production change.

The same discipline applies to judging whether an AI feature is working for its users. Use of the tool is not the same as value for the user. Measure tasks completed, time saved and complaints, and connect those results to the same traces.

One action for the next 90 days

The closing question was the one practical thing a technology leader should do in the next ninety days. My answer: give your shadow builders a route to production that is easier than the route around it.

People across your business are already building with AI, whether or not anyone signed it off. Telling them to stop does not work. Making the safe route the easy route does.

1. Best practice is the foundation. Build one pipeline with security scans, owners and SLOs, and make it the easiest way to ship.
2. Observability is the key. Trace each AI service from the pull request to production. If you cannot see it, you cannot secure it.
3. Find, then pave. The first step is to find every AI app, every agent and every key. Then put each one on the route.

Where we fit

We do this for our customers. We run the stack an AI service sits on, 24x7, on Datadog, and we keep the record that shows it was running. That is AI Runtime Operations, with incident response built for the ways AI services fail. We are not a security consultancy and we do not pretend to be. We are the SREs who make sure the controls are on, the signals reach a person and the evidence exists.

Sources

• Datadog Live London, 4 November 2026, Datadog Secure track, customer panel "Observability and Security in the AI Era". Panel questions supplied by Datadog; answers from my own speaker notes.
• NCSC, Annual Review 2026: the cyber threat to the UK, published 3 November 2026.