Debugging Production with Cloud CLIs and AI Agents

Table of Contents

The Problem With Incident Debugging

Most production investigations follow the same shape. An alert fires, someone opens five browser tabs, and the next twenty minutes are spent correlating a deployment, a log stream, a metric, and a pull request that were all recorded in different systems with different clocks.

The information is almost always available. The bottleneck is the human effort of collecting it, normalizing it, and holding the whole picture in working memory while under pressure.

That specific bottleneck — gather, normalize, correlate — is what an AI agent is genuinely good at, provided you give it the right tools and the right boundaries.

The right tools already exist on your laptop: az, aws, gcloud, and gh. They are scriptable, they emit JSON, they respect your existing identity and permissions, and they are auditable. That combination makes them close to ideal as agent tools.

Why CLIs Beat Custom Integrations

There is a temptation to build a bespoke MCP server or a custom API wrapper for every incident workflow. For most teams that is premature.

The official cloud CLIs already give you:

• Complete API coverage. Anything the provider supports is reachable, usually before the SDKs and long before third-party tools.
• Structured output. --output json, --format json, and gh api all emit machine-readable data that an agent can parse reliably.
• Existing authentication. The agent inherits your session. There is no new credential to mint, store, or rotate.
• Existing authorization. IAM, Azure RBAC, and GitHub permissions still apply. An agent cannot read what you cannot read.
• Auditability. CloudTrail, Azure Activity Log, and GCP Audit Logs record the calls exactly as they record yours.

The agent’s value is not access. It is orchestration: deciding which five commands to run, in which order, and what the combined output means.

The Read-Only Boundary

Before anything else, establish the rule that makes this practice safe.

The agent investigates. The human remediates.

Give the agent an identity that can read and nothing else. Concretely:

• AWS: attach ReadOnlyAccess or a scoped variant, and assume that role in a dedicated profile.
• Azure: assign the Reader role at the subscription or resource group scope.
• GCP: grant roles/viewer plus specific log viewer roles.
• GitHub: use a fine-grained token with read access to the repositories in scope.

# A dedicated read-only profile the agent is allowed to use
export AWS_PROFILE=incident-readonly
az account set --subscription "prod-readonly"
gcloud config configurations activate prod-readonly

This is not a matter of trusting or distrusting the model. It is the same principle that keeps a debugging session from becoming a second incident: the blast radius of a mistaken command should be zero. A read-only identity guarantees that property regardless of what the agent decides to run.

Then constrain the agent in its own configuration. Most coding agents support an allowlist of permitted commands, and this is where you use it:

# Conceptual agent policy: read verbs only
allow:
  - "aws * describe-* --*"
  - "aws logs filter-log-events --*"
  - "az * list --*"
  - "az monitor log-analytics query --*"
  - "gcloud * describe --*"
  - "gcloud logging read --*"
  - "gh run view --*"
  - "gh pr view --*"
deny:
  - "* delete *"
  - "* update *"
  - "* create *"
  - "kubectl apply *"

Belt and braces. The IAM boundary is the real control; the allowlist prevents wasted turns and accidental noise.

The Four Tools and What They Answer

Each CLI answers a different question during an incident. Knowing which is which is most of the skill.

aws — what is the workload actually doing

# Recent errors from a specific log group, last 30 minutes
aws logs filter-log-events \
  --log-group-name /aws/lambda/checkout-service \
  --start-time $(($(date +%s) - 1800))000 \
  --filter-pattern '?ERROR ?Exception ?timeout' \
  --output json

# Did the service definition change recently?
aws ecs describe-services \
  --cluster prod --services checkout \
  --query 'services[0].deployments' --output json

az — what changed in the subscription

Azure’s Activity Log is the fastest way to answer “did anyone touch this?”.

az monitor activity-log list \
  --resource-group rg-prod \
  --start-time 2026-08-04T14:00:00Z \
  --query "[?contains(operationName.value, 'write')].{op:operationName.localizedValue, by:caller, at:eventTimestamp}" \
  --output json

gcloud — structured log correlation

GCP’s logging filter language is expressive enough that the agent can narrow to a single request path.

gcloud logging read \
  'resource.type="k8s_container"
   AND resource.labels.namespace_name="prod"
   AND severity>=ERROR' \
  --limit 100 --format json --freshness 30m

gh — what humans changed

This is the tool most teams forget, and it is frequently the one that closes the case.

# What shipped in the last day?
gh pr list --state merged --search "merged:>=2026-08-03" \
  --json number,title,mergedAt,author,files

# Did the deploy pipeline actually succeed?
gh run list --workflow deploy.yml --limit 5 \
  --json databaseId,status,conclusion,createdAt,headSha
gh run view 1234567890 --log-failed

A Realistic Investigation

Consider a concrete case: checkout latency climbed at roughly 14:20 UTC, error rate is up, no alert fired on infrastructure metrics.

You give the agent the symptom and the boundaries, not the procedure:

Checkout p99 latency went from 300ms to 4s starting around 14:20 UTC today. Use the incident-readonly AWS profile and the gh CLI on the checkout-service repo. Find what changed. Read-only commands only. Show me the timeline and your evidence for each entry.

A competent agent will work roughly like this:

1. Establish the change window. It queries recent deployments and merged pull requests around 14:20.

gh pr list --state merged --search "merged:>=2026-08-04T13:00:00Z" \
  --json number,title,mergedAt,author
aws ecs describe-services --cluster prod --services checkout \
  --query 'services[0].deployments[].{status:status,created:createdAt,taskDef:taskDefinition}'

2. Confirm or eliminate the deployment. If the newest task definition predates the latency change by two hours, the deploy is not the cause and the agent should say so rather than anchoring on it.

3. Widen to dependencies. It pulls errors from the log group and looks at what the service talks to.

aws logs filter-log-events --log-group-name /ecs/checkout \
  --start-time 1754316000000 --filter-pattern 'timeout' --output json
aws rds describe-db-instances --db-instance-identifier prod-checkout \
  --query 'DBInstances[0].{class:DBInstanceClass,status:DBInstanceStatus}'

4. Cross-reference the human timeline. A config-only pull request merged at 14:15 that reduced a connection pool size will show up in gh pr view --json files even though it never triggered a deployment alert.

5. Report a timeline, not a verdict.

13:58  PR #482 merged — "tune checkout connection pool" (2 files, config only)
14:12  deploy.yml run 9931 succeeded, task definition checkout:214
14:20  connection timeout errors begin in /ecs/checkout
14:21  p99 latency crosses 4s

The agent did not fix anything. It removed the twenty minutes of tab-switching and handed you a correlated narrative. Deciding to roll back PR #482 remains yours.

Prompt Patterns That Work

The quality of the investigation depends heavily on how the task is framed.

Give the symptom, the scope, and the constraint. Not the steps. If you already know the steps, run them yourself. The agent earns its place when it explores a space you have not enumerated.

Demand evidence per claim. Ask for the command output that supports each timeline entry. This makes hallucinated conclusions immediately visible, because a fabricated claim will have no command attached to it.

Force explicit uncertainty. Add “state clearly what you could not determine and what access you would need”. An agent that reports gaps is far more useful than one that produces a confident and wrong root cause.

Constrain the time window. Unbounded log queries are slow, expensive, and dilute the context window with irrelevant data. Always pass an explicit start time.

Ask for the ruled-out list. Knowing that the database, the deployment, and the load balancer were all checked and eliminated is itself valuable, and it prevents the next person from re-checking them.

Failure Modes to Expect

This workflow is useful, not magic. The recurring problems:

Context window exhaustion. A raw gcloud logging read can return megabytes. Always cap with --limit, --freshness, and a --query or --format projection that keeps only the fields that matter. Filtering at the CLI is dramatically better than filtering in the model.

Correlation mistaken for causation. A deployment that landed near the incident is a candidate, not a cause. Ask the agent to explain the mechanism, not merely the coincidence in timing.

Stale CLI knowledge. Model training data lags provider releases, so agents will occasionally invent flags. Expect a --help round trip and let the agent self-correct from the error output.

Cost. aws logs filter-log-events over a wide window scans data and bills for it. An agent iterating quickly can generate real spend. Bound the window.

Silent permission gaps. A read-only role that lacks one specific permission produces an access-denied error the agent may quietly route around, leading to a conclusion drawn from incomplete data. Ask it to surface every failed command.

Where This Fits

Use this pattern for the first fifteen minutes of an investigation: gathering context, building a timeline, eliminating hypotheses, and drafting the incident summary while the details are still fresh.

Do not use it for remediation, for anything touching customer data in logs, or as a replacement for observability tooling. If you are running aws logs filter-log-events on every incident because you have no tracing, the agent is compensating for a gap you should close properly.

The durable version of this practice is unglamorous: a read-only identity, a short list of allowed commands, a prompt that demands evidence, and a human who decides what to do next. The CLIs were already there. The agent is a fast, tireless operator of them — nothing more, and that is enough.