From Azure SRE Agent to HolmesGPT: AIOps Practices in Multi-Cloud Kubernetes Environments

In the multi-cloud Kubernetes era, the pain point for SREs is no longer just “too many alerts,” but rather investigation chains that are too long, context that is too scattered, and troubleshooting costs across clouds that are too high. What truly drains people isn’t glancing at a chart, but constantly switching between multiple cloud platforms, logging systems, deployment records, and ticketing systems.

This is why AI SRE Agents are starting to deliver real value. Their goal isn’t to be a better conversational Copilot, but to proactively take over the highly repetitive first half of the work—“checking logs, finding correlations, guessing root causes, and giving suggestions”—once an alert is triggered.

This article focuses on three representative solutions: Azure SRE Agent, HolmesGPT, and SREWorks, and discusses a more practical question: in environments with multiple tools like AKS, EKS, and Grafana Stack, how should AI operations actually be implemented?

Note: The information in this article primarily comes from official documentation, CNCF resources, and public technical sharing. Some market background information references industry media reports. Data verification cut-off date: 2026-04-17.

1. The 3 AM Alert: Every SRE’s Common Enemy

It’s 3:17 AM. Your phone buzzes. PagerDuty shows: payments-service: HTTP 5xx rate > 5%.

You open your laptop, connect to the VPN, first check Grafana on AKS, and see the error rate started rising 14 minutes ago. Then you switch to Datadog on EKS to investigate database metrics. Finally, you ask on Slack if anyone did a deploy in the last half hour. Three screens, five browser tabs, two cups of coffee, and 40 minutes later, you find the root cause was an exhausted RDS connection pool on EKS.

This isn’t an edge case; it’s the daily reality for multi-cloud SRE teams.

The CNCF 2025 Annual Cloud Native Survey shows that 82% of container users are running Kubernetes in production, 98% of organizations have adopted cloud-native technologies, and among organizations running generative AI inference, about 66% use Kubernetes to manage some or all of their inference workloads.

This is the core problem SRE Agents need to solve: not to draw prettier Grafana dashboards for you, but to complete the entire initial investigation chain for you when an alert triggers.

2. AI SRE Agent Market Landscape

From 2025 to 2026, the AI operations assistant market has taken shape rapidly, but product forms vary significantly.

The first category is native cloud vendor agents. Microsoft’s Azure SRE Agent reached GA in March 2026, billed using Azure Agent Units (AAUs). The fixed cost is 4 AAU per agent per hour, with variable costs related to model and token consumption. AWS DevOps Agent also reached GA at the end of March 2026, positioned as an operations investigation and remediation assistant across AWS services, as well as multi-cloud and on-premises environments.

The biggest advantage of these products is deep integration with their respective cloud platforms. Their biggest limitation is equally obvious: the native control plane is often cloud-first. Once you extend to multi-cloud or on-premises systems, the capability isn’t absent, but the complexity of security boundaries, credential management, permission mapping, and governance increases significantly. The Azure SRE Agent official documentation explicitly supports extension to external systems via MCP and Python tools.

The second category is open-source platforms. Alibaba’s open-sourced SREWorks encapsulates its operations engineering practices, supports multi-cloud Kubernetes cluster management, and is more suitable for large organizations with platform engineering investment capabilities.

The third category is cloud-agnostic AI Agents, which is the focus of this article. HolmesGPT, created by Robusta.dev, was accepted as a CNCF Sandbox project in October 2025. Its positioning is clear: a cloud-native SRE Agent, not tied to a single cloud vendor or a single model provider. Holmes uses LiteLLM to be compatible with multiple model sources, including OpenAI, Anthropic, Azure AI, AWS Bedrock, and locally deployed models compatible with the OpenAI API.

The “38+ built-in integrations” count for HolmesGPT in the table is based on the official installation documentation.

3. Azure SRE Agent: An Enterprise-Grade Choice with Clear Boundaries

What It Can Actually Do

The core value of Azure SRE Agent lies in automating the process of “alert comes in, manual investigation, execute change, write back ticket.”

A typical chain is: PagerDuty triggers an incident, the Agent pulls data from Azure Monitor, Application Insights, code repositories, and change information, generates a root cause analysis, and then, after approval, executes Azure CLI remediation actions like restarting, scaling, or other Azure-side recovery measures. Microsoft’s GA announcement and product documentation emphasize this.

Supported data sources include logs, code, deployments, and events. The Microsoft Learn setup documentation lists integration directions like GitHub, Azure DevOps, Datadog, Splunk, Elasticsearch, Dynatrace, and New Relic. Event and ticket collaboration also covers scenarios like PagerDuty.

Extension Boundaries in Multi-Cloud Scenarios

The diagram below better explains the capability boundaries of Azure SRE Agent in a multi-cloud environment.

graph TD
    subgraph AZ["Azure Cloud / Native Support Zone"]
        A[AKS Cluster] -->|Native Telemetry / Zero Config| B[Azure Monitor]
        C[Azure VMSS] -->|Native Telemetry / Zero Config| B
        B --> D{{Azure SRE Agent}}
        D -->|Native API Auto-Remediation\ne.g., Scale/Restart| A
        D -->|Native API Auto-Remediation| C
    end

    subgraph EXT["AWS / GCP / IDC / MCP Extension Zone"]
        E[EKS Cluster] -.->|Requires manual MCP extension\nor Python tools| D
        D -.->|No native cross-cloud execution guardrails\nCredential management & security boundaries\nare user's responsibility| E
    end

    style D fill:#0078D4,color:#fff
    style E stroke:#FF9900,stroke-dasharray: 5 5

graph TD
    subgraph AZ["Azure Cloud / Native Support Zone"]
        A[AKS Cluster] -->|Native Telemetry / Zero Config| B[Azure Monitor]
        C[Azure VMSS] -->|Native Telemetry / Zero Config| B
        B --> D{{Azure SRE Agent}}
        D -->|Native API Auto-Remediation\ne.g., Scale/Restart| A
        D -->|Native API Auto-Remediation| C
    end

    subgraph EXT["AWS / GCP / IDC / MCP Extension Zone"]
        E[EKS Cluster] -.->|Requires manual MCP extension\nor Python tools| D
        D -.->|No native cross-cloud execution guardrails\nCredential management & security boundaries\nare user's responsibility| E
    end

    style D fill:#0078D4,color:#fff
    style E stroke:#FF9900,stroke-dasharray: 5 5

graph TD
    subgraph AZ["Azure Cloud / Native Support Zone"]
        A[AKS Cluster] -->|Native Telemetry / Zero Config| B[Azure Monitor]
        C[Azure VMSS] -->|Native Telemetry / Zero Config| B
        B --> D{{Azure SRE Agent}}
        D -->|Native API Auto-Remediation\ne.g., Scale/Restart| A
        D -->|Native API Auto-Remediation| C
    end

    subgraph EXT["AWS / GCP / IDC / MCP Extension Zone"]
        E[EKS Cluster] -.->|Requires manual MCP extension\nor Python tools| D
        D -.->|No native cross-cloud execution guardrails\nCredential management & security boundaries\nare user's responsibility| E
    end

    style D fill:#0078D4,color:#fff
    style E stroke:#FF9900,stroke-dasharray: 5 5

The native control plane of Azure SRE Agent is Azure-first. For AKS and other Azure resources, it can directly access the Azure control plane. For AWS, GCP, or IDC resources, although official support exists via MCP and Python tools, the complexity shifts to the user’s own IAM, credentials, network boundaries, and audit design.

The key point here isn’t “can it be extended,” but once extended, who is responsible for the permission model, audit trail, and security liability? In enterprise environments, this often determines whether something can go live more than “feature support.”

Data Residency: A Non-Negotiable Compliance Factor

According to the Learn documentation, the data processing region for Azure SRE Agent is directly tied to the chosen model provider:

• In EU / EFTA / UK, the default model provider is Azure OpenAI.
• Anthropic is an option, not the default, in these regions and is not protected by the EU Data Boundary.
• If Anthropic is chosen, prompts, responses, and resource analysis content may be processed in the US.
• In government clouds like GCC, GCC High, and DoD, Anthropic is unavailable.

Therefore, for regulated industries like finance, healthcare, and government, compliance with Azure SRE Agent isn’t just about “which region the Agent itself is deployed in,” but also who the model provider is and where the data will land.

This is one reason HolmesGPT offers more flexibility regarding data sovereignty: if an organization needs it, a locally deployed model is an option, not an exception path.

4. HolmesGPT: A CNCF SRE Agent Built for Multi-Cloud

Design Philosophy: Not a Copilot, an Agent

The fundamental difference between HolmesGPT and most AI assistants is its emphasis on agentic investigation—proactive, multi-step, iterative investigation.

The Holmes official documentation clearly explains its core mechanism: when a problem is presented to the system, it doesn’t answer in one shot. Instead, it decides which tool to query next, what data to fetch, how to control context size, and then continues reasoning.

This approach can be broken down into three key strategies:

• Aggregations at Source: Perform PromQL or other query filtering as close to the data source as possible.
• Traversable JSON Trees: Expand large API responses on demand rather than stuffing them all into the context at once.
• Output Budgeting: Dynamically control context size to avoid token overflow.

The diagram below more closely represents HolmesGPT’s core workflow.

sequenceDiagram
    participant Alert as Alert Source
    participant Holmes as HolmesGPT Core
    participant Tools as Toolset
    participant LLM as LLM

    Alert->>Holmes: 1. Trigger Alert (e.g., HTTP 5xx > 5%)
    loop Agentic Reasoning Loop
        Holmes->>LLM: 2. Pass current context, request next action
        LLM-->>Holmes: 3. Decision: Invoke specific tool
        Holmes->>Tools: 4. Execute Query
        Note over Tools: Source-side filtering + on-demand expansion\nReturn only high-value compressed data
        Tools-->>Holmes: 5. Return filtered structured data
        Holmes->>LLM: 6. Validate hypothesis, decide whether to dig deeper
    end
    Holmes->>Alert: 7. Output RCA and write back to ticket or Slack

sequenceDiagram
    participant Alert as Alert Source
    participant Holmes as HolmesGPT Core
    participant Tools as Toolset
    participant LLM as LLM

    Alert->>Holmes: 1. Trigger Alert (e.g., HTTP 5xx > 5%)
    loop Agentic Reasoning Loop
        Holmes->>LLM: 2. Pass current context, request next action
        LLM-->>Holmes: 3. Decision: Invoke specific tool
        Holmes->>Tools: 4. Execute Query
        Note over Tools: Source-side filtering + on-demand expansion\nReturn only high-value compressed data
        Tools-->>Holmes: 5. Return filtered structured data
        Holmes->>LLM: 6. Validate hypothesis, decide whether to dig deeper
    end
    Holmes->>Alert: 7. Output RCA and write back to ticket or Slack

sequenceDiagram
    participant Alert as Alert Source
    participant Holmes as HolmesGPT Core
    participant Tools as Toolset
    participant LLM as LLM

    Alert->>Holmes: 1. Trigger Alert (e.g., HTTP 5xx > 5%)
    loop Agentic Reasoning Loop
        Holmes->>LLM: 2. Pass current context, request next action
        LLM-->>Holmes: 3. Decision: Invoke specific tool
        Holmes->>Tools: 4. Execute Query
        Note over Tools: Source-side filtering + on-demand expansion\nReturn only high-value compressed data
        Tools-->>Holmes: 5. Return filtered structured data
        Holmes->>LLM: 6. Validate hypothesis, decide whether to dig deeper
    end
    Holmes->>Alert: 7. Output RCA and write back to ticket or Slack

This is why HolmesGPT is better suited for multi-cloud operations. Its focus isn’t “start with one cloud, then extend outwards,” but rather assumes you are already in a heterogeneous environment: Kubernetes, databases, logging platforms, alerting platforms, ticketing systems, local APIs, and multiple cloud vendors all coexisting.

Security Design: Principle of Least Privilege

The Holmes official documentation emphasizes that most observability-oriented toolsets are designed as read-only. However, this statement shouldn’t be mechanically interpreted as “all tools are read-only.” Holmes also provides a bash toolset, and the current official documentation explicitly states it is enabled by default, with boundaries controlled via allow/deny lists.

A more accurate statement would be: Holmes’ default security philosophy leans towards read-only observability, but actual production deployments still require separate review of toolsets with execution capabilities, such as bash.

The recommended production pattern is to deploy a centralized Holmes instance, give it scoped credentials, and let engineers query production data through this unified entry point, rather than giving everyone a set of high-privilege credentials to directly access production. This aligns with the principle of least privilege in platform engineering.

When using the HTTP connector to interface with private APIs, Holmes also requires explicit declaration of allowed hosts, paths, and HTTP methods. This is a crucial part of its security boundary design:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

toolsets:
  internal-cmdb:
    type: http
    config:
      endpoints:
        - hosts: ["cmdb.internal.company.com"]
          paths: ["/v1/assets/*"]
          methods: ["GET"]
      auth:
        type: bearer
        token: "{{ env.CMDB_TOKEN }}"

38+ Toolset Covering the Entire Multi-Cloud Tech Stack

The Holmes official installation documentation shows it supports 38+ built-in integrations. These tools span metrics, logs, traces, ITSM, CI/CD, Kubernetes, databases, and cloud platforms.

For multi-cloud teams, the significance isn’t just “supporting many tools” itself, but that you can finally put cross-system investigation chains into the same Agent reasoning process, instead of relying on manual mental stitching.

5. Grafana Stack + HolmesGPT: Three-Signal Correlation

For teams already using the Grafana Stack, HolmesGPT’s value isn’t about replacing Prometheus, Loki, or Tempo, but about stringing the three signal types into a single reasoning chain.

graph LR
    subgraph OBS["Multi-Cloud Data Foundation"]
        P[(Prometheus / Mimir
Metrics)]
        L[(Loki
Logs)]
        T[(Tempo
Traces)]
    end

    subgraph HOL["HolmesGPT Intelligent Reasoning Layer"]
        C[Context Manager
Data Summarizer]
        A{{Agentic Router}}
    end

    subgraph DEST["Response & Collaboration"]
        S[Slack / Teams]
        D[PagerDuty / Jira / GitHub]
    end

    P -->|PromQL| C
    L -->|LogQL| C
    T -->|TraceQL| C
    C <-->|Structured Context| A
    A -->|RCA Report / Remediation Suggestions| S
    A -->|Ticket Update / Open PR| D

    style A fill:#8A2BE2,color:#fff

graph LR
    subgraph OBS["Multi-Cloud Data Foundation"]
        P[(Prometheus / Mimir
Metrics)]
        L[(Loki
Logs)]
        T[(Tempo
Traces)]
    end

    subgraph HOL["HolmesGPT Intelligent Reasoning Layer"]
        C[Context Manager
Data Summarizer]
        A{{Agentic Router}}
    end

    subgraph DEST["Response & Collaboration"]
        S[Slack / Teams]
        D[PagerDuty / Jira / GitHub]
    end

    P -->|PromQL| C
    L -->|LogQL| C
    T -->|TraceQL| C
    C <-->|Structured Context| A
    A -->|RCA Report / Remediation Suggestions| S
    A -->|Ticket Update / Open PR| D

    style A fill:#8A2BE2,color:#fff

graph LR
    subgraph OBS["Multi-Cloud Data Foundation"]
        P[(Prometheus / Mimir
Metrics)]
        L[(Loki
Logs)]
        T[(Tempo
Traces)]
    end

    subgraph HOL["HolmesGPT Intelligent Reasoning Layer"]
        C[Context Manager
Data Summarizer]
        A{{Agentic Router}}
    end

    subgraph DEST["Response & Collaboration"]
        S[Slack / Teams]
        D[PagerDuty / Jira / GitHub]
    end

    P -->|PromQL| C
    L -->|LogQL| C
    T -->|TraceQL| C
    C <-->|Structured Context| A
    A -->|RCA Report / Remediation Suggestions| S
    A -->|Ticket Update / Open PR| D

    style A fill:#8A2BE2,color:#fff

Configuration Example

According to the official documentation, if grafana/loki is enabled, the default kubernetes/logs should be disabled; otherwise, the system will have multiple log sources simultaneously, affecting the troubleshooting path selection.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# values.yaml
holmes:
  llmProvider: openai
  openAiApiKey: "sk-..."

  toolsets:
    prometheus:
      enabled: true
      config:
        prometheus_url: "http://kube-prometheus-stack-prometheus.monitoring:9090"

    grafana/loki:
      enabled: true
      config:
        api_url: "http://loki-gateway.monitoring:80"
        external_url: "https://grafana.yourcompany.com"

    grafana/tempo:
      enabled: true
      config:
        api_url: "http://tempo.monitoring:3100"
        grafana_datasource_uid: "tempo-uid"

    kubernetes/logs:
      enabled: false

The officially recommended installation method is:

1
2

helm repo add robusta https://robusta-charts.storage.googleapis.com
helm install holmesgpt robusta/holmes -f values.yaml

Practical Troubleshooting Effect of Three-Signal Correlation

When AlertManager triggers HTTPRequestsErrorRate > 5%, Holmes’ investigation method typically follows this chain:

1. First, determine the time window and check the error rate curve from Prometheus.
2. Then, correlate changes by checking Deployment or release history.
3. Next, dig into logs using Loki to find abnormal patterns.
4. Finally, validate the call chain using Tempo to pinpoint latency or failure locations.

The output conclusion is usually: provide a preliminary RCA, along with next-step remediation suggestions.

This section is closer to a methodological explanation rather than a verbatim retelling of a single official case. Its key point is: HolmesGPT’s value comes from cross-signal correlation, not single-point Q&A.

6. Multi-Cloud Operator Mode: 24/7 Proactive Health Checks

Beyond passive alert response, HolmesGPT also features an Operator Mode. According to the official documentation, it is a Kubernetes-native health check controller system built around two resource types: HealthCheck and ScheduledHealthCheck.

graph TD
    subgraph K8S["Kubernetes Multi-Cloud Management Cluster"]
        SHC[ScheduledHealthCheck CRD
Scheduled Cron Checks]
        HC[HealthCheck CRD
One-time Check Job]
        O[Holmes Operator
Lightweight Controller]
        API[Holmes API Server
Stateless Inference Service]

        SHC -->|Triggers / Generates| HC
        HC -->|Listens for Events| O
        O -->|HTTP Task Delegation| API
    end

    API -->|1. Fetches Multi-Cloud Telemetry| DS[(Prometheus / Loki / AWS RDS / Azure SQL)]
    API -->|2. Pushes Analysis Reports| OUT[Slack / PagerDuty / GitHub]

    style O fill:#2E8B57,color:#fff
    style API fill:#9370DB,color:#fff

graph TD
    subgraph K8S["Kubernetes Multi-Cloud Management Cluster"]
        SHC[ScheduledHealthCheck CRD
Scheduled Cron Checks]
        HC[HealthCheck CRD
One-time Check Job]
        O[Holmes Operator
Lightweight Controller]
        API[Holmes API Server
Stateless Inference Service]

        SHC -->|Triggers / Generates| HC
        HC -->|Listens for Events| O
        O -->|HTTP Task Delegation| API
    end

    API -->|1. Fetches Multi-Cloud Telemetry| DS[(Prometheus / Loki / AWS RDS / Azure SQL)]
    API -->|2. Pushes Analysis Reports| OUT[Slack / PagerDuty / GitHub]

    style O fill:#2E8B57,color:#fff
    style API fill:#9370DB,color:#fff

graph TD
    subgraph K8S["Kubernetes Multi-Cloud Management Cluster"]
        SHC[ScheduledHealthCheck CRD
Scheduled Cron Checks]
        HC[HealthCheck CRD
One-time Check Job]
        O[Holmes Operator
Lightweight Controller]
        API[Holmes API Server
Stateless Inference Service]

        SHC -->|Triggers / Generates| HC
        HC -->|Listens for Events| O
        O -->|HTTP Task Delegation| API
    end

    API -->|1. Fetches Multi-Cloud Telemetry| DS[(Prometheus / Loki / AWS RDS / Azure SQL)]
    API -->|2. Pushes Analysis Reports| OUT[Slack / PagerDuty / GitHub]

    style O fill:#2E8B57,color:#fff
    style API fill:#9370DB,color:#fff

The Holmes Operator primarily handles scheduling and resource management; the actual inference work is performed by the Holmes API service. The official documentation also explicitly states that Operator Mode is still evolving, and production environments should pay close attention to version changes and cost control.

Multi-Cloud Scheduled Health Check Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: holmesgpt.dev/v1alpha1
kind: ScheduledHealthCheck
metadata:
  name: multi-cloud-hourly
spec:
  schedule: "0 * * * *"
  query: |
    Hourly multi-cloud health check:
    - AKS: pod restarts and error rates across all namespaces
    - EKS: database connection pool usage (AWS RDS tool)
    - Check Loki for cross-cluster error spikes in last 60min
    - Identify any stuck rollouts or pending pods
  destinations:
    - type: slack
      config:
        channel: "#platform-health"
    - type: pagerduty
      config:
        integration_key: "${PD_INTEGRATION_KEY}"
  timeout: 180

It’s important to emphasize: Operator Mode is currently a rapidly evolving capability. High-frequency health checks can significantly increase model invocation costs. In production environments, it’s more suitable to start with low-frequency checks rather than immediately implementing high-frequency full scans.

7. Pitfall Guide and Production Recommendations

Configuration Level

• After enabling grafana/loki, disable kubernetes/logs to avoid duplicate log sources.
• When configuring multiple similar toolsets in a multi-cloud environment, ensure clear naming isolation to prevent future maintenance confusion.
• Holmes’ bash toolset is enabled by default; the allow/deny list must be reviewed before production.
• Installation commands, chart paths, and operator fields may change with versions; always refer to the current official documentation before deployment.

Architecture Level

• Start with read-only investigations before considering automated execution.
• Govern the Agent as a new high-privilege entity, not as a regular plugin.
• It is recommended to deploy multiple replicas of the Holmes API service to prevent the investigation chain itself from becoming a single point of failure.

The last three points here are closer to production experience judgments rather than official hard requirements.

8. Decision Guide

If your business is primarily Azure-based with limited multi-cloud expansion needs, Azure SRE Agent is often the more cost-effective choice in terms of operational overhead. Its strengths lie in native execution capabilities and deep control plane integration, but special attention must be paid to the model provider and data processing region, especially in EU / EFTA / UK or stricter compliance scenarios.

If your environment has clearly expanded into EKS, GKE, private clusters, or scenarios with higher data sovereignty requirements, HolmesGPT is the more natural choice. Its value isn’t just “supporting multi-cloud,” but designing for the real-world complexity of multi-cloud, multi-tool, and multi-signal environments as a default premise.

If you need a heavier, platform-oriented operations system and your organization has the sustained capability for platform engineering investment, SREWorks also has its place, though deployment and governance complexity will be higher.

For teams that already have a Prometheus, Grafana, and Loki foundation, HolmesGPT acts more like a low-cost, incremental inference layer. It doesn’t require you to tear down your existing observability stack; its value primarily comes from connecting metrics, logs, traces, and external system information into an automated investigation chain. This assessment is derived from the product architecture and deployment approach, not from official marketing copy.

Conclusion

In 2026, SRE shouldn’t still primarily rely on humans pulling all-nighters for repetitive troubleshooting.

A more realistic direction is to let Agents handle the highly repetitive work of “gathering evidence, connecting context, and generating preliminary RCAs,” while leaving “permission boundary design, system resilience, Runbook quality, and multi-cloud disaster recovery strategy” for humans to lead.

This division of labor is where AI-driven operations truly provides value.

References

• CNCF: HolmesGPT Project Page and Official Blog
• HolmesGPT Official Documentation: Installation, Why HolmesGPT, Bash toolset, Operator, ScheduledHealthCheck
• Microsoft Learn / Azure Official: Azure SRE Agent GA, Model Provider Selection, Anthropic Subprocessor, Setup
• AWS Official: AWS DevOps Agent GA