https://shengxu.pages.dev/en/tags/architecture-evolution/Cloud architecture & DevOps notes by Shengxu: Kubernetes, Cilium, observability, LLM infra, AI agents.Hugo 0.153.2 & FixIt v0.4.0-alpha.3-20251225101113-8ffb9a95enSun, 04 Jan 2026 17:00:00 +0800https://shengxu.pages.dev/en/posts/prometheus-monitoring-architecture-evolution/Sun, 04 Jan 2026 17:00:00 +0800https://shengxu.pages.dev/en/posts/prometheus-monitoring-architecture-evolution/Observability<p>Looking back at the years spent navigating the observability space—especially around building metrics systems—it feels like a long architectural pilgrimage. From the early days of babysitting a standalone Prometheus and worrying about disk space, to introducing Thanos in an attempt to achieve &ldquo;infinite storage,&rdquo; and now rebuilding the entire monitoring hub with Mimir, these experiences are scattered in memory, with some details already starting to blur.</p> <p>Recently, I took some time to systematically revisit the pitfalls I&rsquo;ve encountered and the technical decisions I&rsquo;ve made over the years. Suddenly, it struck me: this isn&rsquo;t just a story of technical iteration; it&rsquo;s a series of philosophical choices made when facing pain points at different scales. What I once thought were &ldquo;upgrades&rdquo; turned out to be fundamentally different species. This post serves as a salvage summary of those fading experiences, discussing what I see as three architectural patterns and why, at a certain scale, Mimir becomes the &ldquo;right&rdquo; choice.</p>