Blueprints · Blueprint 02

Prometheus and Grafana on ECS

A reference architecture: metrics scraped from every service, dashboards the team actually opens, alerts that reach a person.

  1. Servicesexpose /metrics
  2. Prometheusscrape and store
  3. Alertmanagerroute alerts
  4. SNSreach on-call
  5. Grafanadashboards

↳ alerts route through SNS to whoever is on call

A reference design for watching a set of services that run on Amazon ECS. It is the starting point we adapt for a client, not a finished product.

The problem

Teams often have plenty of data and no clear picture. Dashboards nobody opens, alerts that fire all day, and a pager that people learn to ignore. The goal here is a small set of views and alerts that people trust.

How it works

  1. Each service exposes a /metrics endpoint.
  2. Prometheus runs as an ECS service and scrapes every service it can discover.
  3. Alertmanager receives alerts from Prometheus, groups them and routes them.
  4. Alerts that need a person go to SNS, which reaches whoever is on call.
  5. Grafana reads from Prometheus and shows the dashboards.

Decision log

Decision Why
Run Prometheus on ECS The team already runs ECS, so there is one fewer platform to learn.
Persistent storage for Prometheus data A restarted task should not lose its history.
Alert on symptoms first Users feel slow pages and errors. Causes go on dashboards, not pagers.
One runbook page per alert Whoever is paged knows what it means and what to try first.
Send CloudWatch and Prometheus alerts to the same place One on-call path, not two.

Rough cost shape

We do not quote numbers here, because they depend on how many services you run and how long you keep data. The main drivers are:

  • The compute for Prometheus and Grafana, which is small and steady.
  • Storage for metrics, which grows with the number of series and the retention period.
  • Data transfer, if services sit in several Availability Zones.

Confirm any estimate in the AWS pricing calculator before you commit.

Things we check on every build

  • Alerts are reviewed after the first month, and noisy ones are removed.
  • Dashboards each answer a named question.
  • Access to Grafana goes through your existing identity provider where possible.
Sheet 06 · Start here

Got a messy cloud? Let's draw it out.

Tell us what's broken or what you're planning. We reply within 48 hours.

no sales script, just a drawing

What do you need?

We reply within 48 hours.

FILED

Thanks. Your note is on the sheet and we will be in touch.