Overview
Databases are the heartbeat of modern applications, but when they slow down, finding the root cause can feel like searching for a needle in a haystack. You see a spike in P99 latency or cryptic wait events like wait/synch/mutex/innodb—what does it mean, and more importantly, what do you do next?
Grafana Cloud Database Observability already provides rich visibility: RED metrics, execution samples, wait event breakdowns, table schemas, and visual explain plans. Visibility alone, however, is only half the battle. The new Grafana Assistant integration bridges the gap by bringing AI-powered analysis directly into your workflow. It doesn't just show you what happened—it explains why and suggests actionable fixes. Best of all, the assistant operates on your real data: Prometheus and Loki metrics, the exact time window you're investigating, your actual table schemas, indexes, and execution plans. You never need to copy-paste SQL into a separate tool or manually describe your schema.
In this tutorial, you'll learn how to leverage the Grafana Assistant to diagnose and resolve common database performance issues faster. We'll walk through a real-world example, from identifying a slow query to interpreting the assistant's health assessment and acting on its recommendations.
Prerequisites
Before diving in, ensure you have the following:
- Grafana Cloud account with an active stack that includes Database Observability (enabled for your database).
- Prometheus and Loki data sources configured and ingesting metrics from your database.
- Basic familiarity with SQL and the Grafana Cloud interface (navigating dashboards, using Explore).
- Permissions to view query performance data and use AI features (contact your admin if unsure).
No additional software installation is required—the assistant is built into the Grafana Cloud UI.
Step-by-Step Instructions
1. Identify a Slow Query in Database Observability
Open your Grafana Cloud instance and navigate to Database Observability. The overview dashboard displays RED metrics (Rate, Errors, Duration) for recent queries. Look for a query where the duration and error rate are spiking, or one flagged by your monitoring. Click on that query to drill into its details.
You'll see time-series performance data: P50, P95, P99 latency, row examine/return ratios, wait event breakdowns, and more. Spend a moment scanning this data—it's the context the assistant will use.
2. Open the Grafana Assistant for the Query
Within the query detail view, locate the Assistant panel. It typically appears as a chat-like interface with pre-built action buttons. If you don't see it, click the “Open Assistant” icon (often a speech bubble or AI sparkle).
The assistant is already context-aware: it knows your selected time range, the query's SQL text, schema metadata, and execution plan. You don't need to re-explain anything.
3. Use the Pre-Built Action: “Why is this query slow?”
In the assistant panel, you'll find several purpose-built analysis buttons. One of the most useful is “Why is this query slow?” Click it.
The assistant immediately queries your Prometheus and Loki data, then synthesizes a health assessment. For our example, it might report:
- Rows examined vs. returned: The query examines 50 times more rows than it returns—most work is wasted filtering.
- Latency distribution: P99 is 12x the median, indicating intermittent spikes (not a constant bottleneck).
- CPU vs. wait time: CPU time is healthy, but wait events consume 40% of execution time.
- Wait event details: The event wait/synch/mutex/innodb is the main culprit. The assistant translates this: it means threads are contending for InnoDB internal resources, often due to lock contention or inefficient queries.
The assistant doesn't just dump raw wait event names—it explains what they mean in plain language and connects them to your specific query.
4. Interpret the Assistant's Diagnosis
Read the assistant's output carefully. It will highlight the key findings and suggest root causes. In our scenario, the combination of a high examine/return ratio and significant mutex wait events points to a full table scan that causes lock contention as data grows. The assistant might recommend:
- Adding a composite index to speed up filtering.
- Rewriting the query to avoid scanning unnecessary rows (e.g., use more selective WHERE clauses).
- Checking for missing foreign key indexes.
It may also show visual explain plans if available, annotated with the problematic steps.
5. Act on the Recommendations
Before making changes, you can ask the assistant follow-up questions—for example, “What index should I add?” It can generate a candidate CREATE INDEX statement based on the query's access patterns.
After implementing the change (e.g., adding an index), return to the query detail view to monitor the metrics. The assistant can also run a “Did it help?” analysis to compare before/after performance. This iterative process transforms troubleshooting from guesswork into a guided diagnosis.
Common Mistakes
Mistake 1: Ignoring Pre-Built Actions
Many users immediately start typing freeform prompts in the assistant chat box. While that works, the pre-built buttons (like “Why is this query slow?”) are designed by database engineers to ask the right questions using the full context. They often reveal insights you wouldn't think to ask for. Always start with these actions before freeform queries.
Mistake 2: Misinterpreting Wait Event Names
Wait event names are database internals jargon. Don't try to memorize them—let the assistant translate. If you see io/table/sql/handler, it might mean disk I/O bottlenecks; wait/synch/mutex/innodb points to internal contention. The assistant's natural language explanation is more actionable than the raw name.
Mistake 3: Not Verifying the Time Window
The assistant uses the current time window of your dashboard. If you have a different time range in mind (e.g., a past incident), adjust the dashboard's time picker before clicking any assistant button. Otherwise, the analysis will be based on irrelevant data.
Mistake 4: Overlooking Row Examine/Return Ratios
Many users focus solely on latency and wait events. The assistant often flags a high ratio of rows examined to rows returned as a primary cause. This is a strong indicator of missing indexes or inefficient filtering. Treat this metric as a top priority.
Mistake 5: Privacy Concerns
Some teams worry about sending query text and schema metadata to an AI tool. Rest assured: Grafana Assistant uses your data only for the current analysis. Query text and schema metadata are not stored or used for model training. The assistant runs queries against your own data sources, not a third-party service for training.
Summary
The Grafana Assistant integration for Database Observability turns raw metrics into clear, actionable insights. By combining AI-powered analysis with real-time data from Prometheus and Loki, it eliminates guesswork and reduces time-to-diagnosis. You learned how to identify a slow query, open the assistant, use the pre-built “Why is this query slow?” action, interpret wait events and row ratios, and implement fixes. Avoid common pitfalls like ignoring pre-built actions or misreading wait events. The next time your database feels sluggish, let the assistant be your co-pilot—it already knows your database's language.