Monitoring a real-world production app

To clearly observe a system's operation and execution, we use a monitoring and observability approach to generate crucial data that can be useful for debugging. Additionally, it helps to monitor the app performance from users’ perspective.

The approach relies on three major components:

1. Metrics - used to collect quantitative data that helps to evaluate the app performance before it goes to production.
2. Tracing - used to collect individual requests within a complex application with a multi-layer architecture. Tracing provides a vision of how the app operates from the user’s perspective. It can be used to record various system requests as well as time the application needs to complete them.
3. Logging - used to bring together data from the app running in the production environment to a single place where all team members can research and analyze it.

Context

We need to constantly monitor the health and performance of the product that our customers are using.

It is also the only shared source of truth engineering, ops and product have about whether the app is healthy.

There are 2 key pillars that will help us gain visibility into the health of a system:

• Monitoring. System exposes a set of predefined parameters (that we consider important) for Viewing and Alerting purposes. Metrics for tracking are:
  • Request Count, Error Rate and Duration short: RED - used for monitoring “request-response” cycle
  • Memory, CPU and I/O utilization - Brendan Gregg's USE method
• Observability. System exposes information that later can be used for debugging purposes. However, we may not know upfront about what information we’ll need when debugging an issue.

Following techniques will help us perform forensic analysis:

• Logs - log important information around activities within request. It's extremely important to log just enough information to be able to answer the question - "Will this help me debug an issue in the future?"
• Traces - use "log correlation id" for actions that happen within a single “trace”. Trace can span multiple systems, like frontend, backend, microservices, etc.

What components do we want to monitor?

The goal is to understand what is happening to a customer while they use the app. Did they hit an error? Did everything work as expected? Was it responsive?

In order for us to achieve this level of visibility, we need to monitor all the components involved in a specific user action: frontend, backend, any internal service and most importantly integration points around 3rd party APIs.

Tools

There are many options in each category. These are the four we have run in production, and what each one was chosen for.

Rollbar

Used ONLY for error tracking.

Was chosen because of:

• Excellent grouping of common errors under single error (with ability to drill down into individual occurrence)
• Clear stack trace for React / Node.js apps
• Ability to link into Internal tools (like Splunk or Grafana)
• Deployment notifications
• Deployment tracking

Raygun

• Used ONLY for Real User Monitoring (RUM) of the frontend app
• Provides an information from the device(browser) on a real-time performance. These metrics are called Core Web Vitals

Splunk

• Logging tool we all know about and use
• Used for adhoc debugging and forensic analysis
• Ability to configure dashboards to present the most interesting information for your use case

AppSignal

• So far the best APM for Node.js apps
• Excellent integration with Apollo - our GraphQL + React stack
• Ability to see GraphQL request itself and configure custom Spans for Traces
• Correlation “what was happening in our system during incident”
• All the metrics we care about are readily available

FAQ

Q: Is it possible to converge on a single tool in the future - perhaps Splunk? A: Yes and no. Yes, because technically it’s possible to track MOST of the metrics in Splunk and build corresponding dashboards / queries to give a similar level of visibility. No, because building those dashboards takes real effort, time and skill. So when making a decision between Build vs Buy we lean towards Buy. And only build what we can’t buy.

Q: Are we using any forms of alerts from these tools? A: Yes. Rollbar sends deploy notifications to a dedicated Slack channel. Also sends messages for production errors to dedicated Slack channels.