Application Performance Monitoring Flashcards

Question 1

What does application performance monitoring mean for a software team?

Accepted Answer

Application performance monitoring is the continuous collection and analysis of data about how a running application behaves, including its speed, reliability, and resource use. Teams use it to find problems early, understand how code affects users, and check that changes help. It usually relies on metrics, logs, and traces from live systems.

Question 2

What is observability and how does it relate to monitoring?

Accepted Answer

Observability is the ability to understand a system's internal state from the data it emits, such as metrics, logs, and traces. Monitoring is the practice of collecting and watching known signals, usually with predefined thresholds or dashboards. Observability supports questions nobody anticipated in advance, while monitoring answers questions that teams already knew to ask.

Question 3

Why is monitoring performance in production different from testing before release?

Accepted Answer

Pre-release testing uses controlled environments and simulated loads, so it cannot capture every condition real users create. Production monitoring observes actual traffic, real data volumes, third-party dependencies, and unexpected usage patterns. Because conditions keep changing, monitoring is an ongoing activity that complements testing rather than replacing it.

Question 4

What is the difference between a symptom and a root cause in performance problems?

Accepted Answer

A symptom is what users or dashboards observe, such as slow page loads or rising error rates. A root cause is the underlying condition that produces those symptoms, such as a missing database index or an exhausted connection pool. Good monitoring helps teams move from symptoms to root causes, not just visible effects.

Question 5

What is latency in the context of application performance?

Accepted Answer

Latency is the time it takes for an operation to complete, measured from the moment a request starts until its response is finished. In web applications it is reported per request or endpoint. Because latency varies from one request to the next, teams examine its distribution rather than relying on a single measurement.

Question 6

What is throughput and why do teams track it alongside latency?

Accepted Answer

Throughput is the amount of work an application completes in a given period, such as requests per second. Latency and throughput interact, since a system can look fast at low load yet slow down when throughput approaches its capacity. Tracking both shows whether speed holds up as demand grows.

Question 7

What are the four golden signals often used in service monitoring?

Accepted Answer

The four golden signals are latency, traffic, errors, and saturation. Latency measures how long requests take, traffic measures demand on the system, errors measure the rate of failed requests, and saturation shows how full critical resources are. Together they give a compact view of service health.

Question 8

Why are averages a poor choice for describing response times on their own?

Accepted Answer

An average can hide the experience of slow requests, because a few very slow responses can be offset by many fast ones. Response times are skewed, so the mean may look healthy while a meaningful share of users waits longer. Percentiles such as the 95th or 99th show the tail of the distribution better.

Question 9

What is the difference between metrics, logs, and traces?

Accepted Answer

Metrics are numeric measurements sampled over time, such as request counts or memory use, and they are cheap to store and chart. Logs are timestamped records of discrete events, useful for detailed context about a single occurrence. Traces record the path of one request through many components, showing where time was spent.

Question 10

What is the difference between a counter and a gauge?

Accepted Answer

A counter is a value that only increases, such as the total number of requests served. A gauge is a value that can move up or down, such as current memory use or the number of active connections. Choosing the right type matters because rates come from counters, not gauges.

Question 11

What is a time series in monitoring data?

Accepted Answer

A time series is a sequence of values for one metric, each paired with a timestamp, such as request latency recorded every ten seconds. Each unique combination of metric name and labels forms its own series. Charts and alerts are built by querying these series over chosen windows of time.

Question 12

Why should teams agree on consistent metric names and label conventions?

Accepted Answer

Consistent names let engineers find the same signal across services without guessing its meaning. They also allow shared dashboards, alert rules, and queries to work across teams. Without agreement, the same measurement may appear under several names, making comparisons unreliable and incidents slower to diagnose.

Question 13

How do utilization and saturation differ as performance signals?

Accepted Answer

Utilization measures the fraction of a resource's capacity in use, such as CPU at seventy percent. Saturation measures demand that cannot be served immediately, such as queued work or threads waiting for a connection. A resource can be highly utilized without being saturated, but saturation usually signals that performance will degrade soon.

Question 14

What is cardinality in metrics, and why can high cardinality cause problems?

Accepted Answer

Cardinality is the number of distinct label combinations a metric can produce, such as one series per user. High cardinality increases storage, memory, and query cost, because each unique combination creates a separate time series. Teams avoid unbounded values like user identifiers as metric labels and keep them in logs or traces instead.

Question 15

What is a histogram metric used for in latency monitoring?

Accepted Answer

A histogram records observations into predefined buckets, such as counts of requests that finished within 100, 250, or 500 milliseconds. From these bucket counts, a monitoring system can estimate percentiles across many instances and time ranges. This makes histograms suited to latency, since they can be aggregated, unlike a percentile computed per host.

Question 16

Why should teams monitor error rates as a percentage of requests rather than raw counts?

Accepted Answer

Raw error counts rise whenever traffic rises, even if the system is no less reliable than before. A ratio of failed requests to total requests stays meaningful as load changes. Comparing rates also makes it easier to set consistent thresholds and to compare services that handle very different volumes.

Question 17

How do trace context headers help link work across services?

Accepted Answer

Trace context headers carry identifiers, such as a trace identifier and a parent span identifier, from one service to the next in outgoing requests. Each receiving service reads these values and records its own span under the same trace. This lets a monitoring backend reassemble the path of requests that crossed many components.

Question 18

What is a span, and how do spans form a trace?

Accepted Answer

A span represents one timed operation within a request, such as a database query, an outbound HTTP call, or a function execution. Each span records its start time, duration, attributes, and parent link. Spans that share a trace identifier and connect through parent references form a tree that shows the request's execution path.

Question 19

What is the purpose of a trace's critical path?

Accepted Answer

The critical path is the chain of spans that determines the total duration of a request, because shortening anything off that chain will not reduce overall latency. Engineers find it by examining how child spans overlap and wait on one another. Focusing optimization effort on the critical path yields the largest improvement.

Question 20

How can a trace help distinguish a slow database call from a slow application server?

Accepted Answer

A trace shows the duration of each span, so the database call appears as its own timed segment inside the request. If that segment accounts for most of the total time, the database is the likely bottleneck. If the time sits in the server's own spans without dependency waits, the application code deserves review.

Question 21

How should a failed operation be recorded within a span?

Accepted Answer

A failed operation should set the span's status to an error state and attach the error details as an event or attribute. Recording the error type and message lets engineers see which operation failed without reading every log line. The failure then stays visible along the trace path, showing where the request broke.

Question 22

Why can message queues make distributed tracing more difficult?

Accepted Answer

A producer may publish a message and finish its work before a consumer handles it, so the request is no longer one call. Trace context must be carried inside the message headers for consumer spans to join the original trace. Without propagation, the work appears as disconnected traces and hides queue wait time.

Question 23

What is a service map and what information does it show?

Accepted Answer

A service map is a diagram of services and the calls between them, usually derived from trace data. It shows which services depend on which, along with request volume, latency, and error rates on each connection. Teams use it to see how a failure in one component may spread to others.

Question 24

What is an N+1 query problem, and how can tracing reveal it?

Accepted Answer

An N+1 query problem occurs when code runs one query to fetch a list, then another query for each item in that list. The result is many nearly identical database spans within one request. A trace makes this visible as a long run of repeated short calls, a pattern that averaged database metrics often hide.

Question 25

What is an APM agent and what does it do?

Accepted Answer

An APM agent is a piece of software attached to an application, either as a library or a runtime hook, that observes the code as it runs. It records timing, errors, and related context for requests and sends that data to a monitoring backend. Agents usually add small overhead that teams should measure.

Question 26

What is the difference between automatic and manual instrumentation?

Accepted Answer

Automatic instrumentation uses libraries or runtime hooks to capture common operations, such as incoming web requests and outgoing database calls, with little code change. Manual instrumentation means developers add measurements around business logic that automatic methods cannot infer. Most teams combine both, starting with automatic coverage and adding manual spans for important steps.

Question 27

Why should teams measure the overhead that instrumentation adds?

Accepted Answer

Instrumentation consumes some CPU, memory, and network capacity, and poorly chosen instrumentation can noticeably slow the application it observes. Measuring overhead, by comparing load tests with instrumentation on and off, shows whether the cost is acceptable. Teams can then reduce costly settings, such as capturing very large payloads or recording every request in full.

Question 28

What role does a telemetry pipeline or collector play between agents and backends?

Accepted Answer

A collector receives telemetry from applications and agents, then processes it by batching, filtering, redacting, or enriching it before forwarding it to one or more backends. This decouples applications from specific backends, so teams can change destinations without redeploying code. It also gives a central point to apply sampling and data retention policies.

Question 29

What is sampling in tracing, and why do systems use it?

Accepted Answer

Sampling means recording only a subset of requests in full detail, such as a fixed percentage. Systems use it to control the cost of storage, network transfer, and processing in high-traffic environments. Good sampling strategies keep traces for slow or failed requests, since those are the ones most useful for diagnosis.

Question 30

What is the difference between head-based and tail-based sampling?

Accepted Answer

Head-based sampling decides whether to keep a trace at the start of the request, before its outcome is known. Tail-based sampling waits until the trace completes, so it can keep traces based on criteria such as errors or high latency. Tail-based approaches are more informative but need more buffering and processing capacity.

Question 31

Why add custom business attributes, such as order size, to telemetry?

Accepted Answer

Business attributes connect technical measurements to outcomes that matter, such as which customer tier or cart size experienced a slowdown. This lets teams prioritize fixes by real impact rather than by raw error counts. Attributes should be chosen deliberately, since high-cardinality values can increase storage costs.

Question 32

Why must telemetry avoid capturing sensitive personal or payment data?

Accepted Answer

Telemetry is copied to shared dashboards, long-term storage, and third-party tools, so private data placed there spreads beyond its original system. Exposed personal or payment details can create legal, contractual, and trust problems. Teams should redact or drop such fields at the source or in the collection pipeline before data leaves the application.

Question 33

How should a performance dashboard be organized for quick triage?

Accepted Answer

A useful dashboard starts with a high-level overview of user-facing signals, such as latency, traffic, and errors for key services. From there, viewers can drill down into the specific service, endpoint, or dependency showing a problem. Panels should be labeled with units and time ranges so anyone on call can interpret them quickly.

Question 34

What is a service level objective and how does it relate to monitoring?

Accepted Answer

A service level objective is a target for a measurable indicator, such as the share of requests that complete within a set time. Monitoring supplies the indicator data needed to judge whether that target is being met. Setting objectives helps teams decide which signals matter most and when reliability work should come first.

Question 35

Why do many teams alert on user-facing symptoms rather than every internal cause?

Accepted Answer

User-facing symptoms, such as elevated error rates or slow checkouts, directly show that people are affected, so they make reliable paging signals. Internal causes such as a single host's high CPU may be harmless or temporary, adding noise. Cause-based signals are valuable for diagnosis, so they belong on dashboards rather than pager alerts.

Question 36

What is alert fatigue and how can teams reduce it?

Accepted Answer

Alert fatigue happens when responders receive so many notifications, especially low-value ones, that they begin to ignore or delay action on all of them. Teams reduce it by removing alerts nobody acts on, tuning thresholds, grouping related notifications, and requiring each alert to have a clear response. Reviewing alert history keeps the set focused.

Question 37

Why must performance baselines account for daily and weekly traffic patterns?

Accepted Answer

Many applications have predictable cycles, with heavier load at certain times, so one fixed threshold may cause false alarms at peak times. Comparing current measurements against the same period in earlier cycles gives a fairer reference. Baselines should be reviewed as usage grows, since old norms stop describing normal behavior.

Question 38

Why should deployments and configuration changes appear on performance charts?

Accepted Answer

Overlaying deployment and configuration events on charts makes it easier to connect a sudden latency rise or error spike to a recent change. Without these markers, engineers may search through unrelated components before noticing the timing. Markers turn a vague slowdown into a testable hypothesis, such as a query added in the latest release.

Question 39

What is a runbook and what should it contain for performance incidents?

Accepted Answer

A runbook is a written procedure for handling a known type of problem, giving responders a consistent starting point under pressure. For performance incidents it should list key dashboards, the signals to check first, common causes and diagnostic steps, safe mitigations such as rollback or feature flags, and escalation contacts. Teams should update runbooks after each incident.

Question 40

What makes a blameless postmortem useful after a performance incident?

Accepted Answer

A blameless postmortem reviews what happened, why the system allowed it, and what changes would reduce recurrence, without assigning fault to individuals. Its timeline should combine metrics, traces, and logs to show when symptoms began and how responders reacted. Action items need owners and due dates, or the lessons remain unapplied.

Application Performance Monitoring Flashcards

1Core concepts of performance monitoring

2Metrics and measurement fundamentals

3Distributed tracing essentials

4Instrumentation and data collection

5Dashboards, alerts, and incident response

Learning Resources