70 questions / 10 random questions
Random questions, instant feedback, and review for missed questions.
View recommended SRE, Observability & Incident Response resources →
Which availability SLI best represents user experience for a search API?
Answer: The share of valid search requests answered correctly within the deadline
An availability SLI should measure the share of valid user requests that receive correct and timely responses, making it less dependent on internal topology.
For a 99.9% availability SLO over 30 days, approximately how much failure time is allowed?
Answer: About 43 minutes
Thirty days contain 43,200 minutes, and 0.1% is about 43.2 minutes. Production definitions must also state whether the SLI is time- or request-based, exclusions, and the window.
A short outage early in the month consumed 40% of a 30-day error budget. What is the appropriate response?
Answer: Check burn rate and cause and prioritize measures that reduce release risk
Burn rate shows whether continued consumption will violate the SLO. An error-budget policy should trigger proportionate release controls and reliability work.
Which alert design detects an SLO threat early while reducing pages from brief spikes?
Answer: Page when burn rate exceeds thresholds in both a short and a long window
Multi-window burn-rate alerting combines the responsiveness of a short window with confirmation from a longer window, paging on sustained budget threats.
Which condition is most appropriate for paging the on-call engineer?
Answer: User impact is ongoing, immediate action is needed, and a runbook exists
Pages should be reserved for urgent, actionable symptoms. Trends belong in tickets or forecasts, while informational events belong in dashboards or lower-urgency channels.
Database connection errors and API 5xx responses rise together. Which paging principle is best?
Answer: Page on the API success drop and use database metrics for diagnosis
A symptom-based page linked to causal dashboards reduces duplicate pages while connecting user impact to diagnosis.
What is the best basis for assigning alert severity?
Answer: Apply predefined criteria for user impact, urgency, and workarounds
Shared severity criteria enable consistent staffing, notification scope, and update cadence. Internal error volume alone does not represent business impact.
Which HTTP request metric label is most likely to cause a cardinality explosion?
Answer: The per-user user_id
Metrics create a time series per label combination. Unbounded values such as user IDs or raw URLs should be avoided, with detailed lookup moved to logs or traces.
Which set of signals makes up the RED method for an online service?
Answer: Rate, errors, and duration
RED captures service demand and symptoms through request rate, errors, and duration. Resource saturation can be investigated with methods such as USE.
Which combination does the USE method examine for a node resource bottleneck?
Answer: Utilization, saturation, and errors
USE examines utilization, saturation or queuing, and errors for each resource such as CPU, memory, disk, and network.
Which metric type is appropriate for aggregating latency across instances and calculating a global p95?
Answer: A histogram with consistent buckets
Histogram bucket counts can be summed across instances and used to estimate a quantile. Averaging per-instance quantiles does not produce the global quantile.
Which mechanism links a latency spike in metrics directly to representative traces?
Answer: Attach exemplars containing trace IDs to the histogram
Exemplars associate trace context with metric observations, allowing navigation from an anomalous bucket to a concrete request trace.
Tracing samples 10% in normal operation, but rare error traces should be retained. What is appropriate?
Answer: Tail-sample in the collector and keep errors and high latency preferentially
Tail sampling can decide after observing completed trace status and latency. Collector buffering, delay, capacity, and incident traffic still require design.
Which application log is both useful for incident diagnosis and appropriately limits sensitive data?
Answer: Structure timestamp, level, service, trace ID, and non-sensitive identifiers
Structured fields and correlation IDs support search, aggregation, and trace linkage. Secrets and unnecessary personal data should be masked or excluded.
How should asynchronous message processing be associated with the originating HTTP request trace?
Answer: Propagate producer context in the message and record a link on the consumer span
Propagating trace context or span links through message metadata preserves causality including queue delay. Incoming context should be validated across trust boundaries.
Logs from multiple hosts appear in an impossible order because clocks differ. What should be prioritized?
Answer: Monitor clock sync and distinguish event time from ingest time
Monitor synchronization such as NTP and retain both event and ingest timestamps. Use trace or message IDs to supplement causal ordering in distributed workflows.
What is most useful at the top of a service overview dashboard?
Answer: Key SLIs, SLO status, recent deployments, and links to dependency health
An overview should quickly show user impact and recent changes, then provide drill-down paths to diagnostic dashboards.
Internal metrics are healthy, but external users report login failures. Which monitoring addition is most useful?
Answer: Synthetic monitoring of the login flow from external locations
Synthetic monitoring continuously tests a critical journey through external DNS, CDN, TLS, and login paths, covering blind spots in internal monitoring.
Which telemetry measures regional page-rendering latency in real users' browsers?
Answer: Use RUM to aggregate web vitals by region and version
Real user monitoring captures actual device, network, and delivery-path experience. Privacy, sampling, and attribute cardinality must be designed carefully.
During an incident telemetry volume surges and the collector queue fills. Which design is appropriate?
Answer: Monitor collector queues and drops with priority-aware limits
The observability pipeline has capacity and failure modes. Define application backpressure, bounded buffering, priority retention, and visibility into dropped data.
During a major incident many engineers begin making changes at once and conditions worsen. What structure should be established first?
Answer: Name an incident commander and separate operations, communications, and planning
Explicit command and role separation prevent conflicting changes while technical work and communication proceed in parallel. The incident commander retains undelegated responsibility.
When should an incident receive an initial severity declaration?
Answer: Declare provisionally when impact and scope meet criteria, then revise
Severity is an initial decision used to activate response and communication. It should follow shared impact criteria without waiting for root-cause proof.
Which stakeholder update practice is appropriate during an incident?
Answer: Regularly share impact, start time, mitigation in progress, and next update time
Separating facts from unknowns and committing to the next update reduces interruptions to responders and preserves trust.
Payment failures are expanding and the cause is still unknown. What is the first technical objective?
Answer: Contain impact with rollback or traffic control while preserving evidence
During an incident, reducing user impact takes priority over proving root cause. Avoid irreversible actions and preserve timeline, logs, and change evidence.
Error rate rises immediately after a deployment. What preparation enables a fast rollback decision?
Answer: Define guardrail metrics and automatic rollback criteria before deployment
Predefined guardrails and version-aware telemetry distinguish normal variation from regression and speed canary stops or rollbacks.
An external recommendation service outage breaks the entire page. How should a feature flag be used?
Answer: Use a kill switch for recommendations and serve the core page degraded
A feature-scoped kill switch isolates a noncritical dependency and preserves the core journey. Manage authorization, audit, defaults, and stale-flag cleanup.
What should an actionable runbook contain?
Answer: Triggers, impact checks, safe mitigations, escalation, and rollback
A runbook must be safe under pressure and state expected results and abort conditions for each step. Exercises should verify access and effectiveness.
The primary on-call has investigated for 15 minutes without reducing impact. What escalation is appropriate?
Answer: Escalate with context to the owner or commander per predefined criteria
Escalation is a mechanism for adding time and expertise, not a failure. Pass impact, hypotheses, actions, outcomes, and the help needed.
Which practice creates an accurate incident timeline?
Answer: Record alerts, decisions, commands, and observations with time and actor in a shared log
A live incident document gives handoffs, parallel responders, and the postmortem a shared chronological record of facts and decisions.
How should incident command be handed to a responder in another region?
Answer: Review the live document verbally and announce the new commander's acceptance
Explicit acceptance and announcement preserve one commander and one shared state. Handoff includes impact, changes, hypotheses, risks, and next actions.
Which statement should be avoided in a blameless postmortem?
Answer: Declare operator inattention the sole cause and end improvements
Stopping at individual blame leaves conditions that can trap another person. Examine design, tools, reviews, permissions, workload, and missing defenses.
Which postmortem action item is most actionable?
Answer: A specific improvement with owner, due date, completion criteria, and ticket
Actions should be trackable and verifiable. Limit the list to high-value changes that reduce recurrence probability or impact.
A redundant system failed over successfully with no user impact. How should this near miss be handled?
Answer: Analyze the working defense, remaining margin, and conditions that could fail next
A near miss reveals weaknesses without user harm. Verify that defenses behaved as designed and that capacity and recovery margins were sufficient.
Error rate has just returned to normal. What is needed before closing the incident?
Answer: Verify sustained SLIs, cleared backlog, consistency, and removed degradation
Recovery means more than errors stopping; delayed work, consistency, capacity, and temporary mitigations must all return to a stable state.
After database failover, the old primary may resume writes and create split brain. What control is required?
Answer: Fence the old primary and grant writes only to the new one via generation or lease
Failover must exclude the old writer, not merely change routing. Fencing tokens, STONITH, or lease generations reject stale authority.
Backup jobs succeed every day. What additional check demonstrates recoverability?
Answer: Run regular restore drills in isolation and verify measured RTO and RPO
Backup success does not imply restore success. Test keys, access, procedures, dependencies, integrity, and elapsed recovery time through actual restoration.
CPU is normally 45%, and an event may double traffic. What is the appropriate capacity decision?
Answer: Load test the limits and reserve headroom including failure capacity
Validate headroom across database connections, queues, rate limits, scaling speed, and N+1 failure capacity. Load testing reveals nonlinear saturation.
An external address-validation API is down. Which degradation strategy safely continues order intake?
Answer: Accept into a pending state after minimal checks and revalidate after recovery
Where business rules allow, explicit deferred validation preserves the core flow. Design retries, deadlines, user communication, and deduplication.
Several deployments and configuration changes occurred just before an outage. What is the appropriate triage approach?
Answer: Overlay change events on the timeline and compare differences by scope
A recent change is a strong hypothesis but timing alone does not prove causality. Compare version-aware telemetry and unchanged groups, recording each mitigation result.
What is the appropriate purpose and method for a game day in a production-like environment?
Answer: Define recovery hypotheses and success criteria, then inject bounded failures
A game day is a hypothesis-driven exercise. Predefine blast radius, abort conditions, ownership, and rollback, then turn detection, communication, recovery, and verification gaps into actions.
Each layer retries three times, causing request volume to surge during a database outage. What is the core preventive design?
Answer: Centralize retries with backoff, jitter, and a global retry budget
Layered retries multiply attempts and further load a failing dependency. Classify retryability, retry with a budget in one layer, and stop work after the deadline.
Most instances are crash-looping from overload and do not recover when traffic returns to its normal level. What is the first stabilization action?
Answer: Throttle traffic, let caches stabilize, and ramp back gradually with checks
Normal traffic can still overload reduced capacity. Shed load to stabilize surviving instances, warm caches and connections, then restore traffic in controlled steps.
Under high load, slow responses fail liveness probes and trigger cascading pod restarts. What is the appropriate improvement?
Answer: Keep liveness light for unrecoverable failure and use readiness for traffic
Liveness failure causes restart, so tying it to overload or transient dependencies removes more capacity. Use readiness to remove traffic and reserve liveness for unrecoverable states such as deadlock.
A three-availability-zone service must handle peak traffic after losing one zone. Which capacity plan is appropriate?
Answer: Load-test the remaining two zones for peak plus recovery headroom
N+1 design evaluates surviving capacity after a failure-domain loss. Failure tests must include scale-out delay, quotas, cache warm-up, and database or network limits.
Pods take five minutes to become ready after a traffic surge, so autoscaling reacts too late. What is the appropriate improvement?
Answer: Scale earlier on leading signals with minimum capacity and prediction
Autoscaling loops include detection, provisioning, startup, and warm-up delays. Add capacity ahead of demand growth and retain headroom plus admission control for bursts.
A deployment renames a database column while old and new versions coexist during a rolling update. Which procedure is safe?
Answer: Add the new column, support both, backfill, then drop the old one later
Expand-and-contract first creates a schema usable by both versions, migrates data and traffic, and removes old elements later. Define the rollback window explicitly.
Errors increase after deployment, but an irreversible data migration has also completed. Which rollback decision is appropriate?
Answer: Check whether the old version reads the new format; otherwise choose a safe mitigation
Rollback is not always safe. Verify application, schema, data, and event compatibility in the runbook and choose the feasible mitigation that minimizes user harm.
An outage begins immediately after a configuration change is pushed to every instance. What prevents recurrence?
Answer: Version configuration, roll out by canary, and auto-roll back on health gates
Configuration is a release artifact that changes production behavior. Apply schema validation, peer review, staged rollout, audit history, and rapid rollback like code.
A bug keeps applying incorrect balance updates and data corruption is expanding. What is the most appropriate first response?
Answer: Stop or isolate affected writes, preserve evidence, and determine the corruption boundary
In a data-integrity incident, first stop ongoing corruption. Preserve snapshots, logs, events, and change history, identify scope and source of truth, then apply a tested repair.
Request count suddenly drops to zero on a dashboard. How should you distinguish service outage from telemetry failure?
Answer: Compare independent signals such as synthetic checks and LB data with collector export failures
No data and zero are different states. Monitor the user path and telemetry pipeline with independent signals, and alert on missing telemetry itself.
A cumulative request counter decreases after process restart. How should rate calculation handle it?
Answer: Use a reset-aware rate function and correlate with restart metrics
Counters reset to zero on restart. Use reset-aware queries such as Prometheus rate functions, and monitor frequent restarts as a separate signal.
A latency histogram has only 0.1-second and 10-second buckets, so a 0.5-second SLO cannot be measured. What is the improvement?
Answer: Design buckets that include the SLO threshold and evaluate compatibility
A histogram needs a bucket at the threshold to calculate the fraction below it. Choose buckets around representative latency ranges and SLO boundaries without excessive time-series cost.
An OpenTelemetry Collector stops under memory pressure and loses a batch of telemetry. What protection is appropriate?
Answer: Configure a memory limiter and bounded sending queue and monitor drops and queues
The collector is a finite-capacity pipeline. Control backpressure or drops before its memory limit and design signal priority, persistent queues where needed, and expected backend outage duration.
A high-traffic service must reduce log volume without losing errors needed for incident investigation. Which sampling policy is appropriate?
Answer: Retain errors and rare events and sample only repetitive success logs
Value-aware sampling protects diagnostically valuable logs. Record pre/post counts, rule versions, and trace correlation so operators understand what was omitted.
Head-based 1% trace sampling misses most critical errors occurring at 0.01%. What is the improvement?
Answer: Use tail sampling on completed traces to prioritize errors with a volume cap
Tail sampling uses completed trace outcomes to retain rare errors and tail latency. Monitor buffering memory, decision delay, trace completeness, and resulting sample rates.
CPU usage is high, but traces do not identify the expensive code path. Which additional telemetry is useful?
Answer: Use low-overhead continuous profiling to compare CPU by function
Profiling attributes CPU time and allocations to stacks, revealing hot paths not visible in metrics or traces. Correlate profiles with release versions and pod attributes.
Engineers manually repair user accounts with the same procedure every week. What is the appropriate SRE response to this toil?
Answer: Measure frequency, time, and risk, then invest in root-cause removal or safe automation
Toil is manual, repetitive, automatable, tactical work that scales with service growth. Make its volume visible and track reduction through engineering work.
One on-call engineer receives thirty pages overnight and responds slowly to a critical alert. What is the appropriate improvement?
Answer: Review pages for actionability, deduplicate and tune thresholds, and track load
Paging fatigue is a system risk. Page only for user impact requiring immediate action, route lesser signals to tickets or dashboards, and keep rotations sustainable.
A postmortem action added an alert, but it is unclear whether the same failure would now be detected earlier. How should completion be verified?
Answer: Replay historical data or run a controlled failure test to verify firing through response
A postmortem action is complete when risk reduction is validated, not merely when code changes. Test detection through human action, including false positives and notification paths.
An external payment provider's availability is below the checkout SLO. What is an appropriate SRE design decision?
Answer: Measure a journey SLI including the dependency and use fallbacks or queueing
Users experience checkout outcome, not ownership boundaries. Calculate dependency budgets and use redundancy, asynchronous flows, degradation, or contract changes to make the end-to-end SLO viable.
In one observation window, region A succeeds on 900 of 900 requests and B on 90 of 100. What is availability when every request has equal weight?
Answer: Sum successes and divide by total requests, giving 99%.
Overall success is 990/1000, or 99%. Averaging regional percentages gives equal weight to unequal traffic volumes. Keep regional monitoring as well.
The request-success SLO is 99.9%, and the current observation window has a 1% failure rate. What is burn rate, defined as failure rate divided by allowed failure rate?
Answer: Allowed failures are 0.1%, so the burn rate is 10 times.
The allowed failure fraction is 0.1%; 1% divided by 0.1% is 10. This does not alone establish that the monthly budget is already exhausted.
Prometheus stores cumulative request counters per pod. How should a service-wide per-second rate be computed while handling individual pod resets?
Answer: Apply rate to each pod counter, then sum by the required labels.
Apply rate before aggregation, for example sum by (service) (rate(http_requests_total[5m])). Aggregating first can hide one pod's reset behind another pod's growth.
A Prometheus alert uses for: 5m. For the same labels, its condition is true for three minutes, false at one evaluation, then true for three more minutes. No evaluations are missed. What is correct?
Answer: The false result resets pending time; five continuous minutes are needed again.
for requires continuous activity for the same alert element, not cumulative true time. A false evaluation resets its pending period.
One service incident triggers the same alert across many pods. Which Alertmanager feature retains the alerts but combines their notifications by service?
Answer: Use grouping to combine notifications using shared labels such as service.
Grouping combines related alerts into notifications. Choose scope labels carefully so unrelated incidents remain distinguishable.
While a cluster-down alert is firing, dependent notifications for that cluster should be suppressed automatically, but other clusters must remain unaffected. Which setup fits?
Answer: Configure inhibition with matching cluster labels for source and target.
Inhibition conditionally suppresses notifications. Match source and target cluster scope and ensure those labels are reliably present.
A planned maintenance needs one hour of notification suppression for one service only. Metrics and alert evaluation must continue. What should be done?
Answer: Set a scoped, expiring silence and verify its expiry.
A scoped silence suppresses matching notifications for a bounded period while monitoring continues. Record its purpose and verify it expires.
Small-file creation on Linux fails with ENOSPC, although df -h shows free space on the target filesystem. Which resource should be checked next?
Answer: Use df -i to check for exhausted filesystem inodes.
Free bytes and free inodes are separate resources. Exhausted inodes can prevent creation despite free capacity. Check the correct mount and handle files under an approved retention policy.
A large log file was deleted on Linux, but disk usage did not drop and the logging process is still running. What should be investigated?
Answer: Check open references to deleted files and identify the owning process.
An unlinked file can retain space while open. Identify the owner and use its supported log-reopen or controlled recovery procedure so the remaining reference can close.
An asynchronous order queue is short, but a few orders wait long enough to miss their deadline. Which metric complements queue length to detect this?
Answer: Compare the oldest outstanding order's age to its deadline.
Low depth does not imply low delay. Monitor outstanding-order age and completion latency. Broker age metrics may treat retries or dead-lettered items differently, so verify their scope.