Senior Evaluation Rubric

Score meanings

Score	Anchor
0	Missing, incorrect, or dangerously hand-wavy.
1	Beginner: recognizes terminology and happy path with heavy prompting.
2	Mid-level: coherent standard design and common trade-offs, but gaps under failure/scale/evolution.
3	Senior: requirements-driven, quantitative, failure-aware, operable, and defensible.
4	Staff-oriented stretch: reframes the problem, handles organization/platform/migration complexity, and creates decision mechanisms for uncertainty.

Weighted scorecard

Dimension	Weight	1 — Beginner	2 — Mid-level	3 — Senior	4 — Staff stretch
Problem framing	10%	Restates prompt; unclear scope	Identifies core use cases	Prioritizes decision-changing requirements and non-goals	Reframes ambiguous product/risk boundary and negotiates scope strategically
Requirements and invariants	10%	Generic scale/latency list	Captures common functional/nonfunctional needs	Names operation-specific SLOs, consistency, RPO/RTO, security, and critical invariants	Classifies requirements by business risk and creates tiered policies
Quantitative reasoning	10%	No estimates or unit errors	Basic QPS/storage estimate	Peak, bytes, skew, fan-out, failure headroom, bottleneck-linked arithmetic	Sensitivity/cost/recovery model and validation plan under uncertainty
API and data contracts	10%	CRUD endpoints only	Reasonable API/schema	Idempotency, pagination, errors, state transitions, schema evolution, ownership	Cross-team contract governance, compatibility, and migration mechanisms
Architecture and data ownership	12%	Technology list/diagram	Coherent component flow	Clear authority, read/write paths, partitioning, replication, derived views	Simplifies boundaries, compares organizational options, plans platform evolution
Deep technical trade-off	12%	Memorized definitions	Explains standard alternatives	Traces protocol/state, chooses from workload, states limits and rejected options	Challenges premise and creates hybrid/novel approach with proof/experiment
Reliability and degraded behavior	12%	“Add replicas/retries”	Handles basic node failure	Overload, timeout ambiguity, partial failure, fencing, backpressure, recovery	Multi-region/correlated failure policy, game-day/decision automation, graceful product modes
Operability and observability	8%	“Add logs/metrics”	Names basic telemetry	User SLOs, p99, per-shard skew, lag/age, trace IDs, reconciliation, runbooks	Fleet governance, error budgets, capacity/control-plane design, incident learning loop
Security, privacy, abuse	6%	Omitted	Auth and encryption	Object/tenant authz, least privilege, limits, audit, deletion/retention, abuse	Data classification, regional/legal policy, platform guardrails and evidence
Evolution and migration	6%	Big-bang replacement	Mentions backward compatibility	Expand/backfill/validate/cutover/rollback with capacity and dual-state repair	Multi-year migration portfolio, incentives, adoption, decommission and policy
Communication and prioritization	4%	Rambling or silent	Understandable with prompts	Structured, states assumptions, responds to steering, summarizes decisions	Facilitates design review, surfaces disagreement and converges on decisions

Weighted score: sum (dimension score / 4) × weight. Do not hire by arithmetic alone; use it to expose evidence and calibration.

Suggested interpretation

Weighted result	Interpretation
<45%	Foundational gaps or insufficient evidence for the target level
45–62%	Developing/mid-level performance
63–78%	Senior range when no critical red flags exist
79–90%	Strong senior / staff signals depending on scope and independence
>90%	Exceptional performance; verify that depth is real rather than rehearsed

Dimension prompts for self-review

Problem framing

• Did I propose a clear scope and name non-goals?
• Did I ask questions that changed architecture rather than collect trivia?
• Did I identify actors and top journeys?

Requirements and invariants

• What must never happen?
• Which operation needs what consistency and durability?
• What can be stale, delayed, partial, rejected, or dropped?

Quantitative reasoning

• Did I include units and peak assumptions?
• Did I model the largest tenant/key and largest failure domain?
• Did the estimate actually justify a component or partition count?

Contracts

• What does a timeout mean?
• How is retry made safe?
• Is pagination stable? Are errors and async states explicit?
• Which producer owns an event and how does it evolve?

Architecture

• What is authoritative?
• How does one write commit and one read get served?
• Which copies are derived and how are they rebuilt?
• What key partitions each stateful component?

Reliability

• What happens under sustained overload, not only a brief spike?
• What if a node is isolated and still running?
• Which retries amplify failure?
• How do we recover and reconcile after communication returns?

Operability

• What would page the on-call?
• Which dashboard proves user health and data freshness?
• Can an operator pause or shed work safely?
• How do we distinguish process health from data completeness?

Evolution

• Can old and new clients coexist?
• Can schema and data migrate online?
• Where is rollback safe?
• Is temporary dual capacity included?

Critical red flags

Any of these should reduce confidence unless the candidate notices and repairs the mistake:

• Claims exactly-once external side effects without idempotency or reconciliation.
• Treats retries as always safe and ignores timeout ambiguity.
• Uses asynchronous replicas for correctness-sensitive reads without a freshness mechanism.
• Shards without naming the key, skew, cross-shard operations, or resharding.
• Uses a cache without tenant/auth keying or invalidation/stampede behavior.
• Claims CAP means any database permanently chooses two letters.
• Uses average traffic and 100% resource utilization as capacity.
• Adds a queue but never defines backlog SLO, retry, DLQ, or consumer idempotency.
• Stores unbounded arrays/partitions or permits unbounded fan-out.
• Omits authorization/tenant isolation in a multi-user system.
• Proposes a big-bang migration for critical data without rollback.

Positive senior signals

• Converts a vague requirement into a testable invariant.
• Says “I need to choose the consistency of this operation” rather than “the database is strongly consistent.”
• Carries an operation ID through API, database, event, and side effect.
• Calculates net backlog or replica catch-up rate.
• Measures per-shard max/p99 instead of cluster average.
• Uses immutable versions plus an atomic pointer for publication.
• Designs events for speed and reconciliation for completeness.
• Reserves capacity for the largest failure and repair.
• Offers a simpler initial design and a clear evolution threshold.
• Names evidence that would reverse the chosen decision.

Scoring a 45-minute mock

1. Record the answer.
2. Score each dimension immediately, with one quote/evidence.
3. Select only the two lowest high-weight dimensions.
4. Rework a five-minute segment that would raise each by one point.
5. Repeat the same prompt 48–72 hours later without reviewing the old diagram.

Blank scorecard

Prompt:
Date / target level:

Problem framing (0–4):
Evidence:
Requirements/invariants (0–4):
Evidence:
Quantitative reasoning (0–4):
Evidence:
API/data contracts (0–4):
Evidence:
Architecture/ownership (0–4):
Evidence:
Deep trade-off (0–4):
Evidence:
Reliability/degraded behavior (0–4):
Evidence:
Operability/observability (0–4):
Evidence:
Security/privacy/abuse (0–4):
Evidence:
Evolution/migration (0–4):
Evidence:
Communication/prioritization (0–4):
Evidence:

Strongest moment:
Most consequential gap:
Five-minute segment to redo:
One quantitative assumption to validate:
One alternative I should have compared:

Interviewer calibration note

Evaluate the level of independence and the complexity actually handled. A candidate who reaches a sound design after direct hints may demonstrate knowledge but not the same senior signal as one who proactively identifies the issue. Do not reward buzzword density. Reward correct contracts, prioritization, recovery reasoning, and the ability to revise when challenged.