40 questions / 10 random questions
Random questions, instant feedback, and review for missed questions.
What is the appropriate first design step for introducing generative AI to internal support?
Answer: Define target users, business outcomes, tolerated errors, and evaluation metrics
Defining the use case, affected people, success and failure criteria, and risk tolerance provides a basis for model, RAG, and human-review choices.
What baseline helps keep instructions distinct from user input and retrieved documents in a prompt?
Answer: Use roles, explicit delimiters, and rules for treating quoted data
Explicit instruction hierarchy and data boundaries clarify intent, though delimiters alone cannot fully prevent prompt injection.
Which parameter is commonly adjusted first to reduce output variability for the same prompt?
Answer: Lower temperature
Lower temperature reduces sampling randomness but does not guarantee full determinism; critical workflows also need schema and deterministic validation.
Is a JSON-looking LLM response safe to pass directly to a downstream API?
Answer: Use schema-constrained output and still parse and validate in the application
Even with structured-output features, validate types, ranges, business rules, and authorization in application code before execution.
Which design reduces hallucination impact when answering questions about internal policies?
Answer: Retrieve approved documents, cite evidence, and abstain when support is insufficient
Combine grounding, source display, and abstention when evidence is insufficient, escalating consequential decisions to humans.
Why consider RAG before fine-tuning for frequently updated private company knowledge?
Answer: Knowledge is easier to update or delete, and access-controlled evidence can be supplied at answer time
RAG manages knowledge in an external index, supporting updates, citations, and retrieval filtered by user authorization.
What is a primary use of embeddings?
Answer: Represent text or other data as vectors for semantic similarity search
Embeddings encode semantic features for similarity search and clustering. Authorization filters and sensitivity controls remain separate.
What is an appropriate way to choose chunk size for RAG?
Answer: Use document structure and question granularity, comparing size and overlap with retrieval evaluations
Oversized chunks add noise; undersized chunks lose context. Compare recall, precision, answer quality, and cost on representative queries.
Which retrieval approach supports both exact product-code matches and natural-language semantic search?
Answer: Hybrid search combining keyword and vector retrieval
Lexical retrieval excels at identifiers and exact terms, while vector retrieval captures semantic similarity. Evaluate score fusion and filters.
After broad first-stage retrieval, what technique can improve relevance of the top context?
Answer: Rerank candidates with a cross-encoder or similar model
A reranker scores query-candidate pairs more precisely, improving ordering at additional latency and cost, so bound the candidate set.
What is the most important access design for internal RAG where document permissions differ by department?
Answer: Filter retrieval by user identity and document ACL, and enforce authorization at the source
Unauthorized content should never enter model context. Keep ACL metadata and deletions synchronized when updating the index.
How should a team distinguish retrieval failures from generation failures in RAG?
Answer: Evaluate retrieval recall and relevance separately from groundedness and correctness
Separating whether gold evidence was retrieved from whether it was used correctly identifies whether to improve indexing, retrieval, prompting, or the model.
Which evaluation dimension measures whether an answer is supported by the supplied context?
Answer: Groundedness
Groundedness checks whether claims are supported by evidence. Evaluate it separately from correctness, relevance, and completeness.
What makes an appropriate evaluation dataset before production release?
Answer: Version representative queries, edge cases, adversarial inputs, expected answers, and evidence
Maintain a dataset reflecting real traffic and risks, and run it as a regression gate for model, prompt, and index changes.
What is an appropriate caution when using an LLM as a judge?
Answer: Use explicit rubrics, calibrate against human labels, and monitor position and style biases
Judge models have biases and variance. Validate against human gold labels and consider multiple judges and blinded comparisons.
What is a safe implementation when a model selects external functions through tool calling?
Answer: Use allowlisted tools and schemas, enforcing authorization and validation server-side
Treat model output as a proposed plan. A deterministic dispatcher validates tool name, arguments, user permissions, and rate limits.
An agent only summarizes documents. Which tool permission design is appropriate?
Answer: Read-only access only to the documents required
Avoid excessive agency by minimizing functionality, permissions, and autonomy to what the task actually requires.
An agent can propose payments or production deletions. What control is appropriate before execution?
Answer: Show the action, impact, and target, and require explicit approval from an authorized human
Use human-in-the-loop and separation of duties for irreversible or high-impact actions. Bind approval to exact action details and reapprove changes.
Agent retries created the same order twice. What should improve in the tool API?
Answer: Implement idempotency keys, duplicate detection, and explicit result states
Agent orchestration retries after ambiguous timeouts. Make side-effect APIs idempotent so one intent is applied only once.
A web-summarization agent obeyed text in a page saying to send secrets. What is this?
Answer: Indirect prompt injection through untrusted content
Treat retrieved content as data rather than instructions. Layer least privilege, instruction-data separation, output checks, and human approval.
How should LLM-generated HTML be handled before displaying it in a browser?
Answer: Treat it as untrusted output, escape or sanitize it, and apply controls such as CSP
Generated output may contain attack payloads or unsafe code. Use context-appropriate encoding, sanitization, and allowlists.
What data design is appropriate before sending customer-support transcripts to an external model API?
Answer: Minimize to the necessary data, redact PII, and verify provider terms and retention
Apply purpose limitation, minimization, redaction, and review contractual retention, region, and access controls before transmission.
What is an appropriate policy for storing prompts and responses in AI request logs?
Answer: Log necessary structured fields, redact sensitive values, and restrict access and retention
Prompts and responses may contain sensitive data. Define required metadata and sampling, with redaction, encryption, retention, and audit controls.
A multi-tenant AI service exposed tenant A's RAG document to tenant B. What is the core prevention?
Answer: Enforce tenant isolation across indexes, caches, logs, tool credentials, and authorization filters
Derive tenant identity from trusted authentication and propagate it through every data path. Post-retrieval prompt filtering is insufficient.
Model API rate limits cause cascading failures during traffic spikes. What design is appropriate?
Answer: Use concurrency limits, queues, exponential backoff with jitter, and bounded retries
Backpressure and retry budgets protect both provider and service. Also design async processing, priority, fallback, and explicit degraded responses.
What is a critical consideration when caching LLM responses?
Answer: Include tenant and permissions, prompt/model/knowledge versions, and TTL in key and policy
AI caches risk data leakage and stale answers. Account for security context and all input versions, with invalidation and encryption.
Simple classification and complex analysis have different cost and latency needs. What model-selection design fits?
Answer: Use an evaluated task router that prefers smaller models and escalates difficult cases
Choose the smallest model meeting each task's quality gate to reduce cost and latency, and evaluate routing mistakes.
How should a service maintain minimum functionality during a primary model-provider outage?
Answer: Use timeouts and circuit breakers with an evaluated fallback or degraded mode
Pre-evaluate fallback quality and safety, disclose degraded behavior, and verify data residency and API differences.
How should a P95 latency SLO be designed for an AI chat system?
Answer: Break down retrieval, reranking, model, tools, and queueing and allocate an end-to-end budget
AI chat latency is the sum of retrieval, reranking, model generation, tools, and queueing. Allocate the end-to-end SLO across stages and measure time to first token separately from completion.
What metric design helps understand generative-AI unit economics?
Answer: Track token, retrieval, and tool cost per successful task alongside quality
Aggregate token, retrieval, tool, and infrastructure cost per successful business outcome, and track it with quality and latency to optimize without eroding value.
Which observability design best supports root-cause analysis for an AI agent?
Answer: Correlate retrieval, model, and tool traces per request and record versions, latency, tokens, and errors
Generative-AI failures span retrieval, prompts, models, and tools. Correlated traces with prompt, model, index, and tool versions make failures diagnosable.
What is an appropriate way to detect quality degradation in a production RAG system?
Answer: Continuously measure retrieval hits, groundedness, and task success and correlate them with corpus and model changes
Answer quality can fall while HTTP requests still succeed. Combine golden-set checks, sampled review, and user signals, correlated with data, index, prompt, and model changes.
Which release-management practice improves reproducibility and rollback of AI responses?
Answer: Version prompts, models, parameters, indexes, and tool schemas and link evaluations to releases
AI behavior depends on many artifacts beyond code. Version the full release configuration with evaluation evidence to support staged rollout and rollback.
How should user thumbs-up/down feedback be used for improvement?
Answer: Collect task context and reasons, review bias, manipulation, and privacy, and use it as candidate evaluation data
Feedback contains selection bias, possible manipulation, and ambiguous intent. Review it with context and first use it for failure analysis and evaluation-set improvement.
Which principle is appropriate for generative-AI content-safety design?
Answer: Classify use-case risk and layer input and output guardrails, policy, human review, and appeal
Safety has false-positive and false-negative tradeoffs. Use risk-tiered policy and layered controls, evaluating refusal quality, bypass rates, and appeals.
How should fairness and accessibility be evaluated for a multilingual AI assistant?
Answer: Measure quality and failure rates by language, region, assistive technology, and key subgroup, with stakeholder review
Aggregate averages can hide severe subgroup failures. Combine representative test sets with qualitative review, checking task success and harmful disparities beyond translation fluency.
Which control should be prepared first for an incident where an AI agent starts making incorrect external-system changes?
Answer: A kill switch for tool execution, credential revocation, impact scoping, and audit-log preservation
For an agent incident, first contain capabilities and credentials while preserving evidence. Then roll back changes, notify affected users, analyze root cause, and improve controls.
How should model, dataset, and library supply-chain risks in an AI system be managed?
Answer: Inventory provenance, licenses, versions, hashes, evaluations, and known risks and use only approved artifacts
AI artifacts carry tampering, malware, licensing, data-provenance, and vulnerability risks. Pin sources, verify integrity, evaluate in isolation, and monitor updates.
When is fine-tuning an appropriate choice?
Answer: When sufficient quality examples exist to stabilize repeated format, style, or task behavior with holdout evaluation
Fine-tuning suits task-behavior adaptation, but it does not replace fresh knowledge retrieval or security boundaries. Compare quality, safety, and cost against a baseline on a holdout set.
Which lifecycle design is appropriate for generative-AI risk management?
Answer: Continuously govern, map, measure, and manage with owners for monitoring, change reassessment, incidents, and retirement
Generative-AI risk changes with models, data, usage, and the external environment. Define owners and risk tolerance and manage continuously from pre-deployment through monitoring, major changes, incidents, and retirement.