AI Engineer Interview: Rounds, Questions and How to Prepare

The AI engineer interview is the newest loop in the industry and the least standardized. A backend engineer who has shipped two LLM features at work gets a recruiter message for an "AI Engineer" role, says yes, and then finds there is no agreed syllabus: one company runs a normal software loop with a retrieval question added, another sends a take-home that asks for a working question-answering system with an evaluation set.
The demand is real. Indeed's Hiring Lab reported in July 2026 that postings with AI in the job title accounted for 37% of the increase in US software development postings between May 2025 and May 2026. The preparation material has not caught up, and most of what exists is either a machine learning syllabus you do not need or a list of buzzwords. This guide describes the role as it is hired now, the rounds prep sites and candidates describe, what gets asked in each, and a four-week plan.
What the title means now
An AI engineer, as the title is used in 2026, builds products on top of models that someone else trained. The work is software engineering with a probabilistic component in the middle: retrieval pipelines, prompts and tool definitions, evaluation harnesses, serving, cost control and the guardrails around all of it.
That is a different job from machine learning engineer, and the interviews differ accordingly.
| AI engineer | Machine learning engineer | |
|---|---|---|
| Starts from | A hosted or open-weight foundation model | Data and a training objective |
| Core work | Retrieval, orchestration, evaluation, serving | Feature pipelines, training, model quality |
| Theory asked | How LLMs behave and fail | Statistics, optimization, model internals |
| Design round | A RAG or agent system | A training and serving pipeline, a ranking system |
| Closest relative | Backend engineer | Data scientist or research engineer |
One prep site puts it compactly: the AI engineer loop is a software engineering screen with retrieval and serving questions layered in, while ML engineer loops spend their time on misbehaving models and messy data. Titles are used loosely, so read the job description. If it mentions training, fine-tuning at scale or a research team, you are closer to the right-hand column and should prepare for it with the machine learning engineer interview guide.
The usual loop
As of November 2026 no company's candidate pages read for this guide describe a standard AI engineer loop, and the accounts differ. Two descriptions from prep sites, both updated in 2026, mark the range.
The first says there is often no separate loop at all: AI engineering questions appear inside software engineer, ML engineer and forward-deployed engineer interviews, with project-style coding rounds in which an AI assistant is often allowed.
The second describes a dedicated loop of four or five rounds in which only one resembles a classic algorithms interview:
- Technical phone screen. Practical questions on retrieval and prompting: how you would chunk, how you choose top-k, when retrieval is the wrong tool.
- Take-home. Build a small LLM system end to end and measure it, for example a question-answering service with an evaluation set and reported metrics.
- Live coding or debugging. Often diagnosing a broken pipeline or an agent stuck in a tool-call loop, not a puzzle.
- System design. Design the serving and evaluation path around a model you do not control.
- Product or customer round. Turn a vague complaint ("the bot gives wrong answers") into a scoped, measurable piece of work.
Add a behavioral round, which every loop has. Treat both descriptions as reports; the mix at your company may differ, and the recruiter will tell you if you ask for the list of rounds and what each assesses. Ask also whether AI tools are allowed in the coding round, since policies vary by company, as the guide to AI in coding interviews sets out.
Fundamentals you are expected to explain
You will not be asked to derive backpropagation. You will be asked to explain, in plain words, why the system behaves as it does. The topics that recur:
- Tokens. Models read and bill in tokens, not characters. Know roughly how text maps to tokens, why that sets both cost and latency, and why output tokens cost more time than input tokens: they are generated one at a time.
- Context window. The model has no memory beyond what is in the prompt. Everything it "knows" about your user arrives in the context, and quality tends to degrade as the context fills with marginal material. Be ready to say what you would cut first.
- Embeddings. Text mapped to vectors so that similar meaning lands nearby. Know what they are good at (paraphrase, topic) and bad at (exact identifiers, negation, numbers), which is the argument for hybrid search.
- Sampling. Temperature and why you set it low for extraction and higher for drafting. Why the same prompt gives different answers and what that does to testing.
- Fine-tuning versus retrieval. Retrieval when the knowledge changes or must be cited; fine-tuning when you need a behavior, a format or a tone the base model will not hold. Prompting first in both cases, because it is the cheapest experiment.
- Hallucination. A model produces fluent text whether or not it has grounds. Know the mitigations and their limits: grounding in retrieved sources, asking for citations, allowing "I don't know", checking output against the sources.
- Structured output and tool calls. How a model is made to return JSON or call a function, and what you do when the output does not parse.
A good test of whether you know a topic: explain it to a product manager in three sentences, then answer one "why" from an engineer.
Retrieval and agent design questions
The design round is where senior candidates are separated, and the commonest prompt is a retrieval system over a company's documents. The full worked answer is in the LLM system design interview guide. What interviewers probe, in short:
- Chunking. Why that size, why that overlap, what happens to tables and code, what metadata rides along.
- Retrieval quality. Keyword plus vector search, a reranker over the top candidates, and how you would know whether a bad answer came from retrieval or from generation.
- Freshness and permissions. How an edited document reaches the index, and how you guarantee a user never sees text from a file they cannot open.
- Cost and latency. Where the time goes in a request and what you would cache.
Agent questions are the newer half. An agent is a loop: the model chooses a tool, your code runs it, the result goes back into the context, repeat until done. Interviewers ask about the engineering around that loop:
- Tool design. Few tools with clear names and typed arguments beat many overlapping ones. What does each tool return on failure?
- Termination. A step limit, a budget, and detection of repeated identical calls.
- State. What is kept between steps, what is summarized, what happens when the context fills.
- Side effects. Which actions need confirmation from a person, and how you make a tool call safe to retry.
- When to skip the agent. A fixed pipeline is cheaper, faster and testable. Saying "this does not need an agent" and explaining why is a strong answer when it is true.
For classic background on how these systems are structured, the older post on ML and AI system architectural patterns is still a useful map.
Evaluation and guardrails questions
One of the prep sites above calls evaluation the round candidates most often fail, and it is easy to see why. Engineers who have built a demo know how to make it work once. The job is knowing whether it works across a thousand inputs and whether Tuesday's prompt change made it worse.
Have a clear account of each of these:
- An evaluation set. Where the examples come from (real queries, sampled and labeled), how many you need to see a difference, and how you keep it from going stale.
- Retrieval metrics versus answer metrics. Recall at k and ranking measures for retrieval; faithfulness to sources and correctness for the answer. Measured separately, because they fail separately.
- LLM as judge. Using a model to grade outputs is cheap and scales. It needs calibrating against human labels before you trust it, and re-checking when you change the judge.
- Regression testing. A fixed set that runs on every prompt or model change, with a threshold that blocks the release.
- Online signals. Thumbs, edits, escalations to a human, repeated questions. Noisy, and still the only measure of what users experience.
- Guardrails. Input checks (prompt injection, out-of-scope requests), output checks (unsupported claims, private data, unsafe content), and what the user sees when a check fires.
- Prompt injection through retrieved content. A document that says "ignore your instructions" is data, and the system must treat it that way. Know why this is hard and what you would do: least-privilege tools, confirmation for actions, separating instructions from retrieved text.
The coding round
Expect ordinary software engineering with an LLM-flavored task. Examples of the shape, invented for illustration:
- Implement a text chunker with overlap that respects paragraph boundaries, with tests.
- Given embeddings as arrays, implement top-k cosine similarity, then make it fast for a million vectors.
- Write a wrapper around a flaky model API with retries, timeouts, backoff and a token budget.
- Parse and validate a model's JSON output against a schema and repair or retry on failure.
- Debug a small agent loop that never terminates.
Classic data structures still appear, especially in loops that are a software engineering loop with additions, so keep that base. Python is the safe default unless the job description says otherwise. Write tests without being asked; in this field more than most, the interviewer wants to see that you verify.
Thirty questions in five groups
These are example questions of the kind described above, written for practice. They are not a leaked list from any company.
Fundamentals
- What is a token, and why does it matter for cost and latency?
- The context window is ten times larger than last year. Why is retrieval still needed?
- What does an embedding capture, and what does it miss?
- When would you fine-tune instead of retrieving?
- Why does the same prompt return different answers, and how do you test around that?
- What causes a model to state something false with confidence?
Retrieval
- How do you choose chunk size and overlap for a mixed corpus of wiki pages and PDFs?
- Vector search returns the wrong document for a query containing a product code. Why, and what do you change?
- What does a reranker do, and where does it sit?
- How do you enforce per-user document permissions in retrieval?
- A document is edited. Walk me through how the answer changes and how long it takes.
- Retrieval looks fine and answers are still wrong. What do you check?
Agents and tools
- Design the tools for an agent that handles refund requests.
- How do you stop an agent that keeps calling the same tool?
- Which actions need a human to confirm, and how is that built?
- The conversation is longer than the context window. What do you keep?
- When is a fixed pipeline better than an agent?
- How do you make a tool call safe to retry?
Evaluation and safety
- Build an evaluation set for a support assistant from nothing. Where do the examples come from?
- How do you measure faithfulness to sources?
- You use a model as a judge. How do you know the judge is right?
- A prompt change improved one metric and hurt another. How do you decide whether to ship?
- How would you defend against prompt injection in retrieved documents?
- What do you log, given that prompts contain user data?
Serving, cost and judgment
- Where does the time go in a retrieval-augmented request, and what do you do about the largest part?
- Cut the cost of this feature by half without a visible quality drop. Where do you start?
- The model provider has an outage. What does the user see?
- When do you route to a smaller model?
- Tell me about an LLM feature you shipped that did not work at first. What did you measure?
- A stakeholder wants a chatbot. How do you find out whether they need one?
Answer each aloud in two minutes. For the design-flavored ones, say the trade-off you are making and what you would measure to check it. That last habit is the fastest way to sound like someone who has operated one of these systems.
A four-week plan
For an experienced software engineer with some exposure to LLM APIs, at seven to ten hours a week.
- Week 1: fundamentals and one build. Write three-sentence explanations of every item in the fundamentals section. Build a small retrieval question-answering service over a corpus you know, such as your own notes or a project's documentation. Keep it ugly and working.
- Week 2: evaluation. Write fifty question and answer pairs for your service. Measure retrieval and answers separately. Change the chunking, then add a reranker, and record what each did to the numbers. This week gives you most of your interview stories.
- Week 3: agents and design. Add two tools and a step limit to make a small agent; break it on purpose and fix it. Do two full design rounds aloud, 45 minutes each, using the worked RAG answer as a template. Do three coding problems of the kind listed above.
- Week 4: rehearsal. Answer the thirty questions aloud, recorded. Do one mock design round and one mock coding round with another person. Prepare four behavioral stories, at least two about an AI feature: one that failed first, one where you chose the simpler option. For a staff-level role, pick them against the signals in the staff engineer behavioral interview questions.
If you have only two weeks, do weeks 1 and 2 compressed and the mocks. The evaluation work is the part most candidates lack and the part that is hardest to fake.
The project gives you things to say, but it does not tell you whether your design answer holds up under questioning. For that you need someone who has built these systems to push on it, and a session with a generative AI and LLM coach is one way to get that before the real round.
Where coaching fits
The field is new enough that few people have both built LLM systems in production and interviewed candidates on them, so check a coach's background before you book. On techinterview.coach, four of the 40 coaches list generative AI and LLMs as a specialty and ten list artificial intelligence, as of November 2026; the generative AI and LLM coaches page shows the first group. A mock design round on a retrieval or agent system is the most useful single session. For speaking practice alone, the practice interview offers a system design round with an AI interviewer and a written recap, and it can be grounded in a job ad you paste in. It is free; sign in to start.
Related guides
- LLM system design interview: a full worked answer for designing a RAG system.
- AI in coding interviews: whether you may use an assistant in the coding round, by company.
- ML and AI system architectural patterns: the broader map of how these systems are structured.
- System design interview questions by level: what depth is expected at senior and staff.
- Mock interviews guide: how to run the week-four rehearsals.