Sequence 11 - 面试官和团队调研、问题预测、反问问题

这一页只放面试分析，不放自我介绍脚本。自我介绍请看 sequence-08-self-introduction-and-opening-script.html。

1. 面试官信息怎么理解

截图里写的是：

Interviewer(s): Yoray, Asaf

这更像 Workday/HR 系统里的姓名格式：Last name, First name。也就是说，它更可能是一个人：Asaf Yoray，而不是两个面试官 Yoray 和 Asaf。

当前公开信息不足以可靠确认 Asaf Yoray 的个人履历、具体组别、职级、论文或开源贡献。所以不要在面试中假设他一定做过某个具体项目，也不要主动提“我查到你做过什么”。正确准备方式是：

个人背景：公开信息不足，不做确定判断。
团队方向：基于 JD 和 NVIDIA Israel / Networking 公开信息做强推断。
问题预测：按 AI Networking Research / GPU Networking / LLM inference architecture 准备。

2. JD 本身给出的强信号

结论：

这不是纯 Python app / LLM prompt 工程岗位。
这也不是纯 CUDA kernel 岗位。
它是 AI workload + GPU networking + communication library + system architecture 的交叉岗位。

3. 以色列 / NVIDIA Networking 背景信号

NVIDIA 在 2020 年完成对 Mellanox 的收购，Mellanox 的核心能力是高性能网络，包括 InfiniBand、Ethernet、数据中心和 HPC 网络栈。NVIDIA 官方新闻稿也强调这次合并是 compute 和 networking expertise 的结合。

所以，如果面试官来自 Israel / Tel Aviv / networking 相关团队，要默认他们更懂：

InfiniBand / Ethernet / RDMA
UCX / NCCL / MPI
GPUDirect RDMA
NIC/GPU/CPU/memory topology
AI data center scale-out networking
communication library performance

不要把他们当成普通后端面试官。更像是系统架构、GPU networking、AI infrastructure 方向的人。

4. Tel Aviv 相似岗位给出的额外信号

公开 NVIDIA jobs 页面里有相似方向的 Tel Aviv Senior Software Architect, AI Networking。它强调：

multi-node LLM inference across GPU clusters
latency / throughput / cost-efficiency
KV cache handling
tensor / pipeline parallel execution
dynamic batching
memory hierarchy
network topology
modern inference architectures
C++ / Python / CUDA
memory / networking / scheduling / compute orchestration

这和 JR2013092 高度同向。即使不能证明你的面试官来自这个具体组，也足够说明：这次面试大概率不是闲聊 CV，而是围绕 AI networking 和 inference architecture 深挖。

5. 他们可能的视角

flowchart TB
    Interviewer[Possible Interviewer Perspective] --> Arch[System Architecture]
    Interviewer --> Net[GPU Networking / Transport]
    Interviewer --> Inference[LLM Inference Runtime]
    Interviewer --> Perf[Performance Profiling]
    Interviewer --> Prototype[Prototype / Design Validation]

    Arch --> DataPath[Data Path / Control Path / Failure Mode]
    Net --> UCX[UCX / RDMA / GPUDirect / NCCL / NIXL]
    Inference --> KV[Prefill / Decode / KV Cache / Batching]
    Perf --> Metrics[TTFT / TPOT / P99 / Bandwidth / GPU Utilization]
    Prototype --> Evidence[Microbenchmark / Ablation / Design Doc]

5.1 面试官画像：不要猜个人履历，要按岗位画像准备

公开搜索没有可靠证据能确认 Asaf Yoray 的具体公开履历、论文、开源贡献或团队职级。因此不要在面试中说：

I saw that you worked on ...

更稳的说法是：

From the job description, my understanding is that this role is close to AI networking, distributed inference, data movement, and communication-library architecture. I prepared around LLM inference paths, NIXL/UCX/NCCL/GPUNetIO boundaries, GPU/NIC data paths, and prototype-driven validation.

这等于把“我查了你”改成“我认真读了岗位并按团队方向准备”。这样不会因为查错人而翻车。

5.2 这场面试更可能是什么风格

6. 他们会更看重什么

7. 他们可能不太关心什么

1. 不会太关心你会不会写漂亮前端。
2. 不会太关心普通 CRUD 后端经验。
3. 不会太关心泛泛的 LLM agent/prompt。
4. 不会只听业务结果，会追问系统路径、指标和瓶颈。

7.5 深度设计区：数据、结构、路径、指标

这一节是为了补足“只有方向没有具体设计”的问题。面试官如果让你白板设计、追问数据结构、追问怎么测，这里直接拿来用。

7.5.1 多节点 LLM 推理系统设计 / Multi-node LLM inference system design

中文设计：

目标：设计一个多节点 LLM inference system，既要高吞吐，也要低 P99。不能一上来讲 NIXL/UCX API，而要先讲 workload 和 data path。

输入数据：
1. Request：request_id、tenant_id、model_id、prompt_tokens、max_new_tokens、priority、deadline、streaming flag。
2. Model profile：num_layers、hidden_size、num_heads、KV dtype、TP/PP 配置、单 token KV 大小。
3. Cluster state：GPU 数量、GPU memory free、NIC/GPU topology、worker load、queue depth。
4. Runtime metrics：TTFT、TPOT、P50/P95/P99、GPU utilization、KV cache usage、network bandwidth、transfer latency。

核心模块：
1. Gateway：接请求、认证、限流、生成 trace_id。
2. Admission Controller：根据 queue depth、deadline、KV memory 估算是否接收。
3. Scheduler：选择 prefill worker 和 decode worker。
4. Prefill Worker：处理 prompt，生成 KV cache。
5. KV Manager：管理 KV block、引用计数、eviction、transfer task。
6. NIXL/Transfer Layer：搬运 KV/state。
7. Decode Worker：连续 decode，输出 token stream。
8. Metrics/Trace：记录每段 latency 和 failure mode。

English design:

The goal is to design a multi-node LLM inference system with high throughput and low P99 latency. I would not start from a NIXL or UCX API. I would start from the workload and the data path.

The input data includes request metadata, model profile, cluster state, and runtime metrics. Request metadata includes request_id, tenant_id, model_id, prompt tokens, max_new_tokens, priority, deadline, and whether streaming is required. The model profile includes number of layers, hidden size, number of heads, KV dtype, tensor or pipeline parallel configuration, and KV size per token. Cluster state includes GPU count, free GPU memory, GPU/NIC topology, worker load, and queue depth. Runtime metrics include TTFT, TPOT, P50/P95/P99, GPU utilization, KV cache usage, network bandwidth, and transfer latency.

The core modules are gateway, admission controller, scheduler, prefill worker, KV manager, transfer layer, decode worker, and metrics/tracing. The gateway receives requests and assigns trace ids. The admission controller decides whether a request can be accepted based on queue depth, deadline, and KV memory. The scheduler selects prefill and decode workers. Prefill builds KV cache. KV Manager tracks KV blocks, reference counts, eviction, and transfer tasks. The transfer layer moves KV/state. Decode workers generate tokens. Metrics and tracing explain latency and failures.

flowchart TB
    Client[Client Request] --> Gateway[Gateway / Trace ID]
    Gateway --> Admission[Admission Controller]
    Admission --> Scheduler[Scheduler]
    Scheduler --> Prefill[Prefill Worker]
    Prefill --> KVManager[KV Manager]
    KVManager --> Transfer[NIXL / UCX Transfer Layer]
    Transfer --> Decode[Decode Worker]
    Decode --> Stream[Streaming Response]

    Scheduler --> Cluster[Cluster State]
    KVManager --> KVMeta[KV Block Metadata]
    Transfer --> TransferTask[Transfer Task Queue]
    Gateway --> Trace[Metrics / Trace]
    Prefill --> Trace
    Transfer --> Trace
    Decode --> Trace

核心数据结构可以这样讲：

RequestContext {
  request_id;
  tenant_id;
  model_id;
  prompt_tokens;
  max_new_tokens;
  priority;
  deadline_ms;
  trace_id;
  prefill_worker;
  decode_worker;
}

KVBlockMeta {
  block_id;
  request_id;
  layer_id;
  token_range;
  dtype;
  bytes;
  location;       // GPU id / CPU / remote
  ref_count;
  last_access_ts;
  state;          // resident / transferring / evicted
}

WorkerState {
  worker_id;
  node_id;
  gpu_id;
  role;           // prefill / decode / mixed
  queue_depth;
  free_gpu_mem;
  gpu_util;
  nic_affinity;
  active_requests;
}

TransferTask {
  task_id;
  request_id;
  src_location;
  dst_location;
  bytes;
  priority;
  deadline_ms;
  status;         // queued / running / done / failed
}

面试收束：

这个设计的关键不是模块多，而是每个模块都有指标。Gateway 看 admission 和 queueing，Prefill 看 TTFT，Decode 看 TPOT，KV Manager 看 memory pressure 和 eviction，Transfer Layer 看 latency/bandwidth/overlap，最终看 P99 和端到端吞吐。

English close:

The key point is not having many modules. The key point is that every module has measurable signals. Gateway tracks admission and queueing. Prefill affects TTFT. Decode affects TPOT. KV Manager tracks memory pressure and eviction. The transfer layer tracks latency, bandwidth, and overlap. The final decision depends on P99 and end-to-end throughput.

7.5.1.0 设计决策框架：不是简单拆模块 / Design decision framework, not just module listing

这一段是面试里最重要的。面试官真正想看的不是你能不能背出 Gateway、Scheduler、KV Manager，而是你能不能把一个模糊需求变成可验证的系统设计。

中文：

我会按六步做设计：

第一步，定义 workload 和 SLO。
我不会直接说“用 prefill/decode disaggregation”或者“用 NIXL”。我会先问：模型多大、单卡能否放下、prompt 长度分布、output 长度分布、QPS、并发 session、是否 streaming、多租户优先级、TTFT/TPOT/P99 SLO、GPU/NIC topology、是否跨节点。因为不同 workload 的瓶颈完全不同。

例如：
1. 短 prompt chat：更关心 TPOT、batching、streaming latency。
2. 长上下文 RAG：更关心 prefill、KV memory、TTFT。
3. Agent 长输出：更关心 decode fairness、长 session 不要占满 batch。
4. 离线 batch：更关心 throughput 和 GPU utilization，P99 可能不是第一目标。

第二步，区分 control path 和 data path，但不是停留在概念。
Control path 处理 metadata 和决策：认证、租户、admission、routing、scheduler decision、worker heartbeat、trace、failure recovery。它的数据小，可以进数据库或 control-plane service，但不能阻塞 GPU hot path。
Data path 处理真实高频数据：prompt tensors、KV cache、GPU memory、decode tokens、network transfer、streaming output。它要尽量 async、少 copy、少锁、少跨进程 hop。

这么分的原因是：control path 需要正确性、可追踪、可恢复；data path 需要低延迟、高吞吐、低 copy。如果把两者混在一起，慢 DB、慢 RPC、日志写入、调度锁都会影响 GPU 执行，P99 会非常难查。

第三步，判断 prefill/decode 是否要分离。
不是所有系统都应该分离。我的判断条件是：
如果 prompt length 分布很偏、长 prompt 经常拖慢短请求、prefill compute-heavy、decode latency-sensitive，而且 KV transfer 可以和计算 overlap，那么分离可能有价值。
如果模型/请求主要是短 prompt、集群很小、KV transfer 跨节点很贵、topology 不好、或者分离后 orchestration overhead 大于收益，那就不应该分离。

我会用一个简单收益判断：
benefit = saved_queueing_time + improved_decode_TPOT + better_GPU_specialization
cost = KV_transfer_latency + metadata_orchestration_overhead + failure_recovery_cost + extra_memory_fragmentation
只有 benefit 明显大于 cost，并且 P99 变好，才认为 disaggregation 成立。

第四步，选择 parallelism。
如果模型能放进单 GPU，优先 data parallel 做多副本，提高 QPS，系统简单。
如果模型单卡放不下，或者单 token latency 需要多 GPU 协同，考虑 tensor parallel，但要承担 all-reduce/all-gather 通信成本，所以更适合同节点 NVLink/NVSwitch。
如果模型层数多、batch 足够大、吞吐目标更强，可以考虑 pipeline parallel，但要处理 pipeline bubble 和调度复杂度。
如果跨节点 tensor parallel，必须非常谨慎，因为网络通信会进入每个 token 的 critical path，TPOT/P99 可能被放大。

第五步，设计 scheduler，不是只说 dynamic batching。
Admission 先算成本：prefill cost、decode cost、KV memory cost、deadline risk。
Prefill scheduler 按 prompt length bucket、tenant priority、deadline 组织 batch，避免长 prompt 拖短 prompt。
Decode scheduler 做 continuous batching，但要有 fairness：长 output request 不能一直占 batch；短 request 也不能为了凑大 batch 等太久。
Worker 选择可以用打分模型：
prefill_score = queue_wait + estimated_prefill_time + gpu_memory_pressure + allocation_cost
decode_score = active_sequences + estimated_TPOT + KV_memory_pressure + transfer_cost + topology_penalty + deadline_penalty
选择 score 最低且满足 SLO 的 worker。

第六步，验证。
我会先做 mixed baseline，也就是 prefill/decode 不分离。然后做 disaggregated prototype，对比 TTFT、TPOT、P99、GPU utilization、KV memory pressure、transfer latency、failure rate。不能只看 microbenchmark 的 bandwidth，要看 end-to-end request timeline 是否变短、P99 是否稳定。

English:

I would design it in six steps.

First, define the workload and SLOs.
I would not immediately say "use prefill/decode disaggregation" or "use NIXL". I would first clarify the model size, whether the model fits on one GPU, prompt length distribution, output length distribution, QPS, concurrent sessions, streaming requirement, tenant priority, TTFT/TPOT/P99 SLOs, GPU/NIC topology, and whether execution is cross-node. Different workloads have different bottlenecks.

For example:
1. Short-prompt chat cares more about TPOT, batching, and streaming latency.
2. Long-context RAG cares more about prefill, KV memory, and TTFT.
3. Agent workloads with long outputs care about decode fairness and long sessions not monopolizing the batch.
4. Offline batch workloads care more about throughput and GPU utilization, while P99 may be less important.

Second, separate control path and data path in a concrete way.
The control path handles metadata and decisions: authentication, tenant identity, admission, routing, scheduler decisions, worker heartbeat, tracing, and failure recovery. The data is small and can go through control-plane services, but it must not block the GPU hot path.
The data path handles high-frequency data: prompt tensors, KV cache, GPU memory, decode tokens, network transfer, and streaming output. It should be asynchronous, low-copy, low-lock, and avoid unnecessary process hops.

The reason is that the control path needs correctness, traceability, and recoverability, while the data path needs low latency, high throughput, and low copy overhead. If they are mixed together, slow databases, slow RPCs, logging, or scheduler locks can hurt GPU execution and make P99 very hard to debug.

Third, decide whether prefill and decode should be disaggregated.
Not every system should do it. I would consider disaggregation if prompt length distribution is highly skewed, long prompts frequently hurt short-request TTFT, prefill is compute-heavy, decode is latency-sensitive, and KV transfer can overlap with compute.
I would avoid it if requests are mostly short prompts, the cluster is small, cross-node KV transfer is expensive, topology is poor, or orchestration overhead is larger than the benefit.

A simple decision rule is:
benefit = saved_queueing_time + improved_decode_TPOT + better_GPU_specialization
cost = KV_transfer_latency + metadata_orchestration_overhead + failure_recovery_cost + extra_memory_fragmentation
Disaggregation is justified only when benefit is clearly larger than cost and P99 improves.

Fourth, choose the parallelism strategy.
If the model fits on one GPU, I would prefer data parallel replicas for higher QPS and simpler operations.
If the model does not fit on one GPU, or per-token latency requires multiple GPUs, I would consider tensor parallelism, but it introduces all-reduce/all-gather communication and is better within a node with NVLink or NVSwitch.
If the model has many layers and batch size is large enough, pipeline parallelism can improve throughput, but it introduces pipeline bubbles and scheduling complexity.
Cross-node tensor parallelism needs extra caution because communication can enter the critical path of every token and hurt TPOT/P99.

Fifth, design the scheduler, not just dynamic batching.
Admission estimates prefill cost, decode cost, KV memory cost, and deadline risk.
The prefill scheduler groups requests by prompt-length bucket, tenant priority, and deadline, so long prompts do not delay short prompts.
The decode scheduler uses continuous batching with fairness: long-output requests should not monopolize the batch, and short requests should not wait too long just to create a larger batch.
A worker scoring model could be:
prefill_score = queue_wait + estimated_prefill_time + gpu_memory_pressure + allocation_cost
decode_score = active_sequences + estimated_TPOT + KV_memory_pressure + transfer_cost + topology_penalty + deadline_penalty
The scheduler chooses the lowest-score worker that still satisfies the SLO.

Sixth, validate the design.
I would first build a mixed baseline where prefill and decode are colocated. Then I would build a disaggregated prototype and compare TTFT, TPOT, P99, GPU utilization, KV memory pressure, transfer latency, and failure rate. I would not rely only on microbenchmark bandwidth. The end-to-end request timeline and P99 stability must improve.

flowchart TB
    A[Start from Workload and SLO] --> B{Model fits one GPU?}
    B -->|Yes| C[Prefer Data Parallel Replicas]
    B -->|No| D{Need lower per-token latency or model sharding?}
    D -->|Yes| E[Consider Tensor Parallel]
    D -->|Layer throughput target| F[Consider Pipeline Parallel]

    A --> G{Prompt length highly skewed?}
    G -->|Yes| H{KV transfer can overlap and topology is good?}
    H -->|Yes| I[Use Prefill Decode Disaggregation Prototype]
    H -->|No| J[Keep Prefill and Decode Colocated]
    G -->|No| J

    I --> K[Measure TTFT TPOT P99 Transfer Latency]
    J --> K
    C --> K
    E --> K
    F --> K
    K --> L{End-to-end P99 and throughput improve?}
    L -->|Yes| M[Keep Design]
    L -->|No| N[Rollback or Change Scheduler Placement Transfer]

7.5.1.1 逐模块展开：每个模块到底做什么 / Module-by-module deep dive

Gateway

中文：

Gateway 是入口层，不做复杂推理。它负责 HTTP/gRPC 接入、认证、租户识别、限流、request_id/trace_id 生成、基础参数校验，以及把请求转换成内部 RequestContext。Gateway 不能做 GPU 调度，也不能同步等待慢存储。它的目标是让所有请求从一开始就可追踪。

Gateway 产生的数据包括：arrival_ts、tenant_id、model_id、prompt_tokens_len、max_new_tokens、priority、deadline_ms、streaming flag、trace_id。后面的 scheduler、prefill、decode、transfer 都要沿用同一个 trace_id。

English:

The gateway is the entry layer, not the inference brain. It handles HTTP/gRPC ingress, authentication, tenant identification, rate limiting, request_id and trace_id generation, basic validation, and conversion into an internal RequestContext. It should not perform GPU scheduling or synchronously wait for slow storage. Its purpose is to make every request traceable from the beginning.

The data generated by the gateway includes arrival timestamp, tenant id, model id, prompt length, max new tokens, priority, deadline, streaming flag, and trace id. The scheduler, prefill worker, decode worker, and transfer layer should all carry the same trace id.

Admission Controller

中文：

Admission Controller 决定“这个请求现在能不能接”。它不是普通限流，而是结合 GPU/KV memory/queue/deadline 的容量判断。比如 prompt 很长、max_new_tokens 很大、当前 decode queue 已经满、KV memory 不够，就应该 reject、degrade、排队或转到低优先级队列。

它需要估算三个成本：prefill cost、decode cost、KV memory cost。一个粗略模型是：prefill cost 跟 prompt_tokens 相关；decode cost 跟 max_new_tokens 和当前 batch 相关；KV memory cost 跟 layers、heads、head_dim、token_count、dtype 相关。面试里不需要写精确公式，但要说明 admission 不是只看 QPS，而是看 workload shape。

English:

The admission controller decides whether a request can be accepted now. It is not just a generic rate limiter. It makes capacity decisions based on GPU capacity, KV memory, queue depth, and deadline. If the prompt is long, max_new_tokens is large, decode queues are full, or KV memory is insufficient, the request may be rejected, degraded, queued, or moved to a lower-priority queue.

It estimates three costs: prefill cost, decode cost, and KV memory cost. Roughly, prefill cost depends on prompt length, decode cost depends on max_new_tokens and batch composition, and KV memory cost depends on layers, heads, head dimension, token count, and dtype. The important interview point is that admission should consider workload shape, not only QPS.

Scheduler

中文：

Scheduler 做两类选择：第一，选 prefill worker；第二，选 decode worker。选择 prefill worker 时，重点看 GPU 是否适合处理长 prompt、当前 queue depth、prefill batch 是否能合并。选择 decode worker 时，重点看当前 active decode sessions、KV memory、TPOT、P99、是否靠近 prefill worker、KV transfer 成本。

一个合理的策略是：先按 deadline/priority 排队，再按 request shape 分组。长 prompt 和短 prompt 不要随便混，否则短请求的 TTFT 会被长 prefill 拖慢。decode 阶段要做 continuous batching，但要限制某些长输出请求长期占用 batch，避免 unfairness。

English:

The scheduler makes two decisions: selecting a prefill worker and selecting a decode worker. For prefill, it considers whether a GPU is suitable for long prompts, current queue depth, and whether prefill batches can be merged. For decode, it considers active decode sessions, KV memory, TPOT, P99, proximity to the prefill worker, and KV transfer cost.

A reasonable strategy is to queue by deadline and priority, then group by request shape. Long prompts and short prompts should not be mixed blindly, otherwise short-request TTFT can be hurt by long prefill work. In the decode stage, continuous batching is useful, but the scheduler must prevent long-output requests from monopolizing the batch and hurting fairness.

Prefill Worker

中文：

Prefill Worker 处理 prompt tokens，计算每层 attention 所需的 KV cache。它主要影响 TTFT。长 prompt、batch 组织不合理、GPU memory allocation 慢、tokenization/preprocessing 慢，都会让 TTFT 变差。

Prefill 的输出不是最终文本，而是 KV blocks + metadata。metadata 必须记录 request_id、layer_id、token_range、dtype、bytes、location、版本和状态。否则后续 decode worker 不知道哪些 KV 已经可用，也无法做 transfer/replay/debug。

English:

The prefill worker processes prompt tokens and builds KV cache for each attention layer. It mainly affects TTFT. Long prompts, poor batch composition, slow GPU memory allocation, or slow tokenization and preprocessing can all hurt TTFT.

The output of prefill is not final text. It is KV blocks plus metadata. The metadata must record request id, layer id, token range, dtype, size, location, version, and state. Otherwise the decode worker cannot know which KV blocks are available, and transfer, replay, or debugging becomes difficult.

KV Manager

中文：

KV Manager 是这个系统里最容易被低估的模块。它负责 KV block 的生命周期：分配、定位、引用计数、迁移、eviction、释放、失败恢复。没有 KV Manager，prefill/decode 分离很容易变成“到处传指针和 buffer”，最后无法 debug。

KV Manager 要维护两类映射：request -> KV blocks，以及 location -> memory usage。这样 scheduler 才能知道某个 decode worker 是否有足够 KV memory，也能知道把 KV 从 A 搬到 B 的代价。

English:

KV Manager is one of the most underestimated modules in this system. It manages the lifecycle of KV blocks: allocation, placement, reference counting, migration, eviction, release, and failure recovery. Without a KV Manager, prefill/decode disaggregation easily becomes unstructured pointer and buffer passing, which is hard to debug.

KV Manager needs at least two mappings: request-to-KV-blocks and location-to-memory-usage. This allows the scheduler to know whether a decode worker has enough KV memory and what the cost is to move KV from one location to another.

Transfer Layer

中文：

Transfer Layer 执行 KV/state movement。这里才轮到 NIXL/UCX/RDMA 这些技术。它拿到 TransferTask 后，要根据 memory type、src/dst location、size、priority、deadline 选择 backend。它要记录 transfer_start、transfer_done、bytes、bandwidth、retry、error_code，并把状态写回 KV Manager。

最重要的是 transfer 不应该无脑阻塞 decode。能 overlap 就 overlap，不能 overlap 就要知道 decode start delay 是多少。如果 transfer 比本地重新 prefill 还慢，或者复杂度导致 P99 更差，就不能说优化成功。

English:

The transfer layer performs KV or state movement. This is where NIXL, UCX, RDMA, or other transport choices become relevant. Given a TransferTask, it selects a backend based on memory type, source and destination, size, priority, and deadline. It records transfer_start, transfer_done, bytes, bandwidth, retries, error code, and writes the result back to KV Manager.

The most important point is that transfer should not blindly block decode. If it can overlap with compute, it should. If it cannot, the system must measure decode start delay. If transfer is slower than local recomputation or if added complexity hurts P99, then the optimization is not successful.

Decode Worker

中文：

Decode Worker 负责逐 token 生成，主要影响 TPOT。它会反复读取 KV cache，所以 memory bandwidth、KV layout、batch composition、scheduler fairness 都很重要。Decode 不是一次性大计算，而是很多小步循环；每一步都可能受到同步、通信、采样、输出 streaming 的影响。

Decode worker 需要上报 per-token latency、active sequence 数、batch size、GPU utilization、KV memory usage、stall 原因、streaming backpressure。否则只看平均 tokens/s 会隐藏 P99 问题。

English:

The decode worker generates tokens step by step and mainly affects TPOT. It repeatedly reads KV cache, so memory bandwidth, KV layout, batch composition, and scheduler fairness matter. Decode is not one big computation; it is an iterative loop, and each step can be affected by synchronization, communication, sampling, and output streaming.

The decode worker should report per-token latency, number of active sequences, batch size, GPU utilization, KV memory usage, stall reasons, and streaming backpressure. If we only look at average tokens per second, we may hide P99 problems.

7.5.1.2 调度逻辑：不是一句 Scheduler / Scheduling logic in detail

中文：

调度可以按三步讲：

第一步，Admission 判断能不能接。估算 KV memory：如果 prompt_tokens + max_new_tokens 会超过当前可用 KV capacity，直接 reject、排队、降级 max_new_tokens，或转到其他 worker。

第二步，Prefill scheduling。把 prompt length 相近的请求组成 prefill batch，避免一个超长 prompt 拖慢很多短请求。长 prompt 可以进 long-prefill queue，短 prompt 进 short-prefill queue。这样能控制 TTFT fairness。

第三步，Decode scheduling。decode 阶段用 continuous batching，把正在生成的 sequences 合并执行。每个 step 后有的请求结束，有的新请求加入。这里要做 fairness：不能让长输出请求永远占住 batch，也不能为了吞吐无限等待凑 batch。

English:

I would explain scheduling in three steps.

First, admission decides whether the request can be accepted. It estimates KV memory. If prompt_tokens plus max_new_tokens would exceed available KV capacity, the system should reject, queue, degrade max_new_tokens, or route to another worker.

Second, prefill scheduling groups requests with similar prompt lengths into prefill batches. This avoids a very long prompt delaying many short requests. Long prompts can go to a long-prefill queue, and short prompts can go to a short-prefill queue. This helps control TTFT fairness.

Third, decode scheduling uses continuous batching. Active sequences are batched for each decode step. After each step, some requests finish and new requests can join. The scheduler must enforce fairness: long-output requests should not monopolize the batch, and the system should not wait too long just to form a larger batch.

简化伪代码：

on_request(req):
  ctx = build_request_context(req)
  cost = estimate_cost(ctx)
  if not admission_ok(cost):
    reject_or_degrade(ctx)
    return

  prefill_queue = choose_prefill_queue(ctx.prompt_tokens)
  enqueue(prefill_queue, ctx)

prefill_loop(worker):
  batch = pick_similar_prompt_batch(worker.queue)
  kv_blocks = run_prefill(batch)
  for ctx in batch:
    decode_worker = choose_decode_worker(ctx, kv_blocks)
    if decode_worker != worker:
      create_transfer_task(ctx, kv_blocks, decode_worker)
    enqueue_decode(ctx, decode_worker)

decode_loop(worker):
  while true:
    batch = build_continuous_batch(worker.active_sequences)
    tokens = run_decode_step(batch)
    stream_tokens(tokens)
    update_finished_sequences(batch)

面试收束：

如果面试官问我怎么设计 scheduler，我不会只说 dynamic batching。我会说 admission、prefill batching、decode continuous batching、KV memory constraint、deadline/priority、fairness、transfer cost 都要一起考虑。

7.5.1.3 指标怎么读：每个指标对应哪个问题 / How to read metrics

英文解释：

Each metric maps to a different part of the system. TTFT points to queueing, tokenization, prefill, KV allocation, or KV transfer. TPOT points to the decode loop, KV access, batching, GPU kernels, or communication. P99 usually indicates saturation, workload skew, memory pressure, network spikes, or slow dependencies. GPU utilization tells whether the GPU is fed efficiently. KV memory usage tells whether cache pressure or fragmentation is becoming a bottleneck. Transfer latency tells whether state movement is on the critical path.

7.5.1.4 典型异常怎么定位 / Failure scenarios and debugging

Case A: TTFT 高，但 TPOT 正常

中文：

这说明生成开始前慢，生成开始后正常。优先看 queueing、admission、prefill batch、tokenization、KV allocation、KV transfer。比如 long prompt 混进 short prompt batch，会让短请求 TTFT 被拖慢；prefill/decode 分离时，KV transfer 如果阻塞 decode start，也会拉高 TTFT。

English:

High TTFT with normal TPOT means the system is slow before generation starts, but token generation itself is fine. I would check queueing, admission, prefill batching, tokenization, KV allocation, and KV transfer. For example, mixing long prompts with short prompts can hurt short-request TTFT. In a disaggregated setup, KV transfer can also delay decode start.

Case B: TTFT 正常，但 TPOT 差

中文：

这说明请求很快开始生成，但每个 token 慢。优先看 decode loop、KV memory access、batch composition、GPU kernel、tensor parallel communication、streaming backpressure。如果 GPU utilization 低，可能 batch 太小或 CPU feeding 慢；如果 memory bandwidth 高，可能 KV layout 或 paging 有问题。

English:

Normal TTFT with bad TPOT means generation starts quickly, but each token is slow. I would check the decode loop, KV memory access, batch composition, GPU kernels, tensor-parallel communication, and streaming backpressure. If GPU utilization is low, batch size or CPU feeding may be the problem. If memory bandwidth is high, KV layout or paging may be the issue.

Case C: 平均延迟正常，但 P99 爆炸

中文：

平均正常但 P99 爆炸，通常是 workload skew 或资源争用。要按 request type、prompt length、output length、tenant、worker、GPU、node 拆分 P99。可能原因包括长 prompt、某个 tenant burst、queue 堆积、KV eviction、transfer retry、某个 worker/GPU/NIC 拖慢。

English:

Good average latency with bad P99 usually indicates workload skew or resource contention. I would break down P99 by request type, prompt length, output length, tenant, worker, GPU, and node. Possible causes include long prompts, tenant bursts, queue buildup, KV eviction, transfer retries, or a slow worker/GPU/NIC.

7.5.2 KV Cache / NIXL 具体设计 / KV cache and NIXL concrete design

中文设计：

场景：prefill/decode disaggregation。Prefill worker 负责长 prompt 的上下文计算，Decode worker 负责低延迟逐 token 生成。两者分开后，核心问题变成 KV cache/state movement。

为什么需要 NIXL：
1. NCCL 主要解决 collective，不适合表达“某个 request 的 KV state 从 A worker 移到 B worker”。
2. KV movement 是 point-to-point / state transfer，更关心 transfer latency、overlap、memory type、failure recovery。
3. NIXL 的价值是把 CPU/GPU/storage memory 和不同 transport backend 抽象起来，为 inference framework 提供统一 transfer 语义。

English design:

The scenario is prefill/decode disaggregation. Prefill workers process long prompts and build context, while decode workers focus on low-latency token generation. Once these stages are separated, the key problem becomes KV cache or inference state movement.

NIXL is useful because NCCL mainly solves collective communication, while KV movement is request-specific point-to-point or state transfer. The metrics are transfer latency, overlap with compute, memory type, and failure recovery. NIXL provides an abstraction over different memory types and transport backends for inference frameworks.

flowchart TB
    Request[Request] --> Prefill[Prefill Worker]
    Prefill --> KVBuild[Build KV Blocks]
    KVBuild --> Meta[KVBlockMeta]
    Meta --> Select[Select Decode Worker]
    Select --> Transfer[Transfer KV via NIXL]
    Transfer --> Decode[Decode Worker]
    Decode --> Token[Token Stream]

    Transfer --> Metrics[transfer_ms / bandwidth / overlap]
    Decode --> Metrics

设计时必须回答：

面试英文回答：

For NIXL, I would design around the KV movement scenario. The system should not move the model; it moves request-specific KV blocks and metadata from a prefill worker to a decode worker. I would track each KV block with metadata such as request id, layer id, token range, dtype, size, location, reference count, and transfer state. The transfer task would include source, destination, size, priority, deadline, and status.

The success metric is not peak bandwidth only. I would measure whether decode start delay is reduced, whether TTFT and P99 improve, whether GPU idle time decreases, and whether transfer can overlap with compute. If the transfer path adds complexity without end-to-end improvement, I would not push it into the roadmap.

7.5.3 UCX / RDMA / GPUDirect 验证设计 / UCX, RDMA, and GPUDirect validation design

中文设计：

面试官可能不会满足于你说“UCX 是 transport abstraction”。他会追问：你怎么证明它真的走了 RDMA 或 GPUDirect？

验证分四层：
1. 环境层：GPU、NIC、driver、CUDA、UCX version、IOMMU、PCIe topology。
2. UCX 层：available devices、selected transport、memory type support、UCX logs。
3. 性能层：TCP baseline、RDMA baseline、host memory、GPU memory、message size sweep。
4. 端到端层：应用 P99、GPU idle、CPU overhead、network counters、fallback rate。

English design:

The interviewer may not be satisfied with the statement that UCX is a transport abstraction. The real question is how to prove that UCX is actually using RDMA or GPUDirect.

I would verify it in four layers: environment, UCX, performance, and end-to-end behavior. The environment layer checks GPU, NIC, driver, CUDA, UCX version, IOMMU, and PCIe topology. The UCX layer checks available devices, selected transport, memory type support, and UCX logs. The performance layer compares TCP baseline, RDMA baseline, host memory, GPU memory, and message-size sweeps. The end-to-end layer checks application P99, GPU idle time, CPU overhead, network counters, and fallback rate.

flowchart TB
    Env[Environment Check] --> Topo[GPU / NIC / PCIe Topology]
    Topo --> UCX[UCX Device and Transport Check]
    UCX --> Bench[Microbenchmark]
    Bench --> Compare[Compare TCP / RDMA / GDR]
    Compare --> App[End-to-end App Metrics]
    App --> Decision[Use / Tune / Fallback]

具体 checklist：

面试英文回答：

I would never claim RDMA or GPUDirect only from the API. I would prove the data path through topology, logs, counters, and benchmark behavior. I would compare TCP, RDMA with host memory, and RDMA with GPU memory. Then I would connect the microbenchmark result back to end-to-end workload metrics. The final question is not whether a feature is enabled, but whether the actual application path improves.

7.5.4 GPUNetIO 具体路径 / Concrete GPUNetIO path

中文设计：

GPUNetIO 不要讲成“更快网络”的泛词。它更像 GPU-centric packet/data processing path。问题是：什么时候值得让 GPU 直接参与网络数据处理？

适合场景：
1. GPU 直接消费网络数据，CPU copy/parse 成为瓶颈。
2. packet/data rate 很高，CPU-mediated path P99 不稳定。
3. GPU 上的数据处理能和 network ingestion 紧密结合。

不适合场景：
1. QPS 不高，CPU path 不是瓶颈。
2. 数据最后还是要 CPU 复杂控制逻辑。
3. debug/operability 成本大于收益。

English design:

I would not describe GPUNetIO as just faster networking. I would describe it as a GPU-centric packet or data processing path. The key question is when it is worth letting the GPU participate more directly in network data processing.

It is suitable when the GPU directly consumes network data, CPU copy or parsing becomes the bottleneck, packet or data rate is high, CPU-mediated P99 is unstable, and GPU-side data processing can be closely coupled with ingestion. It is not suitable when QPS is low, CPU path is not the bottleneck, data still requires complex CPU control logic, or the debug and operability cost is higher than the benefit.

flowchart TB
    NIC[NIC / RX Queue] --> DOCA[DOCA / GPUNetIO Runtime]
    DOCA --> GPUMem[GPU Memory Buffer]
    GPUMem --> GPUKernel[GPU Processing Kernel]
    GPUKernel --> Output[Processed Data / Signal]
    Output --> App[Application Runtime]

    NIC --> CPUPath[CPU Path Baseline]
    CPUPath --> Compare[Latency / CPU / P99 Compare]
    GPUKernel --> Compare

面试英文回答：

I would use GPUNetIO only after proving that CPU-mediated networking is on the critical path. I would build a baseline CPU path first, then compare it with a GPU-centric path in terms of packet rate, P99, CPU overhead, GPU utilization, memory copies, and debuggability. If the workload does not directly benefit from GPU-side packet or data processing, I would not add GPUNetIO just because it is available.

7.5.5 NCCL All-Reduce 慢的结构化排查 / Structured slow all-reduce debug

中文设计：

All-reduce 慢时不要只说“网络慢”。要把 collective 拆成 message size、rank mapping、topology、transport、algorithm、overlap。

数据：
1. num_ranks、rank_id、node_id、gpu_id、nic_id。
2. message_size、dtype、collective type。
3. algbw、busbw、latency。
4. topology：NVLink / PCIe / NUMA / NIC affinity。
5. NCCL logs：selected algorithm、transport、channel、warnings。

English design:

When all-reduce is slow, I would not simply say the network is slow. I would break the collective into message size, rank mapping, topology, transport, algorithm, and overlap.

The data includes number of ranks, rank id, node id, GPU id, NIC id, message size, dtype, collective type, algorithm bandwidth, bus bandwidth, latency, topology, and NCCL logs such as selected algorithm, transport, channel, and warnings.

flowchart TB
    Slow[Slow All-Reduce] --> Size[Message Size]
    Slow --> Rank[Rank Mapping]
    Slow --> Topology[GPU / NIC / NUMA Topology]
    Slow --> Transport[NCCL Transport]
    Slow --> Algo[Algorithm / Channels]
    Slow --> Overlap[Compute Communication Overlap]

    Size --> Decision[Root Cause]
    Rank --> Decision
    Topology --> Decision
    Transport --> Decision
    Algo --> Decision
    Overlap --> Decision

具体判断：

面试英文回答：

For slow all-reduce, I would first make the result reproducible with nccl-tests. Then I would compare single-node and multi-node results, inspect algbw and busbw across message sizes, check rank mapping and GPU/NIC affinity, and look at NCCL logs for selected transport and warnings. If single-node is good but multi-node is bad, I focus on network, RDMA, topology, and rank placement. If both are bad, I also inspect workload configuration and GPU synchronization.

7.5.6 AI 风控项目的具体数据结构 / Concrete data structures for the AI risk project

中文设计：

如果他们追问你 AI 风控项目的“数据结构”，可以这样讲：我不会把它说成一个模型输入，而是一个可追踪的 case。

RiskEvent {
  event_id;
  user_id_hash;
  event_type;        // wallet / trading / security / behavior
  event_ts;
  raw_payload_ref;
  trace_id;
}

FeatureSnapshot {
  case_id;
  schema_version;
  rule_features;
  statistical_features;
  context_features;
  missing_feature_flags;
  generated_ts;
}

DecisionRecord {
  case_id;
  model_score;
  llm_summary_ref;
  strategy_id;
  guardrail_result;
  action;            // allow / block / review / escalate
  reason_codes;
  latency_breakdown;
  version;
}

ReplayCase {
  case_id;
  event_ref;
  feature_snapshot_ref;
  expected_decision;
  actual_decision;
  diff;
}

English answer:

If asked about the data structures in the AI risk project, I would not describe it as a single model input. I would describe it as a traceable case. A RiskEvent captures event id, anonymized user id, event type, timestamp, raw payload reference, and trace id. A FeatureSnapshot captures schema version, rule features, statistical features, context features, missing-feature flags, and generation time. A DecisionRecord captures model score, LLM summary reference, strategy id, guardrail result, action, reason codes, latency breakdown, and version. A ReplayCase connects the event, feature snapshot, expected decision, actual decision, and diff.

为什么这重要：

这能证明你不是只会说 AI 概念，而是懂生产系统里 correctness、traceability、replay、schema version、latency breakdown 和 versioned decision。

English close:

This shows that I am not only talking about AI concepts. I understand production concerns such as correctness, traceability, replay, schema versioning, latency breakdown, and versioned decisions.

8. 大概率问题预测和可直接回答的答案

这一节不要只看问题，要直接背答案结构。每题都按这个顺序回答：

1. 先给结论。
2. 再给 data path / system path。
3. 再讲指标和 debug。
4. 最后连接 CV 或 JD。

8.1 Q: 讲讲 AI/LLM 平台项目 / Tell me about the AI/LLM platform project.

面试官想看：

你是不是只是在业务系统里调模型 API，还是理解 production inference path、observability、replayability、correctness validation 和 latency/cost tradeoff。

English answer:

The AI/LLM platform project was a production-oriented risk-control and strategy platform. It connected feature generation, online inference, retrieval-assisted LLM analysis, anomaly detection, strategy execution, observability, replayability, and feedback evaluation.

The key point is that it was not just calling a model API. The system had to handle a production request path: events come in, features are generated, model or LLM analysis is triggered, the strategy layer makes a decision, and the result must be traceable and replayable. We also needed observability around latency, correctness, abnormal cases, and effectiveness.

For this NVIDIA role, the connection is the way I reason about inference systems: request path, queueing, latency breakdown, cache/state behavior, validation, and operational visibility. The stack is different, but the systems thinking transfers to LLM serving, GPU inference, and data-movement problems.

中文理解：

这个回答要把项目从“AI 风控业务”升级成“production inference system”。核心词是 request path、observability、replayability、correctness validation、latency breakdown。

如果继续追问：

Q: Where was the bottleneck?
A: 拆成 feature generation、model/LLM latency、retrieval I/O、queueing、downstream strategy dependency、P99 tail。

Q: How did you know it was correct?
A: replay、shadow、case tracing、metric comparison、abnormal sample review、guardrail。

深入跳转：sequence-03-cv-project-deep-dive.html，sequence-06-deep-interview-qa.html

8.2 Q: AI/LLM 系统瓶颈是什么 / What was the bottleneck in your AI/LLM system?

English answer:

I would not describe the bottleneck as a single number. I usually break it down by the request path.

First, I look at queueing and scheduling: whether requests wait before any real computation starts. Second, I separate feature generation, retrieval I/O, model or LLM inference latency, strategy execution, and downstream dependencies. Third, I look at tail behavior, especially P95 and P99, because average latency can hide saturation or interference.

The debug method is to build a baseline, add tracing around each stage, identify the critical path, and then validate whether an optimization actually improves end-to-end latency and correctness. For NVIDIA-style inference systems, I would use the same method but with GPU-specific metrics such as TTFT, TPOT, GPU utilization, memory pressure, communication time, and KV cache behavior.

中文回答：

我不会把瓶颈说成一个孤立数字，而是按 request path 拆。

第一看 queueing 和 scheduling，请求是不是还没开始计算就已经在排队。第二把 feature generation、retrieval I/O、model/LLM inference、strategy execution、downstream dependency 分开看。第三看 P95/P99，因为平均值会掩盖 saturation 和 interference。

我的方法是先建立 baseline，在每个阶段加 tracing，找到 critical path，然后验证优化是否真的改善 end-to-end latency 和 correctness。如果迁移到 NVIDIA inference 场景，我会看 TTFT、TPOT、GPU utilization、memory pressure、communication time 和 KV cache behavior。

8.3 Q: 你怎么验证正确性 / How did you validate correctness?

English answer:

For AI systems, correctness is not only whether the service returns a response. I usually validate correctness at several levels.

At the data level, I check feature consistency, schema changes, missing values, and abnormal distributions. At the model or LLM output level, I check whether the output is traceable and whether risky outputs are guarded. At the system level, I use replay, shadow testing, case review, metric comparison, and feedback loops to compare expected behavior with actual production behavior.

For this role, the same mindset applies to infrastructure prototypes. A benchmark improvement is not enough by itself. I would also check reproducibility, workload assumptions, correctness under edge cases, and whether the optimization still holds end to end.

中文回答：

AI 系统的 correctness 不是服务返回了结果就结束。我会分几层验证。

数据层看 feature consistency、schema changes、missing values、异常分布。模型/LLM 输出层看输出是否 traceable，是否有 guardrail。系统层用 replay、shadow testing、case review、metric comparison 和 feedback loop，对比预期行为和真实生产行为。

迁移到 NVIDIA 这个岗位，prototype 或 benchmark 也不能只看性能数字，还要看 reproducibility、workload assumptions、edge cases，以及优化是否在 end-to-end path 里仍然成立。

8.4 Q: 解释 Prefill 和 Decode / Explain prefill and decode.

English answer:

In LLM inference, prefill and decode are two different phases.

Prefill processes the input prompt and builds the KV cache for the existing context. It is usually compute-heavy, especially for long prompts, and it strongly affects time to first token, or TTFT.

Decode generates tokens step by step. Each step uses the existing KV cache and produces the next token. Decode is latency-sensitive and memory-access-sensitive because it repeatedly reads KV cache. It affects time per output token, or TPOT.

The important point is that prefill and decode have different resource profiles. That is why modern inference systems may separate prefill and decode workers, use different batching policies, and care about KV cache placement and movement.

中文回答：

LLM inference 里 prefill 和 decode 是两个阶段。

Prefill 处理输入 prompt，并为已有上下文建立 KV cache。长 prompt 下 prefill 通常偏 compute-heavy，会明显影响 TTFT，也就是 time to first token。

Decode 是逐 token 生成。每一步都读取已有 KV cache，生成下一个 token。Decode 更 latency-sensitive，也更受 KV cache memory access 影响，会影响 TPOT，也就是 time per output token。

关键点是 prefill 和 decode 的资源画像不同，所以现代 inference system 会考虑 prefill/decode disaggregation、不同 batching 策略，以及 KV cache placement 和 movement。

继续追问：

Q: 为什么要分离 prefill/decode？
A: 因为 prefill 偏计算和长上下文，decode 偏低延迟和反复读 KV。混在一起可能互相干扰，分离后可以针对资源画像分别调度，但代价是 KV/state movement。

8.5 Q: 为什么 KV Cache 重要 / Why is KV cache important?

English answer:

KV cache is important because it turns LLM inference from pure computation into a state and memory-management problem.

During prefill, the model computes key/value tensors for the prompt. During decode, those tensors are reused so the model does not recompute the whole context. This saves compute, but the cache can become very large, especially with long context, large batch size, or many concurrent sessions.

At system scale, KV cache affects GPU memory pressure, scheduling, batching, prefill/decode separation, and data movement. If KV has to move across devices or nodes, then libraries such as NIXL become relevant because the problem is no longer only collective communication; it is inference state transfer.

中文回答：

KV cache 重要，是因为它把 LLM inference 从单纯计算问题变成了 state 和 memory management 问题。

Prefill 阶段会为 prompt 计算 key/value tensors。Decode 阶段复用这些 tensors，避免重复计算整个上下文。这样节省计算，但 KV cache 会占很多显存，尤其是长上下文、大 batch 和高并发 session。

系统层面，KV cache 会影响 GPU memory pressure、scheduling、batching、prefill/decode separation 和 data movement。如果 KV 需要跨 GPU 或跨节点移动，NIXL 这类库就相关，因为这已经不是普通 collective communication，而是 inference state transfer。

8.6 Q: 怎么降低 TTFT / How would you reduce TTFT?

English answer:

I would reduce TTFT by first decomposing it.

TTFT includes queueing time, routing and scheduling time, prefill computation, KV cache allocation, possible retrieval or preprocessing, and the time until the first token is emitted. I would first measure each part rather than optimizing blindly.

Possible optimizations include reducing queueing through better admission control and scheduling, optimizing prefill batching, caching reusable prompt or prefix states, reducing retrieval or preprocessing latency, improving GPU utilization during prefill, and avoiding unnecessary synchronization or data movement.

The tradeoff is that aggressive batching may improve throughput but hurt TTFT for individual requests, so I would evaluate both average and P95/P99 TTFT.

中文回答：

我会先拆 TTFT，而不是直接优化。

TTFT 包括 queueing、routing/scheduling、prefill computation、KV cache allocation、可能的 retrieval/preprocessing，以及直到第一个 token 输出的时间。我会先测每段。

优化方向包括：用 admission control 和 scheduling 减少排队；优化 prefill batching；缓存可复用 prompt/prefix state；降低 retrieval/preprocessing latency；提升 prefill 阶段 GPU utilization；避免不必要同步和数据移动。

tradeoff 是 aggressive batching 可能提升 throughput，但伤害单请求 TTFT，所以要同时看 average 和 P95/P99 TTFT。

8.7 Q: 怎么降低 TPOT / How would you reduce TPOT?

English answer:

TPOT is mainly about the decode loop.

I would look at decode kernel efficiency, KV cache memory access, batch composition during continuous batching, GPU utilization, synchronization overhead, and communication overhead if tensor parallelism or multi-node execution is involved.

The optimization direction depends on the bottleneck. If the GPU is underutilized, scheduling or batching may be the issue. If memory bandwidth is saturated, KV layout and access pattern matter. If multi-GPU communication dominates, I would look at parallelism strategy, NCCL communication time, topology, and whether data movement is on the critical path.

中文回答：

TPOT 主要看 decode loop。

我会看 decode kernel efficiency、KV cache memory access、continuous batching 里的 batch composition、GPU utilization、synchronization overhead，以及如果涉及 tensor parallelism 或 multi-node execution，就看 communication overhead。

优化方向取决于瓶颈。如果 GPU utilization 低，可能是 scheduling/batching 问题。如果 memory bandwidth 饱和，要看 KV layout 和 access pattern。如果 multi-GPU communication 占主导，要看 parallelism strategy、NCCL communication time、topology，以及 data movement 是否在 critical path 上。

8.8 Q: P99 延迟高意味着什么 / What happens when P99 latency is high?

English answer:

When P99 is high, I would first assume it may be a saturation or interference problem, not just a slow average path.

I would check queue depth, request burstiness, batching policy, long-context requests, GPU memory pressure, network or communication spikes, downstream dependencies, and retry behavior. I would also separate P50, P95, and P99 by request type, context length, output length, and node or GPU placement.

The goal is to find whether the tail comes from queueing, workload skew, batching interference, communication, GPU memory pressure, or an external dependency. Only after that would I choose the optimization.

中文回答：

P99 高时，我会先假设这可能是 saturation 或 interference，不只是平均路径慢。

我会查 queue depth、request burstiness、batching policy、long-context requests、GPU memory pressure、network/communication spikes、downstream dependency 和 retry behavior。同时按 request type、context length、output length、node/GPU placement 拆 P50/P95/P99。

目标是判断 tail latency 来自 queueing、workload skew、batching interference、communication、GPU memory pressure，还是外部依赖。确认来源后再优化。

8.9 Q: 解释 NCCL、UCX、NIXL、GPUNetIO、GPUDirect RDMA / Explain NCCL, UCX, NIXL, GPUNetIO, and GPUDirect RDMA.

English answer:

I would separate them by the layer and the problem they solve.

NCCL is mainly for GPU collective communication, such as all-reduce, all-gather, and reduce-scatter. It is commonly used in distributed training and tensor-parallel workloads.

UCX is a high-performance communication framework and transport abstraction. It can use transports such as shared memory, TCP, RDMA, and CUDA-aware paths. It is more about the communication substrate and endpoint/request abstractions.

NIXL is focused on inference data or state transfer, especially cases such as KV cache or inference state movement. It is not the same as NCCL collectives. The mental model is inference data movement rather than collective reduction.

GPUNetIO is about letting GPU participate more directly in network packet or data processing paths, typically with DOCA and NVIDIA networking. It is useful when CPU involvement becomes a bottleneck in high-throughput packet/data paths.

GPUDirect RDMA allows an RDMA-capable NIC to access GPU memory directly, reducing CPU copies and kernel involvement. It depends on hardware, driver, memory registration, and topology support.

中文回答：

我会按层次和解决的问题区分。

NCCL 主要做 GPU collective communication，比如 all-reduce、all-gather、reduce-scatter，常用于 distributed training 和 tensor parallel workloads。

UCX 是高性能通信框架和 transport abstraction，可以覆盖 shared memory、TCP、RDMA、CUDA-aware path，更像底层通信 substrate 和 endpoint/request abstraction。

NIXL 聚焦 inference data/state transfer，典型是 KV cache 或 inference state movement。它不是 NCCL collectives，应该按 inference data movement 来理解。

GPUNetIO 是让 GPU 更直接参与 network packet/data processing path，通常和 DOCA/NVIDIA networking 相关。当 CPU packet path 成为瓶颈时，它可能有价值。

GPUDirect RDMA 是 RDMA-capable NIC 直接访问 GPU memory，减少 CPU copy 和 kernel involvement。它依赖硬件、driver、memory registration 和 topology。

8.10 Q: All-Reduce 很慢怎么排查 / How would you debug a slow all-reduce?

English answer:

I would debug a slow all-reduce from several layers.

First, I would check the benchmark and workload: message size, number of ranks, algorithm bandwidth versus bus bandwidth, and whether the result is reproducible. Second, I would check topology: GPU placement, PCIe or NVLink, NIC affinity, NUMA, and whether ranks are mapped correctly. Third, I would check transport and library configuration: NCCL logs, selected transport, RDMA availability, and whether the expected network path is used.

Then I would compare single-node versus multi-node performance. If single-node is good but multi-node is slow, the issue is likely network, transport, topology, or rank placement. If both are slow, I would also inspect GPU utilization, synchronization, and workload configuration.

中文回答：

我会分层 debug slow all-reduce。

第一看 benchmark 和 workload：message size、rank 数、algorithm bandwidth、bus bandwidth，以及结果是否可复现。第二看 topology：GPU placement、PCIe/NVLink、NIC affinity、NUMA、rank mapping。第三看 transport 和 library config：NCCL logs、selected transport、RDMA availability，以及是否走了预期网络路径。

然后对比 single-node 和 multi-node。如果单机好、多机慢，大概率是 network、transport、topology 或 rank placement。如果两者都慢，再看 GPU utilization、synchronization 和 workload config。

8.11 Q: 怎么确认 UCX 走了 RDMA 或 GPUDirect / How do you know UCX is using RDMA or GPUDirect?

English answer:

I would not assume it from the API alone. I would verify it from configuration, logs, device visibility, and performance behavior.

For UCX, I would check available devices and transports, UCX logs, selected transport, memory type support, and whether CUDA memory is recognized correctly. I would also compare performance against expected TCP and RDMA baselines.

For GPUDirect RDMA, I would check hardware and driver support, GPU/NIC topology, memory registration, whether the path avoids host staging copies, and relevant counters or traces. The key is to prove the actual data path, not just say the feature is enabled.

中文回答：

我不会只看 API 就假设走了 RDMA/GDR，而是从 config、logs、device visibility 和 performance behavior 验证。

UCX 里我会看 available devices/transports、UCX logs、selected transport、memory type support，以及 CUDA memory 是否被正确识别。还会和 TCP/RDMA baseline 对比性能。

GPUDirect RDMA 要看 hardware/driver support、GPU/NIC topology、memory registration、是否避免 host staging copy，以及相关 counters/traces。关键是证明真实 data path，而不是只说功能开了。

8.12 Q: 设计一个高吞吐低 P99 的多节点 LLM 推理系统 / Design a multi-node LLM inference system for high throughput and low P99.

先看完整设计：7.5.1 多节点 LLM 推理系统设计。下面是面试时可以直接讲的详细版。

English answer:

I would design it from workload assumptions and SLOs, not from a library API.

The first thing I would do is classify the workload. I would ask for model size, whether the model fits on one GPU, prompt length P50/P95/P99, output length P50/P95/P99, request rate, concurrent sessions, streaming requirement, multi-tenant priority, TTFT target, TPOT target, P99 target, GPU/NIC topology, and whether the system is single-node or multi-node. This matters because a short-prompt chat workload, a long-context RAG workload, an agent workload with long outputs, and an offline batch workload require different scheduling policies.

Then I would split the system into a control path and a data path in a concrete way. The control path carries small metadata and decisions: authentication, tenant id, request id, trace id, admission result, scheduler decision, worker heartbeat, placement state, and failure recovery state. The control path can use normal RPCs and persistent state, but it must not block GPU execution. The data path carries the hot data: prompt tensors, KV cache blocks, decode tokens, GPU memory buffers, network transfers, and streaming output. The data path should be asynchronous, low-copy, and measured at every stage. The reason for this split is that correctness and recoverability belong to the control path, while latency and throughput belong to the data path. Mixing them causes slow metadata operations to show up as P99 latency.

For the request lifecycle, I would start with gateway and admission. Gateway validates the request, assigns trace_id, extracts tenant/model/deadline/streaming flags, and builds RequestContext. Admission then estimates prefill cost, decode cost, and KV memory cost. A rough KV estimate is proportional to layers * 2 * heads * head_dim * total_tokens * dtype_bytes, where total_tokens is prompt_tokens plus expected output tokens. If the request would exceed memory capacity or violate deadline under the current queue depth, admission should reject, queue, degrade max_new_tokens, or route it elsewhere. This is why admission is not just QPS rate limiting.

Next I would decide whether prefill and decode should be colocated or disaggregated. I would keep them colocated if prompts are mostly short, topology is poor, cross-node KV transfer is expensive, or the cluster is small. I would consider disaggregation if prompt length is highly skewed, long prefill work hurts short-request TTFT, decode needs stable low TPOT, and KV transfer can overlap with compute. The decision rule is: saved queueing time plus better GPU specialization must be larger than KV transfer latency, metadata overhead, failure recovery cost, and memory fragmentation. If end-to-end P99 does not improve, disaggregation is not justified.

For scheduling, I would not only say dynamic batching. I would use admission queues by tenant priority and deadline, then prefill queues by prompt-length bucket. The prefill scheduler should batch similar prompt lengths so a very long prompt does not delay many short prompts. The decode scheduler should use continuous batching, but with fairness controls: long-output requests cannot monopolize the batch, and short requests cannot wait too long just to form a larger batch. For worker selection, I would score workers using queue wait, estimated compute time, free KV memory, active decode sessions, TPOT, topology penalty, transfer cost, and deadline risk.

For parallelism, I would choose the simplest strategy that satisfies the SLO. If the model fits on one GPU, data parallel replicas are usually the simplest and most robust way to increase QPS. If the model does not fit or per-token latency needs multiple GPUs, I would consider tensor parallelism, preferably within a node with NVLink or NVSwitch because all-reduce or all-gather may sit on the token critical path. Pipeline parallelism can help throughput for large models and large batches, but pipeline bubbles and scheduling complexity must be considered. I would be careful with cross-node tensor parallelism because network communication can directly hurt TPOT and P99.

For KV cache, I would treat it as a managed resource, not just a pointer. KV Manager should track request_id, layer_id, token range, dtype, bytes, location, state, ref_count, and last access time. Scheduler needs this metadata to know whether a decode worker has enough memory and how expensive it is to place or move a request. If KV moves between workers, the transfer task should include source, destination, bytes, priority, deadline, status, retry count, and error code. This is where NIXL or UCX becomes relevant, but only after the state-movement problem is clear.

Finally, I would validate the design with a baseline and an A/B prototype. The baseline is colocated prefill/decode with normal batching. The prototype adds disaggregation, KV Manager, transfer layer, and scheduler changes. I would compare TTFT, TPOT, P50/P95/P99, throughput, GPU utilization, KV memory pressure, transfer latency, queue depth, retry rate, and failure rate. I would debug from an end-to-end timeline. High TTFT with normal TPOT points to queueing, tokenization, prefill, KV allocation, or KV transfer. Normal TTFT with bad TPOT points to decode loop, KV access, batching, kernels, tensor-parallel communication, or streaming backpressure. Good average latency with bad P99 points to workload skew, memory pressure, queue buildup, transfer retries, or a slow worker/GPU/NIC.

The main design principle is that every split must pay for itself. Splitting control/data path improves isolation and debuggability. Splitting prefill/decode improves specialization only if KV movement is cheap enough. Adding tensor parallelism helps only if compute or memory pressure dominates more than communication. Increasing batch size helps throughput only until it hurts TTFT, TPOT, or fairness.

中文回答：

我会从 workload assumptions 和 SLO 开始设计，而不是从某个库 API 开始。

第一步我会先给 workload 分类。我会问：模型多大、单卡能不能放下、prompt length 的 P50/P95/P99、output length 的 P50/P95/P99、request rate、并发 session、是否 streaming、是否多租户优先级、TTFT 目标、TPOT 目标、P99 目标、GPU/NIC topology、单节点还是多节点。这个很关键，因为短 prompt chat、长上下文 RAG、长输出 agent workload、离线 batch workload 的调度策略不应该一样。

然后我会具体拆 control path 和 data path。Control path 只走小数据和决策：authentication、tenant_id、request_id、trace_id、admission result、scheduler decision、worker heartbeat、placement state、failure recovery state。Control path 可以走普通 RPC 和持久化状态，但不能阻塞 GPU 执行。Data path 走热数据：prompt tensors、KV cache blocks、decode tokens、GPU memory buffers、network transfer、streaming output。Data path 要 async、low-copy、少锁，并且每一段都要有 tracing。这样拆的原因是：control path 追求正确性、可追踪、可恢复；data path 追求低延迟、高吞吐、低 copy。如果混在一起，慢 metadata RPC、慢 DB、日志、调度锁都会变成 P99 问题。

请求生命周期上，我会先设计 gateway 和 admission。Gateway 做参数校验、生成 trace_id、解析 tenant/model/deadline/streaming flag，然后生成 RequestContext。Admission 估算 prefill cost、decode cost 和 KV memory cost。KV memory 可以粗略按 layers * 2 * heads * head_dim * total_tokens * dtype_bytes 估算，其中 total_tokens 是 prompt_tokens 加预期 output tokens。如果当前 queue depth 和 memory capacity 下这个请求会超过显存或违背 deadline，admission 就应该 reject、排队、降低 max_new_tokens，或者转到其他 worker。所以 admission 不是普通 QPS 限流，而是 workload-aware capacity control。

接着我会判断 prefill 和 decode 是否应该分离。不是所有系统都应该分离。如果大部分是短 prompt、topology 不好、跨节点 KV transfer 很贵、集群很小，我会让 prefill/decode colocated，减少复杂度。如果 prompt length 分布很偏、长 prefill 经常拖慢短请求 TTFT、decode 需要稳定低 TPOT，而且 KV transfer 可以和计算 overlap，我才考虑 disaggregation。判断规则是：saved queueing time + better GPU specialization 要大于 KV transfer latency + metadata overhead + failure recovery cost + memory fragmentation。只要 end-to-end P99 没变好，就不能说这个拆分成功。

调度上，我不会只说 dynamic batching。我会先按 tenant priority 和 deadline 建 admission queues，再按 prompt length bucket 建 prefill queues。Prefill scheduler 把相近 prompt length 放一起，避免一个超长 prompt 拖慢一批短 prompt。Decode scheduler 用 continuous batching，但要加 fairness：长 output request 不能长期占住 batch，短 request 也不能为了凑大 batch 等太久。Worker 选择上我会打分：queue wait、estimated compute time、free KV memory、active decode sessions、TPOT、topology penalty、transfer cost、deadline risk。选择满足 SLO 且 score 最低的 worker。

Parallelism 方面，我会选能满足 SLO 的最简单方案。如果模型能放在单 GPU，优先 data parallel replicas，提升 QPS，系统也最简单。如果模型放不下，或者单 token latency 需要多 GPU，就考虑 tensor parallel，但它会引入 all-reduce/all-gather，最好在同节点 NVLink/NVSwitch 内做。如果模型很大、batch 足够、吞吐目标更强，可以考虑 pipeline parallel，但要处理 pipeline bubble 和调度复杂度。跨节点 tensor parallel 要非常小心，因为通信会进入每个 token 的 critical path，直接影响 TPOT 和 P99。

KV cache 我会当成一个被管理的资源，而不是一个裸 pointer。KV Manager 要记录 request_id、layer_id、token range、dtype、bytes、location、state、ref_count、last_access_ts。Scheduler 依赖这些 metadata 判断某个 decode worker 是否有足够显存，以及把请求放过去要付出多少 transfer cost。如果 KV 要跨 worker 移动，TransferTask 需要包含 source、destination、bytes、priority、deadline、status、retry count、error code。这时才引入 NIXL 或 UCX，因为问题已经明确是 inference state movement，而不是泛泛地“用通信库”。

最后我会验证设计。先做 baseline：prefill/decode colocated，普通 batching。再做 prototype：加入 disaggregation、KV Manager、transfer layer、scheduler 改造。对比 TTFT、TPOT、P50/P95/P99、throughput、GPU utilization、KV memory pressure、transfer latency、queue depth、retry rate、failure rate。我会从 end-to-end timeline debug。TTFT 高但 TPOT 正常，优先看 queueing、tokenization、prefill、KV allocation、KV transfer。TTFT 正常但 TPOT 差，优先看 decode loop、KV access、batching、kernel、tensor-parallel communication、streaming backpressure。平均正常但 P99 爆炸，优先看 workload skew、memory pressure、queue buildup、transfer retry、某个 worker/GPU/NIC 拖慢。

核心原则是：每一次拆分都要证明收益。拆 control/data path 是为了隔离和可调试；拆 prefill/decode 是为了资源画像不同，但 KV movement 必须足够便宜；加 tensor parallel 只有在 compute/memory 压力大于通信代价时才合理；加大 batch 只有在不伤 TTFT、TPOT 和 fairness 时才合理。

8.12.1 Q: 这个 LLM serving 系统慢了，你怎么用 nsys/ncu 定位 / How would you use nsys and ncu to debug a slow LLM serving system?

English answer:

I would not start by profiling a random kernel. I would start with the serving metric that is bad: TTFT, TPOT, P99, throughput, GPU utilization, or failure rate.

If TTFT is bad, I use request traces and Nsight Systems to break down queueing, tokenization, scheduling, prefill, KV allocation, and KV transfer. I would add NVTX ranges around gateway, admission, scheduler, prefill, transfer, decode start, and streaming. If the GPU is idle before prefill starts, the problem is probably queueing, CPU preprocessing, scheduler delay, or data movement. If the prefill kernel itself dominates, then I use Nsight Compute on that kernel.

If TPOT is bad, I use Nsight Systems to inspect the decode loop. I check whether each token step is dominated by CUDA kernels, communication, sampling, stream synchronization, or output streaming. If the decode kernel dominates, then Nsight Compute can explain whether the issue is KV cache memory access, low occupancy, warp stalls, memory transactions, or instruction mix. If there are gaps between decode steps, the issue may be scheduler, communication, sampling, or streaming backpressure rather than the kernel.

If P99 is bad, I compare fast and slow request timelines. I would split by prompt length, output length, tenant, priority, worker, GPU, node, and whether KV transfer happened. P99 is often caused by workload skew, memory pressure, queue buildup, transfer retries, or a slow GPU/NIC/worker. Nsight Compute is only useful after the slow request has been reduced to a specific kernel bottleneck.

The key rule is: Nsight Systems finds the critical path; Nsight Compute explains a dominant kernel. I would not use kernel-level metrics to explain a system-level P99 problem unless the timeline proves the kernel is on the critical path.

中文回答：

我不会一上来 profile 一个随机 kernel，而是先看哪个 serving 指标坏了：TTFT、TPOT、P99、throughput、GPU utilization，还是 failure rate。

如果 TTFT 差，我会用 request trace 和 Nsight Systems 拆 queueing、tokenization、scheduling、prefill、KV allocation、KV transfer。我会加 NVTX ranges 标记 gateway、admission、scheduler、prefill、transfer、decode start、streaming。如果 prefill 前 GPU 是 idle，问题大概率是排队、CPU preprocessing、scheduler delay 或 data movement。如果 prefill kernel 本身占主导，再用 Nsight Compute 深挖这个 kernel。

如果 TPOT 差，我会用 Nsight Systems 看 decode loop。每个 token step 到底是 CUDA kernel 慢、communication 慢、sampling 慢、stream sync 慢，还是 output streaming 慢。如果 decode kernel 占主导，再用 Nsight Compute 看 KV cache memory access、occupancy、warp stall、memory transaction、instruction mix。如果 decode step 之间有 gap，问题可能是 scheduler、communication、sampling 或 streaming backpressure，而不是 kernel。

如果 P99 差，我会对比 fast request 和 slow request 的 timeline，按 prompt length、output length、tenant、priority、worker、GPU、node、是否发生 KV transfer 来拆。P99 常见原因是 workload skew、memory pressure、queue buildup、transfer retry、某个 GPU/NIC/worker 拖慢。只有当 slow request 已经收敛到某个 kernel bottleneck 时，Nsight Compute 才是下一步。

核心规则是：Nsight Systems 找 critical path；Nsight Compute 解释 dominant kernel。除非 timeline 证明 kernel 在 critical path 上，否则不要用 kernel-level metrics 解释 system-level P99。

8.12.2 Q: ncu 发现 decode kernel 慢，你怎么优化 / ncu shows the decode kernel is slow. How would you optimize it?

English answer:

I would first classify the limiter.

If the decode kernel is memory-bound, I look at KV cache layout, memory coalescing, memory transactions, L2/DRAM throughput, and whether the access pattern is fragmented by paging or batching. Possible fixes are better KV layout, contiguous block placement, reducing layout conversion, improving cache reuse, and avoiding unnecessary reads or writes.

If the kernel is compute-bound, I look at instruction mix, tensor core usage, data type, fusion opportunities, and whether the kernel does too much per token. Possible fixes are kernel fusion, using more efficient data types, tensor core-friendly shapes, and reducing redundant computation.

If occupancy is low, I check whether it is limited by registers, shared memory, block size, or launch configuration. I do not blindly chase 100 percent occupancy. I check whether low occupancy actually causes latency-hiding problems.

If warp stalls dominate, I map the stall reason back to source code. Memory dependency points to global memory or KV access. Barrier stalls point to synchronization or shared-memory tiling. Execution dependency points to long dependency chains. Branch divergence points to control-flow variation across tokens or requests.

After any kernel change, I re-run the serving benchmark. A kernel can look better in ncu but still hurt TTFT or P99 if it increases memory pressure, reduces batching flexibility, or adds synchronization.

中文回答：

我会先判断 limiter。

如果 decode kernel 是 memory-bound，我会看 KV cache layout、memory coalescing、memory transactions、L2/DRAM throughput，以及访问模式是否因为 paging 或 batching 变得碎片化。可能的优化是更好的 KV layout、连续 block placement、减少 layout conversion、提高 cache reuse、避免不必要读写。

如果是 compute-bound，我会看 instruction mix、tensor core usage、data type、fusion opportunity，以及每个 token 是否做了过多重复计算。可能的优化是 kernel fusion、更合适的数据类型、tensor core-friendly shape、减少 redundant computation。

如果 occupancy 低，我会看是不是 registers、shared memory、block size 或 launch configuration 限制。我不会盲目追 100% occupancy，而是看低 occupancy 是否真的导致 latency hiding 不足。

如果 warp stalls 占主导，我会把 stall reason 映射回源码。Memory dependency 指向 global memory/KV access；barrier stall 指向同步或 shared-memory tiling；execution dependency 指向依赖链；branch divergence 指向 token/request 间控制流差异。

任何 kernel 改动后，都必须重新跑 serving benchmark。一个 kernel 在 ncu 里变快，不代表 TTFT/P99 一定变好，因为它可能增加 memory pressure、降低 batching 灵活性，或者引入额外同步。

8.12.3 Q: 如果 nsys 显示通信或 KV transfer 在 critical path 上，你怎么办 / What if nsys shows communication or KV transfer on the critical path?

English answer:

I would first decide whether the communication is necessary, whether it can be avoided, and whether it can be overlapped.

For KV transfer, I would check the placement decision: why was prefill assigned to one worker and decode to another? Was the transfer cost included in scheduler scoring? Would colocating prefill and decode be faster for this request shape? I would compare colocated baseline versus disaggregated path.

Then I would inspect transfer size, message count, memory type, source and destination GPU/NIC topology, backend selection, retry behavior, and whether the path falls back to host staging or TCP. For NIXL or UCX, the question is not just API usage; the real question is selected backend, memory registration behavior, topology, and whether transfer overlaps with compute.

For NCCL communication, I would check message size, rank mapping, GPU/NIC affinity, selected transport, single-node versus multi-node performance, and whether communication blocks decode steps. I would use NCCL logs and nccl-tests together with nsys. Nsight Compute is usually not the first tool unless a packing/copy kernel itself is slow.

The optimization choices are: change placement, colocate prefill/decode for some requests, batch or coalesce transfers, overlap transfer with compute, change topology/rank mapping, tune backend selection, or adjust the parallelism strategy. I would accept the optimization only if end-to-end TTFT/TPOT/P99 improves.

中文回答：

我会先判断这个通信是否必要，能不能避免，能不能 overlap。

对于 KV transfer，我会先看 placement decision：为什么 prefill 放在一个 worker，decode 放在另一个 worker？scheduler 打分里有没有包含 transfer cost？对这种 request shape 来说，prefill/decode colocated 会不会更快？我会对比 colocated baseline 和 disaggregated path。

然后看 transfer size、message count、memory type、source/destination GPU/NIC topology、backend selection、retry behavior，以及是否 fallback 到 host staging 或 TCP。对于 NIXL/UCX，问题不是“有没有调用 API”，而是实际选了什么 backend、memory registration 行为是什么、topology 是否合理、transfer 是否和 compute overlap。

对于 NCCL communication，我会看 message size、rank mapping、GPU/NIC affinity、selected transport、single-node vs multi-node performance，以及 communication 是否阻塞 decode steps。我会把 NCCL logs、nccl-tests 和 nsys 一起看。Nsight Compute 通常不是第一工具，除非 communication 前后的 packing/copy kernel 本身慢。

优化选择包括：改变 placement、对某些请求 colocate prefill/decode、batch/coalesce transfers、让 transfer overlap compute、改变 topology/rank mapping、调整 backend selection，或者改变 parallelism strategy。只有 end-to-end TTFT/TPOT/P99 变好，才接受这个优化。

8.13 Q: 如果加入，前 60 天你会做什么 / If you join, what would you do in the first 60 days?

English answer:

In the first 30 days, I would focus on understanding the team's real workloads, codebase, performance metrics, and data paths. I would map where NIXL, UCX, GPUNetIO, NCCL, GPU memory, scheduling, and networking topology appear in the system.

In the first 60 days, I would try to own a small but measurable task: a benchmark, a profiling investigation, a data-movement prototype, or a debug checklist. The goal would be to produce something concrete: a baseline, a bottleneck analysis, a prototype result, or a design recommendation.

I think that is the fastest way to contribute without pretending to know all internal details on day one.

中文回答：

前 30 天，我会重点理解团队真实 workload、codebase、performance metrics 和 data path，画清 NIXL、UCX、GPUNetIO、NCCL、GPU memory、scheduling、network topology 在系统里的位置。

前 60 天，我希望 ownership 一个小但可度量的任务：benchmark、profiling investigation、data-movement prototype 或 debug checklist。目标是产出具体结果：baseline、bottleneck analysis、prototype result 或 design recommendation。

这是我认为最快产生贡献的方式，同时也不会假装第一天就懂所有内部细节。

8.14 Q: 你最相关的项目是哪一个，为什么 / Which project is most relevant to this role and why?

English answer:

The most relevant project is the AI risk-control and LLM strategy platform, because it can be explained as a production AI inference path rather than a pure business project. The path was event ingestion, feature generation, online inference or LLM analysis, strategy decision, guardrail, action, feedback, and replay.

The reason it is relevant is that this role also cares about making AI workloads measurable and optimizable in production. The concrete technologies are different, but the system concerns are similar: latency breakdown, state or cache behavior, observability, correctness validation, replayability, fallback, and measurable prototype validation.

I would use the quant and GPU-adjacent projects as supporting evidence. Quant shows event-driven data paths, risk gates, replay, and hot-path thinking. Wuxia/Diamond/Blender show GPU workload and resource-lifecycle awareness.

中文回答：

最相关的是 AI risk-control 和 LLM strategy platform，因为它可以讲成 production AI inference path，而不是单纯业务项目。路径是 event ingestion、feature generation、online inference/LLM analysis、strategy decision、guardrail、action、feedback 和 replay。

相关点在于这个岗位也需要把 AI workload 放进可度量、可优化的生产系统。具体技术栈不同，但系统问题相似：latency breakdown、state/cache behavior、observability、correctness validation、replayability、fallback 和 measurable prototype validation。

Quant 和 GPU-adjacent 项目作为辅助证据：Quant 证明 event-driven data path、risk gate、replay、hot-path 思维；Wuxia/Diamond/Blender 证明 GPU workload 和 resource lifecycle 意识。

8.15 Q: 讲一个你解决过的性能问题 / Tell me about a performance problem you solved.

English answer:

I usually approach performance problems by first refusing to guess. I break the system into a data path and a control path, then add measurements at each stage. For AI or LLM systems, that means queueing, feature generation, retrieval, inference, strategy execution, and downstream action. For communication systems, that means request ingress, routing, queueing, transport, backpressure, worker execution, and response.

The most important part is separating average latency from tail latency. A system can look fine on average but still have bad P99 because of queueing, burstiness, long requests, retries, or downstream dependencies. My method is baseline, trace, bottleneck isolation, targeted change, and end-to-end validation.

If I apply the same method to NVIDIA inference systems, I would measure TTFT, TPOT, P95/P99, GPU utilization, memory pressure, communication time, queue depth, and failure or fallback rate.

中文回答：

我处理性能问题时不会先猜。我会先把系统拆成 data path 和 control path，然后给每一段加测量点。AI/LLM 系统里就是 queueing、feature generation、retrieval、inference、strategy execution、downstream action。通信系统里就是 ingress、routing、queueing、transport、backpressure、worker execution、response。

最重要的是区分 average latency 和 tail latency。平均值正常不代表 P99 正常，P99 可能来自排队、突发流量、长请求、retry 或下游依赖。我的方法是 baseline、trace、bottleneck isolation、targeted change、end-to-end validation。

迁移到 NVIDIA inference 系统，我会看 TTFT、TPOT、P95/P99、GPU utilization、memory pressure、communication time、queue depth 和 failure/fallback rate。

8.16 Q: 如果让你设计 NIXL benchmark，你怎么做 / How would you design a NIXL benchmark?

English answer:

I would not start from the API. I would start from the inference scenario. For example, prefill and decode are disaggregated, and KV cache or inference state needs to move from a prefill worker to a decode worker.

The benchmark should have at least two layers. The first layer is a microbenchmark: transfer size, memory type, source and destination placement, concurrency, latency, bandwidth, and CPU overhead. The second layer is an end-to-end serving benchmark: TTFT, TPOT, P99, throughput, GPU idle time, and whether communication overlaps with compute.

The success criteria should not be only peak bandwidth. It should answer whether NIXL improves the actual inference path, whether it reduces decode start delay or P99, and whether the complexity is justified.

中文回答：

我不会从 API 开始，而是从 inference scenario 开始。例如 prefill/decode disaggregation 下，KV cache 或 inference state 需要从 prefill worker 移到 decode worker。

benchmark 至少两层。第一层是 microbenchmark：transfer size、memory type、source/destination placement、concurrency、latency、bandwidth、CPU overhead。第二层是 end-to-end serving benchmark：TTFT、TPOT、P99、throughput、GPU idle time，以及 communication 是否能和 compute overlap。

成功标准不能只看 peak bandwidth，而是要回答 NIXL 是否改善真实 inference path，是否降低 decode start delay 或 P99，复杂度是否值得。

8.17 Q: 你怎么区分 NCCL、UCX、NIXL、GPUNetIO 的层次 / How do you separate NCCL, UCX, NIXL, and GPUNetIO by layer?

English answer:

I separate them by communication pattern and layer.

NCCL is mainly GPU collective communication: all-reduce, all-gather, reduce-scatter. UCX is a lower-level communication framework and transport abstraction: shared memory, TCP, RDMA, CUDA-aware paths. NIXL is more inference-state movement, such as KV cache or request state movement between inference components. GPUNetIO is more GPU-centric networking, where the GPU can participate more directly in packet or data processing paths.

So I would not say one replaces the other. I would ask what data is moving, between which components, with what memory type, over what transport, and what metric we are optimizing.

中文回答：

我按通信模式和层次区分。

NCCL 主要是 GPU collective communication，比如 all-reduce、all-gather、reduce-scatter。UCX 是更底层的通信框架和 transport abstraction，比如 shared memory、TCP、RDMA、CUDA-aware path。NIXL 更偏 inference-state movement，比如 KV cache 或 request state 在 inference components 之间移动。GPUNetIO 更偏 GPU-centric networking，让 GPU 更直接参与 packet/data processing path。

所以我不会说谁替代谁。我会先问：什么数据在移动、从哪里到哪里、memory type 是什么、底层 transport 是什么、优化指标是什么。

8.18 Q: 如果他们质疑你没有真实 RDMA 集群经验怎么办 / What if they challenge your lack of real RDMA cluster experience?

English answer:

I would acknowledge it directly. I have not owned a large RDMA cluster in production, and I would not claim that.

What I can bring is a disciplined systems-debugging method. For RDMA or GPUDirect, I would verify hardware support, driver support, topology, memory registration, selected transport, logs, counters, and benchmark behavior. I would compare TCP baseline, RDMA baseline, host-memory path, and GPU-memory path, and then connect that to end-to-end workload metrics.

So the gap is real, but the ramp-up path is concrete and measurable.

中文回答：

我会直接承认。我没有 owning 过大规模 RDMA 集群生产系统，不会假装有。

我能带来的是严谨的系统排查方法。对 RDMA/GPUDirect，我会验证硬件支持、driver、topology、memory registration、selected transport、logs、counters 和 benchmark behavior。我会对比 TCP baseline、RDMA baseline、host-memory path、GPU-memory path，然后再连接到 end-to-end workload metrics。

所以这个 gap 是真实存在的，但我的 ramp-up 路径是具体、可测量的。

8.19 Q: 如果他们问你最担心入职后哪里上手慢 / What part would take you longest to ramp up?

English answer:

The slowest ramp-up would probably be the internal NVIDIA-specific codebase and production constraints around NIXL, GPUNetIO, and real cluster deployments. Public concepts are not the same as internal workload details.

My way to reduce that risk is to start with the data path and metrics. I would map where each component sits, what workloads matter, what benchmark is trusted by the team, what logs and counters are used, and what failure modes are common. Then I would take a small measurable task such as a profiling investigation, a benchmark improvement, or a debug checklist.

中文回答：

上手最慢的可能是 NVIDIA 内部 codebase，以及 NIXL、GPUNetIO、真实集群部署里的生产约束。公开概念和内部 workload 细节不是一回事。

我的降低风险方式是从 data path 和 metrics 开始：画清每个组件在哪里，团队关心什么 workload，信任哪个 benchmark，用哪些 logs/counters，常见 failure modes 是什么。然后 ownership 一个小但可度量的任务，比如 profiling investigation、benchmark improvement 或 debug checklist。

8.20 Q: 你会怎么问澄清问题 / How would you ask clarification questions in a system design interview?

English answer:

I would first clarify the workload before designing the architecture. For LLM inference, I would ask about model size, number of GPUs, context length distribution, output length, request rate, TTFT/TPOT SLO, P99 target, streaming requirements, multi-tenancy, and topology.

Then I would clarify the optimization objective: throughput, latency, cost, GPU utilization, memory efficiency, or reliability. After that, I would propose the data path, control path, metrics, failure modes, and validation plan.

中文回答：

我会先澄清 workload，再设计架构。LLM inference 场景下，我会问 model size、GPU 数量、context length distribution、output length、request rate、TTFT/TPOT SLO、P99 target、是否 streaming、多租户、topology。

然后澄清优化目标：throughput、latency、cost、GPU utilization、memory efficiency 还是 reliability。之后再提出 data path、control path、metrics、failure modes 和 validation plan。

9. 他们可能会追你的短板：答案要更直接

Q: 你没有直接 NIXL/GPUNetIO 生产经验，为什么匹配 / You do not seem to have direct NIXL/GPUNetIO production experience. Why are you a fit?

English answer:

I would not claim direct production ownership of NIXL or GPUNetIO. My fit comes from system architecture, performance diagnosis, AI inference path understanding, communication-path debugging, and GPU profiling methodology. For NIXL and GPUNetIO, my ramp-up is not just reading APIs. I place them into the data path: what data movement problem they solve, where they sit relative to NCCL and UCX, what metrics matter, and how I would validate performance. I can contribute through prototypes, benchmarks, design analysis, and debug checklists, which I believe is highly relevant to an architect role.

中文回答：

我不会把 NIXL/GPUNetIO 说成自己过去 production owning 的系统。我的匹配点是系统架构、性能诊断、AI inference path、通信路径和 GPU profiling 方法论。对于 NIXL/GPUNetIO，我的学习方式不是只背 API，而是把它放到 data path 里：它解决什么数据移动问题，和 NCCL/UCX 的边界是什么，关键指标是什么，慢了怎么验证。我能快速做 prototype、benchmark 和 debug checklist，这和 architect role 需要的能力是匹配的。

Q: 你的 GPU 经验是 Vulkan/WebGPU，不是 CUDA，这够吗 / Your GPU experience is Vulkan/WebGPU, not CUDA. Is that enough?

English answer:

I would distinguish transferable GPU systems experience from CUDA-specific ownership. Vulkan/WebGPU gave me hands-on practice with GPU timelines, memory layout, command scheduling, synchronization, and profiling-driven optimization. CUDA has different APIs and a deeper kernel-level programming model, so I am closing that gap with focused CUDA experiments around memory coalescing, pinned memory, streams, occupancy, and Nsight Compute. I would not pretend the APIs are the same, but the performance reasoning is transferable.

10. 反问他们的问题

这些问题适合 AI Networking / NVIDIA Israel 方向面试官。反问不是为了显得聪明，而是要判断这个岗位到底偏 research、prototype、library internals，还是 end-to-end product integration。

10.1 Q: 这个岗位更偏推理数据搬运、通信库架构，还是端到端 AI 系统架构 / For this role, is the main focus closer to inference data movement, communication-library architecture, or end-to-end AI system architecture?

你为什么问：

这个岗位描述横跨 NIXL/UCX/GPUNetIO、Dynamo、AI inference、system architecture。你要判断真正的工作重心。

你希望听到什么：

如果对方说 inference data movement：重点补 KV cache、prefill/decode、NIXL、serving。
如果对方说 communication library architecture：重点补 UCX/RDMA/NCCL/transport。
如果对方说 end-to-end AI system architecture：重点展示系统拆解、prototype、benchmark、roadmap。

10.2 Q: 团队怎么定义 NIXL/UCX/GPUNetIO 原型成功 / How does the team define success for prototypes around NIXL, UCX, or GPUNetIO?

你为什么问：

Architect/research/prototype 岗位最怕不知道 success metric。你要知道他们看重 paper idea、benchmark、product integration，还是 workload enablement。

你希望听到什么：

好的答案会提到 measurable baseline、real workload、benchmark reproducibility、integration path、customer/internal user impact。
如果只说“explore new ideas”，说明可能更 research；如果强调 shipping/integration，说明工程落地更重。

10.3 Q: 多节点推理最常见瓶颈是什么 / What are the most common bottlenecks in multi-node inference workloads?

你为什么问：

这个问题能让面试官暴露团队真正面对的技术痛点。你也能顺势接你的准备：TTFT/TPOT/P99、KV、NIXL、UCX、NCCL、GPU memory。

你希望听到什么：

如果对方说 KV movement：重点是 NIXL、prefill/decode disaggregation、GPU memory。
如果对方说 scheduling：重点是 batching、admission control、fairness、tail latency。
如果对方说 topology/network：重点是 RDMA、GPUDirect、UCX、NIC/GPU placement。
如果对方说 collectives：重点是 NCCL、parallelism、rank mapping、algbw/busbw。

10.4 Q: 团队如何和模型、运行时、编译器、网络团队跨区域协作 / How does this team collaborate with model/runtime/compiler/networking teams across regions?

你为什么问：

这个岗位是系统架构岗，不会只在一个模块里闭门写代码。你要判断它是否需要跨团队沟通、设计文档、prototype、benchmark 和 roadmap。

你希望听到什么：

好的答案会提到跨 runtime、model serving、networking、driver/library、infra、product team 的协作。
这也能让你强调自己过去 cross-layer collaboration、diagnostic tooling 和 architecture documentation 的经验。

10.5 Q: 新人前 60 天最该补什么技术 / For someone joining this team, what would be the most important technical area to ramp up in the first 60 days?

你为什么问：

这是最实用的问题。对方的回答可以直接指导你入职前和入职后的学习重点。

你希望听到什么：

如果对方说 NIXL/Dynamo：优先补 inference data movement。
如果对方说 UCX/RDMA：优先补 transport、topology、GPUDirect。
如果对方说 workload：优先补 LLM inference、prefill/decode、KV、serving scheduler。
如果对方说 codebase：说明入职后工程 ownership 很快，需要更快进入 C++/Python/Linux/debug。

11. 面试前最后 30 分钟复习清单

1. 背熟 sequence-08 的 60 秒英文自我介绍。
2. 准备一个最相关项目：AI/LLM 平台。
3. 准备一个系统项目：高并发通信系统。
4. 准备一个 GPU 项目：Vulkan/WebGPU profiling。
5. 复述 UCX/NCCL/NIXL/GPUNetIO 区别。
6. 复述 prefill/decode/KV/TTFT/TPOT/P99。
7. 准备一套“我没有直接 production ownership，但我怎么验证和 ramp up”的回答。
8. 准备 3 个反问问题。

12. 参考来源

• NVIDIA Workday JD JR2013092: https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/System-Software-Architect--AI-and-GPU-Networking_JR2013092
• NVIDIA Jobs - Senior Software Architect, AI Networking, Tel Aviv: https://jobs.nvidia.com/careers/job/893382766050
• NVIDIA completes acquisition of Mellanox: https://nvidianews.nvidia.com/news/nvidia-completes-acquisition-of-mellanox-creating-major-force-driving-next-gen-data-centers
• NVIDIA Networking overview: https://www.nvidia.com/en-us/networking/
• NVIDIA NIXL blog: https://developer.nvidia.com/blog/enhancing-distributed-inference-performance-with-the-nvidia-inference-transfer-library/
• NVIDIA DOCA GPUNetIO docs: https://docs.nvidia.com/doca/sdk/doca-gpunetio/index.html