Sequence 07 - 实战实验和验证证据

实验不是为了刷题，而是为了回答面试追问。

1. 实验总图

flowchart TB
    Labs[Practical Labs] --> CUDA[CUDA/Nsight Labs]
    Labs --> VLLM[vLLM Serving Labs]
    Labs --> NCCL[NCCL Labs]
    Labs --> UCX[UCX/RDMA Labs]
    Labs --> Design[Design-only Labs]

    CUDA --> C1[Memory coalescing]
    CUDA --> C2[Pinned memory copy]
    CUDA --> C3[nsys timeline]
    CUDA --> C4[ncu kernel metrics]
    CUDA --> C5[LLM-style profiling story]

    VLLM --> V1[Input length vs TTFT]
    VLLM --> V2[Output length vs TPOT]
    VLLM --> V3[Request rate vs P99]

    NCCL --> N1[all_reduce_perf]
    NCCL --> N2[algbw/busbw interpretation]

    UCX --> U1[ucx_info]
    UCX --> U2[transport/device boundary]

    Design --> D1[NIXL KV transfer design]
    Design --> D2[GPUNetIO path design]

1.1 必做实验：nsys / ncu profiling 闭环

目标：

不是证明你会打开工具，而是证明你知道：
1. 先用 nsys 找系统 critical path。
2. 再用 ncu 深挖单个 kernel。
3. 最后回到 end-to-end 指标验证优化是否成立。

建议环境：

检查命令：

nvidia-smi
nvcc --version
nsys --version
ncu --version

实验 A：nsys 看 timeline

要跑什么：

任选一个 CUDA sample、自己的 CUDA 小程序、PyTorch CUDA 脚本、vLLM benchmark。
目标是生成 timeline，观察 CPU、CUDA API、memcpy、kernel、stream、sync、GPU idle。

命令模板：

mkdir -p reports
nsys profile \
  --trace=cuda,nvtx,osrt \
  --cuda-memory-usage=true \
  --force-overwrite=true \
  -o reports/timeline_baseline \
  ./your_cuda_app

如果是 Python：

nsys profile \
  --trace=cuda,nvtx,osrt \
  --cuda-memory-usage=true \
  --force-overwrite=true \
  -o reports/python_timeline \
  python your_script.py

怎么看：

面试怎么讲：

I use nsys first because it tells me whether the bottleneck is even inside a CUDA kernel. If the GPU is idle, optimizing a kernel is the wrong target. I look for CPU feeding gaps, CUDA API overhead, serialized copies, stream synchronization, communication wait, and GPU idle time.

实验 B：ncu 看单 kernel

要跑什么：

先用 nsys 找到最重的 kernel，再用 ncu 指定 kernel 深挖。
不要对整个复杂 app 无脑 --set full，否则会很慢。

命令模板：

ncu \
  --set full \
  --target-processes all \
  --launch-skip 5 \
  --launch-count 1 \
  --force-overwrite \
  -o reports/kernel_deep_dive \
  ./your_cuda_app

如果知道 kernel 名：

ncu \
  --set full \
  --kernel-name regex:your_kernel_name \
  --launch-skip 5 \
  --launch-count 1 \
  --force-overwrite \
  -o reports/target_kernel \
  ./your_cuda_app

怎么看：

面试怎么讲：

After nsys identifies the dominant kernel, I use ncu to explain the kernel limiter. If it is memory-bound, I check coalescing, layout, reuse, and traffic reduction. If it is compute-bound, I look at tensor core usage, instruction mix, and fusion. If occupancy or stalls are the issue, I check register pressure, shared memory, block size, dependencies, and barriers.

实验 C：把 profiling 连接到 LLM serving

即使本地没有完整生产 serving，也要能讲这个映射：

flowchart TB
    Metrics[Serving Metrics] --> TTFT[TTFT]
    Metrics --> TPOT[TPOT]
    Metrics --> P99[P99]

    TTFT --> Queue[Queueing / Tokenization / Scheduling]
    TTFT --> Prefill[Prefill Kernel / KV Allocation]
    TTFT --> Transfer[KV Transfer]

    TPOT --> Decode[Decode Kernel]
    TPOT --> KVAccess[KV Cache Access]
    TPOT --> Comm[Tensor Parallel Communication]
    TPOT --> Stream[Streaming Backpressure]

    P99 --> Skew[Workload Skew]
    P99 --> Memory[Memory Pressure]
    P99 --> SlowWorker[Slow GPU / NIC / Worker]

    Queue --> NSYS[nsys Timeline]
    Prefill --> NCU[ncu Kernel Metrics]
    Decode --> NCU
    Comm --> NSYS
    Transfer --> NSYS

面试可讲总结：

For TTFT, I start from request trace and nsys: queueing, tokenization, scheduling, prefill, KV allocation, and transfer. For TPOT, I inspect the decode loop: per-token kernels, KV cache access, batching, communication, and streaming. For P99, I compare fast and slow request timelines by prompt length, output length, tenant, worker, GPU, and node. I use ncu only after a specific kernel is proven to dominate.

2. CUDA memory coalescing

目标：

证明你知道 GPU memory access pattern 会影响性能。

实验：

• kernel A：out[i] = in[i]
• kernel B：out[i] = in[(i * stride) % n]

看什么：

面试回答：

Coalesced access lets adjacent threads access adjacent memory, which allows the GPU to combine memory transactions. Strided or scattered access increases transactions and lowers effective bandwidth.

3. Pinned memory copy

目标：

证明你理解 CPU-GPU data movement，也能连接到 GPUDirect/RDMA/NIXL。

实验：

• pageable host memory + H2D/D2H
• pinned host memory + H2D/D2H
• 可选：pinned + async copy + stream overlap

看什么：

4. vLLM benchmark

目标：

把 serving 指标和 runtime 阶段对应起来。

实验矩阵：

看什么：

5. NCCL all-reduce

目标：

证明你能用 microbenchmark 抽离通信问题。

命令：

git clone https://github.com/NVIDIA/nccl-tests.git labs/nccl_tests/nccl-tests
cd labs/nccl_tests/nccl-tests
make MPI=0
NCCL_DEBUG=INFO ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 1

看什么：

6. UCX/RDMA capability check

目标：

诚实判断本机能验证什么，不能验证什么。

命令：

ucx_info -v
ucx_info -d
nvidia-smi topo -m
ibv_devinfo || true

判断：

7. NIXL KV transfer 设计实验

本地没有 NIXL 环境也要会设计：

flowchart TB
    Baseline[Baseline serving] --> Measure[Measure TTFT/TPOT/P99]
    Measure --> Disagg[Prefill/Decode Disaggregation]
    Disagg --> KV[KV Blocks Produced]
    KV --> Transfer[Transfer via NIXL-like path]
    Transfer --> Decode[Decode Worker]
    Decode --> E2E[End-to-end Metrics]
    E2E --> Compare[Compare with baseline]

要看：

• transfer latency
• decode start delay
• GPU idle time
• TTFT/P99 impact
• memory usage
• failure/fallback behavior

8. GPUNetIO 设计实验

本地没有 DOCA/NVIDIA NIC，也要能画：

flowchart TB
    Packet[Network Packet] --> CPU[CPU Networking Stack]
    CPU --> HostBuf[Host Buffer]
    HostBuf --> GPUCopy[Copy to GPU]
    GPUCopy --> GPU[GPU Processing]

    Packet --> RNIC[RNIC/DOCA]
    RNIC --> GPUNetIO[GPUNetIO]
    GPUNetIO --> GPU2[GPU Processing]

    CPU --> Cost1[CPU overhead]
    GPUCopy --> Cost2[Copy latency]
    GPUNetIO --> Risk[Complexity/debug risk]

面试结论：

I would only consider GPUNetIO if CPU-mediated networking and CPU-GPU copy are on the critical path and the GPU directly consumes the network data. Otherwise the added complexity may not be justified.