PyTorch/CUDA Expert for Audio AI Pipeline
Publicada el 2026-07-20
Descripción de la oferta
Job Description: I'm running a self-hosted, real-time conversational voice AI pipeline (STT → LLM → streaming TTS over WebSocket) on a RunPod GPU pod (CSM-1B TTS model, using the davidbrowne17/csm-streaming fork with Sesame's Mimi/moshi audio codec, PyTorch torch.compile). I need an experienced PyTorch/CUDA engineer to find and fix a reproducible latency bug that's blocking production readiness. The problem: At sentence/turn boundaries in a multi-turn conversation, generation stalls for ~7-8 seconds of dead air. I've already localized this precisely via wall-clock instrumentation: the stall occurs specifically inside Mimi's audio_tokenizer.encode() call, when encoding a previous turn's generated audio into context — and specifically only when encoding an audio length/shape the process hasn't seen before (repeat lengths are ~12ms; new lengths cost ~7-8s, every time). What's already been ruled out (to save you time re-testing these): - torch.compile(mode='reduce-overhead') vs mode='default' — switched to default, no change - moshi's internal CUDAGraphed wrapper — disabled via no_cuda_graph() around both the generation loop and the context-tokenization loop — no change - moshi's torch_compile_lazy decorator — disabled via NO_TORCH_COMPILE=1 — no change - torch._dynamo.config.cache_size_limit (default 8) — raised to 256 — no change - Shape bucketing/padding to fixed sizes — implemented, not conclusively re-validated after a related crash fix - Standalone isolated reproduction (Mimi-only, no CSM, alternating encode/decode across simulated turns) — does NOT reproduce the stall, meaning something about the full server process (FastAPI/asyncio/ThreadPoolExecutor architecture, or CSM+Mimi coexisting in one process) is required to trigger it Scope of work: 1. Reproduce the stall in my environment (RunPod SSH access provided) or your own equivalent setup 2. Use direct profiling (py-spy, torch.profiler, or nsys) to find the actual root cause — not another disable-and-guess cycle 3. Implement and verify a fix, confirmed via a provided gap-test script showing zero >500ms inter-chunk gaps across a multi-turn conversation 4. Document the root cause and fix clearly enough for me to maintain it going forward Required skills: - Strong PyTorch internals experience: torch.compile, CUDA graphs, dynamo/inductor caching behavior - Experience with audio codec models (RVQ/quantizer architectures) a strong plus — this is Kyutai's Mimi codec specifically - Comfortable profiling GPU workloads (py-spy, torch.profiler, nsys/nvprof) - Async Python (FastAPI, asyncio, ThreadPoolExecutor) — the bug may be process-architecture-related, not purely model-relate
Skills
Fuente original: freelancer