Debugging OOM and rank hangs on dual DGX Spark with DeepSeek V4.1 Flash
A developer traces two failure causes on a dual DGX Spark setup: concurrent long prompts OOM rank 0, which tokenizes alone and holds ~1.4GB per 1M-token request, and when one rank dies the other hangs forever; their branch detects the broken pair and restarts both ranks, noticing a dead rank in about 1s and recovering in under a minute.
Mine was doing similar things until I tracked down two causes: several long prompts at once can OOM rank 0 (rank 0 alone tokenizes, and held ~1.4 GB per 1M-token request until the reply finished; fix in MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#40), and when one rank dies the other just hangs forever. My branch (#30, or wojo/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks main) detects a broken pair and restarts both ranks automatically, about 1 s to notice a dead rank and under a minute to recover. What do your logs show when it crashes? A memguard kill, a CUDA/NCCL error, or does it just hang?