Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo

NVIDIA introduces Shadow Engine Recovery for Dynamo, a technique to restore LLM inference capacity in seconds instead of performing lengthy cold restarts that require reloading weights and recompiling kernels.

When an LLM engine process fails, the standard recovery path involves a cold restart. This requires loading weights into HBM from storage, compiling kernels,...