Allocate failed due to rpc error: code = Unknown desc = no free node, which is unexpected

[mingkai-yang]

我在集群同一批机器上使用 [单卡共享]、[单卡独占]、[多卡独占] 混合任务时，某些机器上vcua core满足，pod调度成功，但创建失败，gpu-manager 日志在一直提示 no free node，我猜测可能是机器上混合使用不同模式导致的，是不是需要不同模式任务 要做隔离呢？

pod状态：UnexpectedAdmissionError
创建失败的pod事件信息：Allocate failed due to rpc error: code = Unknown desc = no free node, which is unexpected

[mingkai-yang]

谁有遇到过吗

[xiaoertong]

我也遇到了相同的问题，看日志如下，日志中显示的资源总和是够的

[xiaoertong]

@mingkai-yang 问题解决了吗？

[huanghuangzym]

I0220 09:55:09.776328 4096971 allocator.go:385] Tree graph: ROOT:0
|---SYS (aval: 0, pids: [], usedMemory: 0, totalMemory: 687194767360, allocatableCores: 0, allocatableMemory: 0)
| |---CPU (aval: 0, pids: [], usedMemory: 0, totalMemory: 343597383680, allocatableCores: 0, allocatableMemory: 0)
| | |---PXB (aval: 0, pids: [], usedMemory: 0, totalMemory: 171798691840, allocatableCores: 0, allocatableMemory: 0)
| | | |---GPU4 (pids: [], usedMemory: 0, totalMemory: 85899345920, allocatableCores: 100, allocatableMemory: 85899345920)
| | | |---GPU5 (pids: [], usedMemory: 0, totalMemory: 85899345920, allocatableCores: 100, allocatableMemory: 85899345920)
| | |---PXB (aval: 0, pids: [], usedMemory: 0, totalMemory: 171798691840, allocatableCores: 0, allocatableMemory: 0)
| | | |---GPU6 (pids: [], usedMemory: 0, totalMemory: 85899345920, allocatableCores: 100, allocatableMemory: 85899345920)
| | | |---GPU7 (pids: [], usedMemory: 0, totalMemory: 85899345920, allocatableCores: 100, allocatableMemory: 85899345920)
| |---CPU (aval: 0, pids: [], usedMemory: 0, totalMemory: 343597383680, allocatableCores: 0, allocatableMemory: 0)
| | |---PXB (aval: 0, pids: [], usedMemory: 0, totalMemory: 171798691840, allocatableCores: 0, allocatableMemory: 0)
| | | |---GPU2 (pids: [], usedMemory: 0, totalMemory: 85899345920, allocatableCores: 100, allocatableMemory: 85899345920)
| | | |---GPU3 (pids: [], usedMemory: 0, totalMemory: 85899345920, allocatableCores: 100, allocatableMemory: 85899345920)
| | |---PXB (aval: 0, pids: [], usedMemory: 0, totalMemory: 171798691840, allocatableCores: 0, allocatableMemory: 0)
| | | |---GPU0 (pids: [], usedMemory: 0, totalMemory: 85899345920, allocatableCores: 100, allocatableMemory: 85899345920)
| | | |---GPU1 (pids: [], usedMemory: 0, totalMemory: 85899345920, allocatableCores: 100, allocatableMemory: 85899345920)
I0220 09:55:09.776339 4096971 allocator.go:396] Try allocate for 70986335-ee14-4b4f-9cb8-a34d7cc79347(deeepseek), vcore 100, vmemory 0
E0220 09:55:09.776352 4096971 allocator.go:755] no free node 相同问题

[spiritLHLS]

遇到了相同的问题，请问怎么解决的？

有同事说调度器问题，用 schedulerName: volcano 就行了，在corev1.PodSpec加上SchedulerName: "volcano",

[mingkai-yang]

遇到了相同的问题，请问怎么解决的？

有同事说调度器问题，用 schedulerName: volcano 就行了，在corev1.PodSpec加上SchedulerName: "volcano",

是的吗。但是vcuda是有admission的，会拦截使用自己的调度器的，