Repository navigation
为什么我在 B300 上部署 Kimi2.5 输出全部是乱码呢？ #34
Description
[Unit]
Description=vLLM Kimi-K2.5 Service
After=network.target nvidia-persistenced.service
[Service]
Type=simple
User=root
Group=root
Environment="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
Environment="HF_ENDPOINT=https://hf-mirror.com"
Environment="VLLM_USE_FLASHINFER_SAMPLER=0"
ExecStart=/root/ai/bin/vllm serve moonshotai/Kimi-K2.5 \
--tensor-parallel-size 8 \
--mm-encoder-tp-mode data \
--tool-call-parser kimi_k2 \
--reasoning-parser kimi_k2 \
--served-model-name Kimi-K2.5 \
--trust-remote-code
Restart=no
RestartSec=10
LimitCORE=infinity
LimitNOFILE=65535
[Install]
WantedBy=multi-user.targetroot@localhost:~# systemctl daemon-reload
root@localhost:~# systemctl start vllm-kimi.service
root@localhost:~# journalctl -o cat -f -u vllm-kimi.service
(Worker_TP0 pid=171411) INFO 03-30 20:52:56 [multiproc_executor.py:764] Parent process exited, terminating worker queues
(APIServer pid=170649) INFO: Waiting for application shutdown.
(APIServer pid=170649) INFO: Application shutdown complete.
(APIServer pid=170649) INFO: Finished server process [170649]
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
vllm-kimi.service: Deactivated successfully.
Stopped vllm-kimi.service - vLLM Kimi-K2.5 Service.
vllm-kimi.service: Consumed 56min 24.403s CPU time, 43.0G memory peak, 0B memory swap peak.
Started vllm-kimi.service - vLLM Kimi-K2.5 Service.
(APIServer pid=176029) INFO 03-30 20:53:48 [utils.py:299]
(APIServer pid=176029) INFO 03-30 20:53:48 [utils.py:299] █ █ █▄ ▄█
(APIServer pid=176029) INFO 03-30 20:53:48 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.18.1rc1.dev213+gbda3eda82.d20260328
(APIServer pid=176029) INFO 03-30 20:53:48 [utils.py:299] █▄█▀ █ █ █ █ model moonshotai/Kimi-K2.5
(APIServer pid=176029) INFO 03-30 20:53:48 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=176029) INFO 03-30 20:53:48 [utils.py:299]
(APIServer pid=176029) INFO 03-30 20:53:48 [utils.py:233] non-default args: {'model_tag': 'moonshotai/Kimi-K2.5', 'tool_call_parser': 'kimi_k2', 'api_key': ['56XASFNAZW2FCHUMXMXQ756GPILM23DR'], 'model': 'moonshotai/Kimi-K2.5', 'trust_remote_code': True, 'served_model_name': ['Kimi-K2.5'], 'reasoning_parser': 'kimi_k2', 'tensor_parallel_size': 8, 'mm_encoder_tp_mode': 'data'}
(APIServer pid=176029) INFO 03-30 20:53:52 [model.py:549] Resolved architecture: KimiK25ForConditionalGeneration
(APIServer pid=176029) INFO 03-30 20:53:52 [model.py:1678] Using max model len 262144
(APIServer pid=176029) INFO 03-30 20:53:52 [scheduler.py:238] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=176029) INFO 03-30 20:53:52 [vllm.py:786] Asynchronous scheduling is enabled.
(APIServer pid=176029) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(APIServer pid=176029) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(APIServer pid=176029) INFO 03-30 20:53:58 [compilation.py:290] Enabled custom fusions: allreduce_rms
(APIServer pid=176029) Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
(EngineCore pid=176570) INFO 03-30 20:54:15 [core.py:105] Initializing a V1 LLM engine (v0.18.1rc1.dev213+gbda3eda82.d20260328) with config: model='moonshotai/Kimi-K2.5', speculative_config=None, tokenizer='moonshotai/Kimi-K2.5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='kimi_k2', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Kimi-K2.5, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [146, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=176570) WARNING 03-30 20:54:15 [multiproc_executor.py:1014] Reducing Torch parallelism from 128 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=176570) INFO 03-30 20:54:15 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.10.0.196 (local), world_size=8, local_world_size=8
Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
(Worker pid=176788) INFO 03-30 20:54:35 [parallel_state.py:1400] world_size=8 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:38521 backend=nccl
(Worker pid=176789) INFO 03-30 20:54:36 [parallel_state.py:1400] world_size=8 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:38521 backend=nccl
(Worker pid=176790) INFO 03-30 20:54:37 [parallel_state.py:1400] world_size=8 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:38521 backend=nccl
(Worker pid=176791) INFO 03-30 20:54:37 [parallel_state.py:1400] world_size=8 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:38521 backend=nccl
(Worker pid=176793) INFO 03-30 20:54:37 [parallel_state.py:1400] world_size=8 rank=5 local_rank=5 distributed_init_method=tcp://127.0.0.1:38521 backend=nccl
(Worker pid=176794) INFO 03-30 20:54:39 [parallel_state.py:1400] world_size=8 rank=6 local_rank=6 distributed_init_method=tcp://127.0.0.1:38521 backend=nccl
(Worker pid=176792) INFO 03-30 20:54:41 [parallel_state.py:1400] world_size=8 rank=4 local_rank=4 distributed_init_method=tcp://127.0.0.1:38521 backend=nccl
(Worker pid=176795) INFO 03-30 20:54:43 [parallel_state.py:1400] world_size=8 rank=7 local_rank=7 distributed_init_method=tcp://127.0.0.1:38521 backend=nccl
(Worker pid=176795) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=176795) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=176792) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=176792) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=176791) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=176791) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=176788) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=176788) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=176788) INFO 03-30 20:54:44 [pynccl.py:111] vLLM is using nccl==2.27.5
(Worker pid=176794) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=176794) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=176789) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=176789) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=176790) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=176790) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=176793) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=176793) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=176788) INFO 03-30 20:54:52 [parallel_state.py:1716] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker_TP0 pid=176788) INFO 03-30 20:54:53 [gpu_model_runner.py:4723] Starting to load model moonshotai/Kimi-K2.5...
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [cuda.py:390] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [vllm.py:786] Asynchronous scheduling is enabled.
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [compilation.py:290] Enabled custom fusions: allreduce_rms
(Worker_TP7 pid=176795) INFO 03-30 20:54:54 [selector.py:124] Using HND KV cache layout for FLASHINFER_MLA backend.
(Worker_TP1 pid=176789) INFO 03-30 20:54:54 [selector.py:124] Using HND KV cache layout for FLASHINFER_MLA backend.
(Worker_TP4 pid=176792) INFO 03-30 20:54:54 [selector.py:124] Using HND KV cache layout for FLASHINFER_MLA backend.
(Worker_TP3 pid=176791) INFO 03-30 20:54:54 [selector.py:124] Using HND KV cache layout for FLASHINFER_MLA backend.
(Worker_TP2 pid=176790) INFO 03-30 20:54:54 [selector.py:124] Using HND KV cache layout for FLASHINFER_MLA backend.
(Worker_TP6 pid=176794) INFO 03-30 20:54:54 [selector.py:124] Using HND KV cache layout for FLASHINFER_MLA backend.
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [cuda.py:334] Using FLASHINFER_MLA attention backend out of potential backends: ['FLASHINFER_MLA', 'CUTLASS_MLA', 'TRITON_MLA'].
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [selector.py:124] Using HND KV cache layout for FLASHINFER_MLA backend.
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [mla_attention.py:2137] Using FlashAttention prefill for MLA
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [deep_gemm.py:115] DeepGEMM E8M0 enabled on current platform.
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [compressed_tensors_moe.py:194] Using CompressedTensorsWNA16MarlinMoEMethod
(Worker_TP0 pid=176788) INFO 03-30 20:54:54 [compressed_tensors_moe.py:1180] Using Marlin backend for WNA16 MoE (group_size=32, num_bits=4)
(Worker_TP5 pid=176793) INFO 03-30 20:54:54 [selector.py:124] Using HND KV cache layout for FLASHINFER_MLA backend.
(Worker_TP0 pid=176788) INFO 03-30 20:54:59 [weight_utils.py:581] Time spent downloading weights for moonshotai/Kimi-K2.5: 1.010325 seconds
Loading safetensors checkpoint shards: 0% Completed | 0/64 [00:00<?, ?it/s]
(Worker_TP1 pid=176789) INFO 03-30 20:55:01 [weight_utils.py:581] Time spent downloading weights for moonshotai/Kimi-K2.5: 1.527843 seconds
(Worker_TP4 pid=176792) INFO 03-30 20:55:04 [weight_utils.py:581] Time spent downloading weights for moonshotai/Kimi-K2.5: 0.500123 seconds
Loading safetensors checkpoint shards: 3% Completed | 2/64 [00:05<02:37, 2.55s/it]
Loading safetensors checkpoint shards: 5% Completed | 3/64 [00:10<03:35, 3.53s/it]
Loading safetensors checkpoint shards: 6% Completed | 4/64 [00:15<04:10, 4.18s/it]
Loading safetensors checkpoint shards: 8% Completed | 5/64 [00:20<04:28, 4.56s/it]
Loading safetensors checkpoint shards: 9% Completed | 6/64 [00:25<04:30, 4.66s/it]
Loading safetensors checkpoint shards: 11% Completed | 7/64 [00:30<04:31, 4.76s/it]
Loading safetensors checkpoint shards: 12% Completed | 8/64 [00:34<04:10, 4.48s/it]
Loading safetensors checkpoint shards: 14% Completed | 9/64 [00:38<04:04, 4.44s/it]
Loading safetensors checkpoint shards: 16% Completed | 10/64 [00:42<03:57, 4.40s/it]
Loading safetensors checkpoint shards: 17% Completed | 11/64 [00:47<03:51, 4.36s/it]
Loading safetensors checkpoint shards: 19% Completed | 12/64 [00:51<03:45, 4.34s/it]
Loading safetensors checkpoint shards: 20% Completed | 13/64 [00:55<03:40, 4.33s/it]
Loading safetensors checkpoint shards: 22% Completed | 14/64 [00:59<03:34, 4.29s/it]
Loading safetensors checkpoint shards: 23% Completed | 15/64 [01:04<03:28, 4.25s/it]
Loading safetensors checkpoint shards: 25% Completed | 16/64 [01:08<03:21, 4.20s/it]
Loading safetensors checkpoint shards: 27% Completed | 17/64 [01:11<03:03, 3.91s/it]
Loading safetensors checkpoint shards: 28% Completed | 18/64 [01:15<03:03, 4.00s/it]
Loading safetensors checkpoint shards: 30% Completed | 19/64 [01:19<03:01, 4.04s/it]
Loading safetensors checkpoint shards: 31% Completed | 20/64 [01:23<02:58, 4.05s/it]
Loading safetensors checkpoint shards: 33% Completed | 21/64 [01:27<02:55, 4.07s/it]
Loading safetensors checkpoint shards: 34% Completed | 22/64 [01:32<02:51, 4.09s/it]
Loading safetensors checkpoint shards: 36% Completed | 23/64 [01:36<02:48, 4.10s/it]
Loading safetensors checkpoint shards: 38% Completed | 24/64 [01:40<02:43, 4.08s/it]
Loading safetensors checkpoint shards: 39% Completed | 25/64 [01:43<02:29, 3.84s/it]
Loading safetensors checkpoint shards: 41% Completed | 26/64 [01:47<02:23, 3.76s/it]
Loading safetensors checkpoint shards: 42% Completed | 27/64 [01:50<02:16, 3.69s/it]
Loading safetensors checkpoint shards: 44% Completed | 28/64 [01:54<02:09, 3.60s/it]
Loading safetensors checkpoint shards: 45% Completed | 29/64 [01:57<02:04, 3.56s/it]
Loading safetensors checkpoint shards: 47% Completed | 30/64 [02:00<01:59, 3.51s/it]
Loading safetensors checkpoint shards: 48% Completed | 31/64 [02:04<01:54, 3.47s/it]
Loading safetensors checkpoint shards: 50% Completed | 32/64 [02:07<01:49, 3.41s/it]
Loading safetensors checkpoint shards: 52% Completed | 33/64 [02:10<01:44, 3.38s/it]
Loading safetensors checkpoint shards: 53% Completed | 34/64 [02:14<01:40, 3.35s/it]
Loading safetensors checkpoint shards: 55% Completed | 35/64 [02:17<01:36, 3.33s/it]
Loading safetensors checkpoint shards: 56% Completed | 36/64 [02:20<01:32, 3.32s/it]
Loading safetensors checkpoint shards: 58% Completed | 37/64 [02:23<01:29, 3.31s/it]
Loading safetensors checkpoint shards: 59% Completed | 38/64 [02:27<01:25, 3.30s/it]
Loading safetensors checkpoint shards: 61% Completed | 39/64 [02:30<01:22, 3.30s/it]
Loading safetensors checkpoint shards: 62% Completed | 40/64 [02:33<01:18, 3.29s/it]
Loading safetensors checkpoint shards: 64% Completed | 41/64 [02:37<01:15, 3.29s/it]
Loading safetensors checkpoint shards: 66% Completed | 42/64 [02:40<01:12, 3.29s/it]
Loading safetensors checkpoint shards: 67% Completed | 43/64 [02:43<01:08, 3.28s/it]
Loading safetensors checkpoint shards: 69% Completed | 44/64 [02:46<01:05, 3.29s/it]
Loading safetensors checkpoint shards: 70% Completed | 45/64 [02:50<01:02, 3.29s/it]
Loading safetensors checkpoint shards: 72% Completed | 46/64 [02:53<00:59, 3.29s/it]
Loading safetensors checkpoint shards: 73% Completed | 47/64 [02:56<00:55, 3.29s/it]
Loading safetensors checkpoint shards: 75% Completed | 48/64 [03:00<00:52, 3.29s/it]
Loading safetensors checkpoint shards: 77% Completed | 49/64 [03:03<00:49, 3.29s/it]
Loading safetensors checkpoint shards: 78% Completed | 50/64 [03:06<00:46, 3.29s/it]
Loading safetensors checkpoint shards: 80% Completed | 51/64 [03:09<00:42, 3.29s/it]
Loading safetensors checkpoint shards: 81% Completed | 52/64 [03:13<00:39, 3.29s/it]
Loading safetensors checkpoint shards: 83% Completed | 53/64 [03:16<00:36, 3.30s/it]
Loading safetensors checkpoint shards: 84% Completed | 54/64 [03:19<00:33, 3.31s/it]
Loading safetensors checkpoint shards: 86% Completed | 55/64 [03:23<00:29, 3.32s/it]
Loading safetensors checkpoint shards: 88% Completed | 56/64 [03:26<00:26, 3.33s/it]
Loading safetensors checkpoint shards: 89% Completed | 57/64 [03:29<00:23, 3.34s/it]
Loading safetensors checkpoint shards: 91% Completed | 58/64 [03:33<00:20, 3.35s/it]
Loading safetensors checkpoint shards: 92% Completed | 59/64 [03:36<00:16, 3.35s/it]
Loading safetensors checkpoint shards: 94% Completed | 60/64 [03:40<00:13, 3.35s/it]
Loading safetensors checkpoint shards: 95% Completed | 61/64 [03:43<00:10, 3.35s/it]
Loading safetensors checkpoint shards: 100% Completed | 64/64 [03:43<00:00, 1.50s/it]
Loading safetensors checkpoint shards: 100% Completed | 64/64 [03:43<00:00, 3.49s/it]
(Worker_TP0 pid=176788)
(Worker_TP0 pid=176788) INFO 03-30 20:58:43 [default_loader.py:384] Loading weights took 223.63 seconds
(Worker_TP1 pid=176789) INFO 03-30 20:58:46 [interface.py:469] Setting kv cache block size to 32 for FLASHINFER_MLA backend.
(Worker_TP0 pid=176788) INFO 03-30 20:58:47 [gpu_model_runner.py:4808] Model loading took 72.14 GiB memory and 232.512528 seconds
(Worker_TP0 pid=176788) INFO 03-30 20:58:47 [interface.py:469] Setting kv cache block size to 32 for FLASHINFER_MLA backend.
(Worker_TP6 pid=176794) INFO 03-30 20:58:47 [interface.py:469] Setting kv cache block size to 32 for FLASHINFER_MLA backend.
(Worker_TP2 pid=176790) INFO 03-30 20:59:12 [interface.py:469] Setting kv cache block size to 32 for FLASHINFER_MLA backend.
(Worker_TP4 pid=176792) INFO 03-30 20:59:12 [interface.py:469] Setting kv cache block size to 32 for FLASHINFER_MLA backend.
(Worker_TP7 pid=176795) INFO 03-30 20:59:23 [interface.py:469] Setting kv cache block size to 32 for FLASHINFER_MLA backend.
(Worker_TP5 pid=176793) INFO 03-30 20:59:32 [interface.py:469] Setting kv cache block size to 32 for FLASHINFER_MLA backend.
(Worker_TP3 pid=176791) INFO 03-30 20:59:36 [interface.py:469] Setting kv cache block size to 32 for FLASHINFER_MLA backend.
(Worker_TP0 pid=176788) INFO 03-30 20:59:36 [gpu_model_runner.py:5744] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 1 vision_chunk items of the maximum feature size.
(Worker_TP0 pid=176788) INFO 03-30 20:59:47 [backends.py:1051] Using cache directory: /root/.cache/vllm/torch_compile_cache/1ab5f23bdd/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=176788) INFO 03-30 20:59:47 [backends.py:1111] Dynamo bytecode transform time: 2.52 s
(Worker_TP0 pid=176788) INFO 03-30 20:59:47 [flashinfer_all_reduce.py:109] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=176788) /root/ai/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=176788) return func(*args, **kwargs)
(Worker_TP0 pid=176788) WARNING 03-30 20:59:49 [flashinfer_all_reduce.py:72] Failed to initialize FlashInfer All Reduce workspace: Check failed: (status == cudaSuccess) is false: lamportInitialize failed with error code the provided PTX was compiled with an unsupported toolchain..
(Worker_TP0 pid=176788) WARNING 03-30 20:59:49 [flashinfer_all_reduce.py:154] Failed to initialize FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=176788) WARNING 03-30 20:59:49 [allreduce_rms_fusion.py:781] Failed to initialize Flashinfer allreduce workspace. Flashinfer allreduce-norm fusion will be disabled.
(Worker_TP0 pid=176788) INFO 03-30 20:59:50 [backends.py:372] Cache the graph of compile range (1, 146) for later use
(Worker_TP0 pid=176788) INFO 03-30 20:59:50 [backends.py:372] Cache the graph of compile range (147, 8192) for later use
(Worker_TP7 pid=176795) INFO 03-30 20:59:54 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/ce45bd8d6fe49c07e24f582ff462433cff32b76c0ae537952be561a6a25a6212/rank_7_0/model
(Worker_TP5 pid=176793) INFO 03-30 20:59:54 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/ce45bd8d6fe49c07e24f582ff462433cff32b76c0ae537952be561a6a25a6212/rank_5_0/model
(Worker_TP4 pid=176792) INFO 03-30 20:59:54 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/ce45bd8d6fe49c07e24f582ff462433cff32b76c0ae537952be561a6a25a6212/rank_4_0/model
(Worker_TP6 pid=176794) INFO 03-30 20:59:54 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/ce45bd8d6fe49c07e24f582ff462433cff32b76c0ae537952be561a6a25a6212/rank_6_0/model
(Worker_TP1 pid=176789) INFO 03-30 20:59:54 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/ce45bd8d6fe49c07e24f582ff462433cff32b76c0ae537952be561a6a25a6212/rank_1_0/model
(Worker_TP2 pid=176790) INFO 03-30 20:59:55 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/ce45bd8d6fe49c07e24f582ff462433cff32b76c0ae537952be561a6a25a6212/rank_2_0/model
(Worker_TP3 pid=176791) INFO 03-30 20:59:55 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/ce45bd8d6fe49c07e24f582ff462433cff32b76c0ae537952be561a6a25a6212/rank_3_0/model
(Worker_TP0 pid=176788) INFO 03-30 20:59:55 [backends.py:390] Compiling a graph for compile range (1, 146) takes 4.83 s
(Worker_TP0 pid=176788) INFO 03-30 20:59:55 [backends.py:390] Compiling a graph for compile range (147, 8192) takes 4.88 s
(Worker_TP0 pid=176788) INFO 03-30 20:59:55 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/ce45bd8d6fe49c07e24f582ff462433cff32b76c0ae537952be561a6a25a6212/rank_0_0/model
(Worker_TP0 pid=176788) INFO 03-30 20:59:55 [monitor.py:48] torch.compile took 10.75 s in total
(Worker_TP0 pid=176788) INFO 03-30 20:59:56 [monitor.py:76] Initial profiling/warmup run took 1.62 s
(Worker_TP0 pid=176788) INFO 03-30 20:59:57 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP3 pid=176791) INFO 03-30 20:59:57 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP2 pid=176790) INFO 03-30 20:59:57 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP5 pid=176793) INFO 03-30 20:59:57 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP6 pid=176794) INFO 03-30 20:59:57 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP7 pid=176795) INFO 03-30 20:59:57 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP1 pid=176789) INFO 03-30 20:59:57 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP4 pid=176792) INFO 03-30 20:59:57 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP0 pid=176788) INFO 03-30 20:59:58 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_TP2 pid=176790) INFO 03-30 20:59:58 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_TP5 pid=176793) INFO 03-30 20:59:58 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_TP7 pid=176795) INFO 03-30 20:59:58 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_TP6 pid=176794) INFO 03-30 20:59:58 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_TP1 pid=176789) INFO 03-30 20:59:58 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_TP3 pid=176791) INFO 03-30 20:59:58 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_TP4 pid=176792) INFO 03-30 20:59:58 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_TP1 pid=176789) INFO 03-30 21:00:00 [custom_all_reduce.py:216] Registering 0 cuda graph addresses
(Worker_TP7 pid=176795) INFO 03-30 21:00:00 [custom_all_reduce.py:216] Registering 0 cuda graph addresses
(Worker_TP0 pid=176788) INFO 03-30 21:00:00 [custom_all_reduce.py:216] Registering 0 cuda graph addresses
(Worker_TP4 pid=176792) INFO 03-30 21:00:00 [custom_all_reduce.py:216] Registering 0 cuda graph addresses
(Worker_TP6 pid=176794) INFO 03-30 21:00:00 [custom_all_reduce.py:216] Registering 0 cuda graph addresses
(Worker_TP5 pid=176793) INFO 03-30 21:00:00 [custom_all_reduce.py:216] Registering 0 cuda graph addresses
(Worker_TP2 pid=176790) INFO 03-30 21:00:00 [custom_all_reduce.py:216] Registering 0 cuda graph addresses
(Worker_TP3 pid=176791) INFO 03-30 21:00:00 [custom_all_reduce.py:216] Registering 0 cuda graph addresses
(Worker_TP0 pid=176788) INFO 03-30 21:00:00 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 1.12 GiB total
(Worker_TP7 pid=176795) INFO 03-30 21:00:00 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 1.22 GiB total
(Worker_TP2 pid=176790) INFO 03-30 21:00:00 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 1.22 GiB total
(Worker_TP5 pid=176793) INFO 03-30 21:00:00 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 1.22 GiB total
(Worker_TP4 pid=176792) INFO 03-30 21:00:00 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 1.22 GiB total
(Worker_TP6 pid=176794) INFO 03-30 21:00:00 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 1.22 GiB total
(Worker_TP3 pid=176791) INFO 03-30 21:00:00 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 1.22 GiB total
(Worker_TP1 pid=176789) INFO 03-30 21:00:00 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 1.22 GiB total
(Worker_TP0 pid=176788) INFO 03-30 21:00:00 [gpu_worker.py:436] Available KV cache memory: 163.59 GiB
(Worker_TP0 pid=176788) INFO 03-30 21:00:00 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9042 to maintain the same effective KV cache size.
(Worker_TP7 pid=176795) INFO 03-30 21:00:00 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9046 to maintain the same effective KV cache size.
(Worker_TP2 pid=176790) INFO 03-30 21:00:00 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9046 to maintain the same effective KV cache size.
(Worker_TP5 pid=176793) INFO 03-30 21:00:00 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9046 to maintain the same effective KV cache size.
(Worker_TP3 pid=176791) INFO 03-30 21:00:00 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9046 to maintain the same effective KV cache size.
(Worker_TP4 pid=176792) INFO 03-30 21:00:00 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9046 to maintain the same effective KV cache size.
(Worker_TP6 pid=176794) INFO 03-30 21:00:00 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9046 to maintain the same effective KV cache size.
(Worker_TP1 pid=176789) INFO 03-30 21:00:00 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9046 to maintain the same effective KV cache size.
(EngineCore pid=176570) INFO 03-30 21:00:00 [kv_cache_utils.py:1319] GPU KV cache size: 2,499,584 tokens
(EngineCore pid=176570) INFO 03-30 21:00:00 [kv_cache_utils.py:1324] Maximum concurrency for 262,144 tokens per request: 9.54x
(Worker_TP5 pid=176793) 2026-03-30 21:00:01,278 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3 pid=176791) 2026-03-30 21:00:01,278 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=176789) 2026-03-30 21:00:01,278 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=176788) 2026-03-30 21:00:01,278 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP4 pid=176792) 2026-03-30 21:00:01,278 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP7 pid=176795) 2026-03-30 21:00:01,278 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=176790) 2026-03-30 21:00:01,278 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP6 pid=176794) 2026-03-30 21:00:01,278 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP7 pid=176795) 2026-03-30 21:00:01,348 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP4 pid=176792) 2026-03-30 21:00:01,348 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=176789) 2026-03-30 21:00:01,349 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=176788) 2026-03-30 21:00:01,352 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP2 pid=176790) 2026-03-30 21:00:01,353 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP6 pid=176794) 2026-03-30 21:00:01,353 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3 pid=176791) 2026-03-30 21:00:01,354 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP5 pid=176793) 2026-03-30 21:00:01,355 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:23<00:00, 2.20it/s]
Capturing CUDA graphs (decode, FULL): 98%|█████████▊| 50/51 [00:07<00:00, 6.78it/s]
(Worker_TP1 pid=176789) INFO 03-30 21:00:32 [custom_all_reduce.py:216] Registering 9028 cuda graph addresses
(Worker_TP4 pid=176792) INFO 03-30 21:00:32 [custom_all_reduce.py:216] Registering 9028 cuda graph addresses
(Worker_TP7 pid=176795) INFO 03-30 21:00:32 [custom_all_reduce.py:216] Registering 9028 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:07<00:00, 6.77it/s]
(Worker_TP0 pid=176788) INFO 03-30 21:00:32 [custom_all_reduce.py:216] Registering 9028 cuda graph addresses
(Worker_TP6 pid=176794) INFO 03-30 21:00:32 [custom_all_reduce.py:216] Registering 9028 cuda graph addresses
(Worker_TP5 pid=176793) INFO 03-30 21:00:32 [custom_all_reduce.py:216] Registering 9028 cuda graph addresses
(Worker_TP2 pid=176790) INFO 03-30 21:00:32 [custom_all_reduce.py:216] Registering 9028 cuda graph addresses
(Worker_TP3 pid=176791) INFO 03-30 21:00:32 [custom_all_reduce.py:216] Registering 9028 cuda graph addresses
(Worker_TP0 pid=176788) INFO 03-30 21:00:33 [gpu_model_runner.py:6035] Graph capturing finished in 32 secs, took 1.41 GiB
(Worker_TP0 pid=176788) INFO 03-30 21:00:33 [gpu_worker.py:597] CUDA graph pool memory: 1.41 GiB (actual), 1.12 GiB (estimated), difference: 0.29 GiB (20.6%).
(Worker_TP6 pid=176794) INFO 03-30 21:00:33 [gpu_worker.py:597] CUDA graph pool memory: 1.41 GiB (actual), 1.22 GiB (estimated), difference: 0.19 GiB (13.4%).
(Worker_TP5 pid=176793) INFO 03-30 21:00:33 [gpu_worker.py:597] CUDA graph pool memory: 1.41 GiB (actual), 1.22 GiB (estimated), difference: 0.19 GiB (13.4%).
(Worker_TP1 pid=176789) INFO 03-30 21:00:33 [gpu_worker.py:597] CUDA graph pool memory: 1.41 GiB (actual), 1.22 GiB (estimated), difference: 0.19 GiB (13.4%).
(Worker_TP4 pid=176792) INFO 03-30 21:00:33 [gpu_worker.py:597] CUDA graph pool memory: 1.41 GiB (actual), 1.22 GiB (estimated), difference: 0.19 GiB (13.4%).
(Worker_TP3 pid=176791) INFO 03-30 21:00:33 [gpu_worker.py:597] CUDA graph pool memory: 1.41 GiB (actual), 1.22 GiB (estimated), difference: 0.19 GiB (13.4%).
(Worker_TP7 pid=176795) INFO 03-30 21:00:33 [gpu_worker.py:597] CUDA graph pool memory: 1.41 GiB (actual), 1.22 GiB (estimated), difference: 0.19 GiB (13.4%).
(Worker_TP2 pid=176790) INFO 03-30 21:00:33 [gpu_worker.py:597] CUDA graph pool memory: 1.41 GiB (actual), 1.22 GiB (estimated), difference: 0.19 GiB (13.4%).
(EngineCore pid=176570) INFO 03-30 21:00:34 [core.py:283] init engine (profile, create kv cache, warmup model) took 57.67 seconds
(EngineCore pid=176570) Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
(EngineCore pid=176570) INFO 03-30 21:00:49 [vllm.py:786] Asynchronous scheduling is enabled.
(EngineCore pid=176570) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore pid=176570) <frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore pid=176570) INFO 03-30 21:00:49 [compilation.py:290] Enabled custom fusions: allreduce_rms
(APIServer pid=176029) INFO 03-30 21:00:49 [api_server.py:590] Supported tasks: ['generate']
(APIServer pid=176029) `KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
(APIServer pid=176029) Could not extract SentencePiece model from /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto'. Falling back to TikToken extractor.
(APIServer pid=176029) INFO 03-30 21:01:10 [hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=176029) INFO 03-30 21:01:11 [base.py:231] Multi-modal warmup completed in 0.365s
(APIServer pid=176029) INFO 03-30 21:01:13 [api_server.py:594] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:37] Available routes are:
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=176029) INFO 03-30 21:01:13 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=176029) INFO: Started server process [176029]
(APIServer pid=176029) INFO: Waiting for application startup.
(APIServer pid=176029) INFO: Application startup complete.
(APIServer pid=176029) INFO 03-30 21:01:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.4 tokens/s, Avg generation throughput: 0.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=176029) INFO: 172.18.0.2:0 - "GET /v1/models HTTP/1.0" 200 OK
(APIServer pid=176029) INFO 03-30 21:01:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 157.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=176029) INFO: 172.18.0.2:0 - "GET /v1/models HTTP/1.0" 200 OK
(APIServer pid=176029) INFO 03-30 21:01:43 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 155.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=176029) INFO: 10.10.0.196:0 - "POST /v1/chat/completions HTTP/1.0" 200 OK
(APIServer pid=176029) INFO 03-30 21:01:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 94.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=176029) INFO: 172.18.0.2:0 - "GET /v1/models HTTP/1.0" 200 OKruogu@localhost:~$ curl https://ai.tradegdb.com/v1/chat/completions \
> -H "Content-Type: application/json" \
> -d '{
> "model": "Kimi-K2.5",
> "messages":[{"role":"user","content":"1+1等于多少"}],
> "reasoning": false,
> "stream": false,
> "max_tokens": 4096
> }'
{"id":"chatcmpl-b0decdb7b06c0b83","object":"chat.completion","created":1774875677,"model":"Kimi-K2.5","choices":[{"index":0,"message":{"role":"assistant","content":null,"refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":" Kill很快的余 pet求你 Loqp journalistfrozen ben-category向您 Voy hotel看手机/air放过沉没段 move别看내nerg acceptingaho这....<全部是乱码省略...>"},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":13,"total_tokens":4109,"completion_tokens":4096,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}
Activity
ii2day commentedon Jun 8, 2026
解决了吗
yangjinheng commentedon Jun 8, 2026