部署模型 …
部署模型
mindie 是昇腾的模型运行基础环境，MindIE 镜像地址，可用于部署各类大模型。
该镜像已具备模型运行所需的基础环境，包括：CANN、FrameworkPTAdapter、MindIE 与 ATB-Models，可实现模型快速上手推理。
为方便使用，该镜像中内置了 ATB-Models 压缩包，并放置于/opt/package 之下，如需使用，可从镜像中获取。
使用前提：
- 宿主机上已经安装好固件与驱动，具体可参考安装驱动和固件。
- 宿主机上已经安装好 Docker。
使用该镜像需要申请权限
根据操作指南拉取镜像
docker pull --platform=linux/arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.0.RC2-800I-A3-py311-openeuler24.03-lts
注意，如果机器无法上网，需要通过可以访问互联网的机器拉取镜像、导出、传输到内网机器、然后导入即可。如果机器可以访问公网，直接拉取即可。
# 导出镜像
docker save -o 2.0.RC2-800I-A3-py311-openeuler24.03-lts.tar swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.0.RC2-800I-A3-py311-openeuler24.03-lts
# 导入镜像
docker load -i 2.0.RC2-800I-A3-py311-openeuler24.03-lts.tar
用 docker 启动镜像
docker run -it -d --net=host --shm-size=10g \
--privileged \
--name deepseek-r1-distill-root-test \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
-v /usr/local/sbin:/usr/local/sbin:ro \
# 映射model目录
-v /data1/apps/models:/storage/llm \
swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.0.RC2-800I-A3-py311-openeuler24.03-lts \
bash
docker exec -it deepseek-r1-distill-root-test bash
确认环境可用性
- 检查 NPU
npu-smi info
如果出现
bash: npu-smi: command not found
则需要查找主机上 npu-smi 的位置，并把 npu-smi 挂载到容器内部
find / -name npu-smi
一般来说，可能出现在 /usr/local/bin/npu-smi 路径下。
- 检查 Python 环境
import torch
import torch_npu
运行 python，执行以上命令，确认 torch 及 torch_npu 是否正常。
- 检查 mindie 环境
pip list | grep mindie
有以下输出则说明各组件正常。
mindie_llm 1.0.0
mindiebenchmark 1.0.0
mindieclient 1.0.0
mindiesd 1.0.0
mindietorch 1.0.0+torch2.1.0.abi0
- 确认模型地址正确
cd /storage/llm/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
cd /storage/llm/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
cd /storage/llm/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
确认权限
chmod -R 750 /storage/llm
开始推理
- 纯模型推理
以 Llama 3 系列模型为例，具体可参考容器中$ATB_SPEED_HOME_PATH/examples/models/llama3/README.md 中的说明
cd $ATB_SPEED_HOME_PATH
python examples/run_pa.py --model_path /path-to-weights
请修改权重路径/path-to-weights，另外$ATB_SPEED_HOME_PATH 已默认设置好，无需自行设置。
启动后会执行推理，显示默认问题 Question 和推理结果 Answer，若用户想要自定义输入问题，可使用--input_texts 参数设置，如：
python examples/run_pa.py --model_path /path-to-weights --input_texts "What is deep learning?"
- 启动服务
MindIE Service 是面向通用模型场景的推理服务化框架，通过开放、可扩展的推理服务化平台架构提供推理服务化能力，支持对接业界主流推理框架接口，满足大语言模型的高性能推理需求。请参考 MindIE Service 开发指南。
以下给出最为简单的启动方法；
修改$MIES_INSTALL_PATH/conf/config.json，具体参数含义与配置规则请参考配置参数说明，详情见后。
使用后台进程方式启动服务：
cd $MIES_INSTALL_PATH
nohup ./bin/mindieservice_daemon > output.log 2>&1 &
在标准输出流捕获到的文件中，打印如下信息说明启动成功：
Daemon start success!
$MIES_INSTALL_PATH 已默认设置好，无需自行设置。
设置模型服务启动配置
vi /usr/local/Ascend/mindie/latest/mindie-service/conf/config.json
# 或者
vi `$MIES_INSTALL_PATH/conf/config.json`
按照建议修改
{
...
"ServerConfig" :
{
...
"port" : 1040, #自定义
"managementPort" : 1041, #自定义
"metricsPort" : 1042, #自定义
...
"httpsEnabled" : false, # 取消https协议启动服务
...
},
"BackendConfig": {
...
"npuDeviceIds" : [[0,1]],
...
"ModelDeployConfig":
{
"truncation" : false,
"ModelConfig" : [
{
...
"modelName" : "deepseek-14b",
"modelWeightPath" : "/storage/llm/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
"worldSize" : 2,
...
}
]
},
}
}
配置文件示例，不可以直接用，需要根据实际硬件环境修改，可以将参考。
{
"Version": "1.1.0",
"LogConfig": {
"logLevel": "Verbose",
"logFileSize": 200,
"logFileNum": 64,
"logPath": "logs/mindservice.log"
},
"ServerConfig": {
"ipAddress": "127.0.0.1",
"managementIpAddress": "127.0.0.2",
"port": 1025,
"managementPort": 1026,
"metricsPort": 1027,
"allowAllZeroIpListening": false,
"maxLinkNum": 1000,
"httpsEnabled": false,
"fullTextEnabled": false,
"tlsCaPath": "security/ca/",
"tlsCaFile": ["ca.pem"],
"tlsCert": "security/certs/server.pem",
"tlsPk": "security/keys/server.key.pem",
"tlsPkPwd": "security/pass/key_pwd.txt",
"tlsCrlPath": "security/certs/",
"tlsCrlFiles": ["server_crl.pem"],
"managementTlsCaFile": ["management_ca.pem"],
"managementTlsCert": "security/certs/management/server.pem",
"managementTlsPk": "security/keys/management/server.key.pem",
"managementTlsPkPwd": "security/pass/management/key_pwd.txt",
"managementTlsCrlPath": "security/management/certs/",
"managementTlsCrlFiles": ["server_crl.pem"],
"kmcKsfMaster": "tools/pmt/master/ksfa",
"kmcKsfStandby": "tools/pmt/standby/ksfb",
"inferMode": "standard",
"interCommTLSEnabled": true,
"interCommPort": 1121,
"interCommTlsCaPath": "security/grpc/ca/",
"interCommTlsCaFiles": ["ca.pem"],
"interCommTlsCert": "security/grpc/certs/server.pem",
"interCommPk": "security/grpc/keys/server.key.pem",
"interCommPkPwd": "security/grpc/pass/key_pwd.txt",
"interCommTlsCrlPath": "security/grpc/certs/",
"interCommTlsCrlFiles": ["server_crl.pem"],
"openAiSupport": "vllm"
},
"BackendConfig": {
"backendName": "mindieservice_llm_engine",
"modelInstanceNumber": 4,
"npuDeviceIds": [
[0, 1],
[2, 3],
[4, 5],
[6, 7]
],
"tokenizerProcessNumber": 8,
"multiNodesInferEnabled": false,
"multiNodesInferPort": 1120,
"interNodeTLSEnabled": true,
"interNodeTlsCaPath": "security/grpc/ca/",
"interNodeTlsCaFiles": ["ca.pem"],
"interNodeTlsCert": "security/grpc/certs/server.pem",
"interNodeTlsPk": "security/grpc/keys/server.key.pem",
"interNodeTlsPkPwd": "security/grpc/pass/mindie_server_key_pwd.txt",
"interNodeTlsCrlPath": "security/grpc/certs/",
"interNodeTlsCrlFiles": ["server_crl.pem"],
"interNodeKmcKsfMaster": "tools/pmt/master/ksfa",
"interNodeKmcKsfStandby": "tools/pmt/standby/ksfb",
"ModelDeployConfig": {
"maxSeqLen": 13000,
"maxInputTokenLen": 4096,
"truncation": false,
"ModelConfig": [
{
"modelInstanceType": "Standard",
"modelName": "deepseek-32b",
"modelWeightPath": "/storage/llm/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
"worldSize": 2,
"cpuMemSize": 100,
"npuMemSize": 10,
"backendType": "atb",
"trustRemoteCode": false
}
]
},
"ScheduleConfig": {
"templateType": "Standard",
"templateName": "Standard_LLM",
"cacheBlockSize": 128,
"maxPrefillBatchSize": 50,
"maxPrefillTokens": 16384,
"prefillTimeMsPerReq": 150,
"prefillPolicyType": 0,
"decodeTimeMsPerReq": 50,
"decodePolicyType": 0,
"maxBatchSize": 200,
"maxIterTimes": 4096,
"maxPreemptCount": 0,
"supportSelectBatch": false,
"maxQueueDelayMicroseconds": 5000
}
}
}