DeepSeek deployment¶
Copyright © Quectel Wireless Solutions Co., Ltd. 2026. All rights reserved.
DeepSeek Overview¶
DeepSeek is a high-performance open-source large language model developed by a domestic team, with strong natural language understanding, code generation, and logical reasoning capabilities. This document demonstrates how to deploy the DeepSeek-R1-Distill-Qwen-1.5B distilled model on the NPU of the Quectel PI M2 (RK3576) using the RKLLM (Rockchip LLM) toolchain. The full DeepSeek-V3/R1 (671B parameters) cannot run on embedded boards; this solution deploys the official distilled small model, retaining R1’s reasoning (thinking) capability, suitable for edge-side Q&A and reasoning scenarios.
Deployment Solution¶
Item |
Detail |
|---|---|
SDK |
RKLLM SDK v1.2.1 (built into external/rknn-llm/) |
Model |
DeepSeek-R1-Distill-Qwen-1.5B, W4A16 quantized |
Inference hardware |
RK3576 NPU (6 TOPS), single core |
Board system |
Debian 13 (trixie), 4GB RAM |
Overall Architecture¶
┌────────────────────────── PC Side (x86_64) ──────────────────────────┐
│ 1. rkllm-toolkit (Python) │
│ 2. Download original model from HuggingFace │
│ 3. Generate quantization calibration data (data_quant.json) │
│ 4. Model conversion + quantization → .rkllm file │
│ 5. Cross-compile on-board demo (llm_demo) │
└──────────────────────────────────┬─────────────────────────────────┘
│ adb push
┌────────────────────────── RK3576 Board ────────────────────────────┐
│ 6. librkllmrt.so (on-board runtime, aarch64 prebuilt) │
│ 7. llm_demo loads .rkllm model → NPU inference │
└───────────────────────────────────────────────────────────────────┘
Environment Preparation¶
Hardware and Connection¶
Item |
Description |
|---|---|
Development board |
Quectel PI M2 (RK3576) |
Memory |
4GB LPDDR4X |
NPU |
6 TOPS, max frequency 950MHz |
Connection |
USB ADB (board adbd enabled, root) |
PC system |
Ubuntu 22.04, Python 3.10 |
SDK Built-in Resources (No Extra Download)¶
external/rknn-llm/
├── rkllm-toolkit/ # PC-side model conversion tool (whl included)
├── rkllm-runtime/ # On-board runtime
│ └── Linux/librkllm_api/
│ ├── aarch64/librkllmrt.so # aarch64 prebuilt runtime library
│ └── include/rkllm.h # C/C++ API header
├── examples/
│ └── DeepSeek-R1-Distill-Qwen-1.5B_Demo/
│ ├── export/ # Model conversion scripts
│ └── deploy/ # On-board demo
├── scripts/fix_freq_rk3576.sh # NPU/CPU frequency lock script
└── doc/ # RKLLM SDK documentation PDF
Detailed Setup Steps¶
Install rkllm-toolkit (PC Side)¶
# Create venv, inherit system-installed torch/numpy
python3 -m venv --system-site-packages /home/q/rkllm-venv
# Install only rkllm-toolkit itself (no dependency pull)
/home/q/rkllm-venv/bin/pip install --no-deps \
<SDK>/external/rknn-llm/rkllm-toolkit/rkllm_toolkit-1.2.1-cp310-cp310-linux_x86_64.whl
# Install missing small dependencies (use domestic mirror)
/home/q/rkllm-venv/bin/pip install -i https://mirrors.aliyun.com/pypi/simple/ \
colorlog transformers==4.51.3 sentencepiece tiktoken einops safetensors \
tqdm accelerate protobuf tabulate jinja2 jsonlines flatbuffers jsonschema datasets
# Verify
/home/q/rkllm-venv/bin/python -c "from rkllm.api import RKLLM; print('OK')"
Gotcha: The wheel metadata hardcodes versions like torch==2.3.0. A direct pip install of the whl will download the full dependency set (2GB+). Use –no-deps with the system torch to avoid this.
Download the Original Model¶
Download from the HuggingFace mirror (hf-mirror is fast domestically):
mkdir -p /home/q/deepseek-1.5b && cd /home/q/deepseek-1.5b
for f in model.safetensors config.json generation_config.json tokenizer.json tokenizer_config.json; do
wget -c "https://hf-mirror.com/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B/resolve/main/$f" -O "$f"
done
Model files: model.safetensors (~3.4GB) + 4 config files.
Generate Quantization Calibration Data¶
cd /home/q/deepseek-1.5b
/home/q/rkllm-venv/bin/python \
<SDK>/external/rknn-llm/examples/DeepSeek-R1-Distill-Qwen-1.5B_Demo/export/generate_data_quant.py \
-m /home/q/deepseek-1.5b \
-o /home/q/deepseek-1.5b/data_quant.json
Produces data_quant.json (21 samples). CPU inference takes about 20~40 minutes.
Model Conversion (Key Step)¶
Key parameters of the conversion script (must be modified for RK3576; official example defaults to RK3588):
# export_rkllm_rk3576.py (based on official export_rkllm.py, modified)
from rkllm.api import RKLLM
import os
os.environ['CUDA_VISIBLE_DEVICES']='-1' # No GPU, force CPU
modelpath = '/home/q/deepseek-1.5b'
llm = RKLLM()
ret = llm.load_huggingface(model=modelpath, device='cpu',
dtype="float32", custom_config=None, load_weight=True)
dataset = "/home/q/deepseek-1.5b/data_quant.json"
target_platform = "RK3576" # ★ Official example is RK3588, must change
quantized_dtype = "W4A16" # ★ W4A16 halves memory, faster
quantized_algorithm = "grq" # ★ grq algorithm pairs with W4A16
num_npu_core = 1 # ★ RK3576 single core
ret = llm.build(do_quantization=True, optimization_level=1,
quantized_dtype=quantized_dtype, quantized_algorithm=quantized_algorithm,
target_platform=target_platform, num_npu_core=num_npu_core,
dataset=dataset, hybrid_rate=0, max_context=4096)
ret = llm.export_rkllm(f"/home/q/deepseek-1.5b/deepseek-1.5b_W4A16_RK3576.rkllm")
Run:
/home/q/rkllm-venv/bin/python /home/q/deepseek-1.5b/export_rkllm_rk3576.py
Output: deepseek-1.5b_W4A16_RK3576.rkllm (~1.3GB). Takes about 75 minutes (CPU quantization).
Cross-Compile the On-Board Demo¶
TC=<SDK>/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu
# Copy deploy directory to workspace (avoid polluting the SDK repo)
cp -r <SDK>/external/rknn-llm/examples/DeepSeek-R1-Distill-Qwen-1.5B_Demo/deploy /home/q/llm_demo_deploy
# In CMakeLists.txt, change RKLLM_API_PATH to an absolute path:
# set(RKLLM_API_PATH "/home/q/rkllm-runtime/Linux/librkllm_api")
mkdir -p /home/q/llm_demo_deploy/build && cd /home/q/llm_demo_deploy/build
cmake .. \
-DCMAKE_SYSTEM_PROCESSOR=aarch64 -DCMAKE_SYSTEM_NAME=Linux \
-DCMAKE_C_COMPILER=$TC-gcc -DCMAKE_CXX_COMPILER=$TC-g++ \
-DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON
make -j4 && make install
Output: /home/q/llm_demo_deploy/install/demo_Linux_aarch64/, containing llm_demo + lib/librkllmrt.so.
Deploy to the Board¶
adb shell mkdir -p /root/llm_demo/lib
cd /home/q/llm_demo_deploy/install/demo_Linux_aarch64
adb push llm_demo /root/llm_demo/
adb push lib/librkllmrt.so /root/llm_demo/lib/
adb push /home/q/deepseek-1.5b/deepseek-1.5b_W4A16_RK3576.rkllm /root/llm_demo/
adb shell chmod +x /root/llm_demo/llm_demo
NPU/CPU Frequency Locking (Required Before Running the Model)¶
The official fix_freq_rk3576.sh hardcodes NPU at 1GHz, but this board’s NPU max is 950MHz — needs adaptation:
# On board (root)
echo userspace > /sys/class/devfreq/27700000.npu/governor
echo 950000000 > /sys/class/devfreq/27700000.npu/userspace/set_freq # max 950MHz
# CPU big core (policy4) locked to max 2208MHz:
echo userspace > /sys/devices/system/cpu/cpufreq/policy4/scaling_governor
echo 2208000 > /sys/devices/system/cpu/cpufreq/policy4/scaling_setspeed
Run¶
adb shell
cd /root/llm_demo
export LD_LIBRARY_PATH=./lib
sh fix_freq_rk3576_debian.sh
export RKLLM_LOG_LEVEL=1 # Print performance statistics
./llm_demo deepseek-1.5b_W4A16_RK3576.rkllm 2048 4096
# Parameters: 2048=inference batch size limit, 4096=max context length
After entering interactive mode, type 0 to test the chicken-rabbit problem, or enter a custom question.
Measured Results¶
Functional Verification¶
The chicken-rabbit problem (14 heads, 38 legs) is answered correctly: 9 chickens, 5 rabbits, with full thinking/reasoning process.
Performance Data (NPU 950MHz Locked)¶
Metric |
Measured Value |
|---|---|
Model load time |
1.54 s |
Prefill |
103.31 tokens/s (40 tokens / 0.39s) |
Generation speed |
10.29 tokens/s (459 tokens / 44.6s) |
Peak memory |
1.10 GB (plenty of headroom in 4GB) |
Quantization Scheme Comparison (RK3576 Official Benchmark, 1.5B)¶
Quantization |
Memory Usage |
Generation Speed |
|---|---|---|
W8A8 |
1665 MB |
8.51 tokens/s |
W4A16 |
932 MB |
14.45 tokens/s |
Conclusion: W4A16 is better for RK3576 (4GB RAM) — half the memory, ~70% faster.
FAQ¶
Problem |
Cause & Solution |
|---|---|
pip install rkllm-toolkit hangs |
Wheel depends on large packages like torch 2.3.0; use –no-deps with system torch |
rkllm.h: No such file or directory |
After copying the deploy directory, CMakeLists relative path breaks; change to absolute path |
aarch64-linux-gnu-g++: command not found |
Use the full SDK prebuilts toolchain |
Frequency lock to 1GHz fails |
This board’s NPU max is 950MHz; use echo 950000000 |
failed to open model on board |
Model file path wrong or not pushed |
libgomp.so.1 not found |
Install libgomp1 on the board |
Future Extensions¶
OpenAI-compatible API service: SDK includes examples/rkllm_server_demo, can start an HTTP service for remote calls
Firmware integration: Package llm_demo + model into rootfs, auto-start on boot
Larger models: 7B W4A16 ~4GB is tight for 4GB RAM, not recommended; 1.5B is the current best choice