Gesture Recognition Algorithm¶

This guide walks through building a desktop GUI gesture-recognition project from scratch on Quectel Pi M2 Debian, using the gesture-recognition algorithm SDK.

The interface uses a three-column layout:

Left column: Algorithm modules (currently “Gesture Recognition”)
Center column: Title, live display, and controls
Right column: Device status (RAM / ROM / CPU)

Two input modes are available:

USB camera: Open a USB camera and recognize gestures in its live video stream.
Local image: Select a local image and recognize gestures offline.

Algorithm Capabilities¶

Item	Description
Algorithm type	Cascaded hand detection and gesture classification
Detection model	board6_RK3576_Ubuntu_hand_det.engine (640² input, hand bounding boxes)
Classification model	board6_RK3576_Ubuntu_gesture_cls.engine (224² input, 13 classes)
Input	Contiguous BGR memory (cv::Mat.data, rows × columns × 3)
Output	One DetRes per hand (xywh box, class label, and confidence)

Example gesture labels: like, dislike, call, fist, four, mute, ok, one, palm, three, yeah, and other.

Prerequisites¶

Obtain the Resources¶

Name	Description
gesture_RK3576 (algorithm SDK package)	Gesture-recognition package containing the public header, shared libraries, third-party runtimes, models, and sample images
gesture_project (demo project)	Example project from this guide, including source code, Makefile, and three scripts for copying dependencies, building, and running

Key directories in the gesture_RK3576 SDK package:

gesture_RK3576/
├── include/gesture.h            # Public algorithm header (the only header required for integration)
├── lib/libgesture.so            # Main algorithm library (cascaded detection and classification)
├── 3rd_party/
│   ├── modelinfer/              # Inference engine libinference.so + authorization libauthorization.so
│   ├── rknn/librknnrt.so        # RKNN hardware runtime
│   └── opencv/                  # Bundled OpenCV world library (headers + libopencv_world.so)
├── models/                      # Detection and classification .engine models
├── images/hand_input.jpg        # Sample image for acceptance testing
└── example.cpp                  # Official minimal usage example

Hardware and System Requirements¶

Item	Requirement
Main board	Quectel Pi M2
System	Debian / Ubuntu aarch64 with a desktop environment (Wayland or X11)
Peripheral	USB camera (required for live recognition)
Connection	The device can be accessed from a PC using adb shell

Software Requirements¶

The demo is built natively on Quectel Pi M2. Make sure the following software is installed:

g++ --version          # C++17 compiler
make --version
pkg-config --exists Qt5Widgets Qt5Gui Qt5Core && echo "Qt5 OK"   # Qt5 (LGPL) for display
v4l2-ctl --version     # Enumerate and troubleshoot USB camera nodes

UI implementation: The display layer uses the system-provided Qt5, so it displays directly on the desktop and supports full-screen mode. Capture, inference, and drawing use the project-bundled OpenCV world library, isolated from the system OpenCV to avoid version conflicts.

Project Structure¶

gesture_project/
├── src/
│   └── gesture_cam_demo.cpp     # Main demo (Qt UI + inference worker thread)
├── include/                     # Populated by setup: SDK public header gesture.h
├── lib/                         # Populated by setup: libgesture.so
├── third_party/                 # Populated by setup: modelinfer / rknn / opencv runtimes
├── models/                      # Populated by setup: detection and classification .engine models
├── images/                      # Populated by setup: sample image hand_input.jpg
├── Makefile                     # Build rules
├── build.sh                     # One-command build
├── setup_from_sdk.sh            # Copy dependencies from the SDK package into this project
├── run.sh                       # Configure environment variables and launch
└── README.md                    # Documentation

include/, lib/, third_party/, models/, and images/ are generated by setup_from_sdk.sh from the original SDK package.

Core Code Walkthrough¶

The demo uses the Gesture class exposed by the SDK’s public gesture.h header. Its lifecycle is: construct →set_param(set thresholds) →init(load models and authorize) →inference(infer each frame) →uinit(release resources).

The source file is src/gesture_cam_demo.cpp. The following sections explain key snippets in call order.

Initialization¶

Gesture g;
float conf = 0.25f, iou = 0.45f, det_thr = 0.0f, cls_thr = 0.20f;
g.set_param(conf, iou, det_thr, cls_thr);          // Four reference parameters: detection confidence / NMS / detection filter / classification threshold

char det_model[] = "models/board6_RK3576_Ubuntu_hand_det.engine";
char cls_model[] = "models/board6_RK3576_Ubuntu_gesture_cls.engine";
char license_buf[512] = "authorization.lic";
if (const char* license_path = std::getenv("GESTURE_LICENSE_FILE")) {
    if (license_path[0]) std::snprintf(license_buf, sizeof(license_buf), "%s", license_path);
}
// p_auth is the algorithm authorization code, supplied through GESTURE_AUTH_CODE or the command line (not bundled with the SDK).
char auth_buf[512] = {0};
const char* auth = std::getenv("GESTURE_AUTH_CODE");
char* p_auth = nullptr;
if (auth && auth[0]) { std::snprintf(auth_buf, sizeof(auth_buf), "%s", auth); p_auth = auth_buf; }
GESTURE_RES_CODE rc = g.init(det_model, cls_model, license_buf, 3, p_auth);
if (rc != GESTURE_PROCESS_OK) { /* Initialization failed: check the authorization code and model paths. */ }

Parameter descriptions:

set_param(conf, iou, det_thr, cls_thr): All four parameters are passed by reference.
- conf: Hand detection confidence threshold; default 0.25. Candidates scoring below this value are discarded during detection. Increase it to reduce false positives or decrease it to detect more hands.
- iou: IoU threshold for detection-box NMS (non-maximum suppression); default 0.45. When two boxes overlap beyond this threshold, only the higher-scoring box is retained, preventing duplicate boxes for one hand.
- det_thr: Output-filter threshold for detection results; default 0.0 (no filtering). Applies a second lower-bound check to the final det_score.
- cls_thr: Classification threshold; default 0.20. If the top-1 classification score is below this value, label is set to other (uncertain gesture).
init(p_det_model, p_cls_model, p_license_file, module, p_auth_code) loads both models and performs authorization. The five arguments are:
- p_det_model: Path to the hand detection model (hand_det.engine, 640² input).
- p_cls_model: Path to the gesture classification model (gesture_cls.engine, 224² input, 13 classes).
- p_license_file: Writable path for the algorithm license file. The project’s run.sh supplies an absolute path inside the project through GESTURE_LICENSE_FILE.
- module: Platform identifier; pass 3 for RK3576.
- p_auth_code: Algorithm authorization code (not bundled with the SDK); supply it through GESTURE_AUTH_CODE or an argument.
Return codeGESTURE_RES_CODE: 0 = GESTURE_PROCESS_OK (success); -1 = initialization failed (authorization/model path); -2 = not initialized; -3 = inference failed; -4 = invalid image/parameters. Check for GESTURE_PROCESS_OK before running inference.
get_status(opt) queries status: get_status(0) returns initialization status (0 means success); get_status(1) returns authorization status (0 means fully authorized).

Inference¶

Call inference once per frame (from either the camera or a local image). The SDK performs the full detection → crop → classification cascade and returns one result per hand.

std::vector<DetRes> results;
// Arguments: BGR data pointer, rows (height), columns (width), channels (must be 3), output result vector
GESTURE_RES_CODE rc = g.inference(frame.data, frame.rows, frame.cols, frame.channels(), results);
if (rc != GESTURE_PROCESS_OK) { /* Inference failed. */ }

for (const auto& r : results) {
    // r.x, r.y, r.width, r.height : Detection box (xywh in original-image pixels)
    // r.det_score                 : Detection confidence
    // r.class_id                  : Classification ID (0–12)
    // r.label                     : Gesture label (for example, "fist")
    // r.cls_score                 : Classification confidence
    cv::rectangle(frame, cv::Rect(r.x, r.y, r.width, r.height), cv::Scalar(0,255,0), 2);
}

Input requirements:

Input must be contiguous BGR memory (cv::Mat is contiguous by default; if isContinuous() is false, call clone() before passing .data).
channels must be 3. Convert grayscale images to BGR with cv::cvtColor first.
The SDK automatically handles BGR → RGB, letterbox resizing to 640², /255 normalization, and int8 quantization. No preprocessing is required in the caller.

Cascade performed inside the SDK:

Run the full image through hand_det to detect all hands, then apply NMS to the candidates.
Crop each hand with 10% surrounding context, resize it to 224², and normalize with ImageNet mean and standard deviation.
Run gesture_cls to get 13-class softmax scores and select top-1. If its score is below cls_thr, label the result other.

Output structureDetRes (one per hand):

Field	Meaning
x / y / width / height	Detection box in original-image pixels (top-left corner + width and height)
det_score	Detection confidence (confidence in the hand box)
class_id	Gesture class ID (0–12)
label	Gesture name (for example, fist, ok, or palm)
cls_score	Classification confidence (top-1 softmax score)

Coordinates are mapped back to the original image and can be drawn directly with cv::rectangle / cv::putText. An empty results vector means no hand was detected in the current frame.

Release¶

g.uinit();   // Release internal engine resources

Deployment and Run¶

Transfer the Files¶

The PC-side examples use Windows PowerShell. First push the directories to /tmp on the device with adb push:

# Push the directories to /tmp on Quectel Pi M2 first.
adb push gesture_RK3576 /tmp/
adb push gesture_project /tmp/

After the push completes, open a terminal in the Quectel Pi M2 desktop session and run the following as a regular user.

# $HOME and $$USER are expanded by the device shell. Do not run this in a root adb shell,
# where they would refer to the root account.
sudo mv /tmp/gesture_RK3576 /tmp/gesture_project "$HOME"/
sudo chown -R "$$USER:$$USER" "$$HOME/gesture_RK3576" "$$HOME/gesture_project"
cd "$HOME/gesture_project" && chmod +x *.sh

Install the Build Environment and Tools¶

Run on the Quectel Pi M2:

# Install make, g++, and other development tools.
sudo apt update && sudo apt install -y build-essential pkg-config qtbase5-dev

Copy Dependencies¶

setup_from_sdk.sh copies headers, shared libraries, third-party runtimes, models, and sample images from the SDK package into the corresponding project directories:

cd "$HOME/gesture_project"
sh setup_from_sdk.sh "$HOME/gesture_RK3576"   # Omit the argument to use ~/gesture_RK3576 by default.

Build¶

sh build.sh
# Output: $HOME/gesture_project/gesture_cam_demo

build.sh invokes make: it uses the system Qt5 for the UI, then links the project-local libgesture.so and bundled OpenCV world library to produce the executable.

Run¶

Run from a desktop terminal on Quectel Pi M2:

./run.sh                # Automatically detect a USB camera
./run.sh /dev/video73   # Specify a USB camera node (skips automatic detection)

We recommend launching from a terminal in the device’s desktop session. If launching through adb shell, switch to the desktop login user and make sure XDG_RUNTIME_DIR, WAYLAND_DISPLAY, or DISPLAY/XAUTHORITY matches the desktop session.

Use the UI¶

The window has three columns: algorithm modules on the left, display and controls in the center, and device status on the right.

Control	Description
[Open Camera]	Open/close the USB camera and recognize gestures in real time; closing it clears the display.
[Local Image]	Open a file picker and run offline recognition on a selected image.
[Full Screen]	Toggle between full-screen and windowed mode.
[Quit]	Close the program.
Keyboard shortcuts	Esc/Q to quit; F for full-screen mode.

The device status panel on the right refreshes once per second:

Metric	Description
RAM	Used percentage and used/total memory (MB)
ROM	Root filesystem usage percentage and used/total capacity (GB)
CPU	CPU utilization percentage

Authorization¶

The SDK uses two authorization layers, engine first and algorithm second. The integrator only needs to provide the algorithm authorization code (p_auth_code); the other components are fixed in the package:

Supply the code through GESTURE_AUTH_CODE (already exported by run.sh; replace its value with your own code).
GESTURE_LICENSE_FILE specifies the license-file path. By default, run.sh sets it to the absolute path authorization.lic inside the project. The engine license file ql_ans_license.txt is also stored in the project. These files are generated after the first successful online authorization and reused on later launches.
A valid algorithm code grants full authorization (get_status(1)=0).
If authorization returns 100010 and the SDK confirms that the authorization server is unreachable, the algorithm enters a 30-minute trial (get_status(1)=100010; expiration maps to 100002).
An empty code or an incorrect code while online causes init to fail; neither case enters the algorithm trial.

Status queries: get_status(0) returns algorithm initialization status (0 means success); get_status(1) returns authorization status.

Demo Video¶

gesture-demo

Troubleshooting¶

Issue	Possible cause	Solution
Initialization fails with rc=-1	Authorization code is empty/incorrect, or a model path is wrong	Set GESTURE_AUTH_CODE; verify that both .engine files exist under models/.
Window does not appear	Not running in a desktop session / QT_QPA_PLATFORM does not match	Run in the device’s desktop session, or let run.sh detect Wayland/xcb automatically.
Camera cannot be opened	USB camera is disconnected / specified node is wrong	Automatic mode probes for USB capture nodes. Use v4l2-ctl –list-devices to check, then run ./run.sh /dev/videoN.
Link or runtime error: .so not found	LD_LIBRARY_PATH is not set	Launch with run.sh (which configures it), or export the project lib and third_party/* paths manually.
No recognition results	No gesture in view / confidence is too low	Keep the hand clearly in frame; adjust the detection confidence and classification threshold in set_param if needed.