ChatGPT Deployment¶

ChatGPT Overview¶

ChatGPT is a conversational large language model product from OpenAI based on the GPT (Generative Pre-trained Transformer) architecture. Representative models include GPT-3.5-turbo, GPT-4, GPT-4o, GPT-4o-mini, etc. It supports multi-turn conversation, code generation, logical reasoning, multimodal understanding, and is currently one of the most widely adopted commercial LLMs.

This document demonstrates how to deploy ChatGPT locally on the Quectel Pi M1 smart controller.

Deployment Resource Overview¶

Since ChatGPT is a closed-source cloud service, “deploying ChatGPT” essentially means running a local service gateway on the device that calls the OpenAI API, so the device and any clients on its local network can perform Q&A through this gateway.

ChatGPT Model Overview¶

Mainstream models currently available through the OpenAI API:

Model	Context Window	Description	Use Case
`gpt-4o`	128K	Latest multimodal flagship model; supports text and image input	Complex reasoning, multimodal
`gpt-4o-mini`	128K	Smaller 4o variant; fast and low-cost	General conversation (recommended default)
`gpt-4-turbo`	128K	GPT-4 long-context version	Long document analysis
`gpt-4`	8K/32K	Classic GPT-4	High-quality reasoning
`gpt-3.5-turbo`	16K	Veteran budget model	Low-cost simple tasks

For embedded-device scenarios, gpt-4o-mini is recommended as the default for the best balance of quality, speed, and cost.

Deployment Model Performance Metrics¶

Since the deployment uses API calls, no model weights are loaded locally. The performance metrics mainly depend on the network and the server side:

Metric	Typical Value	Notes
Local memory usage	~30–50 MB	Python process + dependencies
Local CPU usage	< 5%	Request forwarding only
Time to first token (TTFT)	0.5–2 s	Depends on network and server load
Generation speed	30–80 tokens/s	Generated server-side; limited by network
Max single input	128K tokens (4o series)	Limited by model context window
Concurrency	Limited by API rate limits (RPM/TPM)	Depends on account tier
Offline availability	Not available	Internet connection required

Installation¶

Dependency Installation¶

The ChatGPT gateway service depends on Python 3 and the official openai library. M1 comes preinstalled with Python 3.13.5 and pip3, so only Python dependencies need to be installed.

# 1. Verify Python and pip
python3 --version    # Expected: Python 3.13.5
pip3 --version

# 2. Upgrade pip and install the official openai library (API client) and flask (local gateway)
pip3 install --upgrade pip
pip3 install openai flask

# 3. Verify installation
python3 -c "import openai; print('openai', openai.__version__)"

If pip install fails with a certificate error, add --trusted-host pypi.org --trusted-host files.pythonhosted.org. For offline installation, run pip download openai flask -d ./wheels on a host with internet access, then adb push the wheels directory to the device and run pip3 install --no-index --find-links=./wheels openai flask.

Tool Installation¶

Install the curl tool:

apt install curl
curl --version | head -1    # Verify curl is available

If you want to run streaming visualization tests on the device, optionally install jq (for JSON processing):

apt-get install -y jq       # The device runs Debian; apt is available

Model Selection¶

Important note: ChatGPT (GPT-4 / GPT-3.5, etc.) is a closed-source commercial model from OpenAI. The model weights cannot be downloaded, nor can they be run locally. Therefore, in this section “Model Selection” refers to choosing which OpenAI API model to call, not downloading a model file.

Selection recommendations:

gpt-4o-mini(default recommendation) — Fast, low-cost, supports 128K context, suitable for most conversational scenarios on embedded devices.
gpt-4o(high-quality reasoning) — For complex logic or code tasks; higher latency and cost.
gpt-3.5-turbo(low-cost simple tasks) — Legacy budget model.

If you need to run a GPT-style model fully offline, consider an open-source alternative (such as GPT-2 117M, Qwen2.5-0.5B, etc.), loaded as GGUF files via llama.cpp on the device. These options do not require a network but lag behind ChatGPT in quality.

Launch¶

Background Deployment and Configuration¶

The following deploys an OpenAI-compatible local gateway service under /data/chatgpt/ on the device. It exposes an HTTP interface, forwards requests to the OpenAI API, and supports multi-turn conversation as well as streaming output.

Directory structure:

/data/chatgpt/
├── config.env          # Configuration file (API Key, model, port, etc.)
├── chatgpt_server.py   # Gateway service main program
└── chatgpt.service     # systemd unit file (persistent background service)

Step 1: Create the directory and configuration file

mkdir -p /data/chatgpt
cat > /data/chatgpt/config.env <<'EOF'
# ===== ChatGPT Gateway Configuration =====
# OpenAI API Key (required; replace with your own Key)
export OPENAI_API_KEY="sk-paste-your-openai-key-here"

# API base URL: official by default; change if you use a relay/proxy service
# Official: https://api.openai.com/v1
# Proxy example: https://your-proxy.example.com/v1
export OPENAI_BASE_URL="https://api.openai.com/v1"

# Default model
export CHATGPT_MODEL="gpt-4o-mini"

# Gateway bind address and port (0.0.0.0 allows LAN access)
export SERVER_HOST="0.0.0.0"
export SERVER_PORT="8000"

# System prompt (defines the assistant persona)
export SYSTEM_PROMPT="You are an intelligent assistant running on an embedded device. Please answer concisely and professionally."

# Sampling parameters
export TEMPERATURE="0.7"
export MAX_TOKENS="1024"
EOF
chmod 600 /data/chatgpt/config.env   # Protect the secret

Step 2: Write the gateway service main program

cat > /data/chatgpt/chatgpt_server.py <<'PYEOF'
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""ChatGPT Gateway Service: exposes an HTTP interface and forwards to the OpenAI API.
   Two endpoints are provided:
     /v1/chat/completions  -- OpenAI-compatible endpoint (callable by any OpenAI client)
     /ask                  -- Minimal Q&A endpoint (GET/POST, easy for embedded calls)
"""
import os, json, uuid
from flask import Flask, request, Response, jsonify
from openai import OpenAI

# ---- Load configuration ----
CFG_PATH = "/data/chatgpt/config.env"
if os.path.exists(CFG_PATH):
    for line in open(CFG_PATH, encoding="utf-8"):
        line = line.strip()
        if line and not line.startswith("#") and "=" in line and line.startswith("export "):
            k, v = line[len("export "):].split("=", 1)
            os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'"))

API_KEY     = os.environ["OPENAI_API_KEY"]
BASE_URL    = os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1")
MODEL       = os.environ.get("CHATGPT_MODEL", "gpt-4o-mini")
HOST        = os.environ.get("SERVER_HOST", "0.0.0.0")
PORT        = int(os.environ.get("SERVER_PORT", "8000"))
SYS_PROMPT  = os.environ.get("SYSTEM_PROMPT", "You are a helpful assistant.")
TEMPERATURE = float(os.environ.get("TEMPERATURE", "0.7"))
MAX_TOKENS  = int(os.environ.get("MAX_TOKENS", "1024"))

client = OpenAI(api_key=API_KEY, base_url=BASE_URL)
app = Flask(__name__)

# In-memory session store (for demonstration; use persistent storage in production)
sessions = {}

def build_messages(sid, user_msg):
    if sid not in sessions:
        sessions[sid] = [{"role": "system", "content": SYS_PROMPT}]
    sessions[sid].append({"role": "user", "content": user_msg})
    return sessions[sid]

@app.route("/ask", methods=["POST"])
def ask_simple():
    """Minimal endpoint: POST /ask  body={"q":"your question","sid":"optional session id"}"""
    data = request.get_json(force=True, silent=True) or {}
    q   = data.get("q", "").strip()
    sid = data.get("sid", "default")
    if not q:
        return jsonify({"error": "missing 'q'"}), 400
    try:
        msgs = build_messages(sid, q)
        resp = client.chat.completions.create(
            model=MODEL, messages=msgs,
            temperature=TEMPERATURE, max_tokens=MAX_TOKENS)
        ans = resp.choices[0].message.content
        sessions[sid].append({"role": "assistant", "content": ans})
        return jsonify({"answer": ans, "model": MODEL})
    except Exception as e:
        return jsonify({"error": str(e)}), 500

@app.route("/v1/chat/completions", methods=["POST"])
def chat_completions():
    """OpenAI-compatible endpoint; supports stream."""
    body = request.get_json(force=True, silent=True) or {}
    msgs = body.get("messages", [])
    stream = body.get("stream", False)
    model  = body.get("model", MODEL)
    # Auto-prepend system prompt if absent
    if not msgs or msgs[0].get("role") != "system":
        msgs = [{"role": "system", "content": SYS_PROMPT}] + msgs
    try:
        if not stream:
            r = client.chat.completions.create(
                model=model, messages=msgs,
                temperature=body.get("temperature", TEMPERATURE),
                max_tokens=body.get("max_tokens", MAX_TOKENS))
            return jsonify(r.model_dump())
        # Streaming
        def gen():
            r = client.chat.completions.create(
                model=model, messages=msgs, stream=True,
                temperature=body.get("temperature", TEMPERATURE),
                max_tokens=body.get("max_tokens", MAX_TOKENS))
            for chunk in r:
                yield f"data: {json.dumps(chunk.model_dump(), ensure_ascii=False)}\n\n"
            yield "data: [DONE]\n\n"
        return Response(gen(), mimetype="text/event-stream")
    except Exception as e:
        return jsonify({"error": {"message": str(e)}}), 500

@app.route("/v1/models", methods=["GET"])
def list_models():
    """Return the list of available models (OpenAI-compatible)."""
    try:
        return jsonify(client.models.list().model_dump())
    except Exception as e:
        return jsonify({"error": str(e)}), 500

@app.route("/health", methods=["GET"])
def health():
    return jsonify({"status":"ok","model":MODEL,"base_url":BASE_URL})

if __name__ == "__main__":
    print(f"[ChatGPT Gateway] model={MODEL} base={BASE_URL} listen={HOST}:{PORT}")
    app.run(host=HOST, port=PORT, threaded=True)
PYEOF
chmod +x /data/chatgpt/chatgpt_server.py

Step 3: Configure the systemd background service (auto-start on boot)

cat > /data/chatgpt/chatgpt.service <<'EOF'
[Unit]
Description=ChatGPT Gateway Service on SC200U
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/data/chatgpt
EnvironmentFile=/data/chatgpt/config.env
ExecStart=/usr/bin/python3 /data/chatgpt/chatgpt_server.py
Restart=on-failure
RestartSec=5
StandardOutput=append:/data/chatgpt/chatgpt.log
StandardError=append:/data/chatgpt/chatgpt.log

[Install]
WantedBy=multi-user.target
EOF

# Install and enable the service
cp /data/chatgpt/chatgpt.service /etc/systemd/system/chatgpt.service
systemctl daemon-reload
systemctl enable chatgpt.service

Starting the ChatGPT Service¶

You must configure the API Key before the first start:

# Edit the configuration and fill in your real Key
vi /data/chatgpt/config.env
# Modify OPENAI_API_KEY="sk-your-real-key"

Start and management:

# Start the service
systemctl start chatgpt.service

# Check status
systemctl status chatgpt.service

# View logs
tail -f /data/chatgpt/chatgpt.log

# Restart / Stop
systemctl restart chatgpt.service
systemctl stop chatgpt.service

Manual foreground debug (bypassing systemd):

cd /data/chatgpt
source ./config.env
python3 ./chatgpt_server.py

After a successful start, the log will show:

[ChatGPT Gateway] model=gpt-4o-mini base=https://api.openai.com/v1 listen=0.0.0.0:8000
 * Running on http://0.0.0.0:8000

Application Demo¶

Health Check¶

curl http://127.0.0.1:8000/health
# {"model":"gpt-4o-mini","status":"ok","base_url":"https://api.openai.com/v1"}

Simple Single-Turn Q&A¶

curl -s -X POST http://127.0.0.1:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"q":"Briefly introduce the SC200U chip in one sentence"}' | python3 -m json.tool

Expected output:

{
    "answer": "SC200U is a Quectel IoT communication module based on Qualcomm Snapdragon, supporting 4G all-network access and edge intelligence.",
    "model": "gpt-4o-mini"
}

Calling the Service from an Embedded Program (C / curl)¶

A C program or shell script on the device can call the service via curl:

# One-line shell call
ANSWER=$(curl -s -X POST http://127.0.0.1:8000/ask \
  -H "Content-Type: application/json" \
  -d "{\"q\":\"${QUERY}\"}" | python3 -c "import sys,json;print(json.load(sys.stdin)['answer'])")
echo "$ANSWER"

Use Cases¶

Scenario	Description
Smart customer-service terminal	Provide natural-language Q&A on SC200U smart gateways / POS devices to handle user inquiries
Industrial on-site voice assistant	Combined with ASR/TTS, engineers can verbally ask for equipment troubleshooting advice
Code / script generation	Generate shell/Python snippets on-site during operations and maintenance

FAQ¶

Model Too Large¶

ChatGPT itself cannot be run locally. However, if the scenario requires fully offline operation or zero API cost, you can deploy small open-source models on the M1 as alternatives. The minimal feasible options on this device are:

Alternative Model	Parameters	GGUF Quantization	Memory Usage	Inference Speed (SC200U)	Quality
GPT-2	124M	F16	~0.5 GB	~15–30 tok/s	Low (English short text only)
Qwen2.5-0.5B	0.5B	Q4_K_M	~0.6 GB	~8–15 tok/s	Medium
DeepSeek-R1-Distill-Qwen-1.5B	1.5B	Q4_K_M	~1.6 GB	~3–8 tok/s	Medium-high (strong reasoning)
Qwen2.5-1.5B	1.5B	Q4_K_M	~1.6 GB	~3–8 tok/s	Medium-high
Qwen2.5-3B	3B	Q4_K_M	~2.6 GB	~1–3 tok/s	High (requires swap; somewhat slow)

7B and larger models are not viable on a 3.7 GB memory device (will trigger OOM).