llama.cpp CUDA GTX 1080 LXC

Running Qwen3.5-9B on GTX 1080

Deploy llama.cpp with GPU acceleration inside an LXC on Proxmox. Two deployment options: systemd service or Docker container - identical performance, choose one.

June 23, 2026 / GTX 1080 8GB / CUDA 12.4

Performance

~30 t/s

Generation Speed

~570 t/s

Prompt Processing (cold)

16K

Context Window

5.8GB

VRAM Usage

KV Cache: Repeat requests reuse cached prompt tokens. Second run of a 10K-token prompt processes only ~500 new tokens - rest served from cache instantly.

Hardware & Environment

GPU

NVIDIA GTX 1080 (8GB, Pascal / compute 6.1)

Host

Proxmox VE, Debian LXC (unprivileged)

CUDA

12.4 - CUDA 13+ dropped Pascal support

Model

Qwen3.5-9B-Q4_K_M (bartowski, ~5.8GB)

CUDA version: GTX 1080 is Pascal (compute 6.1). CUDA 13+ removed Pascal support. Use CUDA 12.4. Download on another machine and transfer if the proxy blocks nvidia.com.

Deployment: Choose One

Option A: System Daemon (systemd)

Direct binary execution via systemd service. No container overhead. The built-in llama-ui is available at http://192.168.2.50:8080/

Step 1: Prepare staging directory

mkdir -p /root/llama-docker/staging
cp /root/llama.cpp/build/bin/llama-server /root/llama-docker/staging/
cp /root/llama.cpp/build/bin/lib*.so* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libstdc++.so.6* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libgomp.so.1* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libcrypto.so.3* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libssl.so.3* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libm.so.6* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libz.so.1* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libzstd.so.1* /root/llama-docker/staging/ 2>/dev/null

Step 2: Create systemd service

cat > /etc/systemd/system/llama-server.service << 'EOF'
[Unit]
Description=llama-server (llama.cpp + CUDA)
After=network.target[Service]
Type=simple
User=root
WorkingDirectory=/root/llama-docker/staging
Environment="LD_LIBRARY_PATH=/root/llama-docker/staging:/usr/local/cuda-12.4/targets/x86_64-linux/lib"
ExecStart=/root/llama-docker/staging/llama-server 
  -m /opt/models/qwen3.5-9b-uncensored-q4_km.gguf 
  -ngl 99 
  -c 16384 
  -n 2048 
  --reasoning off 
  --host 0.0.0.0 
  --port 8080 
  -t 4
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF

Step 3: Start

systemctl daemon-reload
systemctl enable llama-server
systemctl start llama-server
systemctl status llama-server

Option B: Docker

Same performance, isolated environment. Container auto-restarts on failure. The built-in llama-ui is available at http://192.168.2.50:8080/

Step 1: Prepare staging directory

mkdir -p /root/llama-docker/staging
cp /root/llama.cpp/build/bin/llama-server /root/llama-docker/staging/
cp /root/llama.cpp/build/bin/lib*.so* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libstdc++.so.6* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libgomp.so.1* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libcrypto.so.3* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libssl.so.3* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libm.so.6* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libz.so.1* /root/llama-docker/staging/ 2>/dev/null
cp /usr/lib/x86_64-linux-gnu/libzstd.so.1* /root/llama-docker/staging/ 2>/dev/null

Step 2: Run container

Save as /root/llama-docker/docker-compose.yml and run:

cd /root/llama-docker
docker compose up -d

Or use docker run (one-liner, no file needed)

docker run -d \
  --gpus all \
  --runtime nvidia \
  --network host \
  --name llama-server \
  --restart unless-stopped \
  -v /root/llama-docker/staging:/app \
  -v /usr/local/cuda-12.4/targets/x86_64-linux/lib:/usr/local/cuda/lib64:ro \
  -v /opt/models:/models:ro \
  -e LD_LIBRARY_PATH=/app:/usr/local/cuda/lib64 \
  nvidia/cuda:12.6.0-runtime-ubuntu24.04 \
  bash -c 'export LD_LIBRARY_PATH=/app:/usr/local/cuda/lib64 && exec /app/llama-server -m /models/qwen3.5-9b-uncensored-q4_km.gguf -ngl 99 -c 16384 -n 2048 --reasoning off --host 0.0.0.0 -t 4'

Step 3: Manage

docker logs llama-server      # view logs
docker stop llama-server      # stop
docker restart llama-server   # restart
docker rm llama-server       # remove (after stop)

Alternative: docker-compose.yml

Save as /root/llama-docker/docker-compose.yml and run with docker compose up -d

services:
  llama-server:
    image: nvidia/cuda:12.6.0-runtime-ubuntu24.04
    container_name: llama-server
    runtime: nvidia
    environment:
      - LD_LIBRARY_PATH=/app:/usr/local/cuda/lib64
    volumes:
      - /root/llama-docker/staging:/app
      - /usr/local/cuda-12.4/targets/x86_64-linux/lib:/usr/local/cuda/lib64:ro
      - /opt/models:/models:ro
    ports:
      - '8080:8080'
    command: >-
      bash -c "export LD_LIBRARY_PATH=/app:/usr/local/cuda/lib64 &&
               exec /app/llama-server -m /models/qwen3.5-9b-uncensored-q4_km.gguf
               -ngl 99 -c 16384 -n 2048 --reasoning off --host 0.0.0.0 -t 4"
    restart: unless-stopped

Flag Reference

Flag	Value	Description
`-m`	model path	Path to GGUF model file
`-ngl 99`	99	Offload all layers to GPU (full GPU offload)
`-c 16384`	16384	Context window. ~10K tokens usable. Fits most PDFs.
`-n 2048`	2048	Max tokens to generate. Long answers without cut-off.
`--reasoning off`	-	Disables Qwen thinking mode. Required for this model.
`--host 0.0.0.0`	-	Bind to all interfaces. Accessible over LAN.
`--port 8080`	8080	HTTP port for REST API.
`-t 4`	4	CPU threads for preprocessing.

API: OpenAI-Compatible

curl -X POST http://192.168.2.50:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/models/qwen3.5-9b-uncensored-q4_km.gguf",
    "messages": [{"role": "user", "content": "What is 2+2?"}],
    "max_tokens": 200
  }'

Works with OpenWebUI, AnythingLLM, and any OpenAI API client. No API key required on local network.

Build from Source

git clone https://github.com/ggml-org/llama.cpp.git /root/llama.cpp
cd /root/llama.cpp
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=on -DGGML_CUDA_FORCE_MMAP=on
make -j$(nproc)

Binary output: /root/llama.cpp/build/bin/llama-server. Copy binary + all lib*.so* files to the staging directory for deployment.

Download Model

# HuggingFace (login may be required) export HF_ENDPOINT=https://hf-mirror.com huggingface-cli download bartowski/Qwen_Qwen3.5-9B-Q4_K_M_GGUF \ Qwen_Qwen3.5-9B-Q4_K_M.gguf --local /opt/models/ModelScope (no login)

wget "https://www.modelscope.cn/models/bartowski/Qwen_Qwen3.5-9B-Q4_K_M_GGUF/resolve/master/Qwen_Qwen3.5-9B-Q4_K_M.gguf" -O /opt/models/Qwen_Qwen3.5-9B-Q4_K_M.gguf

Troubleshooting

Container keeps restarting (exit 127)

Missing shared libraries. Run ldd /app/llama-server inside the container and copy missing libs to the staging directory.

Zombie GPU memory after killing process

nvidia-smi --gpu-reset

Or reboot the LXC. The ~2.1GB stale allocation is a CUDA driver context, not a memory leak.

PDF exceeds context size

With -c 16384, prompts up to ~10K tokens work. For larger PDFs, split the file or reduce layers to free VRAM for bigger context (-ngl 35 -c 32768 - slower).

File Locations

Path	Description
`/root/llama.cpp/build/bin/llama-server`	Built binary (source output)
`/root/llama-docker/staging/`	Deployment staging (binary + libs)
`/opt/models/*.gguf`	Model files
`/etc/systemd/system/llama-server.service`	Systemd unit (Option A)
`/usr/local/cuda-12.4/`	CUDA 12.4 libraries

Running Qwen3.5-9B on GTX 1080: llama.cpp + CUDA (Systemd or Docker)

Running Qwen3.5-9B on GTX 1080

Performance

Hardware & Environment

GPU

Host

CUDA

Model

Deployment: Choose One

Option A: System Daemon (systemd)

Step 1: Prepare staging directory

Step 2: Create systemd service

Step 3: Start

Option B: Docker

Step 1: Prepare staging directory

Step 2: Run container

Step 3: Manage

Alternative: docker-compose.yml

Flag Reference

API: OpenAI-Compatible

Build from Source

Download Model

ModelScope (no login)

Troubleshooting

Container keeps restarting (exit 127)

Zombie GPU memory after killing process

PDF exceeds context size

File Locations