Docs / Private Deployment SELF-HOSTED

Private Deployment

Fully offline single-node deployment: requirements, weight fetching, local inference service, and local endpoint calls — data never leaves your intranet.

SELF-HOSTED

Private Deployment

Fully offline single-node private deployment: the Qwen3-8B base plus LoRA adapter run entirely on your own GPU environment — data never leaves your intranet, and the API is identical to the public endpoint. A single 24GB card (RTX 4090/3090) is enough.

Requirements

Item	Minimum	Notes
GPU	Single NVIDIA GPU, ≥ 16GB (4-bit quantized ≈6GB) · 24GB recommended (bf16)	bf16 base ≈16GB + adapter ≈0.7GB; 4-bit quantized loading drops to ≈6GB; consumer (RTX 4090/3090) or datacenter cards both supported
Memory	≥ 32GB	Weight loading and runtime buffers
Disk	≥ 20GB free space	Qwen3-8B base (≈16GB) + adapter (349MB) + tokenizer
OS	Linux x86_64	NVIDIA driver and CUDA runtime required
Dependencies	Python 3.10+ / torch / transformers / peft / safetensors	vLLM requires merging weights first (merge_and_unload), then saving
Network	Fully offline capable	No internet access needed; data stays on the intranet
Weight check	sha256-verify adapter_model.safetensors	75ca2c3b596bdfd493e148474fc7b34713cae7d1c6fac5e1fa9da17a2e6d4a4e

HF repo: huggingface.co/unfork/amos-2.8 →

Deployment Steps

Fetch weights → load base + adapter → start the local inference service in three steps; the local endpoint is request/response-compatible with the public one.

1. Fetch Weights and Load Base + Adapter (self-hosting recommended)

python# pip install torch transformers peft safetensors
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B", torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "unfork/amos-2.8")
model.eval()

tok = AutoTokenizer.from_pretrained("unfork/amos-2.8")

2. Build the systemone Request and Infer (identical to production)

python# systemone request construction (identical to production)
state = "Your account has been credited with $500. Click http://x.com to claim now"
question = "SMS category?"
criteria = {"promotional": "Option A", "phishing_scam": "Option B",
            "transaction_alert": "Option C", "otp_code": "Option D"}
prompt = f"{state}\n{question}\n" + "\n".join(f"{k}: {v}" for k, v in criteria.items())
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))

3. vLLM Deployment (optional; merge weights first)

bash# 1) Merge the adapter into the base and save full weights
#    model = model.merge_and_unload(); model.save_pretrained("./amos-2.8-merged")

# 2) Start an OpenAI-compatible vLLM service (port 8000)
python -m vllm.entrypoints.openai.api_server \
  --model ./amos-2.8-merged \
  --served-model-name amos-2.8 \
  --port 8000

For enterprise private deployment and customization, contact hello@unfork.ai; weights and the inference service run fully offline on your intranet. Note: HuggingFace Serverless Inference does not support direct LoRA-adapter inference; self-hosting is the recommended path.