OpenDrop

Universal open-weight local AI aggregator — drop a model link, run it locally.

---

What is OpenDrop?

OpenDrop is a single piece of software where you paste any Hugging Face (or compatible) URL pointing to an open-weight model and get a fully running, hardware-optimized, locally-served AI — with optional post-training via a dropped dataset — no ML knowledge required.

Inspired by the end-to-end philosophy of ds4.c — one model, one chip, validated deeply — OpenDrop generalizes that: any open model, every hardware target, zero configuration friction.

---

Quickstart

# Install
pip install -e ".[dev]"          # development (from a clone)
pip install "git+https://github.com/fernandogarzaaa/OpenDrop"   # latest from GitHub
# NOTE: do NOT run `pip install opendrop`: that name on PyPI belongs to an
# unrelated project (an AirDrop implementation). OpenDrop is not on PyPI yet.

# Pull a model from Hugging Face
opendrop pull https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

# Search models on Hugging Face
opendrop search "llama 3.1 instruct"

# Run a single model server (default port: :11401)
opendrop run llama-3.1-8b-instruct

# Or run the multi-model OpenAI-compatible gateway on :11400
opendrop serve

# Chat via any OpenAI client (serve on :11400), e.g.:
curl http://localhost:11400/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.1-8b-instruct","messages":[{"role":"user","content":"Hello!"}]}'

# If using `opendrop run`, use port :11401 instead.

# Fine-tune with a local dataset
opendrop fine-tune llama-3.1-8b-instruct --data my_data.jsonl

# Terminal dashboard
opendrop tui

# What hardware do I have?
opendrop hardware

---

Features

Feature	Status
HuggingFace URL model pull	✅
Direct GGUF URL download	✅
Hardware auto-detection (Apple Silicon, NVIDIA, AMD, CPU)	✅
Automatic quantization selection	✅
SafeTensors → GGUF conversion	✅
OpenAI-compatible server (/v1/chat/completions)	✅
Streaming responses	✅
LoRA fine-tuning	✅
QLoRA fine-tuning (low VRAM)	✅
MLX fine-tuning (Apple Silicon)	✅
Dataset format auto-detection	✅
SQLite model registry	✅
Textual TUI dashboard	✅
Embedded Web UI	✅
Multi-model concurrent serving	✅
Disk KV cache (long context)	✅
License detection	✅
Model re-quantization	✅

---

Architecture

┌─────────────────────────────────────────────────────┐
│                    User Interfaces                   │
│         CLI        TUI        Web UI        API      │
└────────────────────────┬────────────────────────────┘
                         │
┌────────────────────────▼────────────────────────────┐
│                   OpenDrop Core                      │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────┐  │
│  │ Model        │  │ Hardware     │  │   Model   │  │
│  │ Resolver     │  │ Profiler     │  │ Registry  │  │
│  └──────┬───────┘  └──────┬───────┘  └─────┬─────┘  │
│         │                 │                │         │
│  ┌──────▼─────────────────▼────────────────▼──────┐  │
│  │              Orchestration Engine               │  │
│  │  (quant decision · download · conversion)      │  │
│  └───────────────────────┬─────────────────────────┘  │
└───────────────────────── ┼ ──────────────────────────┘
                           │
        ┌──────────────────┼─────────────────┐
        ▼                  ▼                 ▼
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│  Inference   │  │  Fine-Tune   │  │  Conversion  │
│  Backends    │  │  Pipeline    │  │  Pipeline    │
│  llama.cpp   │  │  LoRA/QLoRA  │  │  HF → GGUF   │
│  ds4.c (fast)│  │  MLX-LM      │  │  llama.cpp   │
└──────────────┘  └──────────────┘  └──────────────┘

---

Supported Model Sources

Source	Example
HuggingFace model page	https://huggingface.co/org/model
HuggingFace GGUF direct file	https://huggingface.co/org/model/resolve/main/model.Q4_K_M.gguf
Local GGUF file	/path/to/model.gguf
Local SafeTensors directory	/path/to/model/
HuggingFace model ID	org/model

---

Hardware Support

Hardware	Backend	Quantization range
Apple Silicon (M-series)	Metal via llama.cpp	Q2_K – fp16
NVIDIA GPU (CUDA)	CUDA via llama.cpp	Q2_K – fp16
AMD GPU (ROCm)	ROCm via llama.cpp	Q2_K – Q8_0
Intel Arc (Vulkan)	Vulkan via llama.cpp	Q4_K_M – Q8_0
CPU only	llama.cpp CPU	Q4_K_M – Q8_0

---

Fine-Tuning

# LoRA (GPU/CUDA)
opendrop fine-tune my-model --data train.jsonl --method lora --epochs 3

# QLoRA (low VRAM, 8 GB+)
opendrop fine-tune my-model --data train.jsonl --method qlora --epochs 3

# Apple Silicon (MLX)
opendrop fine-tune my-model --data train.jsonl --method mlx --epochs 3

# Full fine-tune (small models, large VRAM)
opendrop fine-tune my-model --data train.jsonl --method full

# Models that ship custom Python code (modeling_*.py) need an explicit opt-in
opendrop fine-tune my-model --data train.jsonl --trust-remote-code

--trust-remote-code lets the model repo's own Python code run on your machine (arbitrary code execution by whoever controls that repo). It is off by default; only enable it for repos you trust.

Supported dataset formats:

• JSONL (instruction/response, prompt/completion, conversations)
• CSV
• HuggingFace dataset ID (org/dataset)
• Alpaca format
• ShareGPT format
• Raw text (for continued pre-training)

---

CLI Reference

opendrop pull <url>              Pull and prepare a model
opendrop search <query>          Search models on Hugging Face
opendrop run <model-id>         Start inference server for a model
opendrop serve                  Serve all registered models
opendrop list                   List all models in registry
opendrop info <model-id>        Show model details
opendrop rm <model-id>          Remove a model from registry + disk
opendrop fine-tune <id>         Fine-tune with a dataset
opendrop convert <path>         Convert local SafeTensors → GGUF
opendrop tui                    Launch terminal dashboard
opendrop hardware               Show hardware profile
opendrop config                 Show/edit configuration

---

API server security

opendrop serve is meant for your own machine. Web pages you visit can still send requests to 127.0.0.1, so the server does not trust a request just because it is local:

• Origin check. Browser requests whose Origin is not a loopback origin (http(s)://localhost, 127.x.x.x, [::1], any port) get 403. CORS only allows those origins. Add more with OPENDROP_CORS_ORIGINS="https://my-app.example,https://other.example".
• API key (optional). Set OPENDROP_API_KEY=<random string> to require Authorization: Bearer <key> (or X-API-Key: <key>) on every route except /health and the web UI page. For the web UI, store the key once in the browser console: localStorage.setItem('opendrop_api_key', '<key>').
• Public binds need a key. opendrop serve --host 0.0.0.0 (or any non-loopback address) refuses to start unless OPENDROP_API_KEY is set.
• POST /v1/pull requires Content-Type: application/json.

OpenAI clients pass the key as their API key, e.g. OpenAI(base_url="http://localhost:11400/v1", api_key=os.environ["OPENDROP_API_KEY"]).

---

Configuration

Config file: ~/.config/opendrop/config.toml (Linux/macOS) or %APPDATA%\opendrop\config.toml (Windows).

[server]
host = "127.0.0.1"
port = 11400
cors = true

[storage]
models_dir = "~/.local/share/opendrop/models"
registry_db = "~/.local/share/opendrop/registry.db"

[inference]
context_size = 8192
disk_kv_cache = true           # Long-context disk-backed KV state
gpu_layers = -1                # -1 = auto (all layers to GPU)

[training]
default_method = "lora"
lora_rank = 16
lora_alpha = 32
learning_rate = 2e-4
batch_size = 4
gradient_accumulation = 4

---

Design Principles

1. End-to-end quality — not just "runnable", but validated per architecture
2. Disk KV cache as first-class — SSDs are fast enough for long context
3. Quantization transparency — always show what was chosen and why
4. Zero cloud calls during inference — fully offline after download
5. Fail loud — explain hardware bottlenecks clearly

---

Roadmap

Beta (current)

• Core pull/run/fine-tune/list/rm CLI
• llama.cpp Metal + CUDA backends
• LoRA + QLoRA training
• Textual TUI + embedded Web UI

v1.0

• Vision model support
• Embedding model server
• Model re-quantization UI
• Windows DirectML backend

v1.1

• Distributed multi-node inference
• Model sharing / export
• Community model driver plugins

---

Acknowledgements

OpenDrop is built on the shoulders of:

• llama.cpp — the inference engine that makes local AI possible
• ds4.c — the inspiration for end-to-end local model quality
• HuggingFace Hub — the open-model ecosystem
• Textual — the TUI framework

---

License

MIT — see LICENSE

---

Public Beta Readiness Docs

• Beta release gates
• Beta readiness report
• Troubleshooting
• Known limitations
• Beta launch playbook
• Changelog