Open-Weight and Local Deployment

Outcome. You can distinguish open-weight from open-source, estimate whether a model fits available hardware, and compare local deployment with a hosted API using total cost and risk.

An open-weight model makes trained parameters available to download under a license. That does not automatically provide training data, full training code, or unrestricted rights to use, modify, and redistribute the system. Read the model card and license. The Open Source AI Definition sets a stronger standard than merely downloadable weights.

Local or self-hosted inference can keep data inside controlled infrastructure, enable offline use, support customization, and remove a vendor’s per-token invoice. It does not remove per-request cost: hardware, cloud rental, energy, memory, engineering, monitoring, upgrades, security, and idle capacity remain. Hosted APIs can offer data controls, regional options, and private networking, so compare actual requirements rather than assuming “API means public.”

Memory depends on parameter count, numerical precision, quantization, context/KV cache, batch size, and serving software. Quantization can reduce requirements with possible quality loss. Benchmark the exact task and deployment configuration; a model that fits on a laptop may still be too slow or inaccurate for the product.

Mental model. Open weights transfer control and operational responsibility from a provider to you; the right choice depends on license, data, quality, hardware, and total cost.

Evidence trail — reviewed 23 July 2026. Open Source AI Definition 1.0: https://opensource.org/ai/open-source-ai-definition. OLMo as a substantially open training-artifact example: https://arxiv.org/abs/2402.00838. GPTQ quantization: https://arxiv.org/abs/2210.17323.