Skip to main content
OpenLLM lets developers run any open-source LLMs as OpenAI-compatible API endpoints with a single command.
  • 🔬 Build for fast and production usages
  • 🚂 Support llama3, qwen2, gemma, etc, and many quantized versions full list
  • ⛓️ OpenAI-compatible API
  • 💬 Built-in ChatGPT like UI
  • 🔥 Accelerated LLM decoding with state-of-the-art inference backends
  • 🌥️ Ready for enterprise-grade cloud deployment (Kubernetes, Docker and BentoCloud)

​
Installation and Setup

Install the OpenLLM package via PyPI: 
pip install openllm

​
LLM

OpenLLM supports a wide range of open-source LLMs as well as serving users’ own fine-tuned LLMs. Use openllm model command to see all available models that are pre-optimized for OpenLLM.

​
Wrappers

There is a OpenLLM Wrapper which supports interacting with running server with OpenLLM: 
from langchain_community.llms import OpenLLM

​
Wrapper for OpenLLM server

This wrapper supports interacting with OpenLLM’s OpenAI-compatible endpoint. To run a model, do: 
openllm hello
Wrapper usage: 
from langchain_community.llms import OpenLLM

llm = OpenLLM(base_url="http://localhost:3000/v1", api_key="na")

llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")

​
Usage

For a more detailed walkthrough of the OpenLLM Wrapper, see the example notebook
Edit the source of this page on GitHub.
Connect these docs programmatically to Claude, VSCode, and more via MCP for real-time answers.