OpenAI Compatible API for Open Source LLMs

We provide an OpenAI compatible API for open source LLMs with the help of ModelZ-LLM (opens in a new tab). You could easily integrate new open source LLMs (e.g. LLaMA (opens in a new tab), ChatGLM (opens in a new tab)...) into your existing applications without any code changes if you are using OpenAI python package (opens in a new tab) or langchain (opens in a new tab).

Features

OpenAI compatible API: ModelZ LLM provides an OpenAI compatible API for LLMs, which means you can use the OpenAI python SDK to interact with the model.
Self-hosted: ModelZ LLM can be easily deployed on either local or cloud-based environments.
Open source LLMs: ModelZ LLM supports open source LLMs, such as FastChat, LLaMA, and ChatGLM.
Cloud native: We provide docker images for different LLMs, which can be easily deployed on Kubernetes, or other cloud-based environments (e.g. ModelZ (opens in a new tab))

Supported Models

Bloomz 560M (CPU)Fastchat T5 3B

Use OpenAI python SDK

Then you can use the OpenAI python SDK to interact with the model:

import openai
openai.api_base="https://<deployment-key>.modelz.io"
# Your ModelZ API key
openai.api_key="mzi-xxx"
 
# create a chat completion
chat_completion = openai.ChatCompletion.create(model="any", messages=[{"role": "user", "content": "Hello world"}])

Integrate with LangChain

You could also integrate modelz-llm with langchain:

import openai
openai.api_base="https://<deployment-key>.modelz.io"
# Your ModelZ API key
openai.api_key="mzi-xxx"
 
from langchain.llms import OpenAI
 
llm = OpenAI()
 
llm.generate(prompts=["Could you please recommend some movies?"])

Supported APIs

/completions
/chat/completions
/embeddings
/engines/<any>/embeddings
/v1/completions
/v1/chat/completions
/v1/embeddings

Any Framework Templates