Load GGUF inside the BEAM.
Load GGUF models inside the BEAM.
ex_llama is a Rustler NIF around llama.cpp so Elixir can infer local GGUF models in-process — no Python sidecar.
hex 0.2.3
MIT
15 GitHub stars
820 Hex downloads
Alpha
{:ok, llama} = ExLLama.load_model("model.gguf")
thread = [
%{role: :user, content: "Say hello from the BEAM."}
]
{:ok, response} = ExLLama.chat_completion(llama, thread, %{seed: 2})Why in-process
|> In-process GGUF
A Rustler NIF wrapping the llama_cpp crate. Load a .gguf in the same VM as your Elixir app — not an HTTP sidecar, not a Python process.
|> Sessions + chat completion
Load a model, open a session, run chat completion or token completion, and stream tokens back to the calling process.
|> Embeddings + special tokens
ExLLama.Model.embeddings/3 for vectors. Special tokens (bos, eos, eot, infill markers) are first-class on the model.
In practice
Chat completion — load a GGUF, pass a thread
{:ok, llama} = ExLLama.load_model("./models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf")
thread = [
%{role: :user, content: "Say Hello. And only hello."},
%{role: :assistant, content: "Hello"},
%{role: :user, content: "What did you just say?"}
]
{:ok, response} = ExLLama.chat_completion(llama, thread, %{seed: 2})
Sessions — advance context, complete, stream
{:ok, options} = ExLLama.Session.default_options()
{:ok, session} = ExLLama.create_session(llama, %{options | seed: 2})
ExLLama.advance_context(session, "<|user|>\nSay Hello.</s>\n<|assistant|>\n")
{:ok, text} = ExLLama.completion(session, 512, "</s>\n*")
# Stream tokens to the calling process
ExLLama.Session.start_completing_with(session, %{max_tokens: 512})
Embeddings + special tokens
{:ok, vectors} = ExLLama.Model.embeddings(llama, ["hello BEAM"], options)
{:ok, bos} = ExLLama.Model.bos(llama)
{:ok, eos} = ExLLama.Model.eos(llama)Know before you ship
- Alpha. Public, useful, not a finished product.
- NIF crash can take the VM. A fault in the native llama.cpp path is not isolated the way a sidecar process is.
- Chat templates. Auto-detect still defaults to Zephyr unless you pass
meta[:template].
Get started
mix.exs
defp deps do
[
{:ex_llama, "~> 0.2.3"}
]
end
Then
mix deps.get
# Rustler builds the llama.cpp NIF on first compile