Load GGUF inside the BEAM.
Load GGUF models inside the BEAM.
Run local models through a direct C++ NIF for llama.cpp. Generate text, create embeddings, and classify text with Jev-Omni’s separate decision head.
{:ok, llama} = ExLLama.load_model("model.gguf")
thread = [
%{role: :user, content: "Say hello from the BEAM."}
]
{:ok, response} = ExLLama.chat_completion(llama, thread, [seed: 2])
Why in-process
|> In-process GGUF
A direct C++ NIF wrapping llama.cpp. Load a .gguf and run inference in the same VM as your Elixir application.
|> Sessions + chat completion
Load a model, open a session, run chat completion or token completion, and stream tokens back to the calling process.
|> Embeddings + special tokens
ExLLama.Model.embeddings/3 for vectors. Special tokens (bos, eos, eot, infill markers) are first-class on the model.
In practice
{:ok, llama} = ExLLama.load_model("./models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf")
thread = [
%{role: :user, content: "Say Hello. And only hello."},
%{role: :assistant, content: "Hello"},
%{role: :user, content: "What did you just say?"}
]
{:ok, response} = ExLLama.chat_completion(llama, thread, [seed: 2])
{:ok, options} = ExLLama.Session.default_options()
{:ok, session} = ExLLama.create_session(llama, %{options | seed: 2})
ExLLama.advance_context(session, "<|user|>\nSay Hello.</s>\n<|assistant|>\n")
{:ok, %{content: text}} = ExLLama.completion(session, 512, "</s>\n*")
# Stream tokens to the calling process
ExLLama.Session.start_completing_with(session, %{max_tokens: 512})
{:ok, vectors} = ExLLama.Model.embeddings(llama, ["hello BEAM"], options)
{:ok, bos} = ExLLama.Model.bos(llama)
{:ok, eos} = ExLLama.Model.eos(llama)
The classifier reads the final token’s hidden state and applies the FP32 head. Each question runs a separate forward pass; it returns probabilities rather than sampled text.
{:ok, classifier} = ExLLama.load_classifier(
"/models/Jev-Omni-Q8_0.gguf", "/models/decision-head.bin",
model_id: "akhilaaa3/Jev-Omni",
model_options: [n_gpu_layers: 0],
context_options: [n_ctx: 2048, n_batch: 256, n_ubatch: 256]
)
request = %GenAI.Classifier.Request{
input: "The meeting starts at 10 AM. It is now 9 AM.",
model: ExLLama.Classifier.model(classifier),
questions: [%GenAI.Classifier.Question{
id: "started", type: :boolean,
instructions: "Has the meeting started?"
}]
}
{:ok, result} = ExLLama.classify(classifier, request)
[answer] = result.answers
answer.value # Boolean decision
answer.distribution # Probabilities ordered [false, true]
Choice questions retain your option IDs and ordered probabilities. Score questions accept numeric rubrics and return a weighted expected_score separately from the winning grade.
Provision a compatible GGUF and export its matching head before loading. The library does not bundle weights. The publisher’s GGUF/head files and the setup and export guide describe the required formats, hashes, build tools and memory considerations.
akhilaaa3/Jev-Omni is a local model independent of TypeSafe’s hosted Jev service. Hosted Jev adapters and conversation classifier pipelines belong to GenAI. This adapter executes text only.
- Alpha. Public, useful, not a finished product.
- NIF crash can take the VM. A fault in the native llama.cpp path is not isolated the way a sidecar process is.
- Model-specific templates. Use a chat template supported by your model. The classifier builds its own Jev-Omni prompt.
- Classifier scope. Text only; model weights and the decision head are separate downloads. Image, audio and video classification are not supported here.
Get started
Install from Hex — the published package is 0.4.0 and includes the classifier API shown below.
defp deps do
[
{:ex_llama, "~> 0.4.0"}
]
end
mix deps.get
mix compile
# Requires Erlang headers, C++17 and CMake.
# elixir_make builds the C++ NIF; pinned llama.cpp is fetched at build time.