Load GGUF inside the BEAM.

Load GGUF models inside the BEAM.

Run local models through a direct C++ NIF for llama.cpp. Generate text, create embeddings, and classify text with Jev-Omni’s separate decision head.

source 0.4.0 published Hex 0.2.3 MIT C++ NIF Alpha 
{:ok, llama} = ExLLama.load_model("model.gguf")

thread = [
  %{role: :user, content: "Say hello from the BEAM."}
]

{:ok, response} = ExLLama.chat_completion(llama, thread, [seed: 2])
Install current source Text classifier GitHub

Why in-process

|> In-process GGUF

A direct C++ NIF wrapping llama.cpp. Load a .gguf and run inference in the same VM as your Elixir application.

|> Sessions + chat completion

Load a model, open a session, run chat completion or token completion, and stream tokens back to the calling process.

|> Embeddings + special tokens

ExLLama.Model.embeddings/3 for vectors. Special tokens (bos, eos, eot, infill markers) are first-class on the model.

In practice

Chat completion — load a GGUF, pass a thread 
{:ok, llama} = ExLLama.load_model("./models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf")

thread = [
  %{role: :user, content: "Say Hello. And only hello."},
  %{role: :assistant, content: "Hello"},
  %{role: :user, content: "What did you just say?"}
]

{:ok, response} = ExLLama.chat_completion(llama, thread, [seed: 2])
Sessions — advance context, complete, stream 
{:ok, options} = ExLLama.Session.default_options()
{:ok, session} = ExLLama.create_session(llama, %{options | seed: 2})

ExLLama.advance_context(session, "<|user|>\nSay Hello.</s>\n<|assistant|>\n")
{:ok, %{content: text}} = ExLLama.completion(session, 512, "</s>\n*")

# Stream tokens to the calling process
ExLLama.Session.start_completing_with(session, %{max_tokens: 512})
Embeddings + special tokens 
{:ok, vectors} = ExLLama.Model.embeddings(llama, ["hello BEAM"], options)
{:ok, bos} = ExLLama.Model.bos(llama)
{:ok, eos} = ExLLama.Model.eos(llama)
Jev-Omni — classify text with a separate decision head

The source classifier reads the final token’s hidden state and applies the FP32 head. Each question runs a separate forward pass; it returns probabilities rather than sampled text.

{:ok, classifier} = ExLLama.load_classifier(
  "/models/Jev-Omni-Q8_0.gguf", "/models/decision-head.bin",
  model_id: "akhilaaa3/Jev-Omni",
  model_options: [n_gpu_layers: 0],
  context_options: [n_ctx: 2048, n_batch: 256, n_ubatch: 256]
)

request = %GenAI.Classifier.Request{
  input: "The meeting starts at 10 AM. It is now 9 AM.",
  model: ExLLama.Classifier.model(classifier),
  questions: [%GenAI.Classifier.Question{
    id: "started", type: :boolean,
    instructions: "Has the meeting started?"
  }]
}

{:ok, result} = ExLLama.classify(classifier, request)
[answer] = result.answers
answer.value         # Boolean decision
answer.distribution  # Probabilities ordered [false, true]

Choice questions retain your option IDs and ordered probabilities. Score questions accept numeric rubrics and return a weighted expected_score separately from the winning grade.

Provision a compatible GGUF and export its matching head before loading. The library does not bundle weights. The publisher’s GGUF/head files and the setup and export guide describe the required formats, hashes, build tools and memory considerations.

akhilaaa3/Jev-Omni is a local model independent of TypeSafe’s hosted Jev service. Hosted Jev adapters and conversation classifier pipelines belong to GenAI. This adapter executes text only.

Know before you ship
  • Alpha. Public, useful, not a finished product.
  • NIF crash can take the VM. A fault in the native llama.cpp path is not isolated the way a sidecar process is.
  • Model-specific templates. Use a chat template supported by your model. The classifier builds its own Jev-Omni prompt.
  • Classifier scope. Text only; model weights and the decision head are separate downloads. Image, audio and video classification are not supported here.

Get started with current source

These examples use source version 0.4.0 at the pinned revision below. The published Hex package is 0.2.3; it does not include the new classifier API.

mix.exs 
defp deps do
  [
    {:ex_llama,
      git: "https://github.com/noizu-labs-ml/ex_llama.git",
      ref: "d9a9d96a76a17c0e3cb8bd62177bfe9e979006f2",
      submodules: true}
  ]
end
Then 
mix deps.get
mix compile
# Requires Erlang headers, C++17 and CMake.
# elixir_make builds the C++ NIF and pinned llama.cpp.
Hex 0.2.3 package → Classifier setup → Hex 0.2.3 docs → Star on GitHub →