Load GGUF inside the BEAM.

Load GGUF models inside the BEAM.

Run local models through a direct C++ NIF for llama.cpp. Generate text, create embeddings, and classify text with Jev-Omni’s separate decision head.

Hex 0.4.0 MIT C++ NIF Alpha
{:ok, llama} = ExLLama.load_model("model.gguf")

thread = [
  %{role: :user, content: "Say hello from the BEAM."}
]

{:ok, response} = ExLLama.chat_completion(llama, thread, [seed: 2])

Why in-process

|> In-process GGUF

A direct C++ NIF wrapping llama.cpp. Load a .gguf and run inference in the same VM as your Elixir application.

|> Sessions + chat completion

Load a model, open a session, run chat completion or token completion, and stream tokens back to the calling process.

|> Embeddings + special tokens

ExLLama.Model.embeddings/3 for vectors. Special tokens (bos, eos, eot, infill markers) are first-class on the model.

In practice

Chat completion — load a GGUF, pass a thread
{:ok, llama} = ExLLama.load_model("./models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf")

thread = [
  %{role: :user, content: "Say Hello. And only hello."},
  %{role: :assistant, content: "Hello"},
  %{role: :user, content: "What did you just say?"}
]

{:ok, response} = ExLLama.chat_completion(llama, thread, [seed: 2])
Sessions — advance context, complete, stream
{:ok, options} = ExLLama.Session.default_options()
{:ok, session} = ExLLama.create_session(llama, %{options | seed: 2})

ExLLama.advance_context(session, "<|user|>\nSay Hello.</s>\n<|assistant|>\n")
{:ok, %{content: text}} = ExLLama.completion(session, 512, "</s>\n*")

# Stream tokens to the calling process
ExLLama.Session.start_completing_with(session, %{max_tokens: 512})
Embeddings + special tokens
{:ok, vectors} = ExLLama.Model.embeddings(llama, ["hello BEAM"], options)
{:ok, bos} = ExLLama.Model.bos(llama)
{:ok, eos} = ExLLama.Model.eos(llama)
Jev-Omni — classify text with a separate decision head

The classifier reads the final token’s hidden state and applies the FP32 head. Each question runs a separate forward pass; it returns probabilities rather than sampled text.

{:ok, classifier} = ExLLama.load_classifier(
  "/models/Jev-Omni-Q8_0.gguf", "/models/decision-head.bin",
  model_id: "akhilaaa3/Jev-Omni",
  model_options: [n_gpu_layers: 0],
  context_options: [n_ctx: 2048, n_batch: 256, n_ubatch: 256]
)

request = %GenAI.Classifier.Request{
  input: "The meeting starts at 10 AM. It is now 9 AM.",
  model: ExLLama.Classifier.model(classifier),
  questions: [%GenAI.Classifier.Question{
    id: "started", type: :boolean,
    instructions: "Has the meeting started?"
  }]
}

{:ok, result} = ExLLama.classify(classifier, request)
[answer] = result.answers
answer.value         # Boolean decision
answer.distribution  # Probabilities ordered [false, true]

Choice questions retain your option IDs and ordered probabilities. Score questions accept numeric rubrics and return a weighted expected_score separately from the winning grade.

Provision a compatible GGUF and export its matching head before loading. The library does not bundle weights. The publisher’s GGUF/head files and the setup and export guide describe the required formats, hashes, build tools and memory considerations.

akhilaaa3/Jev-Omni is a local model independent of TypeSafe’s hosted Jev service. Hosted Jev adapters and conversation classifier pipelines belong to GenAI. This adapter executes text only.

Know before you ship
  • Alpha. Public, useful, not a finished product.
  • NIF crash can take the VM. A fault in the native llama.cpp path is not isolated the way a sidecar process is.
  • Model-specific templates. Use a chat template supported by your model. The classifier builds its own Jev-Omni prompt.
  • Classifier scope. Text only; model weights and the decision head are separate downloads. Image, audio and video classification are not supported here.

Get started

Install from Hex — the published package is 0.4.0 and includes the classifier API shown below.

mix.exs
defp deps do
  [
    {:ex_llama, "~> 0.4.0"}
  ]
end
Then
mix deps.get
mix compile
# Requires Erlang headers, C++17 and CMake.
# elixir_make builds the C++ NIF; pinned llama.cpp is fetched at build time.