llm | An ecosystem of Rust libraries for working with large language models

llm is a Rust library. llm has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can download it from GitHub.

llm is an ecosystem of Rust libraries for working with large language models - it's built on top of the fast, efficient GGML library for machine learning. Image by @darthdeus, using Stable Diffusion. The primary entrypoint for developers is the llm crate, which wraps llm-base and the supported model crates. Documentation for released version is available on Docs.rs. For end-users, there is a CLI application, llm-cli, which provides a convenient interface for interacting with supported models. Text generation can be done as a one-off based on a prompt, or interactively, through REPL or chat modes. The CLI can also be used to serialize (print) decoded models, quantize GGML files, or compute the perplexity of a model. It can be downloaded from the latest GitHub release or by installing it from crates.io. llm is powered by the ggml tensor library, and aims to bring the robustness and ease of use of Rust to the world of large language models. At present, inference is only on the CPU, but we hope to support GPU inference in the future through alternate backends.