🚀 MAINTAINED FORK: inference-sh/llama-cpp-python – Active, Up-to-date, Contributors Welcome

[okaris]

Since this repo hasn’t been maintained in over 6 months and I couldn’t get in touch with the original author (@abetlen) via issues or socials, I’ve started a maintained fork:
👉 https://github.com/inference-sh/llama-cpp-python

Much respect to @abetlen — their work made Python integration with llama.cpp accessible and clean.

The fork includes:
• Updates to the latest llama.cpp (as of yesterday)
• Fixed bindings, memory layout, and runtime compatibility
• Plans to track upstream closely and add support for new models
• Upcoming improvements for ROCm, cross-platform support, and stability

Contributions welcome. Also looking for co-maintainers with Python/C++ experience and familiarity with repo lifecycle management.

[matichon-vultureprime]

@okaris

Thank you for leading the fork of this project.

The original author is doing very well.
It’s a shame the original author might have personal problems or limited bandwidth to maintain the repository.

By myself float16.cloud, I also use llama-cpp-python as my main inference engine and have extended compatibility with OpenAI APIs, such as streaming function calls with and without reasoning mode, streaming JSON output, and more.

I could submit a PR to support and maintain parts of the OpenAI-compatible interface, including reasoning options, vision input, tools calling, and JSON output.

I hope this will help you maintain the new llama-cpp-python.

[MikeLP]

While it's not available via pip yet:

dependencies = [
    "llama-cpp-python",
]

[tool.uv.sources]
llama-cpp-python = { git = "https://github.com/inference-sh/llama-cpp-python.git", branch = "main" }

[okaris]

Thanks @MikeLP !
I’m also currently using it like that. Will make some updates and publish a package as soon as i stablize a bit. Please leave a fomment with any PRs from here you would like me to merge.

Just updated with the new llama.cpp and gemma3n support along with function calling streaming

[okaris]

Thanks @matichon-vultureprime ! I just merged streaming function calls last night along with support for using gguf templates with function calling parsing. Please check that out. PRs are always welcome!

[MikeLP]

@okaris

Please leave a comment with any PRs from here you would like me to merge.

I saw somewhere PR for re-ranking support (idk how good is it). It would be great to have this functionality. I believe llama-server has some implementation as well.

updated:
Found this
#1820

[kilian-hu]

First, thank you @abetlen for all the great work on this project and making llama.cpp accessible to Python users!

And also thank you @okaris for taking the initiative with this fork. I think this is a good opportunity to make some changes to the project in order to make it easier for contributors to work with and make it generally more pythonic. It's not my intention to go into bikeshedding territory, the following is just a list of well-intended suggestions:

• Convert module constants to IntEnum inference-sh/llama-cpp-python#4 for easier API usage and better type safety
• Automatic code formatting (preferably with ruff) to reduce commit noise and reduce mental overhead when working on the project
• Suggest uv in the readme for setting up the venv
• Enable stricter type checking with e.g., pyright, basedpyright or maybe eventually ty when it's ready

I think these things would make the project more scalable, easier to maintain and more welcoming to contributors. To be honest, some of these are a personal preference of mine but I'm happy to hear what you all think about these suggestions.