Cactus is a "Cross-platform framework for deploying LLM/VLM/TTS models locally in your app". What does that mean?

As small devices become more powerful, they're now able to run Large Language Models (LLMs) on-device. This is crucial for complete privacy and control, as models using private data can run locally without connecting to external servers. Cactus is a lightweight, high-performance, and open-source framework that does exactly this, allowing you to run AI models on mobile devices.

Cactus is also cross-platform, so you can build AI applications using popular frameworks like Flutter, React Native, and Kotlin Multiplatform.

That aside, the rest of its key features are:

• Supports GGUF Models: Works with any GGUF model from Hugging Face, including Qwen, Gemma, Llama, and DeepSeek.

• Multi-Modal AI: Run various models including LLMs, VLMs, Embedding Models, and TTS (Text-to-Speech) models.

• Optimized Performance: Efficiently runs models from FP32 to 2-bit quantized versions.

• Agentic Tool Calling: Use Cactus Agents to integrate custom tools for complex, enhanced workflows.

• Native Support: Includes iOS xcframework and JNILibs for native setups.

• Tiny C++ Build: Ideal for custom hardware deployments.

• Advanced Features: Supports chat templates with Jinja2 and token streaming.

Cactus is also "local first". The implications of not owning your data are widely known, and a local-first approach aims to solve this by creating applications that are responsive and satisfying for users, while also allowing them to run offline. Cactus is especially effective in scenarios where continuous data connectivity isn't guaranteed, or when a completely offline experience is preferred. It can, however, fall back to cloud inference for longer or asynchronous tasks if needed.

The framework is built around several core APIs:

• CactusLM (for Language Models)

• CactusVLM (for Vision Language Models)

• CactusTTS (for Text-to-Speech)

• CactusAgent (for Agentic workflows)

As an example let's consider building a React Native Text Completion App.

To use the CactusLM API in a React Native application for a feature like text completion, you'd follow these steps:

1. Install the cactus-react-native package:

   npm install cactus-react-native
   

2. Initialize

3. Code
   
   

Benchmarks on a range of devices, from the iPhone 16 Pro Max to the OnePlus 13 5G and Redmi K70 Ultra, are positive and publicly available. Official demo chat applications for both Android and iOS are also available on their respective app stores. I've personally downloaded the Android app and found that it performed surprisingly well on my 3-year old Xiaomi device using the default models.

You can also load other models, but be sure your phone has enough power to handle them.

In addition to the chat app, an official Cactus Agent app showcases the CactusAgent class, which you can use to build useful agentic applications. The app can:

• Set Reminders: Create reminders for specific dates and times from your input.

• Write Draft Emails: Generate draft emails with suggested subjects.

• Add To-Dos: Create to-do items from tasks mentioned in your text.

Each tool suggestion appears as a recommended action that you can either accept or dismiss.

As far as supported LLM foundation models go, there's support for 17, all of which can be found in the Hugging Face Cactus section.

While Cactus is designed exclusively for phones and not small devices like a Raspberry Pi or IoT, its open-source nature makes it a possible candidate for experimentation.

So, what are we looking at here? Cactus is an agile, cross-platform framework for building local-first AI applications that run on limited hardware. While still in its early stages, it gives off a very positive vibe, and its roadmap is full of exciting features.

More Information

Cactus Main

Cactus Chat - Github

Cactus Chat - Android Play Store

Cactus Agents - Github

Related Articles

With MCP Docs Servers You'll Never Run Out Of Fresh Documentation 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info