On Device LLM IN Flutter

Help
, , ,
1

I want to integrate on device LLM in flutter like tinyllama for offline access , so let me know if there is any flutter package for that. i have tried llama_cpp , llama_cpp_dart , flutter_gemma but there is no sufficient documentation for that so please let me know any working code github link or something for reference

2

If there isn’t a package, check if these cpp libraries have a simple C API then you can wrap it using FFI and access it from Dart.
It’s not rocket science but you will need a bit of research how to set everything up

1 Like
3

Imho it’s best to try out AI Edge Gallery to assess the performance of what you’re trying to achieve GitHub - google-ai-edge/gallery: A gallery that showcases on-device ML/GenAI use cases and allows people to try and use models locally.

1 Like
4

There is a working example in the flutter_gemma package that should help you get started with on-device LLM integration.

For a practical walkthrough, check out this video tutorial: “FCAIC #6 flutter_gemma Package & Quick Look at Gemini CLI” - https://www.youtube.com/live/QVCpoWWLYZo?si=pZjGNIGXaUnWdw2M

1 Like
6

Yes, Cactus is working :slight_smile: thanks a lot

7

@azeem1410 How did Cactus end up working out for you?

Reaching out to get your feedback on our new release –

We’re launching a new version of the Flutter SDK, built on top of fully custom mobile inference kernels. Faster inference, better energy efficiency, more stable across devices.

Just launched in beta, you can get started on the v1 branch of cactus-flutter.

Would be super grateful for your feedback on the developer experience, performance, or API design.

Thank you in advance :slight_smile: :cactus: