On Device LLM IN Flutter

Help
, , ,
1

I want to integrate on device LLM in flutter like tinyllama for offline access , so let me know if there is any flutter package for that. i have tried llama_cpp , llama_cpp_dart , flutter_gemma but there is no sufficient documentation for that so please let me know any working code github link or something for reference

2

If there isn’t a package, check if these cpp libraries have a simple C API then you can wrap it using FFI and access it from Dart.
It’s not rocket science but you will need a bit of research how to set everything up

3

Imho it’s best to try out AI Edge Gallery to assess the performance of what you’re trying to achieve GitHub - google-ai-edge/gallery: A gallery that showcases on-device ML/GenAI use cases and allows people to try and use models locally.

4

There is a working example in the flutter_gemma package that should help you get started with on-device LLM integration.

For a practical walkthrough, check out this video tutorial: “FCAIC #6 flutter_gemma Package & Quick Look at Gemini CLI” - https://www.youtube.com/live/QVCpoWWLYZo?si=pZjGNIGXaUnWdw2M