What if your app could chat with a real large language model that runs entirely on the phone — offline, private, and free to run? That is exactly what flutter_gemma unlocks: it loads Google's open Gemma models and runs inference directly on-device, so no text ever leaves the handset and there is no per-token cloud bill.

Why on-device AI matters in 2026

Cloud LLM APIs are powerful, but they come with three recurring costs: privacy (user data is sent to a server), latency (a network round-trip per message), and money (you pay per token, forever). For a large class of features — smart replies, summarization, offline assistants, on-device search — a small model running locally is faster, cheaper, and dramatically more private. flutter_gemma brings that capability to Flutter with GPU acceleration and multimodal support.

Installing the package

Add it to your pubspec.yaml:

dependencies:
  flutter_gemma: ^0.9.0

Then place a Gemma model file (for example a quantized .task or .bin Gemma variant) in your assets or download it at runtime. Small quantized models keep the download light while still being genuinely useful.

A minimal on-device chat

Here is the shortest path from "nothing" to "streaming answers on-device":

import 'package:flutter_gemma/flutter_gemma.dart';

// 1. Load a model that lives on the device.
final gemma = FlutterGemmaPlugin.instance;
await gemma.modelManager.installModelFromAsset('gemma-2b-it.bin');

// 2. Create an inference model + a chat session.
final model = await gemma.createModel(
  modelType: ModelType.gemmaIt,
  maxTokens: 1024,
);
final chat = await model.createChat(temperature: 0.7);

// 3. Ask a question and stream tokens back.
await chat.addQuery(Message.text(text: 'Explain Flutter in one sentence.'));
await for (final token in chat.generateChatResponseAsync()) {
  stdout.write(token.token); // append to your chat bubble
}

That is the whole loop: install a model, open a chat, stream the response. No API key, no server, no network permission required. Wire the token stream into a ListView of chat bubbles and you have a private assistant.

Live demo

See a working UI for this pattern here: fluttercook.github.io/demos/flutter_gemma, with the full runnable source on github.com/fluttercook.

Gotchas worth knowing

• Model size vs. device RAM. Pick a quantized 2B-class model for phones; larger models need more memory and warm-up time.
• First-run latency. Loading the model takes a moment — show a warm-up state and keep the model in memory between messages.
• GPU delegate. Enable GPU acceleration where available for a big speed-up over CPU inference.
• Licensing. Review the Gemma model license before shipping a bundled model.

The takeaway

flutter_gemma turns "add AI to my app" from a recurring cloud expense into a one-time, on-device capability. It is one of the most exciting tools in the Flutter AI ecosystem right now, and a perfect starting point if you care about privacy, offline support, or cost. Try the demo, drop the snippet into a project, and you will have a working local chatbot in an afternoon.

FlutterCook is publishing 100 hands-on guides to the best open-source Flutter libraries — AI-first. Follow along and grab every runnable demo at github.com/fluttercook.

Package: pub.dev/packages/flutter_gemma · Tags: Flutter, AI, LLM, on-device AI, Gemma, privacy