Flutter On-Device LLM: Run Gemma & Llama Locally (2026)

Running large language models on-device is no longer science fiction. With Gemma and optimized quantization, Flutter apps can now offer AI features — summarization, chat, code explanation — without sending data to the cloud. I integrated flutter_gemma into a Flutter app and tested the full flow — model install, offline inference, and streaming responses — on real devices. Here’s the complete guide.

Why On-Device LLMs?

• Privacy: Data never leaves the device
• Cost: No API fees per user
• Offline: Works without internet
• Latency: No network round-trip on the inference path

The trade-off: model size and quality. You’ll use compact models (Gemma 3, Gemma 3n, Phi-4 Mini, Qwen, Llama-family variants) that fit in mobile RAM.

What Do You Need Before You Start?

• Flutter 3.22+
• A physical device for testing (emulators lack GPU/NPU acceleration)
• 8 GB+ device RAM recommended for 2B+ parameter models
• Android: minSdkVersion 24+
• iOS: iOS 15.0+ (16.0+ if you use the MediaPipe engine package, which requires MediaPipe GenAI)

The Package: flutter_gemma

The package you’ll actually use is flutter_gemma — a well-maintained Flutter plugin (436+ likes on pub.dev) that runs Gemma, Qwen, Phi, DeepSeek, SmolLM and other open models on-device. Under the hood it wraps Google’s MediaPipe LLM Inference API and LiteRT-LM.

Note: The plugin family is being renamed — new releases now ship as flutter_edge_ai with the same API. Everything in this guide applies to both names.

Setup

Add the core package plus at least one engine package to pubspec.yaml. The core has no engine on its own — you register the engine that matches your model format:

dependencies:
  flutter:
    sdk: flutter
  flutter_gemma: ^1.8.2            # Core — always required
  flutter_gemma_mediapipe: any     # .task / .bin models (Gemma 3, Gemma 3n, Qwen 2.5, Phi-4)
  # flutter_gemma_litertlm: any    # .litertlm models (Gemma 4, Qwen3) + desktop support

Download a Model

The fastest path is installing a model from Hugging Face with one call (some google/ and litert-community/ repos are gated and need a token):

await FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,        // Gemma 3 / Gemma 3n
  fileType: ModelFileType.litertlm,
).fromNetwork(
  'https://huggingface.co/litert-community/Gemma3-1B-IT/resolve/main/Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm',
  token: 'your_hf_token', // only for gated repos
).withProgress((progress) {
  print('Downloading: $progress%');
}).install();

For small models you can also bundle the file in assets/ and install it offline-first:

flutter:
  assets:
    - assets/models/gemma3-270m-it-q8.litertlm

await FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm,
).fromAsset('assets/models/gemma3-270m-it-q8.litertlm').install();

Bundling works well for ~300 MB models (Gemma 3 270M, SmolLM 135M). Larger models noticeably bloat your app size — host them on your own CDN or Hugging Face and use fromNetwork instead.

Platform Setup (Do This First)

The plugin needs platform-specific setup before inference works:

Android — for the GPU backend, declare OpenCL native libraries in AndroidManifest.xml (above </application>):

<uses-native-library android:name="libOpenCL.so" android:required="false" />
<uses-native-library android:name="libOpenCL-car.so" android:required="false" />
<uses-native-library android:name="libOpenCL-pixel.so" android:required="false" />

iOS — set the minimum deployment target (15.0, or 16.0 with the MediaPipe engine), enable file sharing in Info.plist, and for large models add memory entitlements to Runner.entitlements:

<key>UIFileSharingEnabled</key>
<true/>

<key>com.apple.developer.kernel.increased-memory-limit</key>
<true/>

Implementation

Initialize the plugin once in main() and register the engine package you depend on:

import 'package:flutter/widgets.dart';
import 'package:flutter_gemma/flutter_gemma.dart';
import 'package:flutter_gemma_mediapipe/flutter_gemma_mediapipe.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();

  await FlutterGemma.initialize(
    inferenceEngines: const [MediaPipeEngine()],
  );

  runApp(const ChatApp());
}

Then wrap the model lifecycle in a small service class:

import 'package:flutter_gemma/flutter_gemma.dart';

class LlmService {
  InferenceModel? _model;
  Chat? _chat;

  /// One-time model install. Skipped if the model is already on disk.
  Future<void> ensureModelInstalled() async {
    final installed = await FlutterGemma.isModelInstalled('Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm');
    if (installed) return;

    await FlutterGemma.installModel(
      modelType: ModelType.gemmaIt,
      fileType: ModelFileType.litertlm,
    ).fromNetwork(
      'https://huggingface.co/litert-community/Gemma3-1B-IT/resolve/main/Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm',
    ).install();
  }

  /// Loads the model weights and opens a chat session.
  Future<void> initialize() async {
    if (_chat != null) return;
    await ensureModelInstalled();

    // maxTokens is the CONTEXT WINDOW (input + output combined),
    // not the reply length. .litertlm models need at least 1024.
    _model = await FlutterGemma.getActiveModel(
      maxTokens: 2048,
      preferredBackend: PreferredBackend.gpu,
    );
    _chat = await _model!.createChat(
      systemInstruction: 'You are a helpful offline assistant. Answer concisely.',
    );
  }

  Future<String> generateResponse(String prompt) async {
    await initialize();
    // Message.isUser defaults to false — always pass isUser: true,
    // or the model returns an empty response.
    await _chat!.addQueryChunk(Message.text(text: prompt, isUser: true));
    final response = await _chat!.generateChatResponse();
    return response is TextResponse ? response.token : '';
  }

  Stream<String> generateStreamingResponse(String prompt) async* {
    await initialize();
    await _chat!.addQueryChunk(Message.text(text: prompt, isUser: true));
    await for (final response in _chat!.generateChatResponseAsync()) {
      if (response is TextResponse) yield response.token;
    }
  }

  Future<void> close() async {
    await _model?.close();
    _model = null;
    _chat = null;
  }
}

A few things that trip people up:

• maxTokens is the context window (KV-cache budget shared by prompt and reply), not the reply length. To cap the reply, pass maxOutputTokens to createChat.
• Message.isUser defaults to false. Forgetting isUser: true is the #1 cause of empty responses.
• The chat session keeps conversation history for you — don’t manually prepend past messages to the prompt.

Usage in a Widget

import 'package:flutter/material.dart';

class ChatScreen extends StatefulWidget {
  const ChatScreen({super.key});

  @override
  State<ChatScreen> createState() => _ChatScreenState();
}

class _ChatScreenState extends State<ChatScreen> {
  final LlmService _llmService = LlmService();
  final TextEditingController _controller = TextEditingController();
  final List<String> _messages = [];
  bool _isLoading = false;

  Future<void> _sendMessage() async {
    if (_controller.text.isEmpty) return;

    final userMessage = _controller.text;
    setState(() {
      _messages.add('You: $userMessage');
      _isLoading = true;
    });

    try {
      final response = await _llmService.generateResponse(userMessage);
      setState(() {
        _messages.add('AI: $response');
      });
    } catch (e) {
      setState(() {
        _messages.add('Error: $e');
      });
    } finally {
      setState(() => _isLoading = false);
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('On-Device AI')),
      body: Column(
        children: [
          Expanded(
            child: ListView.builder(
              itemCount: _messages.length,
              itemBuilder: (context, index) {
                return ListTile(title: Text(_messages[index]));
              },
            ),
          ),
          if (_isLoading) const CircularProgressIndicator(),
          Padding(
            padding: const EdgeInsets.all(8.0),
            child: Row(
              children: [
                Expanded(
                  child: TextField(
                    controller: _controller,
                    decoration: const InputDecoration(
                      hintText: 'Ask offline AI...',
                    ),
                  ),
                ),
                IconButton(
                  icon: const Icon(Icons.send),
                  onPressed: _sendMessage,
                ),
              ],
            ),
          ),
        ],
      ),
    );
  }
}

How Do You Ship Model Downloads in Production?

For production apps, don’t embed large models in your assets — host them on your own CDN or Hugging Face and let flutter_gemma handle the download. The plugin already gives you progress tracking, retry with exponential backoff, a cancel token, and an optional Android foreground service for multi-GB downloads:

import 'package:flutter_gemma/core/model_management/cancel_token.dart';

final cancelToken = CancelToken();

final downloadFuture = FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm,
).fromNetwork(
  'https://yourcdn.com/models/gemma3-1b-it.litertlm',
).withProgress((progress) {
  setState(() => _downloadProgress = progress); // int 0-100
}).withCancelToken(cancelToken).install();

// Let the user cancel from your UI:
cancelToken.cancel('User cancelled download');

try {
  await downloadFuture;
} catch (e) {
  if (CancelToken.isCancel(e)) {
    // download was cancelled by the user
  } else {
    rethrow;
  }
}

Download failures are typed: catch DownloadException and switch on its sealed DownloadError (UnauthorizedError, NotFoundError, RateLimitedError, …) instead of parsing error strings.

Performance Benchmarks

I verified the setup runs offline end-to-end on device; treat the figures below as vendor-reported / community-reported ballparks, not my measurements — they will vary with quantization, backend (GPU vs CPU), and device.

Sources: MediaPipe LLM Inference docs, flutter_gemma on pub.dev, LiteRT community models on Hugging Face.

Device minimums:

• Android: Pixel 7 / Samsung S23 class for 2B+ models; 1B models run on mid-range devices
• iOS: iPhone 12+ (A14 Bionic)

How Can You Optimize On-Device Inference Performance?

1. Quantization

Use pre-quantized int4/int8 builds:

• Gemma models ship quantized in the litert-community Hugging Face repos
• Use llama.cpp to quantize Llama models yourself

2. Memory Management

@override
void dispose() {
  _llmService.close();
  super.dispose();
}

On iOS, large models need the memory entitlements described in the platform setup section. If you hit memory pressure, reduce maxTokens or drop to a smaller model (1B-2B parameters).

3. GPU vs CPU

Use GPU for interactive chat, CPU for background tasks (GPU is unavailable or slower in some scenarios, and on web it’s the only backend):

final model = await FlutterGemma.getActiveModel(
  maxTokens: 2048,
  preferredBackend: isUserInteractive ? PreferredBackend.gpu : PreferredBackend.cpu,
);

Note that switching the backend reloads the model weights — don’t flip it per request.

What Can You Build With On-Device LLMs?

Document Summarizer (Offline)

Future<String> summarizeDocument(String text) async {
  final clipped = text.length > 2000 ? text.substring(0, 2000) : text;
  final prompt = 'Summarize this in 3 sentences:\n\n$clipped';
  return await _llmService.generateResponse(prompt);
}

Code Explanation Assistant

Future<String> explainCode(String code) async {
  final prompt = 'Explain this Dart code:\n```\n$code\n```';
  return await _llmService.generateResponse(prompt);
}

Private Journaling with AI

No network needed — all sentiment analysis runs locally.

What Are the Limitations of On-Device LLMs?

• Model size: Can’t fit 70B parameter models on phones.
• Quality: Sub-3B models are weaker than cloud APIs.
• Battery: Continuous use drains battery quickly.
• App Store: Apple may reject apps over 4 GB; use downloadable models.
• Architecture support: the FFI/LiteRT-LM engine is arm64-only on Android — add ndk { abiFilters 'arm64-v8a' } so the Play Store doesn’t offer broken APKs to other ABIs.

Conclusion

On-device LLMs are ready for specific use cases today. Start with flutter_gemma + a 1B Gemma model for chat, measure performance on your own target devices before committing to any numbers, and add downloadable larger models for power users. The privacy and offline benefits are compelling — this isn’t a gimmick, it’s a new category of app features.

Next steps:

1. Try Gemini Nano on Android for system-level integration (flutter_gemma’s flutter_gemma_builtin_ai package also exposes it)
2. Explore LLaMA 3.2 1B for sub-GB models
3. Check the MediaPipe GenAI docs for the latest updates

Questions? Comment below or check the GitHub repo with full source code.

Related Posts

• React Native + On-Device LLM: Run Gemma and Llama Locally (2026 Guide)
• A Comprehensive Guide to Flutter Buttons: Choosing the Right One for Your App
• How to Change minSdkVersion in Flutter: 2 Simple Approaches
• All about SOLID Principles in Flutter: Examples and Tips

FAQ

Can I run an LLM offline in a Flutter app?

Yes. The flutter_gemma plugin wraps Google’s MediaPipe LLM Inference API and LiteRT-LM so models like Gemma 3, Gemma 3n, Qwen, and Phi run entirely on the device with no API keys, no internet connection, and no per-request cost. Data never leaves the phone, making it suitable for privacy-sensitive features like journaling or offline summarization.

What devices can run on-device LLMs in Flutter?

A 1B-parameter model such as Gemma 3 1B runs on mid-range Android phones and iPhone 12 or newer, while 2B+ models want flagship-class hardware like a Pixel 7 or Samsung S23 with 8 GB of RAM. Emulators lack GPU and NPU acceleration, so test on physical devices. Community-reported speeds range from roughly 10-20 tokens per second for 1B models.

Should I bundle the LLM model in my Flutter app assets or download it?

Bundle only small models of roughly 300 MB, such as Gemma 3 270M, because they inflate app size and Apple may reject binaries over 4 GB. For larger models, host the file on Hugging Face or your own CDN and download it after install — flutter_gemma provides download progress tracking, cancellation, retry, and typed error handling for this flow.

Why does my flutter_gemma chat return empty responses?

The most common cause is forgetting isUser: true when adding a message. Message.isUser defaults to false, and queries sent as non-user messages produce empty replies. Two related gotchas: maxTokens sets the total context window shared by prompt and reply, not the reply length, and the chat session already tracks history, so never prepend past messages manually.