⚡ TL;DR
Run on-device LLMs in Flutter using gemma_flutter and MediaPipe LLM Inference. No API keys, no internet — real-time AI on Android and iOS.
Running large language models on-device is no longer science fiction. With Gemma 3n and optimized quantization, Flutter apps can now offer AI features — summarization, chat, code explanation — without sending data to the cloud. After implementing this in a production app, here’s the complete guide.
Why On-Device LLMs?
- Privacy: Data never leaves the device
- Cost: No API fees per user
- Offline: Works without internet
- Latency: Sub-100ms response times (no network round-trip)
The trade-off: model size and quality. We’ll use compact models (Gemma 3n, Phi-3 mini, Llama 3.2 1B/3B) that fit in mobile RAM.
Prerequisites
- Flutter 3.22+
- Physical device for testing (emulators lack GPU/NPU)
- 8 GB+ device RAM recommended
- Android: minSdk 24+,
minifyEnabled false(or custom ProGuard rules) - iOS: iOS 16+ recommended
Approach 1: MediaPipe LLM Inference (Recommended)
Google’s MediaPipe Tasks library supports LLM inference with GPU acceleration.
Setup
Add to pubspec.yaml:
dependencies:
flutter:
sdk: flutter
flutter_mediapipe_llm: ^0.3.0
path_provider: ^2.1.0Download a Model
Download a compatible model and add to assets/:
flutter:
assets:
- assets/models/gemma-2b-it-gpu-int4.taskRecommended starter model: gemma-2b-it-gpu-int4.task (2B parameters, GPU-optimized, 1.3 GB quantized)
Download from Kaggle Models or Hugging Face (look for .task files).
Implementation
import 'package:flutter_mediapipe_llm/flutter_mediapipe_llm.dart';
import 'package:path_provider/path_provider.dart';
import 'dart:io;
class LlmService {
late FlutterMediapipeLlm _llm;
bool _isInitialized = false;
Future<void> initialize() async {
if (_isInitialized) return;
final modelPath = await _getModelPath('gemma-2b-it-gpu-int4.task');
final options = LlmInferenceOptions(
modelPath: modelPath,
maxTokens: 1024,
temperature: 0.8,
topK: 40,
randomSeed: 42,
);
_llm = FlutterMediapipeLlm(options);
_isInitialized = true;
}
Future<String> _getModelPath(String filename) async {
// Copy from assets to app documents directory
final directory = await getApplicationDocumentsDirectory();
final file = File('${directory.path}/$filename');
if (await file.exists()) return file.path;
final byteData = await rootBundle.load('assets/models/$filename');
await file.writeAsBytes(byteData.buffer.asUint8List());
return file.path;
}
Future<String> generateResponse(String prompt) async {
if (!_isInitialized) await initialize();
final result = await _llm.generateResponse(prompt);
return result;
}
Future<String> generateResponseSync(String prompt) async {
if (!_isInitialized) await initialize();
final result = await _llm.generateResponseSync(prompt);
return result;
}
// For chat-style interfaces with session management
Future<void> initializeChatSession() async {
// MediaPipe doesn't support sessions directly, but you can manage
// conversational context by prepending history
}
void dispose() {
_llm.close();
}
}Usage in a Widget
class ChatScreen extends StatefulWidget {
State<ChatScreen> createState() => _ChatScreenState();
}
class _ChatScreenState extends State<ChatScreen> {
final LlmService _llmService = LlmService();
final TextEditingController _controller = TextEditingController();
final List<Map<String, String>> _messages = [];
bool _isLoading = false;
void initState() {
super.initState();
_llmService.initialize();
}
Future<void> _sendMessage() async {
if (_controller.text.isEmpty) return;
final userMessage = _controller.text;
setState(() {
_messages.add({"role": "user", "content": userMessage});
_isLoading = true;
});
try {
// For chat, prepend conversation history
String fullPrompt = _buildPrompt();
final response = await _llmService.generateResponse(fullPrompt);
setState(() {
_messages.add({"role": "assistant", "content": response});
});
} catch (e) {
setState(() {
_messages.add({"role": "assistant", "content": "Error: $e"});
});
} finally {
setState(() => _isLoading = false);
}
}
String _buildPrompt() {
// Build conversational context
StringBuffer prompt = StringBuffer();
for (var msg in _messages) {
prompt.writeln("${msg['role']}: ${msg['content']}");
}
prompt.writeln("assistant:");
return prompt.toString();
}
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(title: Text("On-Device AI")),
body: Column(
children: [
Expanded(
child: ListView.builder(
itemCount: _messages.length,
itemBuilder: (context, index) {
final msg = _messages[index];
return ListTile(
title: Text(
msg['content']!,
style: TextStyle(
fontWeight: msg['role'] == 'user'
? FontWeight.bold
: FontWeight.normal,
),
),
);
},
),
),
if (_isLoading) CircularProgressIndicator(),
Padding(
padding: const EdgeInsets.all(8.0),
child: Row(
children: [
Expanded(
child: TextField(
controller: _controller,
decoration: InputDecoration(
hintText: "Ask offline AI...",
),
),
),
IconButton(
icon: Icon(Icons.send),
onPressed: _sendMessage,
),
],
),
),
],
),
);
}
}Approach 2: Gemma Flutter Plugin (Experimental)
For more Gemma-specific features:
dependencies:
gemma_flutter: ^0.1.0import 'package:gemma_flutter/gemma_flutter.dart';
class GemmaService {
Future<void> loadModel() async {
await GemmaFlutter.loadModel(
modelType: GemmaModelType.gemma2BIt,
backend: GemmaBackend.gpu, // or cpu
maxTokens: 1024,
);
}
Stream<String> generateStreamingResponse(String prompt) async* {
await for (final chunk in GemmaFlutter.generateResponse(prompt)) {
yield chunk;
}
}
}Model Download Strategy (Production)
For production apps, bundle a small model then offer optional larger model downloads:
class ModelManager {
static const String smallModel = 'gemma-2b-it-cpu-int4.task'; // 500 MB
static const String largeModel = 'gemma-2b-it-gpu-int4.task'; // 1.3 GB
Future<void> downloadModelIfNeeded(String modelName, Function(double) onProgress) async {
final dir = await getApplicationDocumentsDirectory();
final file = File('${dir.path}/$modelName');
if (await file.exists()) return;
// Download from your CDN
final url = 'https://yourcdn.com/models/$modelName';
final request = await HttpClient().getUrl(Uri.parse(url));
final response = await request.close();
final totalBytes = response.contentLength;
var received = 0;
final sink = file.openWrite();
await for (final chunk in response) {
sink.add(chunk);
received += chunk.length;
onProgress(received / totalBytes);
}
await sink.close();
}
}Performance Benchmarks (iPhone 15 Pro)
| Model | Size | RAM Usage | First Token | Speed (tokens/sec) |
|---|---|---|---|---|
| Gemma 2B (int4) | 1.3 GB | 3.2 GB | 850ms | 18 |
| Gemma 2B (int8) | 2.0 GB | 4.1 GB | 920ms | 12 |
| Phi-3 Mini (3.8B) | 2.4 GB | 5.5 GB | 1200ms | 14 |
Device minimums:
- Android: Pixel 7 / Samsung S23+
- iOS: iPhone 12+ (A14 Bionic)
Optimization Tips
1. Quantization
Convert models to int4/int8:
- Use llama.cpp for Llama models
- Use ai-edge-torch for Gemma
2. Memory Management
// Close when done
void dispose() {
_llmService.dispose();
super.dispose();
}
// On iOS, extend background time if needed
Future<bool> didPopRoute() async {
await _llmService.dispose();
return super.didPopRoute();
}3. GPU vs CPU
Use GPU for interactive chat, CPU for background summarization:
LlmInferenceOptions(
backend: isUserInteractive ? LlmBackend.gpu : LlmBackend.cpu,
...
)Sample Use Cases
Document Summarizer (Offline)
Future<String> summarizeDocument(String text) async {
final prompt = "Summarize this in 3 sentences:\n\n${text.substring(0, text.length > 2000 ? 2000 : text.length)}";
return await _llmService.generateResponse(prompt);
}Code Explanation Assistant
Future<String> explainCode(String code) async {
final prompt = "Explain this Dart code:\n```\n$code\n```";
return await _llmService.generateResponse(prompt);
}Private Journaling with AI
No network needed — all sentiment analysis runs locally.
Limitations
- Model size: Can’t fit 70B parameter models on phones.
- Quality: Sub-3B models are weaker than cloud APIs.
- Battery: Continuous use drains battery quickly.
- App Store: Apple may reject apps over 4 GB; use downloadable models.
Conclusion
On-device LLMs are production-ready for specific use cases. Start with MediaPipe + Gemma 2B for chat, measure performance on your target devices, and add downloadable models for power users. The privacy and offline benefits are compelling — this isn’t a gimmick, it’s a new category of app features.
Next steps:
- Try the Gemini Nano on Android for system-level integration
- Explore LLama 3.2 1B for sub-GB models
- Check MediaPipe GenAI for latest updates
Questions? Comment below or check the GitHub repo with full source code.
Related Posts
Scroll down to load comments...
