Learning local LLMs for an offline voice assistant

With the recent developments in local LLMs, building an offline voice assistant that actually works started to feel like a plausible idea. I was also looking for a project that would teach me how to build LLM-based applications that can run on the edge.

This is a short summary of that journey: from knowing very little about running inference to having a quantized, optimized voice assistant running on my machine.

There will be two parts:

• In this post, I’ll go through the resources I used to learn about neural networks, LLMs, and inference.
• In the next post, I’ll walk through building and optimizing the application.

The first thing I did was pick the paper that started all of this: “Attention Is All You Need.” I used NotebookLM to go deep on it. Highly recommended. Reading that paper opened up a lot of questions, and from there I knew what to look for.

PyTorch was next. I had heard a lot about it, but never actually looked into it. The docs are great. Previous experience with numpy made the ideas easy to pick up, but the best part was the pointer to 3Blue1Brown’s neural networks playlist. That one is essential. It gave me the foundation I needed to read further and actually understand the material.

A lot of reading followed. Most of my focus was on running models on the edge. These were the most useful resources I found:

• Inference
  • What is LLM inference?
  • Training vs. inference
  • What is LLM Inference, its challenges and solutions for it - Hugging Face
  • TensorFlow.js
• Neural networks
  • Why are Transformers replacing CNNs? (Video)
• Models
  • How to Use Hugging Face: Beginner’s Guide to AI Models
  • How to Use Hugging Face Pretrained Model
  • Let’s build GPT: from scratch, in code, spelled out (Video)
  • gguf-docs
  • Reverse-engineering GGUF | Post-Training Quantization (Video)
  • Use and share pre-trained models - Kaggle
  • How LLMs survive in low precision | Quantization Fundamentals (Video)
• Hardware
  • CUDA vs ROCm vs Vulkan vs Metal
  • Mastering LLM Techniques: Training - NVIDIA
• Inference engines
  • How the vLLM inference engine works? (Video)
  • What Is Llama.cpp? The LLM Inference Engine for Local AI (Video)
  • Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales? (Video)
  • The KV Cache: Memory Usage in Transformers (Video)
  • Inference at the edge
  • ONNX Runtime for Inferencing

With that base, I shifted toward voice models, specifically Whisper. The plan was to run this on a small SoC like a Raspberry Pi, so I focused on whisper.cpp from the start. It’s easy to build and run, and the bundled tools make it straightforward to try quantization.

These were the most useful materials on that side:

• Voice models
  • Quantization for OpenAI’s Whisper Models: A Comparative Analysis
  • Speech LLMs: Models that listen and talk back (Video)
  • Voice activity detection (VAD)
  • Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
  • Google Speech Commands
• Projects
  • Moonshine Voice
  • Silero VAD: pre-trained enterprise-grade Voice Activity Detector

These resources did the most to get the theory off the ground. In the next post, I’ll cover the process of building an optimized voice assistant.