Local Multimodal LLM is a native Win64 Unreal Engine 5.8 C++ plugin for building private, offline AI-driven characters. It turns player speech or text into locally generated and voiced NPC dialogue without requiring a cloud inference service.

Youtube Demo:

Metahuman demo level with multiple model configurations. (Demo level assets are not included.)

GitHub Equivalent Release:

Github open source code, documentation, and coding agent integration.

User Guide:

Direct link to the github user guide, additional documentation also included inside that folder.

Optional Downloads:

Prepackaged CUDA accelerator for NVIDIA, Code only, starter models only.

The Starter Core package includes a preconfigured Gemma 4 E2B text model, Parakeet speech recognition, Pocket TTS, four starter voices, CPU and Vulkan inference backends, matching native runtimes, documentation, and applicable licensing notices.

Core features

• Local GGUF inference through llama.cpp

• Separate memory and context sessions for multiple characters using one loaded model

• Streamed subtitles and sentence-level speech generation

• Push-to-talk and optional voice-activity detection

• Native Parakeet speech-to-text and Pocket text-to-speech

• Character sheets, grounded world context, conversation memory, relationships, and controlled dynamic lore

• Typed and allow-listed Unreal function calls validated by game code

• Conversation routing using explicit targeting, facing direction, conversation state, and proximity

• Contextual speech vocabulary, rollback, configurable guard patterns, and provider-neutral speech interfaces

• Blueprint components and a documented public C++ API

• Packaging validation, checksums, model notices, and an AI-agent integration guide

Getting started

Enable Local Multimodal LLM and Unreal Engine’s Audio Capture plugin, compile the project once, and add the Local LLM, Microphone, and Text To Speech components to a persistent actor. Load the included model and create one character session for each NPC. Complete Blueprint and C++ instructions are included above in the documentation or the included starter pack.

Technical scope

• Unreal Engine 5.8.1

• Win64 development and packaged games

• CPU and Vulkan included

• Optional CUDA accelerator distributed separately

• No network connection required at runtime

• No Python runtime required

• Source code and documentation included

• Experimental vision API available with a developer-supplied matching projector

Disclosures:

• Local model output is nondeterministic. Developers remain responsible for authoritative game state, validation of mutating actions, and any additional content moderation required for their intended audience.

• The demonstration environment, MetaHumans, animations, facial-animation setup, Qwen, and NeuTTS are integration examples and are not included. Original plugin source and documentation are licensed under Apache 2.0. Bundled models, voices, and third-party runtimes retain their respective licenses and notices.

• Requires Unreal Engine’s built-in Audio Capture plugin for microphone input. It is automatically enabled by the supplied .uplugin descriptor.

• Development used an AI coding assistant under human direction and review. The package does not include AI-generated visual game assets, characters, environments, or demo scenes.

• Pocket TTS v0.1 initializes one reference voice per Text To Speech component. Use one component per distinct Pocket voice reference; dynamic single-component reference switching is planned.

Contact me through my email: [email protected] or by commenting on my YouTube page if you need assistance (I'll get back as soon as I can). Integration video coming soon if there is interest.

Created by Volley Studios.