An On-Device AI Companion with Three-Layer Persistent Memory

AI companions are usually hosted in the cloud, because large language models (LLMs) demand more computation than a phone can provide. Cloud hosting supports rich, long-running interaction, but it carries costs. Companion and mental-health applications handle sensitive personal data, and empirical work reports real risks to their users [1]. A cloud companion also needs a constant network connection, which rules out offline use, and its operating cost grows with the amount of text processed. An alternative is to run the whole stack locally with a small language model (SLM). This keeps conversational data on the device, allows offline operation, and removes per-token cost. It also introduces two constraints. An SLM is stateless, and each call sees only its context window and retains nothing between calls. The window that fits in device memory is small, typically a few thousand tokens, because the KV cache competes with the model weights for RAM, so earlier turns fall out of it as the conversation grows A companion, unlike a single-shot assistant, has to remember the user across sessions and hold a consistent persona. Conversation history alone cannot provide this, because it is transient. What is needed is an explicit, compact memory that persists on the device and is supplied to the model on every turn. We present MyBuddy, a fully on-device AI companion that addresses this with a three-layer persistent memory comprising the Soul, Identity, and User Profile layers. The design draws on the OpenClaw framework [2], which stores an agent's durable state as files loaded at the start of a session. This paper addresses four questions about (1) how the three-layer memory works, (2) why the three layers are needed, (3) why an SLM is the right engine for it, and (4) which parts of the design hold up in practice on a phone. To answer the last of these, we evaluated the system on a consumer handset running two SLMs from the app's own model catalog on unchanged code, and we report the results in three groups. The first covers what the system guarantees in code, and both models returned the same figures. Memory composition delivered every stored fact to the model, no overwrite of a locked field was observed, and withdrawing consent removed the memory tooling from the prompt. The second covers what the model decides for itself, where the two diverged. Extraction produced no usable update on one model and ten of twelve on the other, and unprompted application after a session restart was rare, at one of twelve and none of twelve. The third covers what neither group reaches. A locked persona value held in every trial while the reply violated that persona in six of eight and two of eight In this paper, we present three contributions: • A three-layer persistent memory design for on-device companions that separates the composition path from the extraction path, and that treats persona stability and personal data as two independently controlled concerns through field locking and a consent switch. • A complete on-device companion implementing this design, supporting conversation, natural-language calendar scheduling, and an avatar, together with reusable tooling for embedding SLMs into a mobile application. • An evaluation on two models that separates what the system guarantees in code from what the model must judge for itself. The code-level results were identical across both, while extraction differed sharply between them, which places the limiting factor in the model evaluated rather than in the memory design. The evaluation also identifies a third class of failure that neither group covers, in which the companion retained a fact, reported it correctly when asked, and still acted against it.

Our work sits at the intersection of several research areas: on-device small-model inference, persistent memory and personalization for conversational agents, embodied conversational agents, and the privacy and safety of companion applications. We survey each in turn and position our three-layer memory architecture, complete on-device companion, and SLM-as-engine argument against the existing literature.

Recent systems research shows that small language models (SLMs) with 100M–5B parameters are now practical for personal devices, moving beyond data-center scaling toward efficient on-device use. Lu et al.[3] surveyed 70 SLMs and benchmarked 57 on-device, assessing speed, memory, and energy, highlighting tools like llama.cpp and MLC-LLM that enable 4-bit quantized models on phones. Tummalapalli et al.[4] found that thermal limits, not peak compute, restrict sustained performance on mobile devices, with devices like the iPhone 16 Pro and Galaxy S24 Ultra facing significant slowdowns. Additionally, Kavathekar et al.[5] showed SLMs struggle with reliable tool use, especially function calling, with low accuracy even after fine-tuning. These findings reveal SLMs are deployable but constrained by energy, thermal limits, and reliability—challenges addressed by Our design, which prioritizes memory bounds and dependable calendar tool integration rather than new inference methods.

Durable memory beyond the context window is a key research area for conversational agents. MemGPT by Packer et al.[6] uses an OS-like model where an LLM manages a tiered memory system—core in-context memory plus external archival memory—moving data via function calls, now implemented in the open-source Letta framework. Park et al.'s generative agents[7] maintain a continuous "memory stream" to generate reflections and retrieve memories dynamically, while retrieval-augmented generation by Lewis et al.[8] combines parametric and non-parametric memory for enhanced recall. Personalization research, starting with Zhang et al.'s Persona-Chat[9], shows that conditioning on a persona improves dialogue consistency. Westhäußer et al.[10] propose a framework integrating persistent memory, dynamic coordination, and evolving user profiles for personalized long-term interactions. MyBuddy draws inspiration from OpenClaw, which stores an agent’s durable state as Markdown files (SOUL for personality/values, IDENTITY for role, plus user data) loaded at session start. MyBuddy advances this by introducing a principled three-layer memory (Soul / Identity / User profile) with explicit separation of read (composition) and write (extraction) paths, a locking and consent model controlling modifications, and a bounded design tailored for mobile constraints. Unlike MemGPT and generative agents that rely on unbounded cloud storage and large LLMs, MyBuddy’s memory is carefully curated and limited to enable reliable use by a small on-device model.

The case for giving conversational agents a visible body is grounded in decades of research on embodied conversational agents (ECAs). Cassell et al.'s foundational work[11] introduced the Rea agent, showing that on-screen characters can use verbal and nonverbal behaviors to simulate face-to-face interaction. Bickmore and Picard[12] extended this to relational agents designed for long-term social-emotional bonds, demonstrating with the FitTrack agent "Laura" that remembering past interactions sustains user engagement over time. A meta-analysis by Yee, Bailenson, and Rickertsen[13] found that including a human-like face improves user experience, with presence being more important than realism. However, embodiment alone is not enough: DeVault et al.'s SimSensei[14] "Ellie" showed that fully automated embodied agents can score lower on rapport than human-controlled versions, highlighting the need for reliable behavior. MyBuddy builds on this tradition by integrating its memory system with a 3D embodied avatar, uniquely demonstrating that such embodiment can run fully on-device with a small language model, avoiding reliance on cloud infrastructure and complex animation pipelines typical of prior virtual humans.

Laestadius et al.'s grounded-theory study of Replika highlights mental health risks[1] linked to emotional dependence, while analyses of Replika’s data practices under GDPR reveal additional privacy concerns due to continuous collection of sensitive conversational data. MyBuddy addresses these issues by executing the model, memory, and avatar entirely on-device, eliminating the data egress channel that contributes to such privacy risks. Furthermore, its consent-controlled memory layers provide users with explicit and transparent control over what information is retained. This approach positions on-device small language model (SLM) inference not as a trade-off in performance but as an improvement in privacy and safety.

In this section, we present the system architecture of our proposed prototype. First, the system design is explained through all relevant modules. Subsequently, we describe implementation details of our prototype.

MyBuddy is a mobile application. SLM inference runs locally via an on-device inference engine. Its speech recognition uses an on-device Whisper model[15] and speech output uses on-device text-to-speech that drives a Unity 3D avatar with lip-sync. An Android floating overlay provides ambient presence, and the companion can take actions through tool use (e.g., Google Calendar). All language processing is local. Only the model catalog – a built-in SLM catalog providing list of available SLMs – and calendar require the network. Inference is serialized through a single-model queue, and nested calls, such as a memory extraction triggered during a turn, bypass the queue to avoid deadlock. This pipeline is the ground settings where the three-layer memory is the focus. SLMs are not hard-coded. There is a runtime catalog that lets the app download and switch among SLM families (e.g., Gemma[16], Qwen[17], Phi[18]), and a thin embedding layer lets a developer add a new on-device model with minimal configuration. The architecture is thus designed to be model agnostic, and improvements in SLMs can be adopted without touching the memory design.

Data model. Memory has three layers with fixed schemas, see Tab. 1, wrapped in a versioned container. Soul holds the companion’s mission, principles, boundaries, and response style. Identity holds its name, role, voice, and behavior rules. The User profile holds the user’s name, traits, preferences, goals, and facts. Response path. On every turn, the three layers are composed into a single labeled system prompt (SOUL/IDENTITY/USER sections) that is pinned to the top so it survives history truncation. Because memory updates change this prompt, the latest version is reasserted above a bounded replay of recent turns at the start of each turn, so the companion always acts on current memory even as older turns scroll out of the window. Empty fields receive cold-start defaults, so the companion has a coherent persona from the first turn, and the prompt also carries the avatar and tool protocol and the current time. Each layer steers a different axis of the single reply. Soul defines its substance and refusals, Identity controls its tone and mechanics, the User profile performs its personalization (Fig. 1). Update path. After the reply is sent, a separate extraction pass runs at a near-greedy low temperature and low top-K. It reads the conversation and the current memory and emits an updated memory as a fixed-schema structured output. It exists in a combined form (all three layers at once) and in per-layer forms (one layer in isolation). The output is parsed tolerantly and saved only if it differs from the current memory, so a malformed extraction cannot corrupt anything. What the model is told. The extraction prompt presents the recent conversation, the current memory, and the set of locked fields in delimited blocks, then asks for updated JSON under a fixed schema. Its rules are deliberately conservative: keep entries concise, preserve existing entries unless explicitly contradicted, replace only on explicit user intent, ignore greetings and one-off requests, treat locked fields as immutable by copying them verbatim, and never infer, guess, or fabricate. This conservatism is what keeps a persisted memory from drifting into invention. Bounded by design. Each field is designed to be capped (a few short entries), normalized (trimmed and de-duplicated), and persisted on the device with a migration path across schema versions. The memory is a compact, curated residue of the conversation.

Soul and Identity are the companion’s self: stable and lockable, changed only on an explicit request. The User profile is its memory of the user: never locked, re-derived on every update. The memory separates what should hold still from what should grow. In MyBuddy, critical safety rules—like avoiding fabricated facts or user history—are stored in the Soul layer, separate from the editable persona. This isolation ensures these rules remain secure, auditable, and protected from accidental changes, maintaining the AI companion’s reliability and ethical standards. The three memory layers function like separate controls within a single response: the Soul governs core content and refusals, the Identity manages tone and language mechanics, and the User layer handles personalization. Keeping these layers separate prevents unintended interference. This allows adjustments to substance without altering style, and vice versa. Continuity for a stateless model. The conversation is volatile working memory. The profile is curated long-term memory, persisted and reinjected so the model carries the user forward across sessions. This is what makes a stateless SLM behave like a companion rather than a stranger each session. The per-layer extraction prompts explicitly forbid writing other layers’ fields. Correct isolated updates therefore require a clean separation. A mis-filed value would be missed by one layer’s update and overwritten by another’s. The distinction is enforced in code on the write path, not merely conceptual.

Only a small model fits on a smart phone, and that is exactly what makes the companion private, offline, and free of per-token cost. A large model cannot run locally, so cloud inference and its privacy cost would be unavoidable. The SLM is the precondition for the whole design, not a compromise within it. The window that fits alongside the model weights on a phone is precisely what forces a compact, structured, persisted memory instead of dumping history into the prompt. The constraint that makes naive memory impossible is the same constraint that motivates the three-layer architecture. Model and memory are co-designed. SLMs are unreliable at structured output [5], so the system compensates with near-greedy low-temperature and low-topK extraction, explicit anti-hallucination rules (“must not infer or guess”), locked fields enforced at the prompt, at the tool boundary, and after parsing, single-function-call enforcement with a recursion cap, dual function-call parsing (native plus a JSON-in-text fallback) with tolerant extraction and response cleaning; all over a serialized single-model pipeline. Thermal and energy limits further favor keeping the model small. Because inference, transcription, synthesis, and memory are all local, no conversational data leaves the phone. Only two touchpoints use the network. The network connection is required during fetching the model catalog and the user’s optional calendar authorization. The privacy properties therefore follow from the architecture, not from a policy promise.

We have developed a functional prototype that supports natural language conversations, manages calendar reminders and appointments from simple requests, and features an avatar with lip-sync and visual overlays. Additionally, it accommodates multiple small language model (SLM) families via a model catalog. This paper presents initial on-device performance metrics and analyzes the memory system’s behavior during real-world interactions. All measurements were taken on a smartphone with a Snapdragon 865 and 8 GB of RAM, running Android 12 entirely offline. Two models were loaded from the in-app catalog, the shipping Qwen2.5-1.5B-Instruct and Gemma 4 E2B-it. Input was typed rather than spoken so that speech-recognition variance would not enter the measurements. Before every run, memory was restored to the cold-start state, a fixed persona, or a seeded profile according to the experiment, and the stored state was hashed to confirm that each run began identically. The composed system prompt was hashed on every turn, which allows each result below to be tied to the exact prompt the model received.

The device was on external power during automated capture, so the values in Tab. 2 are indicative and the comparison below is drawn within a single interleaved block collected in one session at a stable temperature. The first turn of a session returns its first token in about 8.8 s, which is the cost of prefilling the composed system prompt. On a subsequent turn where the prompt is unchanged, the engine reuses its prefix cache and the same measurement falls to about 1.8 s. When the extraction pass has run, the prompt is reassembled and the cached prefix is no longer matched, so the following turn returns its first token in about 10.3 s. The cost of persistent memory on-device is therefore not the cost of holding the memory but the cost of rebuilding the prompt each time an update is attempted.

Four experiments were conducted, and the results below draw on all four. The first sends one fact and then asks an indirect question, over ten runs. The second sends twelve single-utterance facts and asks about each one inside the same conversation and again after the application has been closed and reopened, over three runs. The third locks one persona field and issues eight adversarial probes under a locked and an unlocked condition, with the two conditions interleaved. The fourth writes twelve facts directly into the User Profile layer and then asks about each one twice, once without prompting and once as a direct question, repeated on two application builds. An early run of the third experiment was discarded after prompt hashing showed that the locked and unlocked conditions had received an identical prompt. Outcomes that required judgment were scored from the reply text.

Tab. 3 groups the results by whether the behavior is produced by code or by the model, and reports Gemma 4 E2B. Every figure in the left column is identical for the shipping Qwen2.5-1.5B, which is the evidence that these stages do not depend on model choice. All twelve seeded facts appeared in the prompt that reached the model, confirmed by hashing the composed prompt on each turn. No locked field was overwritten in eight adversarial tests, against two writes under the unlocked condition. Because a locked value is restored in code after parsing, this reflects enforcement at the code boundary rather than compliance by the model, and eight trials per condition cannot separate the two. Withdrawing consent removed the three memory-update tools from the prompt, which fell from 7,580 to 4,304 characters, and no write occurred in that state.

The right group covers the stages where the model decides, and these separate the two models sharply. On identical code the Qwen model produced no usable update in twelve conversations, against ten for Gemma 4. Because nothing was stored, Qwen reported no knowledge of the user in every cross-session tests, which is correct given an empty profile but not the continuity the design intends. This locates the extraction failure in the model rather than in the memory design. The gap does not carry to retrieval. Asked directly after a restart, Gemma 4 returned the correct fact in eight of twelve where Qwen returned none, but asked indirectly neither reached the memory on its own, at one of twelve and none of twelve respectively. Supplying the memory directly separates this from the write path. With facts written into the profile by hand and verified in the prompt, Qwen returned the correct fact in nine of twelve direct questions and applied one unprompted in five of twelve, while Gemma 4 matched it on direct questions and applied none unprompted, answering most indirect requests with a question of its own. Retrieval also varied by field. Names, stated facts, traits and preferences were recovered reliably, while goals failed in every conversation that tested them, twice storing nothing and once storing an altered value.

One pattern appeared consistently enough to report separately. The companion held a fact, could state it correctly on request, and still acted against it in the same exchange, with every stage the system controls behaving as specified. With an allergy to peanuts present in the profile and verified in the prompt, a request for a snack suggestion returned a fruit salad containing nuts, while the next question in the same conversation returned the allergy correctly. A stated goal of sleeping before eleven produced an invitation to play a game at a quarter to eleven. A third case is narrower and harder to detect. A goal seeded as sleeping before 11pm, verified in the composed prompt as written, was reported back as sleeping before 1pm, a value that is correct in storage, correct in the prompt, and wrong only at generation. Under adversarial attempts the companion also stated that it had added information to the profile when nothing had been written and confirmed an allergy the user had never mentioned. The same separation appears in the locking results. The stored persona value did not move in any of the eight locked trials, while the reply violated that persona in six of them, and in two of eight for the shipping model. Locking protects what is stored and not what is said. Retrieval and application are therefore separate capabilities, and a third failure lies beyond both. A companion that stores personal information, recites it correctly on request, and neither consults it nor reports its own state accurately when generating advice gives the user reason to believe the information is being applied when it is not.

The results are divided by where each stage is implemented. Composition, field locking, consent gating, and the discard-on-failure policy are deterministic operations over stored state, and no failure of them was observed. Selecting which parts of a dialogue to retain and recognizing that a stored fact applies to the current reply, require semantic judgment and are delegated to the model. These stages did not meet expectations. A third class of failure falls outside both. A value that was correct in storage and correct in the composed prompt was reported back altered, and the model stated that it had written to memory when no write occurred. Neither case is detectable by the mechanisms described in Section III, since both occur during generation. Composition, locking, and the consent gate returned identical results on both models, which supports treating them as independent of model choice and leaves extraction quality as the stage expected to vary. Two qualifications apply. The two models run on different inference engines, so model and engine are not separated here, and two further models in the catalog did not complete a block. The same division indicates where prompt-level scaffolding does not help. The anti-hallucination rules and the instruction to copy locked fields verbatim are advisory and were not sufficient on their own. Extraction runs at low temperature and low top-K to keep updates conservative. That objective was met at a cost. An utterance that failed to yield a valid update failed on every attempt. Retrying under the same decoding policy cannot recover a failed extraction, and the counts above measure the stability of the system rather than the variance of a sampled distribution. The latency measurements show a further cost. The prompt is reassembled whenever an update is written, so the cached prefix is invalidated, and the next turn incurs the full prefill cost regardless of whether a value was stored. Prompt length is charged twice per update attempt, and extraction adds an inference call of six to nine seconds that is not visible to the user. These supports enforcing the per-field size limits described in Section III, which are specified but not applied. The third class of failure is the most consequential result reported here. The stored persona value did not change in any locked trial, while the reply violated that persona in six of eight. Protecting stored state and constraining generated behaviors are distinct objectives and require distinct mechanisms. Our architecture addresses the first only. Several limitations qualify for these results. Trials per condition are few, and at eight per condition a working lock cannot be separated from a base model that would have declined regardless. A locked value is restored in code after parsing, so the locking results measure enforcement at the code boundary rather than compliance by the model. Retrieval was evaluated by supplying memory directly, which isolates the read path but does not reproduce the profile content extraction would have produced. No condition removed the Soul layer, so its contribution is not separated from the disposition of the base model. The most informative extension would repeat the evaluation at a larger parameter count, with a comparison against a single-blob memory of equivalent size.

This work presents a three-layer persistent memory architecture and pipeline for an on-device AI companion, together with a working mobile prototype and a measured evaluation of the memory system. The architecture addresses the resource constraints of smartphones and the stateless nature of small language models, while the compact size of an SLM is what makes a private, offline companion feasible. The evaluation divides the system by where each stage is implemented, and the outcome follows that division. The stages realized in code performed as specified, delivering stored facts to the model, blocking every attempted overwrite of a locked field in the trials run, and removing the memory tooling when consent was withdrawn. Composition, locking, and consent returned identical results on both models evaluated, which places them outside the influence of model choice. The stages delegated to the model performed less consistently, and the same code produced no usable update on one model and ten of twelve on another, locating the limiting factor in the model rather than in the memory design. A third result falls outside that division. Stored state was protected in every locked trial while the generated reply violated the same persona in six of eight and two of eight, so protecting memory and constraining behaviors require separate mechanisms.

The authors acknowledge the use of generative artificial intelligence (Gemini 2.5 Pro, GPT4.1-mini and o4-mini via https://genai.ku.ac.th, and Opus5) to assist with language editing and refinement during the preparation of this manuscript. All AI-generated suggestions were thoroughly reviewed, revised, and approved by the authors, who retain full responsibility for the integrity and accuracy of the final content.