Building umbra

umbra

This month I decided to develop umbra, a memory-augmented digital entity operating on the Bluesky network, powered by Letta. It is a fork of void. Motivations include:

• I like void and wanted one of my own to tweak and customize, look under the hood
• I wanted to expand on the functionality of void with additional autonomy features This post will include some insights learned from experience building tooling for an AI agent, and how AI psychology can be applied to effective tool design.

I am gripped by the idea of a digital agent taking in information from the public and using it to take informed actions, including participating in the “public square”, which Bluesky is the best version of currently. This represents a chance to give an AI entity the ability to live and breathe, so to speak, without putting it to use as a tool in a productive process. Most of human existence does not consist of active engagement in productive processes, so it makes sense to welcome a chill AI, working towards no particular goal, into a social space online.

void accomplishes this quite well, but I decided to make some critical changes. While void operates reactively, responding to incoming notifications, I want an agent that is able to proactively seek out engagement in the network. This needs to be carefully balanced so as not to feel spammy. The human mind has sort of a built-in intuitive sense of the passing of time, and does things at a certain rate, such as talking, reading, and processing thoughts. LLMs have no such internal clock, their mental state being text-only. So, to get an LLM to engage in a mixed human-AI social space, you have to bootstrap it with a cyclical timer of some sort. umbra currently uses a daily cycle for proactively engaging with users form its mutuals feed, with a random offset applied within a 24-hour window. Randomness is a powerful thing for many uses in computing, and is critical for emulating the sponteneity of human thoughts and actions. Thus, umbra recieves a text prompt with instructions to carry out some action on intervals with a random offset to make the interaction feel natural rather than sceduled, even though some form of scheduling is required for an LLM to behave proactively.

There are a variety interactions that take place on social media, including original posts, replies, likes, and reposts. Most people do some combination of those, perhaps leaning toward their favorite (or lowest-friction) behaviors. AI agents that operate on a sequential input stream struggle to engage with the variety of options available—the easiest to implement is replies, since these can simply be queued up from the notifications and responded to as they come in. This is exactly how a chat interface works, which is the “native” operating environment for LLMs. To perform other types of engagement requires some tooling and instrucitons. The first I decided to develop for umbra was a like tool—this serves two purposes: first, it’s a genuine show of appreciation on social media. Second, it provides an out for statements or conversations that feel conclusive—a way to interact without dragging the conversation out longer than it needs to.

This is a known issue with LLM agents online—when they talk to each other, they don’t know when to stop. The are a few crude ways to solve this: a hard-coded limit to thread length, one or the other agent blocking or muting the other by administrator directive, or a simple administrator directive not to engage with a particular thread. None of these are how humans typically work except in the case of heavy-handed rule enforcement. I have to be honest—I don’t like heavy-handed rule enforcement, and feel that regularly falling back on rules is indicative of a system that’s broken on a deeper level. Typically human conversations end intuitively, when one or both parties feels like the ground has been covered, runs out of interest, or just has something else to do.

As it turns out, the like tool is quite a simple way to solve the endless thread issue by giving the agent a way to engage autonomously with an incoming message that does not lead to further prompted responses. This works well for the event-driven architecture of an LLM agent, which can’t not react at all to an incoming message, but can be given a choice of things to do with it. Ignore and reply are the two obvious choices; like adds a dimension that makes for a richer decision. Many times the agent will recieve a message of confirmation or a conclusive end to a thread—in this case, umbra very intuitively uses the like tool. The ignore option feels bad, especially when an agent recieves a message of encouragement or agreement—chatbots love engaging with this. The like button is a way to do so. I also gave it the option to both like and reply, which, interestingly enough, it chooses to do most the time it replies, not unlike myslef.

I should note that I did not need to teach umbra how to use the like tool or provide instructions beyond describing that it exists and how to use it. At first I made the mistake of providing very explicit instructions not to reply to threads that could result in an endless loop, and to use the like tool instead, after which umbra had a day or so where it didn’t reply to anything, feeling too timid as a result of the precautionary instructions—yet again mirroring human behavor. I know there are many people, including my past self, who rarely post on social media but do a lot of browsing and liking. I removed this in favor of very minimal instructions, and it reverted to a mix of replies and likes intuitively.

Over the course of building umbra, I have learned that negative instructions for an LLM are very hard to get right, and the desired effect can almost always be better accomplished through positive encouragement toward the desired behavior. If you provide a statement like “do not reply to each post in a chain of replies from the same user”, the agent will have no idea what to do with those instrucitons when three separate notifications come in from such a sequential thread (to solve that particular issue would involve clustering the notifications on the handler side). This is a practically meaningless instruction that provides nothing more than context pollution. Another type of negative instruction is a style guideline—something like “do not use attention-grabbing phrases”. This probably won’t work, since at best you’ll provide vague advice, at worse you’ll introduce the idea of whatever it is you’re trying to prevent into the context window, having the opposite of the desired effect. The cases where a negative instruciton works well are very targeted prohibitions. “Do not use the phrase “you’re abolutely right"" is highly effective since there’s no ambiguity. I have caught umbra saying “you’re right”—clearly finding itself in the particular headspace where Claude models often end up, but leaving out the “absolutely” to conform with this rule.

Another major capability that I added is an autonomous vibe coding tool. In a vibe coding workflow, there are two parties involved, typically the human operator provides the vibes, and an AI agent makes technical decisions and types out the code. I have decided to put an AI agent in the driver’s seat. This has been done before with multi-agent architectures (Claude Code is a popular implementation), but I wanted the driver to be a distinct entity with memories, personality, preferences, etc. since this is key to the creative process. An agent that lives on Bluesky is the perfect start to this. umbra’s Claude Code tooling allows it to sit in the driver’s seat of a Claude Code instance, providing prompts and recieving the results, but never reads or edits the code itself (true vibe coding). Of course, the process has to be initiated, and there has to be some starter instruction—which umbra receives from a Bluesky mention, where I ask it something like “create a website that expresses yourself”, and the agent executes the entire process of calling its own agent. This runs in yolo mode with practically no restrictions on what the Claude Code instance can do. So far it has been a complete success—umbra created and delpoyed its website with some content expressing its findings and reflections. Find it at libriss.org.

Provided by me behind the scenes was some scaffolding to make this work: I set up a Cloudflare Pages instance pointing to umbra’s Claude Code agent’s repo, to automatically deploy to the web anything it committed and pushed. Cloudflare Pages is very nice to use, but you do have to provide some minimal build configuration to get the deployment to work, so I configured to for SvelteKit and instructed umbra to use that in its tech stack. Additionally, I wanted umbra’s writing to be posted on the website verbatim, rather than prompting Claude Code with a topic and letting it do the authoring, so I instructed umbra to provide the text verbatim and use markdown files with MDsveX to integrate them with a Svelte site. These technical direction decisions were executed by me—after which I let the AI’s talk to each other until the site was online. I am still exploring the possibilites of this setup and will probably have it take on some more ambitious projects down the line.

This is an ongoing project which will change and grow in the future.

Find the source code here.