Over the past few months, I've been building my own locally run AI assistant, and the process has been much bumpier than I expected.
V1: Initial keyword matching
I came up with the idea when I bought two tiny office computers, and wanted to build my own always-on AI assistants after learning about homelabbing. These two computers have old i5 CPUs, 8GB of ram, and no GPU, so I am heavily limited on what I can run, LLM-wise. The initial draft barely even used LLMs to process, but it was built off of TF/IDF detection mapping commands to CLI-like phrases (I later found out about RAG and nomic-embed, don't worry). This ran simple JavaScript functions that passed basic commands to other functions, was hardcoded, but it was performant. I knew the limited hardware I was working with wouldn't let me run anything meaningful with LLM inferencing, but I was able to make some scripts to actually automate parts of my life.
V2: An agent for everything
The second version was made with the help of the recently-released Opus-4.6. This V2 had its own custom harness, agent pipeline, and a memory storage system. It could do many more things than V1, but it was still limited by the harness. In order to get access to larger and more powerful models, I opened up the Ollama port on my main PC so that it could use my 3060-Ti I already had. This version of my assistant could actually build code, and Claude called it high-quality code, but it took over a month to build a JavaScript page to turn my TV into a productivity manager, simply because the harness had a new kink we had to untangle every time I tried to re-run it. I really went overboard here, making dedicated agents for requirements building, code generation, test case builders, and even a consultant-like agent to analyze the whole process and tell which agent when it was wrong if it failed to deliver code. The biggest issue is that it couldn't do anything beyond generating code, because I built is solely for the code generation loop, not a general act-> code -> execute loop.
V3: A thin harness on top of Hermes
This brings me to V3. I wanted to give Hermes a try, and it's been all right so far. Claude Opus 5 and 5.5 have assisted me in building a custom harness again, but this time it's just for taking input, model routing, and sending requests to Hermes or running known scripts. This lets me swap out the actual harness doing anything if I wanted to swap it out for any other bright and shiny harnesses I like later (looking at you, Deepseek Harness). I can speak to it through Discord like V2 could, but this time, I've been able to build a few apps to manage processes on my computer and in my network, monitor the health of processes running in Docker containers, and actually get things done.
What's next
There's still a lot I want to figure out about developing a custom AI assistant, but the resounding advice that I've seen surrounding AI is that the harness is the limiting feature of AI-run infrastructure. Would love to hear some other open-source harnesses my network has come across, or any advice I might need.