I’ve read some interesting articles and watched a few videos regarding “loops” in AI agents. One video by Anthropic with the tagline “the winners won’t have the smartest model, they’ll have the best loop.”
I realize that this is more related to LLMs but since I have over 80 Aqara devices in my home connected to my HA ecosystem, I wanted to share my experience with “loops”.
As stated, I have a very rich IoT environment, mostly consisting of Aqara devices which are integrated via various communication protocols to Home Assistant. I started using Claude 3 or 4 months back to speed up and clean up my HA automations and processes. I decided to take a stab at the loop approach. I ended up with 4 agents running which perform the following tasks:
Plan Agent - Takes my task and turns it into a detailed development scope.
Build Agent - After approving the scope from the Plan agent, it gets handed off to the Build agent which actually builds and develops the code. I have instructions built in to the agent to do a full HA backup to NAS before writing any changes.
Judge Agent - The judge agent evaluates the code developed by the Build agent and then creates tests to validate the code generated by the Build agent. If any tests in the Judge agent fails, it is handed back off to Build agent until all tests pass.
Research Agent - This agent runs outside of the Plan, Build, and Judge agents and I am using it to research new firmware released by Aqara, new HA Core, OS, services, and HACS integrations and then evaluates my HA environment for improvements, bugs, or enrichment based on those software changes and new functionality. It then passes off the research to the Plan agent where it starts the loop process.
Test Example: I wanted a simple project to test these new agents with so I told the plan agent that I wanted to create a new automation in HA that would alert me via iOS and Alexa announcement when any Aqara Matter devices become unavailable (I already have a similar automation and I chose this so I would have a working solution to compare against). The Plan agent did what it was meant to do and developed a full development plan after reading my HA ecosystem. I approved the plan and it was handed off to the Build agent which created:
Device unavailable automation
Device restored automation
Helper entity for N minutes offline
New technical document (MD file) that outlines the runbook
The Judge agent ran automatically once Build agent was complete. Judge created 12 validation tests, all of which passed. Had the tests not passed, the Judge agent would hand back off to Build agent until all tests pass.
Great insight! It’s a pity that I’m not experienced enough with HA and still a smart home newbie.
I’m looking forward to the day when I can just tell an agent what kind of scenes I want and have it make it happen!
Since you mentioned you already had a similar automation running for the Matter devices, I’m really curious about your final conclusion. How exactly does the AI-generated code compare to your manually created one? Is it cleaner, or did the Judge agent perhaps even catch some edge cases you hadn’t thought of initially?
Also, I have to ask: what did this experiment cost you in terms of API usage? Letting four agents continuously loop through planning, building, and judging sounds like it could burn through tokens quite fast.
I kinda think we are already there! That does sound like a great use case for the P/B/J agents. I’ve been needing to add a few new scenes so “challenge accepted”. I’ll report back here how it did.
Great questions. Question 2 is much easier to answer then question 2, so I’ll start w that.
Q1 - “What did this experiment cost you in terms of API usage”: I am on the standard “Pro” plan w Claude which physically costs me $20/month and I have used 14% of my weekly limit.
Remainder of Q1 comments were generated by CLaude
Plan agent: 72,212 tokens
Build agent: 116,124 + 198,017 tokens (two runs)
Judge agent (substitute): 103,301 tokens
That’s roughly 490K tokens total just for the subagents, on top of whatever this main conversation used. The whole point of the Plan/Build/Judge pattern is to spend more tokens deliberately — giving each phase a clean, unbiased context — in exchange for higher confidence that the result is actually correct. It’s a cost/rigor tradeoff, not a free lunch.
Q2 - “How exactly does the AI-generated code compare to your manually created one”?: This one is a bit more difficult to answer due to the fact that I use orchestrators for repeatable processes, so the iOS and Alexa announcement code was already done in HA and the agents knew that and reused it in their processes.
I have all of my automations and entities classified by process, so everything is grouped well together. Along with that, I have given Claude instructions to always check process documents before starting a project or session and to always update the documentation and perform NAS GIT backups after changes. Each project also has a technical and runbook doc which Claude reads at the beginning of each session. I can say the code was very clean and the process it used is more efficient then the one I initially created. I was using the Watchman service to determine unavailable devices via its report. Here are the tools and tests the judge agent ran for this project.
Read the HA device registry for all Matter configured devices
Read the actual entity states of all Matter sesnsors live via ha_get_state (not just the plan’s assumption)
Read the automation YAML straight off disk via SSH (not Build’s pasted diff) and confirmed the for: value references the helper rather than a hardcoded number
Confirmed the input_number helper file has no initial: key, both in the file on disk and via the live entity’s initial: null attribute
Used ha_eval_template to render the labels dict for entities (Build had only spot-checked 2) and confirmed every physical-location translation was correct
Grepped the automation block specifically (not the whole file) for any Alexa/TTS call — found none
Re-derived the restart-guard condition itself and ran it through ha_eval_template against 3 scenarios: on→off (should fire), unknown→off (should suppress), unavailable→off (should suppress) — all behaved correctly
MD5-diffed the two doc copies (HA vs. local mirror) to confirm they’re byte-identical
Checked git log on both repos to confirm the commits Build claimed actually exist with matching content.
Here is an example of how I have my technical docs arranged in HA
So I decided to knock out a couple items at once. There is a forum post titled “Share Smart Match Day Setup…” where you can share your automations, light scenes, etc. and I wanted to try a new scene creation just by tell Claude what I wanted (as you asked). Claude was able to easily knock out this new scene for me in about 5 minutes. Here was my prompt to the P/B/J agents.
