Recently, I decided to set up an animal detection system using LLM vision models on my Home Assistant setup. This project was born out of a desire to enhance safety in my remote property, which frequently sees wildlife like moose, bears, and coyotes. I wanted to share my journey and some tips for anyone considering a similar project.
Choosing the Right Model
After some research, I settled on llama3.2-vision:latest for its balance of accuracy and performance. I also tested Perplexity, which showed impressive results at 96% accuracy, but unfortunately, it doesn’t support image uploads yet. While llama3.2-vision had slightly lower accuracy at 90%, it was still a solid choice for my needs.
Setting Up the Prompt
Crafting the perfect prompt was a bit of a challenge, but here’s what worked best for me:
text
if there is an animal in the image, respond with the name of the animal. if you think it is a deer, say it is a Moose. If it is a bear, say if it is a Brown Bear or Black Bear. if it is an animal, then respond strictly in the format: ‘There are [number] [animal name] in your front yard.’ Example: ‘There are 3 brown bears in your front yard.’ if there is no animal in the image, then respond strictly in the format: ‘There are no animals in your front yard.’
This prompt ensures consistent and clear notifications, which is crucial for my system.
Testing and Fine-Tuning
I tested the system with around 50 photos of various animals and was impressed by the results. However, I encountered some challenges, such as occasional misclassifications. To mitigate this, I ensured a diverse dataset and fine-tuned the prompt for clarity.
Integration with Home Assistant
Integrating the system with Home Assistant was straightforward. I used Zerotier for remote access to my self-hosted server, which has been rock solid for me. The notifications are now a key part of my daily monitoring routine.
Final Thoughts
This project has been incredibly rewarding. It not only enhances safety but also adds a layer of automation to my property management. I encourage anyone with similar needs to explore this setup. Let me know if you have questions or tips to share!
Happy automating! 