What SillyAgent Does

SillyAgent is an AI assistant that takes control of your Mac and does things for you — you tell it what you want, and it watches your screen, moves the mouse, clicks buttons, and types. It's like having a helpful person sitting at your keyboard who can follow instructions. You might ask it to "search for flights to Tokyo" or "create a folder on my desktop," and it figures out which apps to open, where to click, and what to type to get it done.

How It Works

The agent runs a simple loop: it takes a screenshot of your screen, sends that image to Google's Gemini AI, which decides what action to take next (like "click the Safari icon" or "type this search term"). Then the agent performs that action — moving the mouse, clicking, or typing. It keeps repeating this cycle until the task is complete. Think of it like a person who can see your screen and respond to what they see, one small action at a time.

The project comes with built-in knowledge of over 30 popular Mac apps like Safari, Chrome, Finder, Slack, Notes, and more — so it knows keyboard shortcuts and where buttons typically are. As you use it, the agent remembers your preferences ("I like Safari better than Chrome") and even learns from its mistakes, so it gets smarter over time. If it gets confused or encounters something risky (like a login screen or a delete confirmation), it stops and asks you for help instead of guessing.

Who Would Use This

Anyone who wants to automate repetitive tasks on their Mac without writing code. A startup founder might use it to handle administrative tasks while focusing on product. A PM could delegate screenshots or note-taking. Someone managing multiple projects could ask it to organize files or send scheduled messages. It's designed for non-technical users — you just give instructions in plain English and let it work. The installer is a single command with no environment setup required.

What You Should Know

SillyAgent only works on macOS and needs permission to see your screen and control your keyboard and mouse. It won't work well with real-time games or anything requiring instant reflexes, since the AI takes a few seconds per action. For safety, it stops after 500 steps automatically, refuses to run destructive commands without your approval, and you can pause it anytime by pressing a keyboard shortcut or shaking your mouse. It requires a Google Gemini API key to run — which is free to set up but does send screenshots to Google's servers as part of how the AI understands what's on your screen.