I Built a Screen Recorder That Documents Itself. Here’s How You Can Build Yours Too.

What if every tedious task on your computer could be recorded once — and then replayed forever, with an AI-written explanation of what it actually does?

I do a lot of repetitive things on my computer. Open Chrome, go to Gmail, click Compose, type a subject, type a body, hit Send.

Every Single Day.

I’d tried tools like AutoHotkey or AutoIt. They all had the same problem: they were either too dumb (record-and-replay with no intelligence) or too black-box (vendor-locked, impossible to tweak, expensive). So I built my own.

The result is on GitHub: activity-recorder-replay. It’s a Python desktop app that records what you do, replays it, and uses a local AI to write a plain-English description of what your recording actually does. It costs $0, runs offline, and — best of all — you own every line of the code, so you can bend it to do whatever you want.

In this article I’ll walk you through.

Why build your own recorder?

Three reasons that matter more than you might think.

1. Your needs will change. I started by recording clicks. Then I wanted drag support. Then I wanted it to handle re-sized windows. Then I wanted a small image preview at each click. Then I wanted the AI to summarize my procedures. Every one of those changes took ten minutes because I owned the code. With a vendor tool, that same change would have been a feature request, a support ticket, and a six-month wait.

2. Privacy. My app records everything I do. I run it on my own machine, the recordings stay in a folder I control, and the AI runs locally via Ollama. Nothing leaves my laptop. Most commercial RPA tools ( software applications that use “bots” to mimic human interactions with digital systems) either store in their cloud, or worse, train on your inputs.

3. It is free.

What it actually does

Here’s the feature set:

• Record mouse clicks, drags, scrolls, keyboard input, and the active window. Stored as a .jsonl log file.
• Replay with adjustable speed. Drags work correctly. The recorder tries a one-shot image match per click so it still works if the target window has moved.
• AI-generated procedures. After you stop recording, a local AI (Ollama, free, runs on your CPU) reads the event log and writes a 3–6 sentence summary plus a short title like “Book flight to Tokyo” or “Edit vacation photos.” The result is saved as a Markdown file you can open in any editor.

• Procedures library. A built-in browser lists every procedure you’ve ever recorded. Click one to see its AI summary, replay it, or delete it (the .md file and the matching log are cleaned up).

• Global hotkeys. Ctrl+Shift+R toggles record, Ctrl+Shift+P toggles replay, Ctrl+Shift+O loads a log, Ctrl+Shift+L opens the procedures library. They work from any app on your desktop, not just when the recorder is focused.
• Fail-fast option. If a recorded element can’t be found on screen anymore (the UI has changed since you recorded), the replay shows a friendly popup and stops instead of clicking the wrong place.

One Python file. That’s it.

The entire app is a single Python file — activity_gui.py, around 1,700 lines, MIT-licensed. Clone the repo and you have everything you need to run, modify, or sell it.

The reason it fits in one file is that the heavy lifting is delegated to excellent libraries:

• pynput — captures and injects mouse and keyboard events
• pywin32 — talks to the Windows API (window activation, foreground detection)
• Pillow — screen capture
• OpenCV — template-matching the small image we save with each click
• ollama — the Python client for the local AI
• tkinter — the GUI (ships with Python, no install)

If you want to add a feature, you usually change one function. Want to record microphone audio? Add an import for sounddevice and a callback. Want to OCR what's on screen? Add pytesseract. Want to email yourself the procedure? Drop in yagmail. The architecture is flat enough that an afternoon is plenty for most tweaks.

How the AI piece works

This was the part I was most excited about. After every recording stops, the app sends the event summary to a local Ollama model (I use ministral-3:3b, about 3 GB, fast on CPU) and asks for two things: a short title and a 3-6 sentence description.

The prompt looks like this:

You are an assistant that reads a log of user activity on a computer.
Return a JSON object with exactly two fields:
  "title": a short, specific name (3-7 words, like "Book flight to
          Tokyo" or "Edit vacation photos"),
  "description": a 3-6 sentence summary of what the user did,
          mentioning the apps used and main actions.

The response is parsed as JSON and saved as <title>.md in your procedures folder. You end up with a searchable, human-readable library of everything you've automated. It's like a personal wiki that builds itself.

If you don’t have Ollama installed, the app still works perfectly — recording, replay, the procedures library, hotkeys, all of it. You just don’t get the auto-generated summaries.

What about OpenAI Codex “computer use”?

You might be thinking: “Wait, doesn’t OpenAI have a thing where AI can control your computer?”

Yes. OpenAI’s Codex have a “computer use” mode that lets an AI model take screenshots, move the mouse, and click around to complete tasks. It can also be used to record screen activity, in a sense, by capturing the actions it takes.

It works, but it comes with costs:

• Per-screenshot pricing. Every screenshot the model takes costs a fraction of a cent.
• Per-action pricing. Mouse moves, clicks, and keystrokes each round-trip through the API.
• Privacy trade-off. Your screen content is sent to OpenAI’s servers (or whoever’s API you’re using). For a tool that captures everything you do, that’s a meaningful consideration.
• Always online. No internet, no automation.

For occasional, simple tasks it’s fine. For a tool you’d run dozens of times a day recording your own workflows, the cost adds up fast. The app I built is the opposite trade-off: free, offline, local AI, instant.

If you want both, the two complement each other: use this local app to record and replay your own workflows (free, fast, private), and use Codex computer use for AI-driven tasks where you want a model to figure out the steps (paid, slow, but flexible).

Setting it up (10 minutes)

# 1. Clone the repo
git clone https://github.com/UnicornDay/activity-recorder-replay.git
cd activity-recorder-replay

# 2. Install dependencies
py -3 -m pip install -r requirements.txt

# 3. (Optional) Install Ollama for AI summaries
# Download from https://ollama.com, then:
ollama pull ministral-3:3b

# 4. Run the app
py -3 activity_gui.py

That’s it. Press Ctrl+Shift+R, do something on your computer, press Ctrl+Shift+R or Escagain, and you'll see an AI-written procedure appear in the bottom panel and saved to C:\Users\willi\activity_logs\procedures\.

The app is fully self-contained — there’s nothing else to learn, no separate CLI tools to remember. One file, one command, done.

Modifying it — the part I care about most

Open activity_gui.py in your editor. The structure is roughly:

• Lines 1–100: imports, paths, hotkey plumbing
• Lines 100–1000: core recording and replay functions and AI helpers
• Lines 1000–1700: the GUI class and the if __name__ == "__main__" block

Because the code is MIT-licensed, you can use it in commercial projects, modify it freely, or ship a paid version of your own fork. The whole codebase is on GitHub: UnicornDay/activity-recorder-replay. Fork it, break it, improve it.

If you build something useful from this and want to sell it, the repo also includes a complete Netlify + Stripe checkout flow in the site/ and netlify/functions/ folders.

But the real win is the code itself. Owning the tool that automates your day is one of those things that, once you have it, you can’t imagine going back.

If you build something cool with it, I’d love to hear about it.