rsbot: building an autonomous OSRS bot that actually has a plan

The interesting thing about an Old School RuneScape bot is not the macro that clicks the tree. The macro is solved. What's not solved is what to do next.

Most bots run a single skill forever. Mine has a goal planner. You give it a long-term target — "combat to 70, total wealth to 1M GP, 200+ quest points" — and it picks its own activity based on current stats, inventory, gear, location, and how close it is to each goal. When it finishes one task, it picks the next one without me being there.

It's called rsbot. It's the largest active codebase I have on disk by source-file count (2,073 files). Most of those files are the verbose modular architecture, not bloat.

The shape

Entry:    run_autonomous.py
              │
              ▼
          BotEngine ──→ AutonomousPlayer (script)
              │
              ├─ DetectionBridge ──→ YOLOv8 pipeline
              ├─ AntiBan          ──→ humanlike timing
              ├─ GoalPlanner      ──→ long-term targets
              └─ TaskManager      ──→ short-term actions

Core layout:

bot/
├── core/                state machine, task manager, anti-ban, goal planner
├── modules/
│   ├── autonomous_combat.py
│   ├── autonomous_skilling.py     (woodcutting, mining, fishing,
│   │                               cooking, smithing, crafting)
│   ├── autonomous_banking.py
│   ├── autonomous_travel.py
│   ├── autonomous_questing.py
│   └── gearing_manager.py
├── ai/
│   ├── ai_brain.py                 LLM decision-making
│   └── strategic_advisor.py
├── game/                inventory, player, bank, login
└── input/               RuneLite plugin OR window-constrained mouse

Two input modes: a RuneLite plugin that posts intents over localhost, and a fallback pyautogui mode that constrains itself to a window rect. The latter is what you ship to a friend who won't install RuneLite plugins.

Perception

YOLOv8 does the heavy lifting. I trained it on the obvious classes — trees, ore rocks, fishing spots, monsters, bank booths, doors — using screenshots from my own gameplay. mss grabs frames; OpenCV preprocesses; YOLO outputs bounding boxes; a coordinate transform converts those to game-window pixels.

detection_bridge.py is the layer that turns "raw detections" into "labeled entities I can decide about." It dedupes overlapping boxes, applies confidence thresholds per class, and tags entities with stable IDs across frames so the planner can say "the same chicken I was already attacking, not a different one."

Strategy

ai_brain.py wraps a local LLM (LM Studio) for higher-level decisions. The planner doesn't ask the LLM what to do on every frame — that would be slow and wasteful. It asks the LLM when a decision is genuinely ambiguous: which of three nearby trees is best when one is closer but another is in safer territory; whether to bank or keep killing when the inventory is at 27/28; whether the current skill grind is the best one toward the goal.

The state machine handles everything that's mechanical. The LLM handles everything that's interesting.

Goals, not scripts

The thing I'm most proud of is the goal planner. You set it like this:

goals:
  - { type: skill,  skill: combat,  target: 70 }
  - { type: wealth, target_gp: 1_000_000 }
  - { type: quest,  target_qp: 200 }

Then goal_planner.py computes a heuristic ranking on every cycle:

score(activity) =
    expected_progress_per_minute(activity, current_state) /
        cost_of_setup(activity, current_state) *
        priority_weight(active_goal, activity)

The bot picks the highest-scoring activity, kicks off the appropriate module, and keeps going until the goal is achieved or a higher-scoring activity emerges.

That's why this isn't a fishing bot or a combat bot — it's "the bot for the goals you set." Combat to 70 might mean killing chickens to 30, hill giants to 50, and dust devils to 70. The planner makes those transitions on its own.

State

What works:

  • Combat (recent successful run vs hill giants — full kill loop, prayer flick, loot pickup, restocks)
  • Skilling (woodcutting, mining, fishing, cooking, smithing, crafting all working)
  • Travel (route planner picks the optimal teleport/run path)
  • Antiban (humanlike timing distributions, mouse movement Bezier curves, idle injection)

What's broken right now:

  • Banking. Debug logs from April show task timeouts on bank-opening — 40 minutes stuck on a gear-acquisition task that should take 30 seconds. There are five bank_open_fail screenshots saved as evidence. The current retry logic is too crude (15 retries on timeout); the fix is to detect "bank booth is occluded" vs "I clicked the wrong thing" and recover differently.
  • Questing is skeleton-only. rune_mysteries.py exists but isn't wired into the main loop.

What's next

Three threads I'd pull, in order:

  1. Fix banking. The whole goal-planner falls over when the bot can't reliably bank. Diagnose the failure modes from the saved screenshots, replace the retry loop with state-driven recovery.
  2. Wire questing into the main loop. Once questing is a primitive the planner can pick, the QP goals start working.
  3. Re-enable RL training. There's scaffolding at rl/osrs_env.py and rl/combat_env.py that's disabled. The OpenAI Gym wrapper around the game state would let me train a real RL agent for the combat sub-loop instead of hand-tuning prayer flicks.

There's also a strong consolidation opportunity with The Visual Bridge, which solves the same vision problem from a different angle — it's an MCP server that wraps YOLOv8 + Moondream2 for any OSRS window. Visual Bridge could be the perception layer for rsbot, freeing the bot to focus on planning.

This is a project that taught me how complex an "agent that does one thing autonomously" actually is. Most days I'm not sure if I'm building software or breeding a creature.