The AI sees the game
When template matching and text recognition fall short: use a local model to recognise states, branch on them and read several values in one call.
Sometimes both fail: a template finds nothing because the image looks different every time — and text recognition returns nonsense because there is no text, or the background is too busy. That is what the AI modules are for. They judge an image instead of comparing it.
One thing up front so expectations are right: the AI is slower than a template search, and it is not meant for every click. It pays off at the few places where understanding is needed — "is a fight going on, or am I in a menu?".
A local model on your own machine is enough for these tasks. No provider, no credit card, and no image data leaves the computer.
Step 1 — Set up a local model
Install Ollama and start it.
Pull a model that can read images. In a terminal:
ollama pull qwen2.5vl:7bIn FlowBotCommander: Tools → Provider profiles, then press "Find local models".
It does not have to be Ollama. The search queries llama.cpp (port 8080) and LM Studio (port 1234)
just the same — all three answer the same call, and the app talks to all of them alike.
What does matter is the model: a text-only model cannot see images. With llama.cpp that means the
mmproj file has to be loaded alongside the model file — without it the server still starts, but
takes only the text part, and the AI answers every image with general prose. That looks like a
recognition failure but is none.
llama-server -m qwen2.5-vl-7b-q4_k_m.gguf --mmproj mmproj-qwen2.5-vl-7b-f16.gguf --port 8080
The search queries the usual addresses (Ollama, llama.cpp, LM Studio), creates a ready-made profile for every hit and preselects a model that can read images. Then Save — that was the whole setup.
If the search finds nothing, the server is almost always not running. That is not a guess but the most common case, which is why the message says so explicitly.
Step 2 — The quickest test: a yes/no question
Take Ask image (AI), use the area tool to put the area tightly around what matters, and ask something with only two answers:
Is there an error dialog in this image? Answer with yes or no only.
The module returns the answer three ways: as text (answer), as a number (zahl) and as yes/no (jaNein). For
a branch, connect jaNein to an If/Then module.
Step 3 — Branching with "AI decides"
As soon as there are more than two cases, AI decides is the right module. Write the options one per line:
fight
menu
loading screen
The module then has the outputs wahl1, wahl2, wahl3 — plus unklar (unclear) and fehler (error).
Two things about it matter more than they look:
- You do not add "unklar" yourself. The module appends that option to the question on its own. The reason is an observation from practice: without that escape hatch a model picks a wrong option rather than holding back — in a test with a 40×64 pixel snippet showing nothing at all, it confidently answered with one of the options. With the escape hatch it correctly answered "unclear".
- The answer is checked against your list. If the model invents something of its own or answers ambiguously
("fight or menu"), the flow goes to
unklar. It never goes somewhere just because a word roughly matches.
Connect unklar to a short wait and another attempt — not to the same action as a choice output. "I don't know"
is a useful answer as long as the bot respects it.
Step 4 — Several values in one call
For numbers and names take Read values from image. One field name per line:
gold
level
enemy_name
Each field becomes a data output of the same name. That is one call for three values — with "Ask image" it would be three.
Name the fields as you would to a person: gold is clear, value1 is not. The model reads what the name
suggests. If a value sits in an unusual place, the extra hint helps: "The numbers are in the top bar. Ignore
the chat."
The area matters more than the model
The most effective lever is not a bigger model, it is a tight area. Measured on a real game screenshot at 3440×1474, same question, same model:
| What was asked | Result |
|---|---|
| whole window, shrunk to 1024 px | wrong — values from another window, 4.4 MB unscaled |
| cut-out around the values (750×385) | correct — 54 KB, 0.2 s |
So a tight area is cheaper, faster and more accurate at once. Spend the half minute with the area tool; it saves more than any other setting.
What it costs — and how long it takes
| local | provider (Claude, GPT) | |
|---|---|---|
| money per question | none | by image area and text length |
| duration | 0.2–0.6 s while the model is in memory | depends on the line, usually 1–3 s |
| first call | about 15 s, the model is being loaded | as usual |
| data | stays on your machine | the cut-out goes to the provider |
That load time on the first call is why the default time limit is generous. And because a loop would otherwise make a hundred calls for the same question, there is "Remember answer for": ask the same question about the same image and the remembered answer comes back. Any change to the image counts as a new question.
When it does not work
| What you see | What it usually is |
|---|---|
| Notice "AI is not set up yet" on the module | No provider profile. Press Set up AI…. |
| "Find local models" finds nothing | The server is not running. Start Ollama, then search again. |
The flow always goes to unklar |
The area shows too much or the wrong thing. Make it tighter — and check that the options really fit the image. |
The flow always goes to fehler |
Wrong model name, or the model cannot read images. Pick one with vl, vision or llava in its name. |
| Values get mixed up | The area is too large — see above. The extra hint helps as well. |
| The answer is cut off | Raise maximum answer length in the provider profile. |
| Everything is slow | The first call loads the model. After that it stays in memory — and "remember answer" makes a loop much cheaper. |
Where to go next
- AI modules — all four modules with every field
- Image recognition — if a template does the job, it is always the first choice
- Control API — have an AI like Claude Code build and watch a bot for you