What you’re learning
These products combine different models, interfaces, tool permissions, and hosting choices. A fair comparison keeps the task and success criteria as similar as possible, then records where each product differs.
Work through it
- Pick a safe, repeatable task. For example: “In this disposable folder, read these three plain-text notes and produce a 5-bullet summary with filenames cited.” Make sample files yourself; do not use personal or sensitive documents.
- Write success criteria first. Decide what counts as correct: all three files represented, five bullets, filenames included, no files changed, and no unsupported claims. Keep the exact prompt and sample input the same.
- Run each tool. Use Hermes, OpenClaw, and Codex with comparable models where practical. Start with read-only permissions. If a product requires a different setup, record that as part of the comparison.
- Score and reflect. Give each result a 1–5 score for accuracy, instruction-following, transparency, ease of setup, speed, and cost. Note model/provider, permissions, and whether any data left your computer.
Example command / prompt
Task: Summarize the three sample notes in exactly 5 bullets. Include the source filename for every bullet. Do not edit or create files. State uncertainty clearly.
Keep this in mind. A local model, hosted API, and coding agent do not have identical privacy or tool behavior. Record the actual provider and enabled access for each run; do not assume that an “agent” processes everything locally.
Try thisRepeat the same task once with a deliberately improved prompt. Did clearer criteria help all three tools? Which one made its work easiest to verify?