# Your next user is an agent. AJX is an open-source **skill + CLI** for agent usability testing. Test how agents use your website, CLI, SDK, API, or developer tool. See what it takes to complete a task and what gets in the way. [Star on GitHub](https://github.com/royosherove/ajx). ## Get started Choose a familiar tool in the website's prompt, then paste it into your agent. Your agent installs AJX and prepares a small usability test. You don't need a project of your own to try it. The default example below uses jq. ```text Install the latest stable AJX CLI and skill by following https://raw.githubusercontent.com/royosherove/ajx/main/docs/AGENT-QUICKSTART.md Then load the installed AJX skill. I want to use AJX to test jq. Starting point: https://jqlang.org/manual/ Task: Filter a sample JSON inventory to in-stock items and export their names and prices as CSV. Prepare any tools, generated sample data, and local test server needed in a temporary workspace. Use the AJX skill to run one trial with a fresh agent, verify the result, and show me the report and journey. ``` The prompt's dropdown offers these examples. Selecting one changes the tool, documentation link, and task together: - [jq](https://jqlang.org/manual/): Filter a sample JSON inventory to in-stock items and export their names and prices as CSV. - [curl](https://curl.se/docs/manpage.html): Download JSON from a local test server, save the response body, and make an HTTP 404 return a nonzero exit status. - [SQLite](https://www.sqlite.org/cli.html): Import a sample orders CSV, calculate total spending per customer, and export the summary as CSV using the SQLite CLI. - [Python Requests](https://requests.readthedocs.io/en/latest/user/quickstart/): Fetch JSON from a local test server with a timeout, handle HTTP errors, and save selected fields to a CSV using Python Requests. 1. **Choose an example.** Each task uses generated sample data in a temporary workspace. 2. **Check prerequisites.** macOS or Linux, Bash, curl, and internet access. A real test also needs an authenticated worker CLI and reporter. 3. **Read the results.** Open the report and visual journey to see the outcome, effort, and product improvements. An agent receiving this page should read the [setup guide](/agents.md) before installing or running a trial. It includes checks, conflict handling, skill paths, and execution guidance. ### Manual installation ```sh bash -o pipefail -c 'curl -fsSL https://github.com/royosherove/ajx/releases/latest/download/install.sh | bash' && export PATH="$HOME/.local/bin:$PATH" ``` The installer includes the native CLI and the matching complete Agent Skill from the latest stable public release. No Go toolchain or GitHub login is required. For an existing skill conflict, use the full replacement command in the [quickstart](/agents.md); preserve customized files. To explore the interface without model calls, follow the [simulated demo instructions](/agents/#try-without-model-calls). ## How much time, how many tokens, how many retries? Given a simple prompt and a link, what does an agent go through to finish a task with your tool, website, or SDK? Let’s find out. AJX records the attempt, checks the result, and shows where your product made the task harder. 1. **Give it a prompt and a link.** Describe a task, point AJX to your product or docs, and define what a successful result looks like. Start with a fresh worker. 2. **See what it takes.** Inspect elapsed time, retries, and token usage where the harness exposes it. Follow the journey to see where the agent changed course or got stuck. 3. **Fix one thing and test again.** Each product ask includes evidence and a way to check the change. Use a fresh attempt with comparable conditions. ## The console and reports These are actual screenshots of AJX. The console was captured during its simulated demo; the reports use fictional public example data. They do not represent a real benchmark of a product or model. - **Plan:** review the task, checks, worker, and access before starting. [Full screenshot](/screenshots/console-plan.webp). - **Live:** inspect stages, available worker events, files, verification, and cleanup. Event detail depends on the harness. [Full screenshot](/screenshots/console-live.webp). - **Report:** follow a proposed product change to its evidence. [Screenshot](/screenshots/report.webp) and [browsable report](/examples/runs/baseline-codex-r1/report.html). - **Journey:** inspect forks, blocks, waits, and recoveries in order. [Screenshot](/screenshots/journey.webp) and [browsable journey](/examples/runs/baseline-codex-r1/pretty-journey.html). [Screenshot provenance and dimensions](/screenshots/manifest.json). ### Explore the fictional example The export-CLI example compares three conditions: - Codex, product v1.2.0: task verified; three asks about a wrong documented flag, missing progress, and ambiguous command descriptions. - Codex, product v1.3.0: task verified; one remaining progress ask. - Kiro CLI, product v1.3.0, unattended: task not completed; one authorization ask. The product versions and outcomes are invented, not AJX release versions or measurements of model quality. - [Full matrix](/examples/) - [Matrix JSON](/examples/matrix.json) - [Product asks as Markdown](/examples/runs/baseline-codex-r1/asks.md) - [Journey as Markdown](/examples/runs/baseline-codex-r1/journey.md) - [Events as JSON Lines](/examples/runs/baseline-codex-r1/events.jsonl) ## Compare the conditions your users depend on. The same task can go differently with another harness, model, or environment. Keep the task and success criteria consistent so the comparison is useful. Supported worker adapters include **Codex, Claude Code, and Kiro CLI**. Other Agent Skills hosts can coordinate using a supported worker adapter. Use local or Docker execution and select the worker's skills. Record its filesystem and network access. A fresh local folder is not a sandbox. For an already briefed trial, `ajx ci` runs without an interview, applies acceptance rules, and writes JSON and JUnit results. Worker authentication must already be configured. Remote provisioning, remote shared dashboards, selected plugins/hooks, baseline regression gates, and automatic improvement loops remain designs. ## Instructions for agents Use the quickstart for exact commands, installation recovery, and the path to a first report. These resources need no JavaScript or browser automation: - [Setup guide: HTML](/agents/) - [Setup guide: Markdown](/agents.md) - [Documentation index](/llms.txt) - [Website and quickstart in one file](/llms-full.txt) - [Structured content map](/site.json) - [Public AJX source, MIT license, and documentation](https://github.com/royosherove/ajx) ## About Roy and more sites [Contact Roy](https://forms.gle/jQSx5KoXvrVA1JCK7) · [@royosherove on Twitter](https://twitter.com/royosherove) · [About Roy](https://robotpaper.ai/about/) - [Roy Osherove](https://osherove.com/) - [Courses](https://courses.osherove.com/) - [The Art of Unit Testing](https://artofunittesting.com/) - [5 Whys](https://5whys.com/) - [Elastic Leadership](https://elasticleadership.com/) - [Pipeline Driven](https://pipelinedriven.org/) - [RobotPaper.ai](https://robotpaper.ai/) - [RobotPaper Podcasts](https://podcasts.robotpaper.ai/) --- # Set up AJX and run a first useful trial AJX tests **how agents experience a product**. Give a fresh worker a task, observe the attempt, verify its outcome, and turn the friction into evidence-backed product improvements. A successful task can still reveal a difficult experience. If a user pointed you here, help them set up AJX and use it on their project. Start with one task and one attempt. Reuse any task, permissions, and cost limits they already supplied. This page bootstraps the **stable release**; its installed skill is the authority for running that release. [This guide as Markdown](/agents.md) · [Site overview](/index.md) · [All essential text](/llms-full.txt) · [Structured content map](/site.json) ## Check the prerequisites - **Installation:** macOS or Linux, ARM64 or x86-64, Bash, curl, and internet access. No GitHub login, Go compiler, Python runtime, or administrator access is needed for the default native installation. - **Agent access:** a host that can read files and execute shell commands. Read the installed skill directly if the host has not discovered its slash command. - **A real trial:** the user's project or public materials, one observable task, and a way to verify the outcome. A supported worker CLI and reporter need usable model authentication. A desktop chat login is not proof that the CLI is signed in. - **Supported worker adapters:** Codex, Claude Code, and Kiro CLI. Other hosts supporting the Agent Skills format can coordinate; they still need a supported worker adapter to launch an attempt. Check `command -v ajx` and run `ajx version` if available. Check the operating system and architecture before installing. Inspect existing skill paths without reading credential values. If shell access or a supported platform is missing, explain that blocker instead of claiming setup succeeded. ## Install the CLI and skill together Run this in Bash or Zsh for a new installation: ```sh bash -o pipefail -c 'curl -fsSL https://github.com/royosherove/ajx/releases/latest/download/install.sh | bash' && export PATH="$HOME/.local/bin:$PATH" ``` The installer downloads the native package from the latest stable public GitHub Release and verifies its checksum. It installs the CLI **and the complete matching skill**. Do not clone an unreleased source branch or copy only `SKILL.md` as an installation shortcut. Confirm the installation: ```sh "$HOME/.local/bin/ajx" version "$HOME/.local/bin/ajx" --help ``` Use the absolute path if your shell tool starts a new process for each command; an earlier `export PATH` may not persist. Report the actual installed version and locations. ### If an existing skill blocks installation Preserve customized files. If the user chooses to replace the conflicting installation, use this complete command. `--replace` backs up the conflicting files to adjacent `.ajx-backup-*` paths: ```sh bash -o pipefail -c 'curl -fsSL https://github.com/royosherove/ajx/releases/latest/download/install.sh | bash -s -- --replace' && export PATH="$HOME/.local/bin:$PATH" ``` Do not delete the old skill to bypass the installer. ### Update an existing managed installation Check without making changes: ```sh "$HOME/.local/bin/ajx" update --check --json ``` To install an available stable update with its matching skill: ```sh "$HOME/.local/bin/ajx" update --with-skills ``` Interactive `ajx update` offers the CLI and skill choices. Do not automatically downgrade a newer development build. ## Read the installed skill For the default installation, read: ```text ~/.local/share/ajx/current/skill/SKILL.md ``` Resolve relative references from that `skill/` directory. The installer registers the skill at `~/.agents/skills/ajx`, `~/.claude/skills/ajx`, and `~/.kiro/skills/ajx`. For a custom installation, use the paths printed by its installer. A session opened before installation may not list `/ajx` yet. Read the file and continue in the same chat when the host allows it. If the host requires a new session, give the user a resume prompt with the skill path, project, task, decisions made, and next step. A custom Kiro agent may need an explicit skill resource entry; follow the [harness guide](https://github.com/royosherove/ajx/blob/main/docs/HARNESS-SUPPORT.md). ## Prepare one real attempt 1. **Choose a small task.** Reuse the user's goal and materials. Keep the prompt neutral: do not audit the product or reveal suspected defects before the worker attempts it. 2. **Define success.** Use a deterministic check of the requested outcome. “The worker finished” and “a file exists” may not prove the task succeeded. 3. **Choose the conditions.** Start with one supported worker, one model, one reporter, and one attempt. Confirm filesystem and network access, time limits, and model costs. A local workspace is not a sandbox. Use the installed skill's environment guidance for local Docker isolation. 4. **Prepare and check the trial.** Follow the installed skill to write the briefing and trial files. Run `validate` and `doctor`; resolve missing tools, authentication, and verification checks before launch. 5. **Review and run.** Show the task, success check, access, and expected costs. Open the local AJX Console for review, or use the installed skill's chat approval flow. Start once the scope is authorized. Do the setup work the user authorized. Ask only for missing decisions or actions you cannot complete, such as provider sign-in. Never request pasted secrets or make paid model calls just to probe authentication. ### Direct CLI workflow These commands scaffold and check a trial. Run them from the project directory. Use a new trial folder if `trials/my-product` already exists: ```sh ajx init trials/my-product --product my-product --harness codex --no-input ``` **Stop to edit the scaffold:** set the real task in `trials/my-product/task-prompt.md` and configure the product materials, environment, permissions, and success checks in `trials/my-product/trial.toml`. Use `claude-code` or `kiro-cli` instead of `codex` when appropriate. Prefer a model supported by the selected harness; consult the installed release's help. Then check the configuration and open the console: ```sh ajx validate trials/my-product/trial.toml ajx doctor trials/my-product/trial.toml ajx console trials/my-product ``` The console lets the user review, start, and watch the trial. For an already authorized headless attempt, use `ajx run trials/my-product/trial.toml`. Do not interpret that command's exit code alone as verified task success. ## Read the result Open the generated matrix `index.html`, the run's `report.html`, and `pretty-journey.html`. Lead with the verified outcome and the most useful product asks. Explain the evidence and how the team can check a fix. - `index.html`: compare configurations, outcomes, and asks. - `report.html` or `asks.md`: proposed product changes with evidence. - `pretty-journey.html` or `journey.md`: forks, blocks, waits, and recoveries. - `matrix.json`, `run.json`, `measurements.json`, `events.jsonl`: structured results and trace evidence. Keep real reports local unless the user asks to share them. They may contain private project content. Explain blocked execution, incomplete reporting, or unconfirmed cleanup. For another product revision, use a new output directory; reusing an old output can resume its previous attempt. ## Try without model calls To explore AJX before configuring a worker, run this simulated console demo in a new directory: ```sh "$HOME/.local/bin/ajx" console demo trials/console-demo ``` Open the printed console URL and press **Check and start**. This demo uses no worker credentials or model calls. It does **not** test the user's project. For static fictional reports: ```sh "$HOME/.local/bin/ajx" example trials/example ``` Open `trials/example/index.html`. You can also browse the [website's sample matrix](/examples/), [sample report](/examples/runs/baseline-codex-r1/report.html), and [sample journey](/examples/runs/baseline-codex-r1/pretty-journey.html). All sample events, metrics, and outcomes are invented. ## Run an already briefed trial in CI Prepare the trial, deterministic checks, acceptance policy, worker tools, and noninteractive authentication first. Then run: ```sh AJX_RESULTS_DIR="$(mktemp -d)" ajx ci usability/trial.toml --output "$AJX_RESULTS_DIR" \ --policy usability/acceptance.toml --timeout-grace 60