Skill Assessment continuously measures the AI Collaboration Skills of your engineers from the usage logs of the AI tools they already use in their daily work, and visualizes those results relative to the rest of the organization.

Skill Assessment measures engineers in their normal working environment. Engineers do not need to move to a special editor or exam environment; they simply keep using the AI tools they already use.

The Five AI Collaboration Skills Axes

Skill Assessment evaluates the same five axes used in AI Collaboration format reports.

Evaluation Axis Behaviors Evaluated
Environment Engineering Prepare tools and context so that AI can work appropriately, and establish rules for safe execution
Intent Specification Frame the problem in the engineer's own words and communicate constraints and completion criteria clearly
Co-Reasoning Elicit alternatives from AI and decide whether to adopt them based on rationale and trade-offs
Execution Control Adjust the scope delegated to AI, steer work during execution, and guide convergence after failures
Output Quality Assurance Design verification methods, critically assess AI output, and further improve quality

For detailed definitions of each axis, see AI Collaboration Skills | SEI. For the thinking behind the evaluation criteria, see also July 2026 | Updated AI Collaboration Skills Evaluation Criteria to Better Assess the Depth of AI Use.

What Is and Is Not Used for Evaluation

Skill Assessment evaluates only the prompts an engineer writes and the tool calls made by the AI agent.

AI responses and tool execution results are not used for evaluation. This prevents scores from rising simply because AI models become more capable. By limiting evaluation to the human side of the interaction, changes in a score reflect changes in how the engineer actually works.

What You Can See

Skill Assessment provides views for administrators and a separate view for the engineers being measured.

For Administrators

Screen What you can review
Dashboard Organization average score, active rate, AI tool spend, token usage, AI usage (request count, token consumption, average prompt length), usage categories, languages, working hours, rank distribution, and members whose scores are growing fastest
Member list Usage status per member, such as primary AI tool and last log time. Search by name or email, and filter by status, token usage, or cost
Member detail Tool usage ratio, usage categories, and activity

Administrator screens currently do not show individual members' scores, ranks, trends, or score breakdowns, and the member list cannot be filtered by rank or score. Usage information (token usage, cost, tool usage, working hours, and AI usage) is still shown.

For Engineers Being Measured

Each engineer can review their own overall score, rank, the breakdown of the five axes, and the observed facts behind the evaluation. They can also review their own usage over the past 30 days (active days, request count, token consumption, usage categories, languages, working hours, and more).

Individual scores of other members are not shown to engineers. They see only their own results.

Getting Started

1. An administrator invites the members to be measured

From the Skill Assessment screen, invite the members you want to measure by email address. Employees already registered with HireRoo can be assigned directly without sending an invitation.

2. Members install the hireroo CLI

Following the guidance on their own page, each invited member runs two commands on their machine.

  1. Install the hireroo CLI
    • macOS / Linux: curl -fsSL https://cli.hireroo.io/install.sh | sh
    • Windows (PowerShell): irm https://cli.hireroo.io/install.ps1 | iex
  2. Run hireroo spider setup and approve the authentication shown in the browser

There is no need to copy or paste a token manually. Existing Claude Code / Codex environments are not affected.

3. Measurement begins

After setup, measurement starts as soon as the member uses their AI tools as usual. Progress is shown on the member's own page in three steps.

  1. Install the CLI — setup is complete
  2. Confirm connectivity — the member sends a message in Claude Code or Codex, and the first usage log reaches HireRoo
  3. Wait for measurement to start — the first evaluation result has been generated

Once the third step is complete, the score appears on the dashboard. Scores are then updated as usage accumulates.

Supported Environments

Support
OS macOS / Linux / Windows
AI tools Claude Code / Codex CLI

The type of AI tool credentials or subscription (API key, subscription, Amazon Bedrock, or Google Vertex AI) does not matter. Measurement works with your existing contracts as they are.

How Data Is Handled

AI Tool Behavior Is Not Affected

Skill Assessment does not sit between AI tools and AI providers. Only usage logs are sent to HireRoo; requests to AI are sent directly to the AI provider as before.

As a result, an incident on the HireRoo side does not stop your engineers from working. The only impact is that usage logs for that period are not recorded.

How Prompt Contents Are Handled

Prompt contents and the contents of tool calls are held in temporary storage only for as long as needed to calculate scores, and are removed from temporary storage once calculation finishes.

However, excerpts of the prompts used in the evaluation are stored with the scores as the basis for the evaluation. Other stored data consists of information that does not include any content (token counts, cost, timestamps, the tool used, prompt length, and similar) and the calculated scores. Personal information such as email addresses is removed as soon as logs are received.

Deployment Options

Skill Assessment is currently available in the form where HireRoo hosts the measurement infrastructure (the cloud option). For organizations that cannot allow logs to leave their own environment, we plan to offer an option in which the measurement infrastructure is installed inside your own cloud environment. With that option, prompt contents never leave your environment, and only calculated scores and aggregated values are sent to HireRoo.

Points to Note

  • Measurement depends on configuration on the engineer's machine. If a member changes their AI tool configuration or uninstalls the CLI, measurement stops. Because AI tools continue to work normally in that case, the member is unlikely to notice. You can check whether logs are arriving from the last log time in the member list.
  • Scores are not updated during periods with no AI tool usage. Scores update once a certain volume of usage logs has accumulated, so members with light usage will see longer intervals between updates.

About the Beta

Skill Assessment is currently offered as a beta. We will continue to refine the definition and weighting of the evaluation axes and the score calculation based on customer data.

If you would like to adopt Skill Assessment, please contact your customer success representative or reach out through our contact form.