AI Monitoring in Skill Assessment now lets you drill down from the whole organization to teams and then to individual members. In addition to Claude Code and Codex, you can now measure usage of GitHub Copilot.

Skill Assessment is currently offered as a beta. For details, see Viewing the AI Monitoring Dashboard.

Background

Until now, AI Monitoring had only two levels: the organization-wide summary and each member's screen. Decisions about AI adoption are often made per department or team, yet there was no way to compare how each team was doing.

In addition, only Claude Code and Codex could be measured, so organizations that mainly use GitHub Copilot could not capture their actual AI usage.

What's new

Teams and members on the AI Monitoring Overview

We added Teams and Member List to the AI Monitoring Overview. Both are ordered by token usage in the selected period.

Teams are based on the groups created in Company Settings. In the Teams list, you can see each team's headcount and number of active members, rank, score, and token usage. If no groups exist yet, a prompt to set up groups is shown.

Team details

Select a team in the Teams list to open Team details. There you can compare the following with the organization average.

  • Team score: the team's rank and score, change from the previous period, and the score for each SEI (Software Engineering Index) metric
  • KPIs: active members, requests, AI tool cost, and token usage
  • AI activity: cache hit rate, sessions, max concurrency, usage breakdown, languages, active hours, and more
  • AI tool usage, Skill / MCP usage, and Rank distribution
  • Members: the team's members in order of token usage

For teams with fewer than two evaluated members, the team score and rank distribution are not shown. This prevents individual members' scores from being inferred from the team score. Usage items are shown as usual.

As with the AI Monitoring Overview, members with Admin Authorization or Manager Authorization can open Team details.

Measuring GitHub Copilot usage

You can now measure GitHub Copilot usage in the same way as Claude Code and Codex. Supported surfaces are GitHub Copilot CLI and the GitHub Copilot extension for Visual Studio Code.

Measured usage appears as "GitHub Copilot" in AI tool usage, Skill / MCP usage, and other sections.

To start measuring, update the hireroo CLI on the member's machine to the latest version and run hireroo spider setup.

  • Members setting up for the first time only need to follow the usual setup; GitHub Copilot is measured as well.
  • Members who have already completed setup should update the CLI with hireroo upgrade and run hireroo spider setup again.
  • After setup, measurement starts in GitHub Copilot CLI once a new terminal is opened, and in Visual Studio Code once the window is reloaded.

Scope

  • Measurement of GitHub Copilot CLI does not currently support Windows. The GitHub Copilot extension for Visual Studio Code can be measured on Windows.
  • Teams are based on the groups in Company Settings. To create groups, see Manage Members in Groups.

Availability

A Skill Assessment contract is required to use Skill Assessment. For details, please contact your Customer Success representative.