Overview

Skill Assessment evaluates AI Collaboration Skills from the usage logs of the AI tools you use every day. This article explains which data is sent, what it is used for, and what is retained, as well as who can see the measurement results.

About the beta Skill Assessment is currently offered as a beta.

For a detailed explanation written for company administrators, see How Skill Assessment Measurement Logs Are Handled.

How It Works

Stage What happens
1. Sent from the machine The AI tools configured by the hireroo CLI send usage logs to HireRoo
2. Masking Immediately after receipt and before temporary storage, values such as API keys and email addresses are replaced with ****
3. Temporary storage Only the items used for evaluation are stored temporarily, in masked form
4. Evaluation Once enough logs have accumulated, they are evaluated using an external AI service
5. Deletion and retention After evaluation, the temporarily stored logs are deleted, and the evaluation results and aggregated values are retained

Requests from AI tools to AI providers continue to go directly to the AI provider as before. HireRoo does not sit between AI tools and AI providers.

What Is Sent and Used

The main items used for evaluation and aggregation are as follows.

Type Main items How they are used
Prompt text Instructions you send to the AI tool Evaluation
Tool calls Tool name, arguments (including file paths, commands, and edits), and approval decisions Evaluation, counts by language, and aggregation of MCP and skill usage
Usage Model name, token counts, cost, and timestamps Aggregation of usage
Session Session ID, and the type and version of the AI tool Aggregation of usage

AI response text and tool output are not used for evaluation. Limiting evaluation to human decisions means that changes in the score reflect changes in how you work.

What Is Discarded

The following items, even if sent, are discarded on receipt and never stored.

  • Attributes that identify users or organizations (email address, account ID, organization ID, host name, and similar)
  • AI response text
  • Tool output
  • Other attributes not used for evaluation or aggregation

Masking

Masking is applied to every received value immediately after receipt and before temporary storage. Any part that matches a pattern is replaced with ****.

  • API keys and tokens for various services, and private keys
  • Credentials embedded in URLs, and assignments such as passwords (password= and similar)
  • Bearer tokens and JWTs
  • Email addresses
  • User names in home directories (/Users/<name> and similar)

Names, phone numbers, IP addresses, internal host names, and business data itself are not replaced unless they match a pattern. For the full list of patterns, see How Skill Assessment Measurement Logs Are Handled.

Temporary Storage and Deletion

Items used for evaluation are stored temporarily in masked form. Temporarily stored logs are deleted in any of the following cases.

  • When evaluation finishes
  • 24 hours after the last log was received
  • 5 minutes after measurement is stopped

Evaluation uses an external AI service (the OpenAI API). What is sent includes masked prompt text and tool calls; AI response text and tool output are not sent. Data sent is not used to train OpenAI's models.

What Is Retained After Evaluation

  • Usage aggregates (call counts, token counts, and cost by tool and model; token counts by time of day; number of sessions; active days; and so on)
  • Per-call records (model name, token counts, cost, timestamps, prompt length, and so on)
  • Counts by usage category (code generation, review, debugging, and so on) and by language
  • Skill and MCP usage counts
  • Evaluation results: scores by area, descriptions of the reasoning and observed facts (written by AI), and excerpts of the prompts used as the basis for the evaluation (masked)

Names and email addresses are not stored as Skill Assessment data.

Who Can See the Results

Viewer What they can see
You On My Page: your own score, rank, scores and observed facts for each evaluation axis, and usage
Administrators and managers Organization-wide and team aggregates (organization average score, rank distribution, usage, and so on), and usage for each member (tokens used, cost, primary AI tool, last log, usage breakdown, active hours, and so on)
Other members Nothing
  • Administrator and manager screens currently do not show individual members' scores, ranks, trends, or score breakdowns.
  • Team scores are shown only for teams with two or more evaluated members, so that an individual member's score cannot be worked out from the team score.
  • For what is shown on My Page, see Viewing Your Measurement Results on My Page.