Overview

Skill Assessment receives usage logs from the AI agents your engineers use (Claude Code, Codex CLI, and GitHub Copilot) and evaluates their AI Collaboration Skills. This article explains which logs we receive, what we use them for, and how long we keep them. Please use it for security reviews before adoption and for internal approval processes.

For a detailed item-by-item list, see Log Items Collected by Skill Assessment.

How It Works

Stage What happens
1. Sent from the machine The measurement CLI updates the AI agent's settings so that usage logs are sent to HireRoo
2. Masking Immediately after receipt and before temporary storage, values such as API keys and email addresses are replaced with ****
3. Temporary storage Only the items used for evaluation are stored temporarily, in masked form
4. Evaluation Once enough logs have accumulated, they are evaluated using an external AI service
5. Deletion and retention After evaluation, the temporarily stored logs are deleted, and the evaluation results and aggregated values are retained

Requests from AI agents to AI providers continue to go directly to the AI provider as before. HireRoo does not sit between AI agents and AI providers.

Settings Changed on the Machine

Running hireroo spider setup with the measurement CLI changes only the following settings.

AI agent Setting changed
Claude Code env in ~/.claude/settings.json
Codex CLI [otel] in ~/.codex/config.toml
  • Destination: https://spider-collector.hireroo.io/otel (HTTPS)
  • Authentication: A token issued per user is sent in an HTTP header
  • How to revert: Running hireroo spider uninstall restores the changed settings to their values before installation

Claude Code and Codex CLI send only OpenTelemetry logs (events). Metrics and traces are not sent.

GitHub Copilot (Copilot CLI and VS Code) does not emit logs, so it sends prompt text, tool calls, and token counts as traces. The same masking as for logs is applied to traces. Metrics are not accepted. For an item-by-item list for Copilot, please contact your customer success representative.

What We Receive and Use

The main items used for evaluation and aggregation are as follows.

Type Main items How they are used
Prompt text Instructions the engineer sends to the AI agent Evaluation
Tool calls Tool name, arguments (including file paths, commands, and edits), and approval decisions Evaluation, counts by language, and aggregation of MCP and skill usage
Usage Model name, token counts, cost, and timestamps Aggregation of usage
Session Session ID, and the type and version of the AI agent Aggregation of usage

The following items, even if sent, are discarded on receipt and never stored.

  • Attributes that identify users or organizations (email address, account ID, organization ID, host name, and similar)
  • AI response text
  • Tool output
  • Other attributes not used for evaluation or aggregation

Masking

Masking is applied to every received value immediately after receipt and before temporary storage. Any part that matches a pattern is replaced with ****. The following 25 patterns are covered.

  • API keys and tokens for various services (Anthropic, OpenAI, GitHub, Google, AWS, Slack, Stripe, npm, PyPI, GitLab, DigitalOcean, HireRoo)
  • Private keys
  • Credentials embedded in URLs (user:pass@)
  • Assignments such as passwords (password=, secret=, api_key=, *_token=, and similar)
  • Bearer tokens and JWTs
  • Email addresses
  • User names in home directories (/Users/<name>, /home/<name>, C:\Users\<name>)

Names, phone numbers, IP addresses, internal host names, and business data itself are not replaced unless they match a pattern.

Temporary Storage and Deletion

Items used for evaluation are stored temporarily in masked form. Temporarily stored logs are deleted in any of the following cases.

  • When evaluation finishes
  • 24 hours after the last log was received
  • 5 minutes after measurement is stopped

Sending Data to an External AI Service for Evaluation

Evaluation uses an external AI service (the OpenAI API).

Details
Sent Masked prompt text, tool call names and arguments, approval decisions, model name, and token counts
Not sent AI response text and tool output
Use for training Data sent is not used to train OpenAI's models. OpenAI states that it does not use data sent through the API for training unless the customer opts in (OpenAI "Enterprise privacy"), and HireRoo has not opted in

What Is Retained After Evaluation

Category Items
User User ID and company ID within HireRoo
Usage Call counts, token counts, and cost by tool and model; token counts by time of day; number of sessions; active days; total prompt length
Per-call records 16 items, including model name, token counts, cost, start and end times, prompt length, and session ID
Purpose and language Counts by usage category (code generation, review, debugging, and so on) and counts by language
Skills and MCP Skill name, source, and count; MCP server name, tool name, scope, and count
Evaluation results Scores by area, descriptions of the reasoning and observed facts (written by AI), and excerpts of the prompts used as the basis for the evaluation (masked)
  • Names and email addresses are not stored as Skill Assessment data.
  • Storage location: Google Cloud Tokyo region (asia-northeast1)
  • Retention period: Data is retained until you stop using the service.

Deployment in Your Own Environment

We plan to offer a configuration in which receiving, masking, temporary storage, and post-evaluation storage of logs all take place within your own network. In that configuration, logs are not sent to HireRoo.