AI Collaboration Format Reports

Troubleshooting

Key Report Features

  • Score
  • Playback
  • Submitted Code Review
  • Deep-Dive Question Generation

Score

Scores for the AI Collaboration format consist of two axes: Deliverable Evaluation and AI Score.

The score for individual problems that impact the total test score is reflected directly by the Deliverable Evaluation. The AI Score does not affect the total score; it is provided as reference information to understand the candidate's ability to utilize AI.

Deliverable Evaluation

Evaluation is based on the deliverables submitted by the candidate.

Requirement Fulfillment Evaluates how well the candidate addressed the requirements presented in the problem. Details can be expanded via an accordion menu for each test case, allowing you to check how accurately each requirement—such as request parameter specifications, expected response formats, and calculation logic—was implemented.

AI Score (β)

This score is an automated evaluation by AI of how the candidate utilized AI tools. The evaluation axes vary depending on the problem format.

  • Interactive Problems: Evaluated on three axes: "Communication Ability," "Problem-Solving Ability," and "AI Collaboration Ability." These can be visually reviewed via a radar chart alongside the overall score.
  • Non-Interactive Problems: Evaluated solely on "AI Collaboration Ability."

Each axis is further divided into detailed evaluation items. You can see which turn's prompt resulted in points added or deducted, along with the specific reasons for the evaluation.

For more details, please visit the AI Score Explanation Page.

Playback

This feature allows you to review the logs of the candidate's interactions with the AI agent during the exam. Click the "Open Playback" button to launch the dedicated viewer.

The viewer consists of the following elements:

Top Header Displays the total number of turns, token usage, and cost.

Left Panel Lists the prompts the candidate sent to the AI agent by turn. It shows the points added/deducted per turn, token usage, cost, and the number of tools used. Clicking a turn will navigate to its details. For interactive problems, the chat history with AI stakeholders can also be viewed in the "Chat" tab.

Turn Details / Tool Calls (Center) Review the candidate's prompt and the AI agent's response for the selected turn. The "Turn Details" tab shows the content of the interaction, while the "Tool Calls" tab shows details of the tools executed by the AI during that turn.

Evaluation Perspectives (Right Panel) Displays which AI score evaluation perspectives apply to the selected turn. Reasons for both positive and negative evaluations are provided, allowing you to review not just the final output but the process of how the candidate mastered the AI.

How to Review Submitted Code

There are two ways to review the code submitted by the candidate:

Review on HireRoo

You can view the code directly within the HireRoo interface by clicking the "Open Submitted Code" button on the report page.

Review on GitHub

To review code on GitHub, you must request access in advance. In the "Review on GitHub" section, click the "Request Access" button and enter the GitHub Account ID you will use for the review. Access to the target repository will سپس then be granted. Once access is granted, you can navigate directly to the Pull Request page on GitHub via the "Open PR (GitHub)" button.

Interview Preparation

Automated Deep-Dive Question Generation

Based on the candidate's submission, the AI automatically generates deep-dive questions that can be used during interviews. Because these questions are based on each candidate's specific answers and design decisions, they can be used directly for interview preparation.

How to View

The generated deep-dive questions can be viewed on GitHub. When you navigate to the Pull Request page via the "Open PR (GitHub)" button, you will find comments posted by hireroo-system linked to relevant sections of the candidate's implemented code. Each comment includes a priority label (e.g., P1 critical) and the question is provided in both Japanese and English.

For instructions on how to request GitHub access, please refer to the "Review on GitHub" section above.