Understanding the AI Collaboration Format

Overview

The AI Collaboration format is a question type where candidates work on tasks in partnership with AI tools.

All instructions given by the candidate to the AI Agent are recorded as logs. Based on these logs, you can evaluate the candidate's proficiency in using AI as AI Collaborative Skills. This allows for a comprehensive assessment of engineering capabilities, including not only implementation skills but also the AI utilization skills required in the AI era.

Two Question Types

Interactive Questions

In this format, the information required to complete the task is not fully specified in the problem statement beforehand.

Candidates interact with a virtual stakeholder (business representative) via a chat panel on the task screen to conduct interviews. They must proactively gather current conditions, constraints, and acceptance criteria to progress with the task. Similar to real-world projects, the information provided is incomplete and may contain contradictions.

You can view a list of Interactive questions here.

Non-Interactive Questions

In this format, the requirements, background, and constraints are presented in the problem statement from the start.

While the context and theme vary by problem, the requirements are generally set at a high level of abstraction, requiring candidates to make decisions and evaluate trade-offs. Fulfilling every requirement perfectly is not necessarily required; instead, candidates are expected to decide what to prioritize and what to compromise within the given constraints, and to document the rationale behind those decisions.

You can view a list of Non-Interactive questions here.

Test Environment and Available Tools

Candidates work on tasks in a VS Code environment that runs in the browser. The following tools are available:

Terminal: Execute commands from the Terminal at the bottom center of the screen.
GitHub Copilot Chat: Ask questions or consult with the AI via the chat panel on the right side of the screen.
Claude Code / Codex: Pre-installed in the development environment. These can be launched and used via GitHub Copilot Chat or the Terminal.
Messages (Interactive questions only): Communicate with AI stakeholders via the "Messages" section at the bottom left of the screen to extract requirements and other details.

AI Model Scope and Usage Limit Settings

In the AI Collaboration format, you can set the "Range of Available AI Models" and an "AI Usage Limit" for each question.

Configurable Items

AI Model Tier: Choose between "Standard" and "High Performance." The default for new tasks is "Standard."

AI Usage Limit: Can be set between 1 USD and 30 USD. The default for new tasks is 20 USD.

Candidate View

The available AI model tier and AI usage limit are displayed in the test rules on the candidate's screen.
The candidate's environment will reflect the configured model tier and usage limit.
The available models also apply to GitHub Copilot Chat within the test environment.

Model Tiers

Available models are subject to future updates. The following information is for reference as of July 9, 2026.

Tier	Description	Default Model	Available Models
Standard	A standard set of AI models for general business tasks, implementation, verification, and bug fixes.	claude-sonnet-5	claude-sonnet-5, gpt-5.4, claude-haiku-4-5, gpt-5.4-mini
High Performance	A set of high-performance AI models suitable for advanced implementation, research, design, and complex fixes.	claude-opus-4-8	claude-opus-4-8, gpt-5.5, claude-sonnet-5, gpt-5.4, claude-haiku-4-5, gpt-5.4-mini

Notes:

The "High Performance" tier includes all models available in the "Standard" tier.
The default OpenAI-based models are gpt-5.4 for Standard and gpt-5.5 for High Performance.
In Demo Tests, the available models are limited to claude-haiku-4-5 and gpt-5.4-mini, with an AI usage limit of 1 USD.

Question Limit

Only one AI Collaboration format question can be included per test. This applies whether you choose the Interactive or Non-Interactive type.

Evaluation

Evaluation for the AI Collaboration format consists of two pillars:

Deliverable Evaluation: Evaluating the candidate's actual output, such as code and documentation.
AI Collaboration Skills Evaluation: Evaluating the process of collaborating with AI—how the candidate provided instructions, verified outputs, and ensured quality.

For details on reports and evaluation metrics, please refer to the Report Explanation Page.