Overview

You can create your own AI Collaboration questions and use them in your coding tests and in internal tests in Skill Development. You write the problem statement, the evaluation criteria, and the code given to candidates inside a browser-based authoring environment, together with an AI agent.

Only your company can view and use the questions you create. They are not visible to other companies. Taking, evaluation, and reports work exactly as they do for the AI Collaboration questions HireRoo provides.

Available plans Available on the Professional plan or higher. Using a question in an internal test also requires a Skill Development contract.

From creation to publication

Step What happens
1. Create Choose the question format and the technical stack, then open the authoring environment
2. Author Write the problem statement, the evaluation criteria, and the code in the authoring environment
3. End The contents of the authoring environment are collected and become the version being edited
4. Verify Solve the question yourself in the same environment a candidate gets, and check that it works
5. Publish Set the difficulty and the expected time limit, and publish. The question becomes available for coding tests and internal tests

Creating a question

  1. Open Questions in the sidebar and switch to the Your questions tab.
  2. Click Create a question.
  3. Choose the question format and the technical stack, then click Create a question.

There are two question formats.

Question format What the candidate does
Solved from the statement alone The candidate reads the problem statement and solves it
Solved by drawing information out through chat The candidate chats with a counterpart and draws out the information they need while solving it

The technical stack becomes the runtime environment at the time of taking. You can choose Go, TypeScript / Node.js, Python, Java, Ruby, C#, or PHP. For a question that does not involve writing code, choose Documentation only.

The question format and the technical stack cannot be changed later. If you need different ones, create the question again.

Once created, the authoring environment starts up. It takes a few minutes to become ready.

Authoring in the environment

The authoring environment is a browser-based development environment that runs on the same image candidates take the question in. It has an AI agent, so you can describe what you want and have the agent write it.

You write the following files. Detailed instructions are in Authoring guide on the left of the screen and in README.md inside the environment.

File Contents
title/ The question name
summary/ The summary shown on the question selection screen
description/ The problem statement shown to candidates
testcases/ The evaluation criteria. One directory per criterion
starter/ The initial code given to candidates
solution/ The reference implementation. It is used in evaluation
interactive-agent/ The counterpart and its reference material, for the chat format

The problem statement and the evaluation criteria are required in both Japanese and English. Write the Japanese yourself and have the AI agent produce the English. A question cannot be published while either is missing.

The authoring environment has a time limit of 24 hours. When it runs out, the contents are collected and the environment is closed, so nothing you have written is lost. To continue, open the authoring environment again from the question detail screen.

Ending an authoring session

Click End at the top right of the screen. The contents of the authoring environment are collected, the version being edited is created, and the environment is closed.

The question detail screen shows what that version is missing. To fill in the gaps, open the authoring environment again from Resume editing.

Verifying before publishing

Before you can publish, you have to solve the question yourself in the same environment a candidate gets.

  1. Click Verify question on the question detail screen.
  2. An environment opens with the initial code given to candidates, exactly as a candidate receives it. Solve the question.
  3. Tell the AI agent in the environment to grade your work against the criteria. It reports whether each criterion passed and why. You can run it as many times as you like.
  4. Once you are satisfied that the question works, click Complete verification. To start over, choose Abandon.

Nothing you write in the verification environment is kept. To change the question itself, finish verifying and then open the authoring environment again.

Verification applies to one version. If you author the question again after verifying it, the result is a new version and it has to be verified again.

Publishing

  1. On the question detail screen, show the version being edited.
  2. Click the button to publish that version.
  3. Enter the difficulty and the expected time limit, and publish.

Once published, the question can be added to coding tests and to internal tests in Skill Development. Even after publication, only your company can see it and use it.

Editing after publication

A published question can be changed as well: open the authoring environment from Edit, make your changes, and publish them as a new version.

Coding tests that have already been delivered keep the previous version. To use the new version, select the question again in a coding test.

To abandon an edit in progress, choose to discard it. The published version stays as it is.

Deleting and archiving

State Action
Not yet published Delete draft deletes the question itself
Published Archive question removes it from the questions you can use

Archiving cannot be undone. If candidates have already taken a test containing an archived question, you can still view those reports.

Limitations

  • Only AI Collaboration questions can be created. Other question formats are not supported.
  • You can hold up to three authoring and verification environments at a time, counted together. When you reach the limit, end a session in progress before opening another.
  • The time limit is 24 hours for both the authoring environment and the verification environment.
  • Questions you create cannot be shared with or published to other companies.