What is SEI (Software Engineering Index)?
On this page
SEI (Software Engineering Index) measures an engineer's skills from the behavior observed during their work. Hiring and development teams use it to compare candidates and employees against the same yardstick. This page explains what SEI evaluates and how, and how to read the score, rank, and evaluation coverage.
What is SEI?
SEI is the name of the Skill Index that HireRoo provides as its standard. Its official name is Software Engineering Index. A Skill Index your company creates is built on this SEI. The score in the new report is based on your company's published Skill Index if you have one, and on SEI otherwise.
SEI evaluates the process of the candidate's work. For example, it looks at how the candidate framed the task, what they delegated to AI, and how they verified AI's output. Deliverable Evaluation (requirement fulfillment rate) is available separately from SEI.
Skill Interview (skills evaluation at hiring) and Skill Assessment (evaluation of in-house engineers' AI use) use the same SEI. The skills you see at hiring and the skills you see after the engineer joins are connected by the same yardstick.
Terms used on this page
SEI is built as a tree of skills and evaluation items. This page uses the following terms.
| Term | Meaning |
|---|---|
| Skill Index | The whole system of skills and evaluation items used for evaluation. There is the HireRoo standard and the Skill Index your company creates (your company's) |
| Skill | A summary of the results of the skills and evaluation items beneath it. There are five skills at the top level, with more detailed skills below them |
| Evaluation item | The actual behavior that is evaluated. Each has five levels of evaluation criteria |
The "evaluation axes" of AI Collaboration Skill refer to the skills beneath AI Collaboration Skill.
The five skills SEI evaluates
SEI evaluates the following five skills.
| Skill | What it looks at | Main skills beneath it |
|---|---|---|
| AI Collaboration Skill | Whether the candidate decides what to delegate to AI, examines its proposals, and leads verification and improvement | Environment Engineering, Intent Specification, Co-Reasoning, Execution Control, Output Quality Assurance |
| Problem-solving Skill | How the candidate frames and solves problems | Requirements Analysis, Problem Decomposition, Hypothesis Testing, Logical Thinking, Decision-making |
| Communication Skill | Whether the candidate communicates accurately and understands accurately in dialogue | Intent Understanding, Clear Writing, Confirmation and Summarization |
| Computer Science Fundamentals | Whether the candidate recognizes when data structures and algorithms are effective, chooses them, and covers the key points | Programming Fundamentals, Data Structures, Algorithms |
| System Design Skill | Whether the candidate makes the design decisions that matter for the requirements and shows the reasons and main trade-offs | Architecture, API Design, Data Design, Scalability, Reliability, Security |
AI Collaboration Skill
AI Collaboration Skill looks not at whether a candidate finished the task with AI, but at whether the candidate used AI while taking responsibility for decisions and quality. It consists of the following five evaluation axes and the 13 evaluation items beneath them.
| Evaluation Axis | Behaviors Evaluated |
|---|---|
| Environment Engineering | Prepare tools and context so that AI can work appropriately, and establish rules for safe execution |
| Intent Specification | Frame the problem in the candidate’s own words and communicate constraints and completion criteria clearly |
| Co-Reasoning | Elicit alternatives from AI and decide whether to adopt them based on rationale and trade-offs |
| Execution Control | Adjust the scope delegated to AI, steer work during execution, and guide convergence after failures |
| Output Quality Assurance | Design verification methods, critically assess AI output, and further improve quality |
For detailed definitions of each axis, see AI Collaboration Skills | SEI.
Problem-solving Skill and Communication Skill
Problem-solving Skill looks at how the candidate frames and solves problems. It evaluates behavior such as whether the candidate identifies unclear points on their own and asks questions, breaks the problem down and handles it in order, and chooses a realistic approach that accounts for trade-offs.
Communication Skill looks at how the candidate communicates and understands in dialogue. It evaluates behavior such as whether the candidate accurately grasps the other person's intent, writes key points clearly, and confirms assumptions and gaps in understanding.
Both are evaluated from the conversation with the persona in the test (the party who plays a role such as the client who made the request).
Computer Science Fundamentals and System Design Skill
When AI writes the code, whether the code runs correctly alone does not differentiate candidates. These two skills therefore look at understanding and judgment, not the correctness of the implementation.
- Computer Science Fundamentals: Looks at whether the candidate recognizes when data structures and algorithms are effective and chooses them. It also looks at whether the candidate covers key points such as boundaries, computational complexity, and empty inputs.
- System Design Skill: Looks at whether the candidate makes the design decisions that matter for the requirements. It also looks at whether the candidate shows the reasons for those decisions and the main trade-offs.
The following rules apply to the evaluation.
- The evaluation target is decided by whether the approach is effective for the situation. If the candidate stays with brute force where a better approach is effective, the evaluation is low.
- If the candidate uses a different approach that is equally suitable, there is no deduction.
- System Design Skill is evaluated only when the perspective is needed for the requirements of the task. A comprehensive coverage of every detail is not required.
How SEI evaluates
Evaluation is based on observed behavior
SEI evaluates based on behavior actually observed during the test. For tests that use an AI agent, it is handled as follows.
| Used as evidence | Kept as reference only |
|---|---|
| The candidate's own statements and actions (instructions to AI, checking AI's output, explanations to the persona, and so on) | Code written by AI, test results |
If code written by AI were used as evidence, the quality of the AI would be counted as the candidate's ability. SEI therefore uses only the candidate's own statements and actions, where their judgment shows, as evidence.
Each evaluation item has five levels of criteria
Each evaluation item has five levels of evaluation criteria, L0 to L4. For each evaluation item, SEI determines which level the candidate reached.
Evaluation is combined per test
SEI evaluates per test, not per problem. When a test has multiple problems, the results of all AI Collaboration problems are combined into a single score, rank, and evaluation coverage.
If each problem were evaluated separately, skills that cannot be evaluated in a given problem would be handled inconsistently from problem to problem. Combining the whole test produces a single evaluation across problems.
How to read the score, rank, and evaluation coverage
Score
The score ranges from 0 to 100. It combines the levels reached on the evaluation items according to their weights at each level of the skill hierarchy. It shows how far the candidate reached against the evaluation criteria.
Besides the overall score, each of the five skills and the skills beneath them also has a score.
Rank
The rank is the candidate's position relative to other candidates.
| Rank | Criteria |
|---|---|
| S | Top 5% |
| A | Top 25% |
| B | Top 50% |
| C | Top 75% |
| D | Below that |
- The rank and top percentage are determined when the report is generated. They do not change afterward.
- When fewer than 30 candidates were compared, the rank and top percentage are not shown. With so few candidates, a difference of one candidate moves the rank a lot.
The score shows how far the candidate reached against the criteria, and the rank is a comparison with other candidates. Depending on the distribution of candidates at the time the report was generated, the same score can result in a different rank.
Evaluation coverage
Evaluation coverage shows how many of the Skill Index's evaluation items could be evaluated. For example, if 40 of a Skill Index's 100 evaluation items could be evaluated, the coverage is "40 / 100." Coverage rarely reaches 100%.
Items that could not be evaluated are not included in the score calculation. They are not counted as zero, either.
Read the score and evaluation coverage separately
The score shows "how far the candidate reached against the criteria within the range that could be evaluated." Evaluation coverage shows "how wide a range the score is based on."
When coverage is narrow, read the score as a result for only the range that could be evaluated. For example, a Computer Science Fundamentals score reflects judgment within the range of data structures and algorithms that the test asked about. A high score with narrow coverage does not mean that every skill is high.
Skills evaluated vary by problem
Skills whose prerequisites are missing are not evaluated. For example, in a problem with no conversation with a persona, Problem-solving Skill and Communication Skill, which are evaluated from conversation, cannot be evaluated.
| Condition | Skills not evaluated |
|---|---|
| Algorithm problem | System Design Skill |
| System design problem | Computer Science Fundamentals |
| Problem with no conversation with a persona | Problem-solving Skill, Communication Skill |
Skills that do not fall under the conditions above are also not evaluated if there is no opportunity to evaluate them. For example, if a candidate gives AI no instructions and leaves everything to it, Computer Science Fundamentals is not evaluated.
Skills whose prerequisites are missing are not included in the calculation, so adding a problem that cannot measure a skill never lowers SEI.
Trace the evidence
To find out why a score is what it is, drill down through the skill hierarchy from the top.
- Use the overall score and evaluation coverage to grasp the overall level and how wide a range it is based on.
- Use the five skills and the skills beneath them to find strengths and weaknesses.
- Use the evaluation items to check which level the candidate reached.
- Use the judgment for each problem to check which behavior at which point served as evidence. You can see the reason for the judgment and the observed facts. Evaluation items that had no opportunity to be evaluated show "Unobserved."
The behavior that served as evidence can also be used to probe deeper in interviews. For example, for an evaluation item with a low rating, you can ask the candidate why they made that decision at that point.
Relationship to your company's Skill Index
If your company has created and published its own Skill Index, the score from your company's Skill Index is shown first. Otherwise, the score from the HireRoo standard Skill Index (SEI) is shown.
Even when your company has its own Skill Index, evaluation against the HireRoo standard is also performed. You can switch between your company's and the HireRoo standard to compare the skill hierarchy. Because the HireRoo standard score is not affected by the contents of your company's Skill Index, it can be used as a common yardstick with other companies.
A draft Skill Index that has not been published is not used for evaluation. An evaluation uses the Skill Index published at the time it is evaluated, so tests evaluated before publication do not receive a score from your company's Skill Index.