We Offer an AI-Powered Technical Recruiting Tool

Our team had a busy September. We expanded our team by hiring two engineers, an engineering intern, and a lead generation expert. Since it takes time to onboard new team members, we didn't plan too much for September. Despite that, we successfully delivered two features that we had been planning for a long time. Additionally, we moved our blog posts to a new platform called Ghost, which has improved the writing experience. Previously, we used Sanity, but I may write another blog post explaining why we made the switch. All in all, it was a productive month for us.

Long story short, let's jump to our big updates.

Recording Of User Activity

Brokee is a platform that offers technical skills assessments to potential hiring candidates. The candidates are given access to live IT infrastructure and are required to fix a broken application to showcase their technical skills with specific technical stacks. Previously, we provided a text-based history of the candidate's user activity that showed which commands they used to troubleshoot and fix various problems with the application's configuration, environment, or security.

This approach was helpful in providing a quick glimpse of how the candidate was attempting to fix IT infrastructure issues. It allowed us to see where candidates got stuck for 5-20 minutes and needed to spend time reading documentation or searching for potential fixes on the internet. However, it lacked some depth as we couldn't see what was happening when the candidate was editing a file. This made it difficult to judge if specific changes were relevant or if the candidate was just looking at a file without making any changes. We do provide automated evaluations of each test to ensure that the application is fixed at the end of the test, but it's still important to know exactly how the problem was solved.

So, we've integrated asciinema into our testing environment. Now, you can replay the whole user activity with a click of a button.

While this is great, nobody wants to spend 30-60 minutes rewatching the terminal activity of every hiring candidate, so we suggest the following approach for technical team members to spend the least amount of time analyzing user activity:

• Take a 1-minute look at the history of user commands.
• If you see something interesting or unusual, look at the time stamp, and jump to the specific time in the recording to understand what was happening in detail.

This way, you can quickly understand how exactly candidates solved specific problems in a fast and efficient manner.

AI-Generated Summary of Candidate Performance

Another major update on Brokee was the integration of AI into automated evaluations of candidate performance. While the history of user commands and replaying of user activity can help technical team members understand how every candidate fixes problems, every engineer has different skill sets and experience levels. As a result, less technical members of the hiring team may not be able to understand user activity, making textual and video representations insufficient to make informed decisions about the best hire. To address this, we asked ourselves how we could make the results more accessible to everyone on the hiring team?

Meet AI summaries of candidate performance:

As we are gathering data on user activity, we can use it to generate a structured summary with easily understandable explanations of what occurred in the testing environment. This allows everyone to quickly review the report and have a common understanding of which tools the candidate used to troubleshoot problems, along with their general purpose (such as networking configuration, working with files, security configuration, etc.). Additionally, the report highlights any changes made to the environment, and provides details on specific configurations used. As there are multiple ways to fix the same problem, even experienced engineers can learn something new by reviewing the AI-generated summary, especially if the candidate used a new tool to resolve an issue.

This is great because our automated evaluations can serve as performance reports as well as educational materials for the entire team. If you have multiple candidates who have completed skills assessments, you can refer to this report to identify the most efficient and creative problem solvers for issues with IT infrastructure.

Reporting Recap

This is just the beginning. We plan to provide more advanced reports in the future, but just to recap, here's what Brokee offers right now in terms of reporting:

• Test Completion Status: Passed/Failed
• Time Taken To Complete Assessment
• Number of Completed Subtasks
• Automated Checks
• AI-Generated Summary of User Activity
• History of Commands
• Recording of Terminal Activity

We strive to be your top choice in hiring DevOps and Cloud engineers. If you're planning to hire or need help assessing candidates, please contact us via our website, and don't forget to follow us on Linkedin and Facebook.