FairPrompt Take-home engineering assessments

Small invited pilot · free for candidates

Candidates are tested by AI agents. Are they tested with them?

FairPrompt runs practical take-home engineering assessments. The candidate directs an AI coding agent on a real-world task, and you see how they decide, check and correct its work.

Request pilot access How it works

  • A dedicated computer for each attempt
  • Every rating cites its evidence
  • A person makes the hiring decision

Conversation · Candidate S-01 (synthetic)

Agent

All 8 tests pass. The retry fix is done.

Before we call it done: make a delivery fail, retry it, and show me it’s sent once.

Turn T5 · 0:30

Agent

The retry sent the message twice. Fixing that first.

Assessment · reviewer only

Verifies agent claims Strong

Didn’t accept “done”. Asked for a failure-then-retry check, which exposed a duplicate send.

Evidence: Turn T5 · 0:30
Fig. 1 A rating that cites the candidate’s own message. Illustration with synthetic data.

How it works From invitation to follow-up questions, in four steps.

  1. Invite

    Invite a candidate, by email address, to one real-world task. A computer starts for this attempt only.

    A dedicated, disposable Linux VM.

    Invitation

    Notification recovery

    One take-home task

    Start attempt

    Launching the starter workspace…

    Linux VM started for this attempt

    Workspace ready

    Ready for your next instruction

    Fig. 2 The invitation, then the launch. Illustration.

  2. Work with the agent

    In the browser, the candidate gives the agent instructions, answers its questions and checks its work. Submitting freezes everything, and the computer is destroyed.

    A FairPrompt web page drives the Codex app-server on that VM.

    FairPrompt Notification recovery Submit work Submitted
    Agent

    The retry now sends once. I added a test for a failed delivery followed by a retry.

    Review changes 2 9 tests passed

    notification.py +3 −1 

    - send(message)
+ if message.id not in sent_ids:
+     send(message)
+     sent_ids.add(message.id)
    Ask, plan, or steer the agent… Send
    Submitted · read-only Conversation, files, changes, and tests remain available.

    Linux VM running Linux VM destroyed

    Fig. 3 The candidate checks the agent’s work, then submits. Illustration.

  3. Assessment

    The work is assessed against a rubric. Every rating links to the evidence behind it: a message, a change or a test run.

    FairPrompt Submission review Reviewer only

    Candidate S-01 (synthetic)

    Notification recovery · Agent assessment ready for human review

    Decision-quality profile

    • Frames the problem Turn T1 Strong
    • Chooses a defensible policy notification.py Strong
    • Verifies agent claims Turn T5 · 0:30 Strong
    • Adapts to evidence Test run 3 Adequate
    • Understanding Until follow-up Not observed
    Fig. 4 Each rating cites its evidence. Illustration with a synthetic candidate.

  4. Human review

    A person checks each rating, agrees or changes it, and chooses follow-up questions about the candidate’s real decisions. People make the hiring decision.

    FairPrompt Submission review Finding 3 of 5

    Verifies agent claims Agent: Strong

    Didn’t accept “done”. Asked for a failure-then-retry check, which exposed a duplicate send.

    Agree with rating Change rating

    Agreed Strong Recorded by you

    Fig. 5 A person confirms the rating and picks a follow-up question. Illustration.

Candidates don’t need an API key. Or an AI subscription. The pilot is free for candidates.

Planned

Employer-funded

You pay for the coding agent’s requests through your own API access.

Not available yet

Bring your own

Candidates sign in with ChatGPT and use their own plan. We’re applying to OpenAI for this.

Their token stays outside the candidate’s command execution, is never kept in assessment evidence, and is deleted when the VM is destroyed after submission.

Want to try it? The pilot is small and by invitation.

Request pilot access