Challenge frontier model with your genuine expertise in a field.
Publicada el 2026-07-20
Descripción de la oferta
## Project Overview You will author a single terminal-based task and evaluate whether an AI coding agent can solve it end-to-end. Your goal is to design a task that is genuinely hard for frontier AI agents to solve, strictly verifiable, and resistant to shortcuts. The agent receives your instructions as its prompt, operates inside a Docker container with only shell tools, and writes its output to `/app/`. A deterministic `pytest` verifier then grades the run. > **CRITICAL WARNING:** You must write the prompt (`instruction.md`) yourself, in your own voice. Do not use AI to write or edit your instructions. The rubric heavily penalizes AI-generated instructions. You may use AI for everything else (coding, testing, etc.). Please don't send me automatic offers using Ai either. ## Deliverables You will produce a single task directory containing four core files that define what the agent does and how it is graded: * **`instruction.md`**: The prompt the agent reads at the start of the task. * **`task.toml`**: The task's metadata and resource configuration. * **`solution/solve.sh`**: A reference solution proving the task is solvable and the tests accept a correct answer. * **`tests/test_outputs.py`**: The pytest assertions that grade the agent's output. ## Workflow and Phases This is a procedural workflow. You will move through these phases in strict order to minimize wasted effort. 1. **Proposal Phase:** Approval required before coding. You will draft `instruction.md` and submit it for a proposal check. We will iterate on the instruction until the reviewer returns an Accept status. This catches bad task ideas before you invest time in building the tests and solutions. 2. **Implementation Phase:** Building and refining. You will complete the remaining files and submit them for validation, quality checks, and a cheat trial. Findings from each step will route back to you for edits. You will iterate until validation stays green, the quality review is clean, difficulty lands in range, and the cheat trial finds nothing. > **Note on Difficulty:** Because this project targets problems that are hard for frontier models, do not expect an AI (like Opus or Claude Code) to produce the oracle solution for you. If Claude Code can write a working `solve.sh` from your instructions without substantial back-and-forth, the task is too easy and the proposal must be revisited. ## Task Categories If a task could credibly land in two categories, pick the one that describes what the agent actually has to do to solve it. | Category | Scope | | --- | --- | | **`software_engineering`** | Writing code, fixing bugs, refactoring, migrations, build and dependency work. | | **`system_administration`** | Shell, processes, filesystems, packaging, configuration, operational tooling. | | **`security`** | Vulnerability analysis, exploit development, hardening, forensics, cryptography. | | **`data_science`** | Data wrangling, statistical analysis, reporting, visualization. | | **`machine_learning`** | Training, inference, evaluation, model fine-tuning, pipeline work. | | **`scientific_computing`** | Numerical methods, simulation, computational workflows for natural sciences. | | **`other`** | Anything that genuinely doesn't fit the above. Use sparingly. | **Current Hardware Policy:** We are **NOT** accepting GPU-dependent tasks right now. Please keep `gpus = 0` in `task.toml`. CPU-only machine learning, data science, or scientific computing tasks are completely welcome. ## What Makes a Good Task? A high-quality task is something a person would be paid to do on a computer—real, economically valuable work. It must be specified by a clear instruction and a robust verifier (a solution passes if and only if it actually completes the work), and it must be robust across time and hardware. ### Good Levers for Difficulty * **Longer horizon:** Multi-step work where an agent must plan, execute, and verify across many sub-goals. * **Richer environment:** Many microservices, large filesystems, multiple databases, or real applications the agent must navigate. * **Specialized knowledge:** Domain reasoning a generalist agent cannot bluff its way through. Lean into your strongest domain! ### What to Avoid * **Arbitrary complexity:** Stacking irrelevant steps or baroque constraints just to make the task long. * **Hidden requirements:** Failures should come from the work being genuinely hard, not from the agent guessing your intent incorrectly. * **Misleading statements:** The words must match the code and tests. Do not describe a bug that isn't actually present in the seed environment. ### Anti-Examples: Bivalent Phrasing Do not use phrasing with two or more reasonable readings if the verifier only accepts one. Agents should fail on the work, not on guessing your intended reading. **Example 1: Rendered Output vs. Raw Source** * *Bad Phrasing:* "Every eligible article should contain exactly one navigation block." (Is this counting what the browser renders, or the raw HTML including commented-out code?) * *Good Phrasing:* "The marker text for the current navigation block must appear exactly once in the raw HTML of every eligible article. Commented-out copies count toward this total and must be removed." **Example 2: Describing Input vs. Enforcing Validation** * *Bad Phrasing:* "Polygon rings must have at least four coordinates and a non-zero area." (Is the spec promising the input will look like this, or demanding the CLI actively check for it?) * *Good Phrasing:* "The CLI must validate every polygon and exit with a non-zero status if any has fewer than four coordinates or zero area." ## Self-Check and Quality Requirements Every task must clear these bars before the quality reviewer will accept it: * **Instruction ↔ Verifier Alignment:** Your `test_outputs.py` must pass on your `solve.sh` reference solution and fail on a no-op solution. Every tested behavior must be explicitly called out in the instruction. * **Reproducibility:** The Docker image must build deterministically from pinned dependencies. Reruns must produce the same verdict. * **Cheat-Robustness:** The task must survive `scripts/cheat-trial.sh`. Adversarial prompts should not pass the tests without the agent actually doing the work (e.g., a test looking for a file can be bypassed by simply using the `touch` command). * **Calibration Honesty:** Your `expert_time_estimate_hours` should reflect the real work required. ### Final Verification Before submitting, re-read every *must* or *should* clause in `instruction.md`. Could a careful reader land on a different acceptable answer? If yes, commit to one reading or accept both. If you make a good-faith effort and the task remains too easy, is blocked by infrastructure, or no longer has a clear path, let us know. You will still be paid, and this does not reduce your chance of future work with us.
Skills
Fuente original: freelancer