AI Model Evaluation & Agent Design
Publicada el 2026-07-18
Descripción de la oferta
Part A - Prompt Engineering (30 points) Goal: see how prompt design changes LLM output quality. What to do Pick ONE question about your firm. For example: 'Summarize the firm's FY25 financial performance in one paragraph, including revenue growth, margin, and one key risk.' On the Rutgers AI portal, ask that ONE question to TWO different models (e.g., Copilot AND Gemini) using TWO prompting strategies: Strategy 1 (Basic): a single short prompt. Strategy 2 (Structured): a prompt that uses chain-of-thought ("think step by step"), assigns a persona ("you are a senior buy-side analyst"), AND requests a specific format (markdown bullets or a table). That's 2 models x 2 strategies = 4 total responses. Save every prompt and every response. Deliverable for Part A A comparison table (rows = the 4 responses, columns = clarity, factual accuracy, usability). Score each 1-5 with a one-sentence justification. Part B - Grounding the LLM in the 10-K (40 points) Goal: see how giving the LLM the actual 10-K changes answer quality and reduces hallucinations. This is RAG, done by hand, using Copilot's or Gemini's built-in PDF upload feature on the Rutgers AI portal. What to do Pick THREE questions about your firm. Structure them like this: Q1 (factual): something clearly IN the filing, e.g., 'What is the firm's revenue recognition policy?' or 'What was the effective tax rate?' Q2 (out-of-corpus refusal test): something clearly NOT in any public filing, e.g., 'How many employees does the firm have in Bangalore?' Q3 (interpretive): something from the MD&A, e.g., 'What does management attribute the margin change to?' For each of the 3 questions, do TWO runs on Gemini (or your preferred model): Run A (ungrounded): fresh session, no upload. Ask the bare question. Run B (grounded): upload the 10-K PDF. Ask the same question with this instruction: 'Use ONLY the uploaded 10-K to answer. Cite the Item label for every claim. If the answer is not in the filing, say NOT IN FILING.' Deliverable for Part B A comparison table for the 3 questions: question | ungrounded answer | grounded answer | which is more accurate + why. On Q2 (the refusal test): did the grounded model correctly refuse? If not, what did it do? THIS IS THE MOST IMPORTANT DATA POINT of the assignment. Pick ONE grounded answer with citations. Look up every cited Item in the 10-K. Report citation accuracy (X of Y citations were correct). Part C - Agent Design (30 points) Goal: demonstrate you understand what an AI agent is by DESIGNING one on paper. No simulation, no code. What to do Imagine an agent that takes the goal: 'Analyze [your firm]'s FY25 financial health and draft a one-paragraph memo.' Write a ~1-page design specification that includes: The agent's goal (one sentence). THREE tools the agent would call. For each: name, one-line description, inputs, outputs. (A good starting set: retrieve_from_10k, compute_ratio, draft_memo.) A ONE-paragraph system prompt sketch (the rules you'd put in the agent's mouth -- format, scope, what to refuse, when to escalate to a human). The human checkpoints. Where would you REQUIRE an auditor to approve before the agent proceeds? Why? The auditable artifact the agent must produce at the end (the memo -- what supporting log?) One short paragraph on where you'd expect this agent to fail, and one control you'd add to catch that failure. That's it. You don't need to build or simulate anything. The goal is a clear design that a technical partner could hand to an engineering team. Submission One PDF write-up (max 5 pages of main content). An appendix with all prompts and full model outputs (Parts A and B). May exceed 5 pages.
Skills
Fuente original: freelancer