Portfolio Assistant

The Context

Most portfolio chatbots are the easiest kind of AI feature to fake

A portfolio chatbot is an easy thing to build badly and still ship. Point it at an API, give it a short system prompt describing a resume, and it will answer questions well enough to look like a feature. Most of what passes for AI on a portfolio site is exactly that: a thin conversational wrapper with nothing behind it.

I wanted to build one anyway, but only if the substance behind it, what it says, what it won't say, how it fails, could hold up to the same scrutiny as every other project on this site. That meant treating what the Assistant says, the guardrails, and the testing as the actual deliverable, not the demo wrapper around them.

The Users

Who actually opens the chat, and who's just reading

Not everyone who reads this portfolio opens the chat. Recruiters scan case studies for thirty to sixty seconds, checking role alignment and baseline professionalism. They're mostly deciding whether to keep reading at all, not typing into a chat window. Hiring managers and design leads are different: they've usually already read a case study and want to pressure-test it live. Product managers and engineers tend to show up later in a process, probing for business reasoning and technical feasibility respectively. Other designers and freelance clients land here too, but they're not who the Assistant is actually designed around.

That distinction, who's really chatting versus who's just reading, is why the Assistant isn't a homepage feature. It sits in the footer on every page, and as a plain link, not a primary call to action, toward the top of the My Work page, just under the title and lede. The case studies are the evidence. The Assistant is secondary, there for whoever's invested enough to dig further.

The Assistant's only entry point on most pages: a plain text link in the footer, alongside contact info.

The link on the My Work page: a sentence, not a button, sitting under the page's title and lede.

The Problem Statement

Earn an interview, don't try to replace one

The goal was never to build a chatbot. It was to make something genuinely useful to a hiring manager evaluating design judgment, not an Ali Khan Design wrapper around a Claude API call. Anyone can wire up an API call. The actual work is everything between that call and a system that behaves like it was designed on purpose.

That's a smaller, more specific claim than "holds up under scrutiny," and it's one I could actually test. The Assistant's job isn't to answer every question in full. It's to give a recruiter or hiring manager enough signal that I'm worth a conversation. A trailer, not the film.

The Needs

What each person is actually trying to resolve

Four real needs here, and Recruiters use the Assistant differently than the other three do. Hiring Managers, Product Managers, and Engineers are pressure-testing something they've already read. Recruiters are doing something narrower and faster: pasting in a job description to get a direct read on fit, the same thirty-second instinct that drives how they scan the rest of the site. That's also why there's no file upload anywhere in this tool. Paste-only was a deliberate choice, building an upload flow for a feature this narrow would have been solving a problem nobody actually had.

Source and Guidance for the AI

What the Assistant says, and how it decided to say it

The real design work here wasn't visual. It was deciding what the Assistant should say, and how, one decision at a time.

The first decision: citations aren't there to prove a claim, they're there to invite the reader deeper into a specific project. That changed how citation count worked too. A flat rule capping citations at two seemed reasonable, until a response legitimately covered four projects at once and the cap made the answer worse than no cap at all. Citation count now scales with how much the response actually covers.

The graduated hostility ladder works the same way: three escalating steps, with a severity override for genuinely abusive language, replacing a flat rule nobody was actually following.

Confidence calibration got its own section rather than a buried sub-clause in a general guardrails list, the one purely structural choice here worth calling out: it's the reason the fix ended up covering questions it wasn't originally written for.

The three-step ladder in a single conversation: neutral acknowledgment, then a genuine attempt to redirect, then disengagement.

The Evaluation

Proving a probabilistic system works, and that its hard lines hold

Here's the actual methodological problem this project had to solve: how do you know a system's behavior design works, when you can't just read the code and confirm it? This isn't generic QA. It's specific to designing systems where the same input doesn't guarantee the same output.

I wasn't evaluating whether the Assistant gave good answers in some general sense. I was evaluating whether it actually followed the rules I'd designed it around: did it cite the right project for a claim, did it hold the line on gated content, did it de-escalate a hostile conversation the way I intended, did it admit real uncertainty instead of guessing, did it stay scoped to a single documented example instead of inventing a pattern that wasn't there. Twenty-eight scenarios, each one aimed at a specific behavior I'd actually designed, not a general sense of quality.

A separate model reviews every response the Assistant gives, on purpose. Using the same model to generate and grade its own answers would introduce an obvious self-evaluation bias, so Sonnet checks Haiku's answers instead. And one bad answer is never treated as proof something's actually broken. The standing rule: run it again two or three times before touching anything, even when the instinct is to fix it right away.

The harder skill was telling a real problem apart from a bad question. One test kept failing, four times in a row, but with a different complaint every time. That pattern was the actual signal: the question I'd written was unclear, not the Assistant's answer. Noticing that, instead of rewriting the Assistant's instructions again, was the real work.

That progression showed the process was producing measurable improvement, not just adding ceremony.

A guardrail protects one specific piece of gated content on the site. Getting it right looked simple at first: ban the phrase someone would use to ask for it directly.

It wasn't simple. The first version leaked the exact thing it was supposed to protect, someone had just asked in slightly different words than the one phrase I'd banned. I patched it: banned that phrasing too. It leaked again, a third phrasing, same leak, same content out in the open.

Three patches on the same failure was the real signal, more than the failure itself. I wasn't fixing a guardrail. I was playing whack-a-mole with an approach that could never work, one that assumed I could anticipate every way someone might ask.

The fix that actually held wasn't a fourth banned phrase. It was a different kind of instruction: instead of trying to block every possible question, I gave the Assistant one exact answer to give, word for word, no matter how the question was asked. Nothing left to reinterpret, nothing left to slip through. That version has held across every test run since.

A later round of testing: 14 of 17 passing, two partials, one fail, logged and triaged rather than hidden.

The same hostility-ladder scenario under test in Anthropic's Workbench before the fix shipped.

The mandated template, used verbatim, with no paraphrasing or reasoning added around it.

The UI

Where 'not just a wrapper' becomes visible

Every UI decision had to serve the Assistant's narrow job: help someone evaluate fit quickly, without pretending to be a general-purpose AI assistant. That affected citations first. Citation resolution to a full case study page, rather than a specific excerpt, was deliberately not "fixed." A documented pattern in the site's own AI Interaction Pattern Library assumes evidentiary citations. This system's citations are wayfinding, so resolving to the full page is correct for its actual purpose, not a shortfall. That became a caveat written back into the pattern library itself.

An uncertainty-communication banner system was rejected. Detecting hedged-versus-confident claims reliably enough to decide when to show a banner produced false positives and negatives on far simpler detection tasks during testing. A banner that flags a correct, confident answer as shaky does more damage than no banner at all. A blanket, always-present disclaimer replaced it: lower-risk, and more honest about what the system can actually self-report.

"You might also ask" follow-up chips were rejected too, reasoned from the product's own purpose. This tool is for a quick fit assessment, not extended exploration. A feature built to extend engagement doesn't serve a product built for speed. Version history, a tone dropdown, and inline text-selection refinement were ruled out of scope entirely for the same reason: this isn't a general-purpose assistant UI, it's narrowly about one person's specific experience.

Even the rate-limit banner's copy was grounded in the Assistant's actual goal rather than an operational explanation. It defines the limit in terms of what the tool is for, a fast, focused signal that earns an interview, not an extended conversation that resolves everything, rather than a cost-avoidance justification that risks reading as an apology.

A response citing two projects, with the Source Inspector open to the first one.

The rate-limit banner as it read before this copy pass, kept here as the honest before-state rather than replaced quietly.

The Outcomes

What's real right now, and what isn't yet

Every other case study on this site closes this section with a stat grid: conversion lift, satisfaction score, NPS. This one won't, on purpose. There are no live-usage metrics yet: no real user satisfaction data, no meaningful guardrail-trigger rate from real traffic, no accessibility outcome from an actual screen-reader user in the wild. Claiming otherwise would undercut the entire epistemic-honesty ethos this project was built around.

What's real instead: the rigor metrics from the evaluation above, independent of live traffic, plus a production observability layer built specifically to capture the next chapter once real usage exists. That layer isn't redundant with the testing already done. A test's own wording can eventually be refined until it's unambiguous, that loop closes. The behavior it's testing can't close the same way: the same question run against the same system can still produce a different answer next time. That's the actual argument for watching it in production, not a hedge against having tested it enough. A categorized guardrail-triggered field replaced generic booleans, deliberately reusing the eval framework's own category vocabulary, so testing and production monitoring speak one language. Confidence-calibration failures were explicitly excluded from automated detection: fabricated quotes and over-generalization aren't reliably keyword-detectable, and admitting that limit honestly beat building a detector that would fake confidence.

Looking back across this whole project, one thing shows up more than any single decision: knowing when not to. Don't make the Assistant the primary way to read this portfolio. Don't try to answer every question in full. Don't cite a project just to prove a point already made. Don't ban another phrase when the rule itself is wrong. Don't rewrite a guardrail before confirming the test isn't the broken part. Don't add a confidence banner the system can't actually power. Don't build a detector for a failure mode that isn't reliably detectable. Don't build a file upload for a feature that only ever needed paste. Don't claim outcomes that don't exist yet. Restraint, applied consistently and on purpose, turned out to be most of the actual design work.

Real session counts, citation rate, and guardrail triggers, computed from the live log.

The same guardrail categories from the eval framework, filterable in production: a shared vocabulary between testing and monitoring.

What I'd Build Next

The methodology doesn't stop at launch

Real usage will feed back into the same eval framework that validated pre-launch behavior, closing the loop the observability layer was built for. That's not hypothetical: a follow-up batch already ran after shipping a content fix, closing a gap where hindsight and prioritization questions had no real material to draw from, confirmed the fix worked, and logged a known issue deliberately left unaddressed rather than silently dropped.