Funnel and sentiment
More people reached guidance, and the experience felt easier to understand.
Chapter 03 · Recommendation integrity
Better metrics weren’t enough if the recs weren’t true.
After the relaunch, an engineer and I escalated a recommendation-integrity gap to the CPO. Demographic-only light recommendations improved access but could not meet the accuracy standard Nayya needed. I led the product model that followed—grounding guidance in recognizable life situations and preparing it for approximately 20 client groups this Open Enrollment.
Upcoming production rollout · October–November 2026 · Outcomes pending
01 / The tension
Stats looked better. The recommendation model didn’t.
NPS and engagement suggested the new front door was working. Internally, we could see a harder truth: the light recommendation relied on broad demographic assumptions and could suggest coverage without enough context to know whether it was appropriate.
Demographic-only light recs
Age and household data could establish eligibility, but not intent, anticipated care, or financial priorities.
Accuracy is the brand
Recommendation integrity mattered more than protecting a favorable short-term metric.
02 / Mobilize
Escalate the principle, then turn it into a model.
Escalate the integrity gap
An engineer and I brought the problem to the CPO: recommendation accuracy could not be traded for a cleaner funnel.
Audit the inputs
I reviewed the survey questions and existing data points that materially changed different recommendations, then grouped recurring signals into early persona hypotheses.
Validate the logic
I took the model to Product and Engineering to confirm which inputs were reliable, available, and consequential enough to affect the recommendation.
03 / Evolution
People look for advice from someone like them.
Across recent Relaunch research and years of Choose research, people repeatedly described asking friends, coworkers, or family members who had navigated similar situations. I proposed that recognizable, scenario-based personas could provide a more human way into the recommendation logic.
“Someone like me”
People sought advice from others with familiar needs or life circumstances.
Questions that change the answer
I isolated the survey inputs that materially affected recommendation direction.
Nine personas
Product and Engineering helped refine the data points and validate the model.
Persona-led flow
I translated the model into the persona selection and guidance experience.
Return to the survey
When the system still lacked sufficient context, the experience connected users to the full survey.
04 / Tradeoff
We knowingly risked worse metrics.
Choose was built to block no one. Demographic-only light recommendations raised access. They also recommended coverage the system could not defend. The obvious move was to protect the new metrics. We did not. We put recognizable situations and targeted questions in front of the recommendation, knowing some of the funnel might come back off.
Full recommendation fidelity
Protect the standard that guidance should be grounded in enough context to be defensible.
Supplemental without the push
Use context already known and ask only when an answer could materially change.
~20 client groups · OE
Prepared for production rollout during the October–November enrollment period.
RouteShort-term engagementRecommendation fidelityPrimary risk
Demographic-only light recHigherLowerTrust and accuracy
Persona + targeted contextModerateHigherSome funnel loss
Full survey firstLowerHighestAccess and completion
Evidence statusOpen Enrollment begins in October–November 2026. This case currently documents the decision, system, and production commitment—not post-launch performance. Recommendation quality, engagement, and survey continuation will be evaluated after launch.
05 / Guarding the metrics
Integrity first — then design against a worse funnel.
We were willing to trade some short-term metrics for recommendation fidelity. That did not mean accepting a dead funnel. Alongside the integrity model, we placed three design hypotheses into the test to keep engagement from collapsing.
Modernize the surface without changing the mandate.
Feedback had called the product outdated and rigid — classic B2B2C SaaS. With CTO approval, the integrity test also carried a light visual and interaction refresh that borrowed more from contemporary AI and D2C patterns, aimed at making the experience feel current enough to engage.
One resonant quote — not a cast of characters.
Fancy persona names ask people to learn a character. Instead, each path led with a single quote meant to land immediately. Eligibility and situation came from what we already knew demographically — age group, marital status, and similar signals — so the persona stayed simple on the surface and grounded in real context underneath.
End into conversation, not a dead end.
By then Titan’s AI-native infrastructure existed, even though client rollout was on hold. I connected the end of this experience into the AI chat so people could keep asking questions inside the ecosystem. That also gave legacy clients a lighter path toward Titan as an add-on — engagement first — instead of only a full implementation sell.