Skip to content

Circa

Ask a patient’s chart a question; every sentence of the answer cites the record it came from.

Role
Designer & engineer
Timeline
Jan 2024
A Circa patient chart: a timeline of visits, labs, vitals, diagnoses, medications and procedures for a de-identified MIMIC-IV demo patient, with dates shifted for privacy.
A Circa patient chart: a timeline of visits, labs, vitals, diagnoses, medications and procedures for a de-identified MIMIC-IV demo patient, with dates shifted for privacy.

The problem

Clinicians ask a patient’s chart the same questions every day: the last A1c, whether a patient is still on a statin, any ER visits this year. The answers are spread across hundreds or thousands of records, and a plausible but wrong answer is worse than none.

The usual AI approach, retrieval search, reads only the few records that look most like the question. Similar is not the same as newest, and a handful of records cannot list every medication.

What I did

I built Circa to answer from a patient’s whole FHIR record. Each question becomes an intent and a medical concept, every relevant record is read, and every sentence of the answer cites the record it came from. If the chart has no answer, it says so; it declines treatment decisions.

I measured it on 100 real de-identified hospital patients (the MIMIC-IV demo, 872,006 records) and 130 synthetic ones, with answer keys computed by separate code.

The test questions were worded by someone who never saw the parser, and I kept the first runs, including the bugs they exposed.

Circa answering How has her creatinine changed?: six results with dates, each sentence citing its numbered source records, highlighted on the timeline above.

The outcome

On real patients Circa answered 97.8% of 1,681 questions correctly, worded by someone who never saw how it reads them, with 4 confident wrong answers. Search over the top 20 matching records scored 66.1%, with 481.

Search fell apart where it matters most: 0% on full medication lists and problem lists, against Circa’s 100% and 90%.

Circa results on real patients against the usual search approach, with counts of confident wrong answers for each.

Next project

Mender

A code-repair agent with a benchmark that catches cheating.