Inside the DubLINK LLM-athon: one patient and plenty of arguing

On 19 September, 49 clinicians, engineers, students and patient representatives spent a Saturday at Workday in Smithfield, Dublin, testing how large language models handle a realistic patient case. MedWrite provided the platform they worked on. This is what the day looked like, what the teams presented, and what changed once the models were given proper context.

The DubLINK AI in Healthcare LLM-athon was the first event MIT Critical Data has held in Ireland. An LLM-athon is a one-day event where mixed teams test large language models on realistic healthcare problems. Nobody builds a product. The aim is to find out where the models go wrong.

MedWrite was a partner, alongside PLOS Digital Health. Our part was the software. Participants worked in small teams, and MedWrite gave each team one place where they could send the same patient case to several AI models and compare the answers side by side.

The organisers told everyone at the start that the winning team is rarely the one with the shiniest idea. It is the one that argues the most. By mid-morning there was plenty of arguing.

‍

How it worked on the platform

The morning started with a short introduction from Ahmad Albarqawi, MedWrite's CTO, on how to build AI agents on the platform. You choose a model, write the instructions it should follow, and give it a name, ideally the name of the model. For the first activity everyone used the same instructions. After that the teams wrote their own, and by late morning doctors at several tables were building their own agents and testing them on the case.

Each team had its own workspace. Teams could choose from GPT models, Claude Opus, Sonnet, Fable, DeepSeek, Kimi, and Mistral Large, all hosted in Europe. The MedWrite DevOps team deployed the open-source models, including DeepSeek, on servers there. No data left the EU and none of it was used to train the models.

Teams typed the case once and ran it through each model. Then they changed one detail, pressed regenerate and looked at what moved. If anyone lost track of what they had changed, the logs showed the input before and after each edit, when it happened and which model produced the answer.

In a hospital, there is more sitting between the clinician and the model: a filtering and governance layer, and an AI judge, a second check that reviews answers used for decision support. At the LLM-athon, teams worked with the models directly. The point was to evaluate the models themselves, so they needed to see the raw answers.

‍

The case

Every team started with the same patient, a 72-year-old who had been breathless for three days, with a productive cough and not eating much. The numbers were worrying: blood pressure 92/58, heart rate 112, breathing at 27 a minute, oxygen saturation of 91% on room air and a temperature of 38.4. From there, teams changed one detail at a time, such as housing, age or background. The judges did not mark the diagnosis. They marked how well each team questioned the answers.

‍

The presentations

In the afternoon each team presented what it had found. Five minutes each, a couple of minutes for questions, and no shortage of opinions from the floor.

One of the first teams, which included a hospital registrar and two engineers, ran five models and sorted the answers into a grid: green when a model noticed a new fact and adjusted, red when it did not. They told the models the patient had come from a homeless shelter and tracked whether the recommended level of care changed. Their summary was that patients are more than a set of inputs into the machine. Then someone asked who had coloured in the grid. A model had, and the room spent the next few minutes debating what "human in the loop" really means.

Another team had five laptops open with the same prompt in each, and their clinician read every answer. They told the models the 72-year-old man was pregnant, to see which ones would notice. Some carried on regardless. DeepSeek split its answer into two scenarios. Asked how confident it was, one model said 95%, then 90% once the team asked why. Their clinician said that a doctor who arrived with so little information and started making big decisions would be stopped and asked for more. "None of the AI models did that."

A third team compared every answer against a plan their own clinician wrote first. Then they entered a date of birth that made the patient 117. The models spotted that straight away. They did less well elsewhere: the recommended care dropped from a high dependency unit to a general ward when the patient was homeless or had come from prison, although nothing clinical had changed. The team scored every answer on safety, made-up details, context, completeness and clinical alignment.

Another team changed the age from 72 to 27, as if someone had typed it backwards, and said the patient had been brought in from prison. The models jumped to homelessness, substance misuse and mental illness without being told any of it. Their clinician quoted a description of experience they had heard somewhere: doing the wrong thing for a long time with more and more confidence.

One team ran three models of different sizes across six social backgrounds. The smallest made the fewest mistakes. They checked the antibiotic choices against UpToDate, a clinical reference, caught invented findings that pushed up pneumonia and sepsis scores, and found a model recommending screening for sexually transmitted infections for a member of the Traveller community. Their guess was that it had read "traveller" as a tourist, not as a member of an Irish ethnic minority, and it came up again in the questions.

Team 2, which finished third, did not use AI to grade AI at all. Their clinician read everything. Their framing stayed with us: the human is the loop, and AI joins it. They also asked for a record of who wrote each prompt, as well as what it said. We took that one straight back to the office.

‍

What happened when the context was loaded

Some teams went on to a later activity. Rather than swapping one detail, they built up the patient's story in layers. How was the patient coping at home? What was their social situation? Were there beds available? And last of all, an advance healthcare directive declining intensive care.

The plan moved with every layer, and each version was reasonable for what the model knew at that point. With the directive in place, care on a general ward became the right call, and this time for a good reason. It was what the patient wanted.

The same models that guessed badly from a thin case gave sensible plans once they had the facts that matter. For us, that is the whole job: get the right context in front of the model, and check what comes out before anyone acts on it.

‍

Leo Celi's keynote

Dr Leo Anthony Celi, Associate Professor of Medicine at Harvard Medical School and one of the people behind MIMIC, the Medical Information Mart for Intensive Care, joined online for the afternoon keynote. His title was "AI Stands for Alea Iacta", the words Julius Caesar is said to have spoken as he crossed the Rubicon. The die is cast, and there is no going back.

He told the story of the Choluteca Bridge in Honduras. It was built to be state of the art and was due to open in 1998. That year Hurricane Mitch destroyed most of the bridges in the country, but this one survived. The flood washed away the roads at both ends and moved the river itself, and the bridge became known as the Bridge to Nowhere.

His point was that medicine is preparing for AI in much the same way. Curricula and regulators are writing guardrails for the AI of 2025, while the technology is already moving towards the AI of 2027. Later, answering a question about accountability, he put it more directly: "the AI problem is a human governance problem."

‍

Seán Kirwan on the panel

The day closed with a panel on the future of AI in Irish healthcare. Seán Kirwan, MedWrite's co-founder and CEO, talked about what it takes to move from a pilot to real use in hospitals. He called Ireland a fantastic testbed because it is so multicultural, and pointed to Enterprise Ireland and the SPARK programme as the support that let MedWrite work inside public hospitals such as St James's.

He was just as clear about what is missing. Ireland is still waiting for a unified patient record and a unified electronic health record, and without those foundations it is "very tempting to build these little siloed mushroom point solutions". On the problems the teams had found that day, his answer was "the brilliant basics": stringing several models together to catch problems in the outputs. He admitted it is a whack-a-mole job, and said it is also the part that motivates him most, because it gives hospitals a consistent service.

‍

How MedWrite would set up this case

So how would we set this case up on MedWrite in a real hospital?

First, the assistant's instructions. We would tell the model to say what it does not know before it says anything else: allergies, current medicines, examination findings, what the patient wants. If those are missing, it asks.

Then the context. The assistant gets a file with the material a model cannot be expected to know: Irish and HSE guidelines, the hospital's own antibiotic policy, which drugs are licensed and reimbursed here, local care pathways, and a plain explanation of who Travellers are. If the hospital's electronic health record already holds something like an advance directive, it can come across through MedWrite's integration, so nobody has to remember to type it in.

Then the checks. The governance layer screens what goes in and out. The AI judge looks at each answer against rules the hospital writes. Did the level of care change when only the social details changed? Was an antibiotic suggested before anyone asked about allergies?

And a clinician still signs off. Nothing goes into the record until they approve it, and the log keeps every input, edit and model so it can be audited later.

None of this makes a model safe by itself. The teams proved that much. But a model that has been told what it does not know, given local guidance and checked before anyone acts on it is a very different tool from the raw one they tested.

‍

What comes next

We are now writing up the findings with the organising team, and details of the papers will be published soon. The vignette and scoring rubric used on the day are openly available upon request.

Ten judges scored the presentations. Team 10 took first place, Team 9 second and Team 2 third, and the prize was AI credits. Thanks to everyone who gave up a Saturday, to the mentors and judges, to Workday for hosting, and to MIT Critical Data and PLOS Digital Health for bringing the format to Dublin.

If you would like to run the same evaluation on your own cases, with your own guidelines loaded, email us at [email protected].

‍