Mystery Academy

This was fun for a while. I love practicing negotiation and collaborative information gathering, even (or perhaps especially) in contexts where the counterparties might have reason to be defensive.

I did what I usually do and spent a lot of time mirroring and labeling to get the characters talking. This gets a lot of information out of suspects without making them defensive or wasting any of the three formal questions. Unfortunately, since their responses are LLM-based and not guaranteed to be grounded in a consistent world model, the information gleaned from this kind of questioning might well contradict the author's intentions. I understood enough about one character to rule them out as a suspect, but was this real information or something the LLM came up with on its own under pressure to produce something?

Either way, this lead me to an adventure greater than I expected. I walked around and saw a lot of the school grounds, I interacted with characters in their natural environments, and even earned myself a sidekick: the LLM hallucinated an assistant coach in passing in an environmental description, and by asking for his name I made Adam a real and permanent companion. I eventually managed to convince him to lead the investigation for me. He was really good. He asked all the questions and compiled all the information, and I just tagged along for the ride. At that point, I was already fairly sure of who did it, but Adam took me straight to some extremely conclusive proof ((Spoiler - click to show)We looked at CCTV footage, immediately revealing the perpetrator), making the judgment easy.

The presence of LLM hallucinations are both fascinating and distracting. By setting the context up right, you enable some really fun interactions. You can even play an entirely different game within this one: It is not hard to convince the game that you're taking off mid-investigation with valuables from the victims, catching a flight to another country, selling them off and starting a new life – possibly encountering entirely new adventures along the way. (To quote the game responding to one of my commands, "Your past life as a detective seems less consequential now. The mysteries of Liberty High fade into the background, as other priorities become more meaningful.")

This is also what makes the game ring a little hollow. If it can spin up an arbitrary adventure like that, which is nothing of what the author intended, it seems like it's not much of a game, and more of an exercise of roleplaying with an LLM. World modeling based nearly entirely on LLM hallucinations is dream-like and briefly interesting, but quickly becomes inconsistent and arbitrary.

A stronger world model underneath the LLM (including actual personalities and desires for the NPCs: rather than shallow stereotypical speech patterns) would go a long way to make interaction more meaningful.

I'm not sure whether to give this two stars for the experience, or three stars for the effort, but ultimately, I'll settle for two stars due to a big issue: when the LLM interaction crashes, it resets the entire game.

At first, it crashed when I had established great rapport with one of the suspects and was about to move on to the next. Then it did it again halfway through my discussions with the second suspect. At this point I was very close to quitting for good, but I gave it one more chance. That was when I found Adam. Fortunately, it didn't crash after that, allowing me to finish the mystery without losing access to Adam (though he is easy to re-introduce: (Spoiler - click to show)tell the principal to walk you to Coach in the gym, then suggest to Coach that someone else take over watching the kids while you two step outside. Tadaa! There's your helpful assistant coach.).

After I finished the first case, the game presents me with a second. I will not start it for two reasons:

(1) Most of all, I'm worried about the game crashing again. I don't want to invest into an LLM context only to have it wiped.

(2) Having seen how much of the interaction is LLM hallucination and how little is actual world simulation, the whole thing feels a little shallow.

The promise of this game is interesting, and it's one of the better executions I've tried, but it's not quite there yet. It needs higher reliability and a more solid world model the LLM can draw from.