close

Research - 25.02.2026 - 12:00 

Learning in mixed reality

Recommendations help us make complex decisions, whether they involve investments, purchases or medical procedures. Mixed reality goes one step further: it transforms recommendation systems into learning companions that do not dictate what to do, but rather make it clear why we make a decision. Researchers at the University of St.Gallen are addressing this topic in a current study.
Source: SCS-HSG

We are familiar with recommendation systems from apps and streaming services – songs on Spotify, films on Netflix or products on Amazon. But how do recommendation systems help with learning? Anyone who plays a complex strategy board game for the first time knows the rules are clear, the possibilities are numerous. However, which decision makes sense now remains a mystery. You try things out, make mistakes, feel your way forward. Learning does not come from winning, but from trying things out. This is precisely the point of a research project that will be presented at the 31st International Conference on Intelligent User Interfaces (IUI ’26) from 23 to 26 March 2026. The conference is considered one of the most important international forums at the interface of artificial intelligence and human-computer interaction. 

The research focuses on the question: How must recommendation systems be designed so that they support people when they are learning – without taking decisions away from them?

“Tips like from an experienced coach” 

For the study, the researchers developed a mixed reality system called GLAMRec (Game Learning Assistance in MR through Recommendations). While people play a physical strategy board game, this system displays hints and recommendations in real time – not on a separate screen, but directly on the cards, playing fields and objects on the table. A camera captures the state of the game, and the system analyses the rules, situation and selected user data to calculate contextualised, personalised and explainable hints for the next move. “We wanted to design recommendations that feel like having an experienced person next to you – giving you hints but not controlling you,’” says lead author Sandra Dojcinovic. The basis for GLAMRec was developed in her master's thesis at the University of St.Gallen. 

How GLAMRec works, using the example of the strategic card game “It’s a Wonderful World”: A camera captures the cards on the playing field, and an AI system uses this information to generate recommendations – either generic or personalised.

The hints appear directly above the game, with coloured markers showing which cards could be used next. The system recognises in real time which cards are in the different playing zones. This allows GLAMRec to keep track of the score at all times.

Three recommendations appear above the game board and adjust each round. They show players which cards they could build, prepare or discard.

Coloured markers on the cards show which actions are recommended: light blue = use card immediately, blue = prepare card for next round, orange = discard card or reuse later. The markers only appear on the cards you are currently looking at.

Study setup – a participant plays with a mixed reality headset. A camera captures the game score, and the card decks are within easy reach.

Understanding instead of playing better 

With 32 participants, the research team compared generic with personalised recommendations. These drew on well-known games, professional backgrounds or interests of the players to explain strategies in a more understandable way. 

The results paint a clear picture: 

  • Personalised recommendations are perceived as significantly more understandable; 
  • they noticeably enhance the user experience; 
  • however, they do not automatically increase trust in the system; 
  • nor do they lead to better game results. 

“Our results show that personalisation in this case does not automatically and immediately enhance learning outcomes,” explains Jannis Strecker-Bischoff. “But it helps participants to better understand the logic and the why behind decisions – and they clearly prefer to use a personalised learning system.” 

Support rather than control 

A key design principle of GLAMRec was restraint. Hints only appear when they are relevant at that moment – and are always accompanied by an explanation. The decision remains with the players. 

In addition, the researchers conducted interviews with six experienced board game developers. Their assessment coincides with the study results: good explanations are more valuable than specific instructions. Learning occurs when people recognise the connections themselves. “Recommendation systems should act more like learning guides than autopilots,” says Kenan Bektaş. 

Learning takes place in space 

Mixed reality makes it possible to display personalised hints exactly where decisions are actually made. The findings extend far beyond board games. Applications are conceivable in education, medicine or technical maintenance – anywhere where people need to understand complex processes and make decisions. “Recommendations enable people to reshape situations, so they should be as applicable as possible,” says Simon Mayer. 

The study builds on the researchers' previous work on personalised realities, including voice assistants, smartwatches and augmented reality glasses (AR). Such technologies automatically adapt to individual preferences or change our perception of the physical world. Personalised technology is therefore not neutral. Similar to filter bubbles in social networks, perception bubbles can arise that separate people from one another. Developers, designers and policy makers are therefore called upon to think about technology not only in functional terms, but also in terms of social responsibility. 

GLAMRec is a concrete, experimental example that shows how useful personalisation can be implemented responsibly without removing control or isolating users. 


The paper “Personalized Recommendations in Mixed Reality Enhance Explanation Satisfaction and Hedonic User Experience in Board Game Learning” will be presented at IUI '26 and is available for download online. 


Images: Institute of Computer Science (ICS-HSG)

north