"We should be wary of a silent obligation to use this technology."

Dr Mathis, you wear glasses. Is their augmentation of reality already digitally enhanced?

No, they are normal glasses. However, I have often tried on mixed reality glasses in my research. In other words, glasses that present me with a mixture of the real world and virtual artefacts that I can also interact with.

What are you researching in the field of mixed reality (MR)?

My research at HSG focuses on the development of applications that support people with disabilities, especially visual impairments, in their everyday lives. We are working on solutions that convert visual information into audio information and haptic information (using the sense of touch) in order to make reality more accessible for people with visual impairments. In addition, we are also developing systems that are useful for a wide range of users, not just people with disabilities. For example, we are currently developing and investigating a concept and developing prototypes where we can translate foreign-language text on street signs or menus directly using glasses. This means you no longer have to pick up your smartphone and a translation app but can translate all the information in your real environment with a simple hand gesture. 

These seem like very useful applications. How else can MR benefit us humans?

Education, for example, could also benefit greatly from MR because learners can engage with certain processes or objects in a very realistic and vivid way that they would otherwise not be able to see in the real world, like a confined classroom.

Apple has now launched MR glasses on the US market this week. Ten years ago, Google wanted to launch similar glasses, but failed. Why was that?

Yes, that's a good question. Sometimes tech companies simply push ahead to show that they are on the market before the product is really ready for the masses. The breakthrough is usually only achieved when the technology is accepted by society and the applications are understandable. Of course, the purchase price and possible running costs also play a major role.

And is the price right with Apple?

Compared to a smartphone, the new Apple glasses are still massively more expensive at around CHF 3,000. And I think there is still a lack of widespread acceptance that people really see added value compared to their smartphones and other technical devices that justifies this price. On the other hand, the Apple Glasses are still quite large, which currently makes them less suitable for everyday use. At the moment, they still look more like ski goggles.

So what can you do with Apple Glasses?

For end users, they will initially be used primarily in the field of virtual reality. For example, you can use them to immerse yourself in the virtual world of games or films or hold a meeting with friends in a virtual location. There will be many more applications in the near future. In research, you see very innovative prototypes and possibilities from time to time, but many of them often don't see the light of day in everyday life. Research is often years ahead, so I'm very excited to see what the first everyday applications by end users will be. Technology and research alone cannot decide this.

And what about augmented reality (AR)? Will end users be able to walk around a city with Apple glasses and see information about people on the street or buildings?

Theoretically, yes. Such applications are already being tested in research. However, if you decide to purchase such glasses and want to try them out, it is advisable to have realistic expectations in order to avoid possible disappointment. Existing applications are still very limited, and of course there are still legal hurdles for such functions, especially when it comes to personal information. And corresponding apps are therefore not yet available for end users, but I think there will be a major change be it in a year or five years' time. But it would already be possible, for example, for me to place a virtual piece of furniture in my living room, manipulate it and view it together with another person in another country.

But I can already do that with a smartphone app. Does this mean that the glasses can still do very little for me as an end user in terms of augmented reality?

The AR function is there, but at the moment it will still be used more like a VR device, as the applications for AR are still very limited. But it will be extremely exciting to see where the AR and MR sector develops over the next two or three years.

You also co-wrote an essay that looks at the social risks of MR and AR. Where do you see the main risks?

In addition to privacy and data protection issues, it is certainly worth considering whether there will also need to be regulations in the future regarding the extent to which users in AR are allowed to digitally manipulate each other. For example, the technology would allow a user to digitally remove the clothes of everyone around them or even hide them completely because they want to walk around a city alone. I see this as extremely critical, especially if the technology theoretically makes it possible but there are no clear regulations yet.

Why would that be so bad? In contrast to published deepfakes, this would not correspond to an image of me that all people could see, but only this one user.

Well, that's just an individual question as to how much that would bother someone. Many people consciously want to be perceived from the outside in a certain way and buy expensive clothes for it, for example. They probably wouldn't care if other people dressed them differently or even exposed them virtually. There should therefore be a regulation that protects people and their privacy by default. Such augmented reality manipulations should only be permitted if explicitly authorised by the respective target person. Compared to deepfakes, it is important to mention that we are no longer on the internet here, but in the real world.

Where else do you see dangers?

You also have to be aware that AR basically gives users the opportunity to customise their everyday world. For example, poverty could be pushed out of social consciousness because people could digitally beautify poor areas. Phenomena such as fake news and filter bubbles could also leave the confines of social media through AR and spill over into reality. Similar to what we see in social media today, AR could influence individual perceptions of reality through the algorithms of platform operators. This would take advertising and the manipulation of everyday decisions to a whole new level. Furthermore, photorealistic changes and extensions of our reality, which no longer only appear digitally on a smartphone but seamlessly merge into our reality, could be very difficult to expose. In the worst-case scenario, these perceptual changes to our reality could even have serious consequences. For example, they could make it difficult for us to distinguish whether it is a real bridge or just a virtual bridge that is supposed to beautify the landscape. In such a situation, a misjudgement can have significant negative, even life-threatening consequences.

But is this a realistic scenario? Many people want to know what is real and what is not and then simply take their glasses off most of the time?

Sure, it's likely that people would do that, at least now. The problem is simply that if we think about the future and a large part of our world were also designed for AR and MR, reality would lose a lot of its appeal without the virtual enhancements and changes.

How would such a world be set up?

Imagine, for example, that in this world the real physical shelves in a shopping centre are empty and I can only see products there with mixed reality technology, which I then place digitally in my shopping basket and only get physically spit out by a machine at the checkout. In a world like this, it will be impossible to go shopping without MR technology. We are seeing similar developments with the spread of smartphones. For people without smartphones, certain services are now difficult to access. And we should definitely be wary of this kind of silent obligation to use technology in the MR sector.

Is the regulation of AR and MR already underway?

No, unfortunately not yet in the way we would all like, even though we can expect huge developments in this area over the next few years. But I fear that we, including politicians, could once again be overwhelmed by developments, as is the case with artificial intelligence. A proactive approach on the part of politics, industry and society is crucial to develop ethical standards, data protection guidelines and security measures. In conclusion, I firmly believe that the new era of spatial computing can be particularly helpful in our everyday lives, but we need to take early action to minimise potential risks and negative impacts.