~ 6 Minute Read.

To­day I had my fi­nal pre­sen­ta­tion of a vir­tu­al re­al­i­ty es­cape room project I did at the Uni­ver­si­ty of Kon­stanz. To­geth­er with Daniel Bo­gen­rieder and Rox­anne Low, we built this project in the Uni­ty En­gine, us­ing a va­ri­ety of tech­nol­o­gy: Leap Mo­tion, the HTC Vive Pro, redi­rect­ed walk­ing, text to speech via IBM Wat­son, 3D scan­ning and 180° im­ages.

The goal of our es­cape room was avert­ing the us­er’s ex­pec­ta­tions: pro­voke the play­er to free him­self of self im­posed re­stric­tions that s/he as­sumes are mapped from re­al­i­ty. That ranges from weight of ob­jects to walk­ing through walls and ex­pand­ing hu­mand ca­pa­bil­i­ties.

I was lucky to have Daniel and Rox­anne on my team since with the projects at Vhite Rab­bit and my job at KN­IME I was not ful­ly com­mit­ted to the project and they re­al­ly did the big­ger por­tion of work here. I will high­light some spe­cial fea­tures of the project here any­way and make sure to give cred­it where cred­it is due.

The Set­ting

At uni­ver­si­ty we have a room called the “Me­dia Lab”. It’s a room packed full with fan­cy tech­nol­o­gy: full-body track­ing set­up, Vive, a huge mul­ti­touch screen and pow­er­ful com­put­ers.

We came up with the idea of us­ing the Vive Pro aug­ment­ed re­al­i­ty (passthrough mode) to be able to over­lay some “per­fect aug­ment­ed re­al­i­ty” on that. It would al­so al­low us to switch to full VR mode and use a VR head­set with­in VR! 1

A quick test of the passthrough showed that it wasn’t very im­mer­sive, though. The res­o­lu­tion of the cam­eras does not seem to be suf­fi­cient to match the res­o­lu­tion of the Vive’s dis­play.

In­stead we opt­ed for repli­cat­ing this room in vir­tu­al re­al­i­ty, do­ing a full 3D scan of it and matched the track­ing re­gion to the re­al-world track­ing area.

The set­ting then was that you locked your­self in­to this room, be­cause you were work­ing on an as­sign­ment for too long. Now you have 15 min­utes left to hand off the as­sign­ment and need to es­cape. All you got is an as­sis­tant of which you don’t re­al­ly know if she wants to help you or not. We called her “evil Cor­ta[…]” (you get the idea).

Text To Speech

I’m ab­so­lute­ly blown away by the op­por­tu­ni­ties giv­en by cur­rent state of tech­nol­o­gy. With Web APIs you have a wide range of ser­vices that of­fer any­thing you can dream of.

In this case one of the first things we im­ple­ment­ed af­ter scan­ning the room was “Text to Speech” via the Wat­son API and their Uni­ty SDK which pro­vide a free con­tin­gent of char­ac­ters you can send to the API (with­out re­quir­ing a cred­it card, which I don’t own).

We quick­ly ap­proached this con­tin­gent so I im­ple­ment­ed an au­to­mat­ic caching sys­tem that wrote out the re­ceived au­dio as wav files and used that when­ev­er it was avail­able.

Here’s a video that should give you a rough im­pres­sion of how all of that sounds like:

Room Scan­ning

Luck­i­ly Daniel had some ex­pe­ri­ence with 3D scan­ning. Af­ter many at­tempts with dif­fer­ent de­vices— Project Tan­go, Stere­o­labs’ ZED track­er, Struc­ture Sen­sor and the HTC Vive Pro—we fi­nal­ly man­aged to get an ac­cep­ti­ble scan with Project Tan­go.

It turns out that most of these more af­ford­able scan­ning tech­nolo­gies work very well with ob­jects, but have prob­lems with scan­ning an en­tire room in one piece. Prob­lems ranged from even get­ting a sta­ble con­nec­tion to soft­ware just not be­ing able to han­dle the amount of da­ta.

In the end I cleaned up the room scan in Blender of which you can see a part as a time­lapse here:

This took me a lot of time and I had to cut off work­ing on it at some point, ba­si­cal­ly re­sult­ing in a rough­ly remeshed room with sep­a­rat­ed ob­jects and a tex­ture that looked like the room sur­vived some fire in­ci­dent.

The first im­age from left top to bot­tom right is the orig­i­nal scan. It still has miss­ing por­tions and very un­even sur­faces. The sec­ond im­age is the cleaned up ver­sion and the third is a bunch of sep­a­rat­ed ob­jects, par­tial­ly remeshed, par­tial­ly left as is.

Since I am es­pe­cial­ly proud of it, the chair de­serves its own im­age (it alone took me 1.5 hours).

Up Next

Since there is a lot to cov­er and my posts re­main dai­ly, I split the doc­u­men­ta­tion of this project over mul­ti­ple posts. Next up to­mor­row is our use of redi­rect­ed walk­ing!

1
An idea which is al­ready used in games like Ac­count­ing for ex­am­ple.

Writ­ten in 45 min­utes, videos and im­ages in 120 min­utes ed­it­ed in 10 min­utes.