Zuckerberg and Bosworth recently (June 3rd) hosted an AMA which you can watch in its entirety here:
The highlights that I found most interesting,
Are Advertisements coming to VR? Yes.
They claim that we won't "hate it" and I can imagine some very useful ways to have advertising in VR. Although I'm sure there will be lots of complaints leading up to it. I mean... I see advertisements everywhere I look in the real world, even when I'm driving down the road or on the highway. Advertisements are part of our lives for anyone living in a populated city or town.
As long as we don't get the terrible pop-up ads that we see in Mobile Apps, then we should be good!
Will we get neural interfaces in VR? Yes.
Zuckerberg: Well, whenever you’re designing a new platform, I think one of the most important aspects of it is input. I think in a lot of ways, how you control it is the most defining aspect of a platform, right? A lot of people think about AR and VR as sort of ‘what’s the output? Like what do you see?’
The bigger thing that defines PCs is you have keyboard and mouse. For phones, it was [that] you have this multi-touch and kind of swipe input.
So the question is — what are you going to use to control this natural interface around AR and VR? Our view is that it’s going to be somewhat of a combination of things, right?
You’ll have voice assistance and that’s going to be neat. But you’re not always going to want to use voice, because there are privacy issues with that – you want to sometimes control things without it telling everyone around you what you’re doing.
Hands are going to be a thing. People want to control hands. But you’re not always going to be walking around through the world with your hands outstretched in front of you doing stuff. So that will work sometimes better than others.
Controllers are going to be one interesting dimension of this too. Because as good as hands can get, if you’re doing something that’s really a micro movement – any gamer can tell you this – actually having a thumb pad and that kind of tactile feedback is super important. So for things like writing, you want a stylus – that’s super helpful to have something physical.
But, in some ways the holy grail of all this is a neural interface, where you basically just think something and your mind kind of tells the computer how you want it to go and that works.
There’s a bunch of research that we and others are doing into this. I think the key insight that our team has had… A lot of people, when they think about neural interfaces, they think about ‘how can we understand what you’re thinking?’
And it’s actually not about that. You don’t want to read the person’s mind. You’re not trying to understand what they’re thinking. What you’re trying to do is give the person an ability to have their brain send signals to the rest of the body about how this works.
And we have a system that does this, right? With motor neurons where your brain basically sends signals to your hands and your body telling them when you want to make movements, how to control it. And it turns out that we all have some extra redundant capacity for that, right? It’s part of the neuro-plasticity. If one pathway gets damaged, your brain can kind of get rewired but you can train those extra pathways to control, for example, a second set of virtual hands, so that way you just kind of think and, like, down the line, you have your virtual hands are typing and controlling what you’re doing in VR and AR, and then you don’t need to actually have a physical controller or anything like that because that’s awesome.
When you get to that, we’re gonna have this whole constellation of inputs, but that is perhaps one of the more ambitious projects that we have going on. But I think it’s really promising long-term and I think the team is making good progress towards it.
A good blurb on Wireless VR required for "presence" (note that this applies to PCVR as well):
Zuckerberg: Yeah, I think there’s two big pieces here, in terms of the experience and how you get it to be accessible. One thing, that I think people probably underrate, is that if you’re delivering a product that’s about presence, you really can’t have wires.
That’s probably not the most obvious place to go with this question, but I do think that if you want this to be something that a lot of people are going to experience, it needs to be a good experience. If you’re trying to deliver a sense of presence, you don’t want a wire wrapped around your neck. It really breaks the whole thing.
I think that is going to be the bar for VR and AR products of high quality, going forward. I think you’ll kind of see the market split into wired experiences (which are maybe going to be less accessible to a broader number of people) and then the things that are going to be the mainstream line of technology, even if it’s a little harder to develop… I think getting on that wireless path is really important.
Digitiser is a great idea 🙂
It's pity that accuracy is not good enough.
I think it may work better with smartphone taped to Touch 😄 So you could make a photo and record 3d coordinates then correct placement errors with photo data.
Or you could set a fixed point and use a lever to make error less. Like this:
So lever could rotate and move in "fixed point" but size of "tip-fixed" much less then "fixed-Touch". (object placed on rotating table)
Or it may help to use 2 of Touch controllers same time taped together.
Were they Q Touch or Rift with trackers? What was the distance?
> It wasn't very accurate though, Touch are good but the rotation of them magnifies the error of the pen tip since it's further from the centre of tracking.
Thank you for info!
It seems that accuracy will not be ok for writing 😞 But could be ok for marker/whiteboard letter sizes. 🙂
Also suggesting that a paper is flat and pen is fixed length could help a little to correct some of errors.
I've done pen tracking in the past using a Wiimote (as an infrared camera) and a whiteboard marker with the tip and insides replaced with an IR LED and a battery and button. Once set up it can control a PC mouse cursor as a virtual whiteboard.
"Were they Q Touch or Rift with trackers? What was the distance?"
CV1 Touch with 3 sensors. Not too far away.
"It seems that accuracy will not be ok for writing"
Yes and no. I was mainly testing for repeatable position tracking (touching the same point on an object several times). That will be mainly reliant on the camera tracking. But writing would benefit from the inertial tracking, which should give better short term precision but not longer term repeatability.
A basic stylus with an IR light at each end would actually be quite trackable with a Rift-S / Quest, if Oculus gave us any way to do it (very unlikely to ever happen). Two tracking points is enough for 5DOF (don't need one axis, unless you wanted calligraphy). As long as it's within the view of two cameras it would be fine. (If only one camera sees it, you'd need more tracking points to work out distance).
Valve Lighthouse tracking development seems pretty cheap (like under $2 per sensor) and is open to people making custom stuff. I've almost got an urge to play around with it, but then I'd have to buy base stations and stuff, and I don't like SteamVR.
> Yes and no. I was mainly testing for repeatable position tracking (touching the same point on an object several times). That will be mainly reliant on the camera tracking. But writing would benefit from the inertial tracking, which should give better short term precision but not longer term repeatability.
I see you point, I think this will be good enough if we do not need ink and real paper.
If we have solid plastic stylus without any ink we need not long term precision at all.
Also the reason we are writing small letter is ... to save paper. In VR we have limitless amount of "paper" (you could press "grip" button to move virtual "paper" to new position and use whole table surface instead of A4 paper size for writing, you could show 2 zoom levels both to see whole "paper" and "work paper" zoomed in).
So we need not small letters (and high precision) for VR math lessons. 🙂
Same time we have VR screen clarity less then ok for small letters (with size like on real paper).
Big letter will do ok with current VR display clarity.
I think it could work even without IR leds. Just a solid stylus taped to Touch. 🙂
But if you get some temporary tracking error it is easy to press "undo" for your last stroke.
Could you test if it is ok for remote learning? Will there be any perspective in this solution for older school/college/university?
I think price and complexity of solution could be barrier in mass adoption. So less IR leds could lead to more use cases and popularity. $300 personal solution for remote education will be fair enough I think. You could teach students from all the world 🙂
ps. it may be possible even to use it for geometry drawing... using ruler and tools same way as one using on whiteboards. Or even have a virtual solution to draw circles and straight lines.
Here's an example, a teacher doing a maths lesson in HalfLife Alyx using the whiteboard markers and windows. 🙂