-
Notifications
You must be signed in to change notification settings - Fork 790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft: Add model dragging in XR headsets by controllers or hands #4643
base: master
Are you sure you want to change the base?
Draft: Add model dragging in XR headsets by controllers or hands #4643
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this is exciting! Now I just need to hunt down a Quest so I can test this personally...
raycaster.ray.direction.set(0, 0, -1).applyMatrix4(tempMatrix); | ||
|
||
const group = this.presentedScene!.model; | ||
// @ts-ignore |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you try to replace these @ts-ignore
s by casting, e.g. const group = this.presentedScene!.model as Group;
or something? That's a lot more specific and helps me to understand the problem.
|
||
const object = intersection.object; | ||
// @ts-ignore | ||
object.material.emissive.setHex(0x333333); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The material might already have an emissive factor - can you save this in userData
and set back to the original instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is to highlight (brighten) the 3D model when the controller ray intersects it. Please check out the video again. ;) This highlight is removed back in the cleanIntersected()
method by calling the code.object.material.emissive.setHex(0x000000)
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I think I get it, you mean that the 3D model can have the emissive property set to a value other than 0x000000 before the controller ray intersects the 3D model? Then a question: can there be more than one 3D model in a scene at the same time?
BTW, highlighting (brightening) the 3D model on the hover is an optional improvement. I mean, we can opt out of that and simplify the implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly, but this caused me to notice a potentially bigger issue. We have one 3D model, but it can be composed of an arbitrary hierarchy of objects. We don't expose those sub-meshes in our public API, so they probably should not be independently selectable. Instead we should move everything when anything is selected. And we already have a highlighting concept in WebXR: the PlacementBox - maybe best to just use that.
I'm interested in what UX you're going for here - it might be nice to make it as similar to our existing AR UX as possible, given controllers. I'm open to other ideas, but I'd like to know how you're envisioning the experience.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please share links to an example where there is more than one 3D model in the one scene at the same time, and an example where one complex 3D model consists of multiple 3D models (parts)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As you can see in the video, a little highlighting (brightening) of the 3D model on the hover is not really needed at all. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like your style 😄. Fair point, though it makes a bit less sense with furniture. I need to think about this. Will this interaction work the same way with hands? Will we only support moving objects within arm's reach? I wish we had a UX person dedicated to thinking these use cases through, but I'm afraid it's just us for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this interaction work the same way with hands?
Yes, hands are also controllers in WebXR. Moreover, on Meta Quest you can switch between controllers and hands within the same XR session. You can put the controllers down and after a few seconds the controller will go to sleep and there will be an automatic switch to the hands. If you pick up the controller, it will wake up and switch back.
Will we only support moving objects within arm's reach?
No, this approach is already essentially standard, it works with both small and large objects, both close and distant objects (the ray is long enough).
I'll try to record other videos for you to make it clearer. But it would be better if you try it yourself (use different apps and games for a few days) when you get the headset. That will clear up most of the questions. :)
I wish we had a UX person dedicated to thinking these use cases through, but I'm afraid it's just us for now.
It's been already researched. Check out e.g. the Mixed Reality Design Language by Microsoft. ;)
Please also note that currently the user has no way to interact with the 3D model in the XR headset at all.
So my goal in this pull request is to add the simplest basic implementation. This will already improve the user experience significantly!
Then we can improve the implementation. Let's move step by step. ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot, that's quite helpful!
private controller2: any; | ||
private controllerGrip1: any; | ||
private controllerGrip2: any; | ||
private intersected: any[] = []; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these new Three.js APIs? Otherwise we should have types for them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I'll try adding types everywhere again.
scene.add(this.controller1); | ||
scene.add(this.controller2); | ||
scene.add(this.controllerGrip1); | ||
scene.add(this.controllerGrip2); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding controllers of the wrong size, you can see further down this file that scene.scale is changed when the object is scaled by pinch on a phone. We may need to add another level of object hierarchy to the AR scene to fix this. However, I believe that scale starts at 1, so if you're seeing a problem from the beginning, it may be a different issue. Can you add a screen recording by chance?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, I just saw your video on the other thread, thanks. Interesting, that does seem like a different problem than scene scale. Position maybe? Let's move discussion from the thread to this PR, FYI @cabanier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it's a position problem, not a scale problem. The controllers seems small because they are far away from the user. E.g. I've found that initially the controller model is in the right place, but then the 3D model drops to the floor and the controller gets lower.
Changing the isFirstView
variable to false
:
// WebXR may return multiple views, i.e. for headset AR. This
// isn't really supported at this point, but make a best-effort
// attempt to render other views also, using the first view
// as the main viewpoint.
let isFirstView: boolean = true;
for (const view of pose.views) {
this.updateView(view);
if (isFirstView) {
this.moveToFloor(frame);
this.processInput(frame);
const delta = time - this.lastTick!;
this.moveScene(delta);
this.renderer.preRender(scene, time, delta);
this.lastTick = time;
scene.renderShadow(this.threeRenderer);
}
this.threeRenderer.render(scene, scene.getCamera());
isFirstView = false;
}
and commenting/removing the following code:
const {theta, radius} =
(element as ModelViewerElementBase & ControlsInterface)
.getCameraOrbit();
// Orient model to match the 3D camera view
const cameraDirection = xrCamera.getWorldDirection(vector3);
scene.yaw = Math.atan2(-cameraDirection.x, -cameraDirection.z) - theta;
this.goalYaw = scene.yaw;
position.copy(xrCamera.position)
.add(cameraDirection.multiplyScalar(radius));
this.updateTarget();
const target = scene.getTarget();
position.add(target).sub(this.oldTarget);
this.goalPosition.copy(position);
fixes the problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you understand why this is? Any recommendation for a better approach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Controllers are added to the scene. The position of the scene changes, the controllers move with it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@FluorescentHallucinogen I tried out the demo. Very cool! The fixed version is already a big improvement for viewing on a Meta Quest.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@FluorescentHallucinogen 👍 great job so far.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried your solution with other models, the "drag and drop" doesn't seem to work for all GLBs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return raycaster.intersectObjects(group.children, **true**);
This should be set to true
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sprengerst But why? From what I can see, it's true
by default. See https://threejs.org/docs/#api/en/core/Raycaster.intersectObjects.
@elalish What are the chances of you getting your hands on Meta Quest? I thought it was a must have for the model-viewer maintainer. :) |
Well, now that we're looking to get serious about adding headset support, I'd agree. I'm going to get one ordered now. |
That's great news!
Please let me know once you order the headset and which model (Meta Quest 3 or Meta Quest Pro). And when you get it in your hands. ;) I expect when we add support for XR headsets and test the code on Meta Quest, we will automatically have Apple Vision Pro support since it's the same WebXR standard here and there. :) |
This pull request fixes a problem (described by me a year ago in the #3969 (reply in thread)) that users can't move and rotate models in the Meta Quest headsets.