U.S. Patent No. 10,395,428: HMD transitions for focusing on specific content in virtual-reality environments

Issued August 27, 2019 to Sony Interactive Entertainment Inc.
Priority Date: June 13, 2016

Summary:
U.S. Patent No. 10,395,428 (the ’428 Patent) relates to bringing objects into focus on the screen of a head mounted display (“HMD”) based on a user’s gaze. The ’428 Patent details a method of an HMD receiving real-world images from the HMD’s forward facing camera(s), or images of a virtual reality (“VR”) scene and while rendering those images, also using sensors in the HMD to detect an HMD user’s gaze direction. The HMD user’s gaze direction is used to bring objects in the rendered images into focus on the HMD’s screen. In some embodiments, the sensors measure the amount of time that a user has been gazing at an object to meet a minimum threshold before adjusting the zoom factor to bring an object into focus. An object in a real-world image from the HMD cameras may be both optically and digitally zoomed, while an object in a VR scene may be digitally zoomed. The zoom factor may be adjusted based on the user’s known vision characteristics. In pre-recorded images, historic data from other users may be used to determine optimal zoom factor settings based on what may have caused other users dizziness or motion sickness. 

Abstract:
Methods and systems for presenting an object on to a screen of a head mounted display (HMD) include receiving an image of a real-world environment in proximity of a user wearing the HMD. The image is received from one or more forward facing cameras of the HMD and processed for rendering on a screen of the HMD by a processor within the HMD. A gaze direction of the user wearing the HMD, is detected using one or more gaze detecting cameras of the HMD that are directed toward one or each eye of the user. Images captured by the forward facing cameras are analyzed to identify an object captured in the real-world environment that is in line with the gaze direction of the user, wherein the image of the object is rendered at a first virtual distance that causes the object to appear out-of-focus when presented to the user. A signal is generated to adjust a zoom factor for lens of the one or more forward facing cameras so as to cause the object to be brought into focus. The adjustment of the zoom factor causes the image of the object to be presented on the screen of the HMD at a second virtual distance that allows the object to be discernible by the user.

Illustrative Claim:
1. A method for presenting an object from a real-world environment on a screen of a head mounted display (HMD), comprising: receiving images of the real-world environment proximal to a user wearing the HMD, the images received from one or more forward facing cameras of the HMD and processed by a processor of the HMD for rendering on the screen; detecting a gaze direction of the user wearing the HMD, using one or more gaze detecting cameras of the HMD that are directed toward one or each eye of the user; analyzing the images to identify an object captured in the images of the real-world environment that correlates with the gaze direction of the user, wherein an image of the object is rendered at a first virtual distance that causes the object to appear out-of-focus for the user; and generating a signal to adjust a zoom factor for lens of the one or more forward facing cameras so as to cause the image of the object to be brought into focus, wherein adjusting the zoom factor results in the image of the object to be presented on the screen of the HMD at a second virtual distance, wherein the signal is generated upon determining the gaze direction of the user directed toward the object lasts at least a pre-defined length of time, and wherein the zoom factor is adjusted to cause the object to be rendered at the second virtual distance for a pre-defined period and upon expiration of the pre-defined period resuming rendering images from the real-world environment without the zoom factor.