The U.S. Patent and Trademark Office today officially issued a series of 62 newly granted patents for Apple. Among them, there are two patents related to Apple's future mixed reality glasses. The first is an augmented reality overlay about coordinating the camera and display system, and the second is foveated controls.
Apple's first patent concerns a head-worn device, such as mixed reality glasses, that could have a camera. During operation, the camera can capture moving images of the real-world environment around the user, allowing control circuitry in the electronic device to display real-world content to the user. In addition, computer-generated content (sometimes referred to as virtual content) may be provided to users.
The headset can operate in a virtual reality mode, in which virtual reality (computer-generated) content is displayed to the user. In this mode of operation, the real content captured by the camera is not displayed. The headset can also operate in mixed reality mode.
In mixed reality mode, virtual content is overlaid on real-world content captured by the camera. The virtual content may be, for example, text, graphics, moving images and/or other content displayed on top of the displayed portion of the real world content.
To ensure content is displayed satisfactorily, display and camera settings can be dynamically adjusted when transitioning between virtual reality and mixed reality modes.
The camera can capture images at the camera frame rate (sometimes referred to as the image capture frame rate), and the display can display content (eg, moving images) at the display frame rate. Camera and display frame rates can be adjusted dynamically when transitioning between virtual reality and mixed reality modes, and operations such as camera reconfiguration can be performed with minimal disruption to system operation.
Figure 1 of Apple's patent is an overview of a system that supports a pair of glasses. In Figure 2, we see support structure #16, which may sometimes be referred to as a housing, which may be configured to form the frame of a pair of eyeglasses.
Apple's granted patent 10,748,340 was originally filed in the second quarter of 2018 and published today by the USPTO.
Glasses: Image Enhancement and Gaze Tracking
Apple's second patent, 10,747,312, covers mixed reality glasses with gaze tracking. The glasses may have a camera and a display. Cameras can capture real-world images, which can then be overlaid with content such as text and graphics to produce mixed reality images.
Apple notes that showing mixed reality content to users can be challenging. For example, the presentation of mixed reality images to the user may be interrupted by head and eye movements. Users may have different visual capabilities, and if not careful, these mixed reality images can cause disorientation or motion sickness and make it difficult for users to identify items of interest.
Electronic devices such as glasses may have a display and a camera. Cameras can be used to capture real-world images. Real-world images can be displayed on the display for use by the user.
Control circuitry in the device can use gaze tracking systems and other sensors to gather information about the user's gaze point. Control circuits can also gather relevant information, such as information about content, motion, and other image properties, by analyzing real-world images. User visual information such as user visual acuity, contrast sensitivity, field of view, and geometric distortion may be provided to the control circuitry.
The input-output circuit may be used by the control circuit to collect user input, such as user preferences and user mode selection commands, and may be used to collect location information and other input information.
Based on the gaze point information and/or other collected information, the control circuitry may display supplemental information on the real-world image on the display. Supplementary information may include augmentations such as icons, text labels, and other computer-generated text and graphics superimposed on the real-world image, and may include augmented image content such as enlarged portions of the real-world image. If desired, real images can be enhanced globally or locally by adjusting contrast, changing image colors, enhancing edges, adjusting for geometric visual distortions, and performing other image enhancements on real images.
Apple's patent FIG. 2 is a diagram showing how gaze tracking system #16 collects eye position information on the user's eyes. In a typical case, a gaze tracking system may include components for monitoring both eyes simultaneously; FIG. 7 is a flow diagram of illustrative operations involved in providing mixed reality content to a user.
In box #84, we see that it is related to people who wear prescription glasses. Apple states: "During the operation of Box #90, user data collected during the operation of Box #84 (e.g., details of the user's visual impairment) may be considered. For example, if the user's line of sight has geometric distortions, it can be Compensating geometric distortion is applied when displaying the image on display #14, and the content can be scaled appropriately if the user's visual spatial acuity is low.
Apple's 10,747,312 patent was originally filed in the first quarter of 2019 and published today by the U.S. Patent and Trademark Office.