Apple has been reported to be developing VR or ar head display for some time. Various reports and patent applications indicate that it may have many research directions. In a pair of patents granted to Apple by the U.S. patent and Trademark Office on Tuesday, Apple’s ar head display or future iPhone can use display light to track the movement of almost any surface, while finger devices may provide details of objects that users may touch to the system.

One of the advantages of AR head display is that it adds cameras to the settings. The same hardware can be used to perform object tracking and view the real-time position change of the object relative to the head display position. However, the resources required for object recognition and object tracking and the completion of orientation determination are a heavy burden for the system, so apple can solve this problem in these two new patents.

In the patent named “AV / VR controller with event camera”, apple put forward such an idea: the camera system does not necessarily need to track all pixels related to the tracked object at all times, and it only needs to track pixels when it only needs a rough check, so as to minimize the number of tracking. Apple proposed that the system would select specific pixels in the image related to the tracked object, and then provide readings of light intensity and other attributes. When the camera or object moves or changes its position, the change of light intensity of these pixels will exceed the set limit, which triggers the whole system to use more resources to search for the change of objects. In addition, the system can simply search for the same light spot instead of trying to detect the object. If there are multiple light points on the object, and the system only tracks these light points, it can use very few resources to determine the change of position and direction.

Although these light spots are easily reflected by ambient lighting on the object’s shell, Apple seems to be considering using light sources generated by other objects, such as LED indicator lights or display screens with display patterns. If it is the latter, as long as the screen is visible, it can provide sufficient positioning and direction data for the system. This light based reference point is not necessarily for visible light, because Apple suggests that the lighting mode can include non visible wavelengths. This will provide a hidden system that will not be seen by users, or a system that can be invisible and use different wavelengths of light, so that the system can work with multiple users or objects.

In the future, Apple's ar head display or iPhone can track finger changes through ray tracing

The second patent relates to data collection, which specifically covers how the system obtains user touch information. Such a system needs to know how to reproduce the feeling that users may feel when touching real-world objects, so that it can simulate the effect with tactile feedback tools. In the patent, “computer system with finger device for sampling object properties,” Apple proposed a device that uses a finger to detect features of real-world objects that users may come into contact with. The sensor in each finger device can read the effect of the real-world object surface, such as surface contour, texture, object color and pattern, acoustic properties and weight. When combined with other elements, such as the camera on the head display, the system will be able to collect a lot of data about the object, and then provide it to the user.

Although worn at the end of the finger, the patent shows that the user can still experience the feeling of touching an object, because the finger pad will remain unobstructed. The sensor can detect by touching protrusions extending beyond the fingertip, or by detecting changes in the way the user’s fingers touch the object. The collected data about objects can be compiled and archived for future use by the system, and may be shared as a database. This information can then be used with tactile feedback elements to simulate touching a user or other user’s object.

Apple submits a large number of patent applications every week, but although the existence of patent applications indicates the areas of interest in Apple’s R & D work, there is no guarantee that these ideas will appear in future products or services.

In the future, Apple's ar head display or iPhone can track finger changes through ray tracing


Editor in charge ajx

Leave a Reply

Your email address will not be published. Required fields are marked *