The reality-virtuality (RV) continuum is a concept introduced by Paul Milgram and Fumio Kishino [1] in 1994. It describes a spectrum that ranges from a purely physical reality (the real world) to a purely virtual reality (VR; a completely computer-generated environment), with various degrees of mixed reality (MR) in between. This continuum is “realized” by different types of displays [2] to encompass different levels of immersion and interaction, allowing for the classification of different types of environments and experiences. The RV continuum helps us understand the varying levels of immersion and interactivity that technology can provide. As technology advances, the boundaries between these immersion levels can become more fluid, and new hybrid experiences can emerge. The continuum is particularly relevant in fields such as VR, augmented reality (AR), and MR, where researchers and developers aim to create more compelling and natural experiences that bridge the gap between the physical and virtual worlds. We used ChatGPT (OpenAI) [3] to gauge the current state of the RV continuum. According to ChatGPT, the continuum is often divided into several main categories (note, we adapted the ChatGPT results and enhanced it with concrete examples, where necessary; Textbox 1 [4]). The original ChatGPT transcript is shown in Multimedia Appendix 1 [3].

What is often overlooked in this concept is the act of diminishing real objects (or persons, animals, etc) from reality, rather than augmenting the reality with virtual things [8,9]. An introduction to the topic can be found in Cheng et al [10]. A reason for this is that diminishing something from reality needs, in general, a sophisticated understanding of the real scene or environment to make the diminishing aspect convincing. In AR, the real world is just overwritten with a virtual object. In diminished reality (DR), however, the real-world part that is augmented or diminished needs to seemingly fit to the reality around it. In addition, this should all be performed in real time when a user is walking around the real world, and an algorithm has to do the following (note that the first 3 items are part of the Extent of World Knowledge axis of the taxonomy by Milgram and Kishino [1]):

All of this has to be done not only in real time but also with very high precision. The inserted virtual object has to fit seamlessly into and make sense with the reality; minor discrepancies will appear to be a glitch and will be noticed immediately by the user, as we recently observed in a DR user study [11]. In fact, we think that diminution and augmentation require fundamentally different technologies. In our opinion, an augmentation may be needed to alter reality at a certain position with regard to other (real) objects (eg, displaying a patient’s tumor as an AR hologram on the patient in front of you, at the real position, such as for needle guidance [12]), but no seamless and semantic fitting is necessary. As soon as a virtual object needs to fit into the scene semantically, we consider this to require diminution. Hence, for augmentation, you only need a volume rendering process with some basic options, such as position, size, and transparency. For diminution, however, additional fundamentally different technologies are needed. The scene has to be analyzed and understood, and a meaningful replacement has to be generated and inserted as an AR hologram. An example could be glasses that are removed from a person in front of you.

In summary, the user has to get the impression that the real, diminished object does not exist at all in reality [13]. Besides sophisticated algorithms, this course of action needs a considerable amount of computing power. Fortunately, there has been tremendous progress in both areas during the last years, with deep learning–based approaches and GPUs that can run these kinds of algorithms, even in real time. As a result, DR has already found its way into some applications [5], such as virtual furniture removal for redecorating purposes (eg, IKEA Kreativ [14]). Other possible applications for DR include the following:

Nevertheless, for all these aforementioned reasons, we think that DR needs to be more prominent on the RV continuum, as shown in Figure 1 [15], without delving deeper into the broad topics of mediated reality [9] or even multimediated reality [16]. This will not only assist in the decision-making for hardware (devices) and software that are needed for new DR applications but also help unfamiliar users to get a better understanding of the entire extended reality (XR) topic (note that we are addressing this revision to the continuum purely from an application or user point of view [POV], not from the POV of an MR researcher or engineer). An example application for DR could be the real-time anonymization of a face via XR. There is a huge difference if a device detects the eye area and simply inpaints a black bar over the eyes (without considering the surrounding facial area) or inpaints the eyes with different or meaningful ones that fit perfectly to the surrounding facial area. The black bar approach can probably be performed on a current smartphone, whereas the second approach needs much more sophisticated hardware and computing power, with an integrated GPU that can run a trained, deep inpainting neural network in real time (note that a user with an XR headset would move around in general, which also changes the POV on the face to be anonymized, so the inpainting algorithms also has to be executed continuously in real time). In this context, we also think that the upcoming Apple Vision Pro will push the limits in DR, because it is a video-see-through device that can enable DR to reach its full potential [17]. In fact, the Digital Crown hardware of the Apple Vision Pro, which also exists for the Apple Watch, should enable us to seamlessly walk along the whole RV continuum (back and forth) and bring medical DR applications to reality, which are still almost nonexistent currently [18]. A potential example of the photo-editing capabilities of newer cell phones as a diminution operation is shown in Figure 2 [15]. In this medical example, DR enables the removal of a skin tumor virtually from a patient’s face before surgery.

Diminished virtuality (DV), however, remains a curiosity, and not even ChatGPT—which has been fed with 300 billion words from books, web texts, Wikipedia, articles, etc [4]—can handle this term as of January 2024 (Textbox 2; the original ChatGPT transcript is shown in Multimedia Appendix 2 [3]).

DV can perhaps be best imagined as starting on the extreme right side on the RV continuum, which contains only virtual objects (virtual environment and VR), and going along the continuum toward the left to reach diminished virtual elements (see the figure of the RV continuum by Iriqat and Vatansever [19], and note that this is especially of interest for computer-generated structures that are very complex and where only some parts need to be diminished). This should be possible with the Digital Crown hardware of the Apple Vision Pro: when fully immersed, the virtuality is reduced to make the real world partly visible. Apple describes scenarios where people might want to increase the pass-through when they intend to interact with a nearby physical object or read text on another device. Another real-world scenario could be a so-called tumor board meeting that is held in VR, where several experts from around the world gather to discuss a rare case and certain virtual elements (images, texts, etc) are diminished because of privacy issues (note that the mixture of reality here is also modifiable): an externally tuned-in expert (who is traveling) can see certain patient information because he or she belongs to the clinic where the patient is located, whereas another externally tuned-in expert who does not belong to the clinic of the patient should not see or hear certain patient information. However, the full potential of DV would unfold when the virtuality is diminished in a way that also fits with the upcoming reality. An example would be a real-world person showing up in VR between virtual objects. Imagine layers of virtual and real context stacked up but still seamlessly interplay with each other for the final output. In fact, this layer-based concept could also extend to mediated and even multimediated realities, where, for example, different sounds (real and virtual) interplay with each other, depending on their location and depth. “Enaudio” (the “hearing” equivalent to envision) hearing real rain outside in the real world that are mixed with virtual sounds while inside a virtual room. Real and virtual input from different senses could also be mixed; for example, there is a (virtual) hole in the VR room and the (real) rain falls inside this virtual room, generating simulated sounds as they hit virtual objects in the VR room. The same also works for outputs and their combinations; for example, the real voice from someone wearing the headset is mixed with virtual sounds, or real smells are mixed into VR by generating the corresponding virtual objects there. Finally, some could argue whether the RV continuum really ends on one side with physical reality [20], because the brain creates reality—comparable to a head-mounted display creating VR—but does not detect it. This could be discussed and explored by the community in the future, and in this regard, we want to propose a diagram of XR that loops back (as opposed to a continuum on a straight line with 2 ends) as an inspiration (Figure 3). In summary, XR is a niche yet growing topic, because more and more consumer devices with better hardware have become available during the last few years. In parallel, artificial intelligence (AI)–based algorithms have made 1 breakthrough after another, which can, for example, be explored to develop novel applications for an intelligent health care metaverse [21].