Over the Threshold

Posted on May 29, 2017 by Dr Lyn Robinson

Mat Collishaw: Thresholds [https://www.somersethouse.org.uk/whats-on/mat-collishaw-thresholds]

Today I experienced my first 6 minutes of immersive, interactive virtual reality (VR), at Thresholds, Mat Collishaw’s artistic interpretation of William Henry Fox Talbot’s first photography exhibition in 1839.

“Using the latest in VR technology, Thresholds will restage one of the earliest exhibitions of photography in 1839, when British scientist William Henry Fox Talbot first presented his photographic prints to the public at King Edward’s School, Birmingham.”

This differed from my previous encounters with VR, which have used Google Cardboard apps. Whereas Cardboard offers a 360 degree visual experience, the sense of presence, or immersion in the unreal world is limited; whilst it is possible to look around at everything filmed by the camera/generated by the app, it is not possible to interact with or impact upon the scripted world.

There is an often-overlooked difference between 360-degree video and virtual reality, in that the latter offers opportunity for participation in a simulated world, alongside a fuller sense of immersion or presence in the simulation. VR requires more computer processing power, hence its association with head mounted displays and earphones (HMDs) connected to a computer, rather than a viewer holding a smartphone against the eyes.

I have, nonetheless, enjoyed Google Cardboard apps immensely, although after today these charming worlds will seem a bit tame.

Thresholds then, employs sophisticated technology to simulate an environment in which the viewer can walk around freely, and in which there is some further sense of presence afforded by the ability to see the hands as orange clouds, by holding them up in front of the headset (there was a short, second or two, time-lag until the ‘hands’ appeared). The virtual hands could interact with documents in cabinets within the environment; swiping at a document caused it to ‘leap’ out so that it could be examined more closely. This did not work for me however, although I managed to summon up one leap, the document pretty much smacked me in the face and then scurried swiftly back to its place in the cabinet. ( I felt a bit like Ron Weasley in Harry Potter, when the spell simply doesn’t work for him..) Further swipes, tried on all the other documents, were ineffective. On querying this with one of the technicians, I was told it was likely to be my bad swiping technique.

There are clearly implications for libraries/archives/museums here however. The short briefing given before we entered the environment, recounted that archivists had been consulted in the design of the program; this type of simulation could allow anyone, anywhere, to examine virtual renderings of rare, fragile documents, at a time and place convenient to them, personally (assuming good swiping technique).

I found the alloted six minutes too short. I really wanted to stay in this unreal world, which although somewhat cartoon-like, was delightful. I dutifully noted features mentioned to us in the briefing – the mice scampering across the floor, the cobwebs in the crevices, the moths fluttering around the lamps and the swirling smog outside the virtual windows. The sounds of the 1839 rioters seemed a bit remote, but I remember hearing them. The fireplace emitted real heat, although to me, the flames appeared as bright green. The background sound of the clock, shown above the entrance, ticking, was somehow comforting.

I didn’t like the heavy headset. We were warned to make sure the contraption was comfortable before we entered the simulation, but even though my apparel seemed comfortable to begin with, I soon felt the need to readjust the way the visor sat against my eyes. I felt I had made the headset too tight, in order to stop it slipping. I soon felt it was pulling at my lower eyelids, and consequently, my vision seemed a little blurry.

Other participants appeared in the simulation as white ghosts, to avoid collisions – there was another time-lag effect here as people appeared (to me) to be either stationary, or to move at lightening speed to another postition.

Another participant asked about glasses – the headsets don’t adjust for vision impairment, and in a short demonstration such as this, I would agree this is a bit too much to hope for. However, vision correction is something that VR designers should think about, as wearing glasses under a headset is annoying and uncomfortable.

I haven’t commented on the exhibition itself; the artist Mat Collishaw did not set out to recreate the original event, but to create something new, based upon original likenesses, documents, and archival materials. I don’t think it matters that we don’t have enough knowledge about the original exhibition to recreate it exactly. We are in a different time now, and the artist’s creative connection with the past was certainly enough to spark interest in the history of photography, and indeed the social context in which photographic developments occurred.

There are parallels in this virtual recreation of Fox Talbot’s first photography exhibition, with attempts to recreate performances from archival documents. Notably, to what extent it is ever possible to recreate an event, or an occasion of any sort? Our DocPerform project considers this question, along with the more fundamental issues of how we define and record documents, and how we approach the processes of documentation. What can technologies such as VR offer in documenting performance?

Leaving the conceptual questions of documentation aside, technology itself raises issues. How can we remove the interface? The face visor is clumsy. It reminds even those of us who are more than willing to jump into virtual worlds, that we have something physical and uncomfortable stuck to our face. How could we improve the design of VR systems? Contact lenses perhaps? Some other small, un-noticeable brain-computer interface?

Further, a more immersive environment could be encouraged by enhanced use of sound, and by employing technologies to replicate smell and touch.

But no matter. Mat Collishaw (@matcollishaw) is to be contragulated on this fabulous installation. Look at what is there. And look at what we see through the headset. It’s not bad.

@lynrobinson cc-by

—

*Read CityLIS student @adafrobinson ‘s account of the Thresholds exhibition! Thresholds and Time Travel.

Documenting Performance: the backstory

Posted on October 29, 2016 by Dr Lyn Robinson

This post first appeared on http://documentingperformance.com on October 16th 2016. It describes the background to the symposium held on 31st October 2016, at City, University of London: “The Future of Documents: documenting performance”

The Twitter hashtag relating to this event is #docperform.

Cross-posted here for reference.

—–

There are now just two weeks to go until ‘Documenting Performance’, our exploratory, interdisciplinary symposium on the concept of performance as a document, and the ideas, theories and practices around the documentation of performance. We are hoping that our initial event will spark further interest to form a longer term project, which we are calling DocPerform.

My initial feelings are positive, as for this event, both the response to our call for papers (27 abstracts, and I had to turn another couple away after the deadline) and the number of registered attendees (75) has been stunning. The event is now sold out, but we are running a wait-list so please email me [lyn@city.ac.uk] if you would like to come but have been unable to secure a ticket. If you are holding a ticket that you know you will not use, please cancel via Eventbrite, so someone else can join us.

We would like to say a very big ‘thank you!’ to everyone who has sent us ideas, and registered for the event. It was hard to make a choice about which papers to include, but I hope that everyone will agree that our Programme, showing the range of approaches to how we currently understand performance as a document, is pretty good! We are very excited about the day, and look forward to meeting new colleagues interested in documents and documentation.

My original idea was to host a series of seminars within our research centre, (Centre for Information Science) to examine how the conceptual view of the document is developing in the 21^st century. The question of what is / is not a document is considered in work of Otlet, La Fontaine, Briet, Buckland, Lund, Latham, Gorichanaz, Robinson and other writers within the field of library and information science, with the earliest papers having been written at the start of the 20th century. The obvious and fascinating question would be ‘what next’? I subsequently felt, however, that it would be helpful to take a step back, and to consider whether any unifying perspective could be applied to the increasing number of entities already extant upon the documentary landscape. Such a framework would be valuable to any discipline concerned with the organisation and preservation of its domain output, and could also be used to help formulate understanding of future document types, real or conceptual.

My working draft of such a framework is shown at the end of this post. Several colleagues contributed to my ideas, and I would like to mention them briefly as part of the background to our forthcoming event.

When my friend and colleague Prof Adrian Cheok joined City a few years ago, I was inspired by his work on the multisensory internet, (transmitting the sensations of taste, smell and touch in addition to sound and vision via the network), and this, plus developments in pervasive computing, wearable technology, human-computer interfaces and virtual reality, brought to mind the idea of the library as the ‘Experience Parlour’. I came across this idea in The Library of the Future, a book by Bruce Shuman, written in 1989. In his series of scenarios for the future library, he suggested one where reading a good book meant actually living, or experiencing it. I wondered if this form of immersive document was one possible future for documents, and consequently for libraries and other cultural, collection orientated institutions.

Whilst science fiction provides us with many depictions of immersive documents, (for example the holosuite in the Star Trek universe), at the present time fully-immersive documents, wherein the reader perceives a scripted unreality as reality, do not exist. However, many of the entities to which we refer as documents offer the reader a partially-immersive, or complex, experience. These documents provide the reader (broadly interpreted to include related terms player, participant, viewer, audience member) with a compelling and realistic world, but one which is delineated to varying extents from actual reality. The reader knows that they, and the document with which they are engaging, are a part of the real world (for want of a better phrase). This is in contrast to the experience delivered by fully immersive-document (as yet theoretical) where the reader cannot distinguish between the unreality and reality, and the interface between human and computer is invisible and frictionless.

A series of encounters over the past 3 years provided insight for developing a framework to help us understand what partially-immersive documents might be, and how they could relate to other documents.

Videogames

In June 2014, Adrian invited me to a seminar on the history of video games at the Daiwa Foundation. The enthusiasm with which players spoke of their interaction with early games indicated how strongly they identified with and enjoyed the ‘unreal’ worlds of the game. Whilst some games offered realistic environments, others offered clearly computer-generated spaces, yet the players still engaged time after time. Video games are an example of partially-immersive documents. They provide the ‘reader’ with a compelling world, but one which is clearly separate from reality. Even the most enthusiastic player knows the game world is constructed.

The playing of historical games was not the only point for consideration during the seminar; the issue of preservation was paramount, and discussion turned towards which characteristics of the games needed to be preserved. The list went beyond saving a copy of the software (computer program), conservation of its associated hardware or simulations of now-extinct computer operating environments, to ensuring that feelings, such as elation, despair, desire to win, anxiety, happiness or nostalgia, all possibly experienced by the player during the game, could also be guaranteed. The question then became more complex; are we attempting to preserve an historical game to be played afresh in a contemporary time, or are we attempting to preserve something more, by including something of the past environment, and even the experience/feelings of the player or players?

So, if we intend to record and preserve ‘experience or feelings’, do we mean that we are attempting to ensure that (re)playing the preserved game will generate the same sorts of feelings that were invoked generally in previous times, or are we attempting to reconstruct individual experiences of a game, exactly as they happened on a previous occasion, so that somehow the players or readers feel exactly the same as they did during a previous, specific occasion (for example the excitment of completing a level, or of a winning goal). If this latter reconstruction were technologically possible, it should also be possible to experience a game from the viewpoint of someone else, by replaying (and experiencing) their memory track, alongside the game timeline.

Experience is hard to define, and even harder to code for access and reuse. This level of enhancement to partially-immersive documents remains theoretical, but it is possible to imagine that a layer of ‘experience’, general or personal, could be added to complex, or partially-immersive document formats. The concept of adding experience or feelings to a document, or record of a document (documentation), introduces the need for us to consider who the document or documentation is for. Whose point of view are we recording? In theory, a record could be made of every individual experience of every individual document, including the perspective creator(s) or author(s).

These concepts are reflected in recent developments in journalism, where 360 degree recording is used to film documentaries. The realistic, ‘immersive’ nature of these films enhances emphathy from the viewer, evoking feelings similar to those felt by those present at the time the recorded events took place. See: Virtual Reality, 360 Video and the Future of Immersive Journalism, by Zillah Watson, 1st July 2015.

At this stage then, we have the concept that video games, or simple copies of video games are partially-immersive documents, but that copies or recordings could also offer an additional layer of general ambience/sentiment, or personally specific thoughts and feelings. These enhanced copies, also partially-immersive documents, would be considered new documents in their own right, although associated with the original video game.

Interactive Narratives

At around the same time that I encountered the video games enthusiasts, I became aware of the convergence of video games with interactive fiction. These latter digital documents, which increased in popularity and number with the ready availability of consumer technologies such as smart phones and tablets, attempt to engage the reader by allowing input to or participation in the script. They offer interactive engagement across a range of platforms, and can reach out to the reader via texts, emails and phone-calls. Innovative software such as that which reads emotion from facial expressions can be used to tailor interactive fiction to the individual reader. The difference between an interactive fiction and a game is hard to specify, although one distinction I came across suggested that although many games have a narrative aspect to them, this is not required.

Interactive fictions are also partially-immersive documents, proferring experiences which are distinguishable from reality, yet which offer varying degrees of participation and immersion in compelling, unreal worlds. Like the worlds of video games, interactive narratives could be recorded with the intent to evoke time or context specific feelings, or indeed the feelings or experiences from a given player at a given instance.

Documentation

Technologies which underpin partially-immersive (and perhaps eventually fully-immersive) documents such as video games and interactive fictions, can also be used to record and preserve them. The most straightforward way to think of this is when making a copy of the software. In some ways, identical copies of video games or interactive fictions can be considered documents which are the same as the originals; compare with FRBR‘s ‘manifestation’ level for books, where the copies differ only at ‘item’ level. If we are thinking, however, also to record the ‘experience’ of the player or ‘readers’, then in adding layers of information to the original computer programs we are creating further new documents. Documents which contain a level of immersion, (experience, feelings) associated with a given reader, player, or creator.

Video games and interactive narratives can also be considered to possess temporality. They arguably exist only whils they are being ‘read’ or ‘played’. Whilst the concept of a book or paper may be considered to exist as long as its physical form is extant, there is the question of whether the video game or the interactive narrative exists as its computer program on some kind of storage media, or whether it only exists when being played. Similary, we could suppose a book only exists either in print or electronic format whilst it is being read, and that the ‘bookness’ is separate from its representative media. This is not a usual interpretation however. The concept of analogue documents, in contrast to digital, partially-immersive and immersive documents is important, and forms the basis of the draft framework suggested below.

Immersive Theatre

The increasing popularity in London and other cities for immersive and participatory theatre added performance to the mix of partially-immersive documents.

Performances, displaying some parallels with video games and interactive narratives, only exist for a given amount of time, and unless one counts their documentary containers, such as the written script, or computer programme and data records, they are intangible forms of document. Like video games and interactive narratives, a performance can offer varying degrees of participation, and feelings of immersion.

Several of my students were/are fans of immersive theatre, attending shows such as Punchdrunk’s ‘The Drowned Man’ and Thomas Otway’s ‘Venice Preserv’d’ several times over. I was invited to go along to a performance of Venice Preserv’d’ where I witnessed first hand the desire of the audience to participate in the show; they readily and willingly suspended reality to a significant extent. Again I thought of documentation, and how a performance could be regarded as a document being ‘read’ by the audience. To some extent, all performance is interactive/participatory, as the audience is reacting internally to the show even if they are sitting as passive observers. Some performance, however, offers the audience higher levels of interactivity, from singing along with the cast, to joining the actors for part of the show, and participating in the performance. Varying degrees of participation in temporal events, also offered by video games and interactive fiction, are traits of partial-immersion, and a performance could therefore also be considered as a partially-immersive document. The boundaries of ‘what is a performance’ is a valuable discussion, but left for another occasion.

In the case of performance, this raises the (unoriginal) question, that if the performance itself is a document, is documentation (recording) of a performance yet another document? And a further, also unoriginal question, does the documentation intended to reconstruct a performance actually create yet another performance? Would it be possible to recreat exactly a performance from the viewpoint of a ‘reader’ or audience member, in the same way as for recreating the exact experience of playing a video game at a particular time and place? We can consider too, the recording of a performance from the perspective of the creator, or of a performer.

When preserving or recording a performance then, are we documenting just the performance per se, or also the thoughts, feelings, and interactions of members of the audience? Should we attempt to garner something from the actors in each performance to improve the validity of the record? A performance is clearly more than something that can be represented by a script, photographs or a video recording. It is necessary, when documenting performance, to say something about temporality and participation (new to me, but of course unoriginal from other disciplinary viewpoints). We must distinguish between a record of somthing which is intended to be experienced for the first time by a reader, and a record which includes something of how it felt to have participated on a previous occasion. The embedding of thoughts and feelings within any sort of document has yet to be fully explored concpetually, as well as technologically.

Fandom

In September 2013, Ludi Price joined the Centre for Information Science as my research student, beginning her PhD on the information behaviour of cult media fans. Our work lead me to appreciate and consider the art of cosplay as a kind of performance, and thus a form of partially-immersive document. Clearly, there are links between cosplay and participatory theatre, and the question of should anything of this be documented, and if so how, appeared again.

Dance

Adrian invited Ludi and I to a talk he had arranged by the artist Choi Ka Fai, who was interested in recording patterns of the electrical signals which generate muscle contractions (is the body itself the apparatus for remembering cultural processes?). His idea was to attempt to record the movement of dancers, and to play them back on another dancer, to see if movement could be recorded and transmitted. If this is ever possible, it would allow one person to almost become another, experiencing not just a recording of a performance, but what it felt like to be part of it. This could in theory be one of the experiential layers added to the concept of the immersive or complex document, but the work so far remains experimental.

Immersive Documents

By this time, I had published two short papers on the concept of the multisensory, immersive document. Sarah Rubidge, Professor Emerita (Dance) at University of Chichester, came across these papers and subsequently contacted me. Sarah had been developing immersive, choreographic installations for two decades, and was interested in how to document such participatory experiential works beyond using words and photographs. Sarah’s ideas of using 360 degree camera recording or virtual reality to represent these forms of performance art were similar to my own ideas; that technologies such as VR, together with multisensory rather than merely multimedia recording, might allow us to more accurately document the experiential nature of performance and related works for future scholars, students and historians.

For the moment however, the work remains conceptual, as although 360 degree recording improves the visual experience of the record, multisensory recording, especially that related to the sensation of movement, is in its infancy.

Clearly if we could make a complete, multisensory recording of a performance, the recorded document could be read either to experience the work for the first time, or to experience it again as either yourself on a previous occasion, or as someone else.

Sarah also suggested that I attend the Digital Echoes Symposium at C-DaRE, the dance research group at Coventry University. Here I met several dancers and researchers, who were interested in the documentation and archiving of dance. One of these participants was PhD researcher Rebecca Stancliffe, who introduced me to a project called Synchronous Objects, where dance movements were converted into data, then into other objects for visual representation. Rebecca had found the work of Paul Otlet, and was working on the concept of what is a document from the discipline of dance, totally unrelated to LIS. The day was fascinating, and I learnt about new things to document, such as body memory, in addition to audience recollections and dancer insights. The way dancers perceive a performance, their work, is totally different from how traditional documentalists think of it.

Performance Art

Over the summer of 2016, I attended a course at the Tate Modern on Framing the Performance, led by Georgina Guy. Georgina led the class for four, weekly sessions, in which a group of us considered how Tate had displayed and documented a range of performance art installations. We were invited as a class to consider what we needed to know about a work, in order to store, archive, preserve, access, use and ultimately understand it. A fascinating field.

Data

My most recent encounter with partially-immersive documents came as I was preparing for one of my own classes, a session for a module called Digital Information Technologies and Applications. The theme was data. In thinking about how to demonstrate data, I came across several artists whose work used a data input to bring into being a constantly changing visual artwork. See for example: http://www.worldprocessor.com/ . Participation then, does not only imply human input, but also that of data.

Interdisciplinarity

Alongside all of this, I undertook some further literature reviews, looking for work relating to documents, and documentation of performance, but from outside the LIS discipline. There was plenty. Performance artists from all fields, theatre, dance, music, performance art, were all represented in the literature on documenting performance. All these previously unimagined colleagues working on documents and documentation from a completely different background to LIS. I started to think about an event to bring the two cohorts together. On mentioning my interest to a friend, Tia Siddiqui, she put me in touch with a colleague of hers, from Rose Bruford college, Joseph Dunne, who had a background in performance, but who was also interested in documentation. I told Joe my idea of a symposium for both LIS and Theatre and Performance advocates, to share ideas on how performance can be regarded as a document, and how we can best record and preserve such partially-immersive entities for reuse if and whenever necessary.

Joe agreed to work with me on the idea, and our first collaborative outcome is this symposium ‘Documenting Performance’. The event sits within the wider consideration of the document by researchers at the Centre for Information Science, and specifically considers performance, although that is not to say video games, interactive fiction and other examples of partially-immersive documents such as performance art or information art do not warrant attention, but perhaps for another day.

Pulling all of these threads together, I think we can understand a partially-immersive document as one which affords the reader a compelling and engaging environment, for which the boundaries between reality and the imaginery, scripted world are blurred. Such documents present in varied media formats, and some of their characteristics may overlap with those of fully-immersive documents. Partially-immersive documents are interesting, because they already exist, and because of the questions they raise in respect of description and indexing, recording, access, preservation and use. They push at the boundaries of traditional documentation and demand that we reconsider our definition of documents in the age of VR, AR and mixed reality. Documentalists need to embrace the characteristics of participation and experience in our work, if we wish to fully maintain the 21st century record of humanity.

Perhaps the most interesting aspect of performance as a document is the capture and recording of the thoughts and feelings of those participating, whether as reader, (audience member), performer or creator, alongside the more usual physical representations such as a script, or a video. Understanding and rendering of this participatory layer is arguably what will allow us to move forward in the documentation of performance, in that it will move us closer to the construct of an acutal performance from a given viewpoint, so that we can offer a reader a more perfect copy of an original experience.

A (Draft) Unifying Perspective for Documents

It is perhaps helpful at this stage to construct a unifying perspective which includes all documents, encompassing also those which are neither partially- nor fully-immersive. I think we need two further categories: physical/analogue documents, and digital documents. The latter category being comprised of counterparts to the physical entities, and also of born-digital works.

As a starting point for discussion, we now have a unifying framework comprising four categories of documents:

physical/analogue

digital

partially-immersive/complex

fully-immersive

Exact definitions of these categories, and the identification of any overlaps or non-sensical implications needs further work. It is likely that the definitive placement of a given type of document within any given category will be problematic, as the interpretation of a document type will suject to context and viewpoint.

At first glance, the main categories seem self-evident. We can, for example, notice two main, straightforward ways in which physical or digital documents may be differentiated from paritally-immersive or immersive works.

Firstly, they are documents which do not change. That is, that a physical book remains the same book over time, as does a digital text, or artwork. No input or participation from the reader is anticipated, or even possible, so the work remains as it was originally created, unaffected by input or participative interaction.

Secondly, physcial/analogue or digital documents do not possess temporal characteristics, apart from those associated with the natural decay which affects all material objects. In contrast to a performance or an exhibition for example, which reach an end point beyond which, arguably, they no longer exist.

The straightforward delineation between the categories of documents becomes subtly problematic however, if we think more closely about the concepts implied by the axes of characterisation. Although the book (physical or digital) demands no active participation, are we not participating by the mere act of reading and construction of the bookish world in our minds? When we refer to a document as a physical entity, are we not implying the container, in contrast to the idea, the informational content, or the ‘bookness’?

Consider also temporality. Is it possible, for example, that the interpretation of a text changes as the reader ages? Is the document or painting encountered as a child the same as that encountered by the same person, but in adult life?

Whilst further thought is necessary, we can suggest that within each of the four categories of documents, further characterisation can be made by placing every document at a point along each of four axes:

temporality: the document exists for a limited time

tangibility: the extent to which the document has a material form

degree of input or participation required: the extent to which interaction is afforded

immersion: the extent to which reality is suspended

The exact understanding of, and the scale or values for these axes are as yet undefined. How does participation relate to immersion? One can surely be immersed in a document, whilst remaining un-participative. There is unquesionably more work to be done to understand the nature of documents and the processes of documentation, and the draft framework above is suggested as a tool with which we can elucidate and explore concepts at a more specific level.

DocPerform

This symposium, focusing on performance, is a part of this work. It is the first of the documentation events at City, although it has grown into a collaborative event, somewhat larger than I originally envisaged.

The symposium is divided into three sessions. Firstly, we look at some existing projects in key memory institutions. Secondly, we examine less traditional aspects of performance which we could try to document, and finally we consider some newer types of performance and ways to understand what we should be documenting. We hope that you enjoy the day, and that it will encourage learning and cooperation from both fields of LIS and Theatre and Performance.

Note: This post was updated on 9/1/17 by LR

On Complex Documents

Posted on March 13, 2016 by Dr Lyn Robinson

Immersive VR … Google Cardboard (+ random biting cat): photo by @lynrobinson cc-by

Immersive Documents

I have written previously on the conceptual and likely practical relevance of immersive documents to the library and information science community. I have defined immersive documents as those which deliver an unreal reality to the ‘reader’. Reader, in this context is a loosely defined term, as the concept of the document is expanded to embrace the type of experience afforded by technologies such as virtual reality (VR), pervasive computing and the multisensory Internet. In this case, the reader may also be described as a viewer, a player, a user or a participant, but in writing from the perspective of LIS, the term reader seems to be an apt, all-encompassing descriptive.

This idea was sparked by my reading, some years ago, of Shuman’s scenario of the library as the ‘experience parlour’, (Shuman,1989).

Immersive documents, wherein reader engagement delivers the perception of an unreal, computer-generated world as indistinguishable from reality, do not yet exist.

This type of intangible document would, like other digital documents, exist only when its overarching computer program executes, and the associated file of coded content is read and processed. An immersive document differs from the familiar digital versions of texts, images, sound and film, (whether born digital or scanned), in that additionally, it allows for varying degrees of real-time reader behavior and interaction data to be processed along with the base content, and to influence the narrative outcome. The scope of participation or interaction could range from passive watching to full body telepresence with complete agency. The term ‘narrative’ can imply that the immersive document delivers the perception of being within a fictional novel, or a game. Whilst this is certainly one example of an immersive document, the ‘narrative’ could be a rendering of an historical event, a travel or news documentary, or a training scenario.

It is possible to regard the computer file, which contains the code for the immersive document, to be a document in its own right. It would have a physical materiality manifested in the computer storage media containing the binary codes. A sort of meta-document perhaps.

Additionally, should the specific outcome/modification from any participatory interaction be recorded, resulting in a new version of the original immersive experience, then this would constitute a related yet different document. This could in theory be played back by another participant, either as a passive, or further interactive experience.

Immersive documents comprise technology, software, and novel narratives, and developments in all three component areas will be needed for such documents to be realized, although an additional, important driving factor will be the strong desire for participatory experiences from readers.

At the time of writing, Spring 2016, we are anticipating the first wave of commercial VR head mounted displays, (HMDs), from Oculus, Samsung, HTC and Sony, which work with compatible computers and software to render stereoscopic, computer generated, virtual worlds, accompanied by sophisticated sound. These environments are compelling, and the ones I have tried deliver a realistic experience of being in a virtual world as an observer.

At this stage however, the reader is not able to fully interact with the environment or content; in programs which support telepresence, this only extends to feeling parts of the body. Doubtless, as technology advances, a fuller sense of body presence in the unreal reality will emerge, but this needs to be matched by the authoring of scripted worlds, to allow for more reader determined behavior and interaction with elements of the unreal world portrayed. [See, for example, http://motherboard.vice.com/read/tribeca-film-festival-2016-virtual-reality-film]

In support of an enhanced feeling of immersion, the HMD interface needs improvement, as at the moment it is rather clumsy and restricts the sense of full immersion, or the suspension of disbelief. Pervasive, wearable technology will undoubtedly improve to the point at which we become less/(un) aware of the interface, and we can look forward to contributions from the fields of neuroscience and psychology in reducing the friction between the reality/unreality interface. EEG headsets already allow merging of brain signals with the machine, brain-machine connection, so it will doubtless be an incremental step for this type of data recording to feed into immersive documents to simulate all five senses. [see for example, http://www.independent.co.uk/news/science/drones-brain-thoughts-controlled-bci-brain-computer-interface-brain-controlled-interface-a6996781.html]

See also, work on implants, leading to body–machine connection; cyborgs and biohacking.

Advances in multisensory transmission over the Internet, i.e. smell, taste and touch, will further enhance our ability to make the unreal, real, and at a distance.

Although fully immersive documents do not yet exist, it would be prudent for the LIS community to consider at this stage, whether the sector should play any part in the handling of these entities, and if so, in which ways. It will be easier to collect and record the documents as they emerge, if frameworks for understanding and description are already in place – thus avoiding the enormous retro-conversion efforts needed to redesign and extend current bibliographic data to enable semantic web functionality and promote discovery.

Partially Immersive Documents

Partially immersive documents do exist, and this prompts enquiry into how these can be recorded, stored, described, discovered, shared and preserved. Whilst LIS related work on these partially immersive entities is scattered amongst other disciplines, and in no way comprehensive, it is a worthwhile source of material relevant to the handling of future immersive documents, and is by its nature surely of interest to the LIS community.

Partially immersive documents may be distinguished from other analogue/physical or digital documents, because, like immersive documents, they allow for, and may even require some level of input or participation, from a data source or a reader. The distinguishing feature of these documents from other digital entities is that they are dynamic, not static.

These partially immersive documents may be divided into two categories.

Firstly, born digital entities, such as: visualizations, simulations, interactive narratives, videogames, virtual worlds, 360 digital video recordings and digital artworks. These documents all furnish the reader with varying degrees of unreal reality.

As with fully immersive documents, the level of participation within partially immersive documents can vary, from almost passive observation through to meaningful interaction – that is interaction which changes some aspect of the documentary experience.

In contrast to fully immersive documents, however, real world elements are present and noticeable, even if the reader is too ‘engrossed’ in the document to notice them.

These partially immersive document entities also exist as computer files containing content together with display or processing instructions, which require specific technological platforms on which to run. They exist only when the content data is acted upon by the software instructions. In many cases, such as with interactive narratives, there is scope for real-time data input from the reader, which generates novel content. Formats such as visualisations, simulations and digital art can all rely on other program data for input as well taking input from a human reader.

This complex, dynamic nature demands a more detailed approach to document handling than that used for digital files representing more conventional (often originally physical) types of static document, such as books, journals, manuscripts, datasets, sound, images or films, even though standard metadata for these more familiar documents may still need to be agreed, and issues of preservation in perpetuity remain.

Secondly, we need to consider partially immersive documents which are ephemeral, temporal, intangible, real world activities. Examples include theatre performances, dance performances and installation art. The level of reader participation may vary between passive reception of the content, to active engagement.

In these cases, the sense of unreality as reality is more related to suspension of reality in the mind, as the readers are at all times perceiving a real world event, even if a fantastical one. These documents need first to be recorded in order to be preserved for future access and understanding.

Augmented reality, and mixed reality events offer yet more time dependent document forms, blending physical world immersive events with the digital.

Complex Documents and the Information Communication Chain

Immersive and partially immersive documents may be thought of as ‘complex documents’, for which an interdisciplinary approach to their information communication chain journey may be beneficial, in contrast to the solely LIS focused efforts to record more usual document forms.

One area in which a significant amount or research has been done is that of preservation, and there are several disciplines which have started to consider how to describe and record complex documents within their domain.

The JISC (Joint Information Systems Committee) organized the POCOS project (Preservation of Complex Objects Symposia) in 2010, which considered complex documents as complex digital objects. To simplify the many types of objects under scrutiny, they were divided into three catagories and investigated from the perspectives of simulations and visualisations, software-based art, and gaming environments and virtual worlds. The original publications from the project are published in “Preserving Complex Digital Objects” edited by Janet Delve and David Anderson, Facet, 2014.

One of the key insights to the description of complex documents comes from consideration of the recording and preservation of dance. Firstly, we need to understand what has to be recorded. At first glance, it seems that we could perhaps merely take a video recording of the performance. But on further consideration, and indeed conversations with dancers, it becomes clear that there each performance (or dance), comprises many layers. It is likely that these layers exist for all complex documents. An initial list of things to be coded, and recorded in description and preservation systems includes:

2D visual recording of the performance

3D visual recording of the performance

coding of the movement, feelings, intention of the performers

sensory, (anatomical, electrical signal) recording from performers

mental, descriptive, narrative from performers

similar recordings from the creator, director

psychological, physical reaction from participants

overall impact of the performance

Perhaps the main challenge to handling complex documents within the information communication chain comes from reader participation.

Participatory behavior can take many forms, from allowing software to read facial expressions, or to measure pulse or heart rate, to allowing full interaction with objects, characters or avatars within the simulated environment. This data is complex to record and process with respect to the base document, but is also complex to add to document description. In some cases, there is more than one participant, adding to the complexity. For the purposes of recording and preservation, there is the question of what is being recorded, and how authentic is the replay?

In some games for example, there are hundreds of participants. Likewise for immersive, participatory theatre. Once we consider participation, we are faced with an additional layer to the recording, description and sharing of the document – do we record the viewpoint/experience of the participant? Can we? There are thus dual (possibly multiple) reader modes; that of a passive observer, a first time reader who is interacting with the document, or that of previous readers, engaging with something previously experienced..

It is perhaps surprising that the question ‘what is a document?’ is still unclear, five and a half thousand years after records began. Humanity nonetheless, still endeavors to record and preserve more and more layers of the human condition.

References

Shuman BA (1989). The Library of the future. Alternative scenarios for the information profession. Englewood CO: Libraries Unlimited.

LR update 2/05/16