Team members:
- Peter Vander Auwera p.vanderauwera@gmail.com
- TJ Thomas John Mcleish tjm@tjmcleish.com
Summary of your improvement idea
Conversations are intricate exchanges embedded with protocols, shaping the dynamics of interaction. We propose that a “great” conversation is where each participant evolves as a learning cognitive system, contributing to existing knowledge and potentially generating novel insights, culminating in actionable outcomes. Put another way, the participant achieves what could not be achieved alone.
In this research, we dissect the anatomy of a “great” conversation, delineating key elements:
- Opening Channel
- Committing to Engagement
- Constructing Meaning
- Evolving Shared Meaning
- Converging on Consensus
- Initiating Action or Transaction
Recognition of a “great” conversation lies in the transformation of shared understanding, persisting across time and contexts.
Our hypothesis is that by enhancing awareness of the elements constituting a “great” conversation, we can elevate the quality of average conversations, whether human-to-human or human-to-machine.
Our focal point is the human experience within the conversation protocol. Our approach emphasizes the embodiment of cognitive systems in spatial and temporal contexts.
To manifest this, we propose a 3-act virtual reality (VR) experiment, enabling participants to both interact and embody agents with various cognitive affordances (including a Machine Learning mode, the Google search engine, personal assistants such as Alexa, Siri, etc.).
Through these immersive experiences, participants can gain deeper insights into the dynamics of conversational protocols within spatial and temporal contexts, ultimately enriching their understanding of cognitive processes and human-machine interfaces.
- For instance, in one scenario, participants can seamlessly transition between subjective viewpoints of different agents, each equipped with unique affordances. In another scenario, participants assume the role of a single agent but possess control sliders, allowing them to adjust their experience and observe the resulting outcomes by manipulating certain parameters within their control. This setup enables them to witness the effects of their agency firsthand.
- Additionally, we envision scenarios involving conversations with authors or other entities serving as internal dialogues. Participants may also engage with non-human agents; for instance, they could embody the Google search engine. In this scenario, other agents initiate conversations, prompting the participant to respond by inputting their queries into Google and relaying the retrieved information. This process spatializes and personifies the search bar, transforming the interaction into a conversational experience akin to the Chinese Room experiment.
Expanding on Venkat’s “I am Protocol” to “I have protocol,” we introduce “I experience protocol” as a pivotal shift.
Meme: “The experience of the protocol is not the protocol”
Q&A
What is the existing target protocol you are hoping to improve or enhance?
The protocol we are hoping to improve or enhance is the protocol for constructing, transforming, and converging on shared meaning among cognitive systems through conversation. Our research is grounded in Gordon Pask’s Conversation Theory as embodied in Dr. Pangaro’s “CLEAT” (Context, Language, Exchange, Agreement, Transaction) framework for guiding human-to-human or human-to-machine conversation
These conversations encompass a wide range of participants, from the conversation I may have with myself, conversations between individuals, and conversations that happen in small to very large groups. These conversational agents possess a wide range of cognitive abilities, including those individuals grappling with temporary or permanent conditions like dementia, those with rapidly developing cognitive abilities like children, entities that have intelligence but do not share human cognitive engines like common household pets, other domesticated animals, or chimpanzees, etc. Furthermore, these exchanges are enabled by an increasingly varied range of media, senses, and timeframes. Including not only voice and language but also search engines like Google, recommendation engines, persuasion systems like social media platforms, and an explosion of generative AI systems interwoven into many common digital tools, like ChatGPT or Midjourney. The mediation of conversation raises questions about the ethics, efficacy, and evolution of communication in a world increasingly permeated by AI and digital technologies requiring thoughtful consideration of how these systems shape our understanding, relationships, and society.
We propose to build on an existing conversational theory protocol and demonstrate its extension through practical application and experience.
What is the core idea or insight about potential improvement you want to pursue?
To improve the adoption of a known conversational protocol, the core insight we build on is that conversational design patterns incorporating CLEAT are not well known, are poorly understood, and there are few working examples of how to implement the patterns in either human-to-human or human-to-machine interactions. We aim to make clear the value of adopting conversational design patterns that deeply engage our cognitive abilities, our ability to learn, ability to reach agreement, and ability to act on that agreement through collaborative action.
As an individual, it is difficult to see the patterns that make up “great” conversations. The very subjective nature of our limited experience may lead us to believe there is no pattern. Or Perhaps what we don’t have are reflexive practices where we pay attention to the patterns of conversation, or we don’t have examples of what the patterns might be? Fortunately, researchers and scientists such as Gordon Pask and Dr. Paul Pangaro have built a strong body of work on experiments studying the nature of conversation and subsequently modeling them. It is this body of work we intend to make clear and accessible through a workshop, experiential experiment, and publication on conversation protocol.
A complementary insight is how new modes of experience, particularly VR, are well suited for explaining, exploring, and experimenting with conversational protocols. New modes of mediating conversation are emerging and with the adoption of these new modes comes a greater need to acknowledge and adapt conversational protocols. Space and time are integral in human cognitive processes. We focus on the exciting new opportunities in VR that provide fertile ground for experimenting with conversational protocols and demonstrating the value of various approaches.
To manifest this, we propose a 3-act virtual reality (VR) experiment, enabling participants to both interact and embody agents with various cognitive affordances (including a Machine Learning mode, the Google search engine, personal assistants such as Alexa, Siri, etc.) Through these immersive experiences, participants can gain deeper insights into the dynamics of conversational protocols within spatial and temporal contexts, ultimately enriching their understanding of cognitive processes and human-machine interfaces.
The enhancement involves fostering protocol design awareness that not all communication (such as sending and receiving messages) necessarily qualifies as conversation.
Furthermore, it underscores that truly “great” conversations, characterized by the mutual offering, acceptance, and evolution of ideas, revolve around agreed-upon human experiences, meanings, and actions.
Meme: “Standing in the protocol, in time and space, at human scale”
What is your discovery methodology for investigating the current state of the target protocol?
Historical analysis of the differences between communication and conversation with Gordon Pask’s Conversation Theory as a starting point. Desk research and consolidation of core ideas and learnings. We are establishing the baseline.
Field-observation: an overview of non-conversational interactions between cognitive systems: humans, search engines, option recommendation engines, and AI engines.
Expert interviews with specialists in fields where the spatial dimension plays a critical role in communication, like UX experts, VR experts, Biologists, etc. to assess possible new virtual ways of conversation and new types of triggers to initiate and keep alive the conversation protocol.
In what form will you prototype your improvement idea?
Draft proposals in writing and presentation decks will be shared with experts for feedback.
Simulation of the 6 elements (Opening Channel, Committing to Engagement, etc.) of the conversation protocol into a VR App version of the 2018 Colloquy of Mobiles installation.
Users of the VR App will be invited to experience different agent configurations and quality levels of conversation protocols. They will “tune” time, space, and estrangement aspects via sliders. The experience will be in a controlled VR experiment in 3 acts:
- Act-1: Shared meaning is established: concept affirmation
- Act-2: Crisis: the shared meaning and its anchoring symbols are lost
- Act-3: Resolution: a new meaning is negotiated by the agents engaged in conversation, and a new stable concept emerges
Note: The original Colloquy of Mobiles installation was shown for the first time at the Cybernetic Serendipity exhibition at the Institute of Contemporary Arts in London in 1968. A faithful visual, behavioral, and experiential replica was constructed by TJ McLeish and Dr. Paul Pangaro in 2018 and exhibited at Centre Pompidou in 2020 and ZKM in 2022 where it is now part of the permanent collection. These interactive experiences invite gallery visitors to engage in immersive, real-time, unexpected, and thought-provoking encounters. At its core, the Colloquy of Mobiles investigates the essence of machine-to-machine and person-to-machine dialogues within a human-scale, immersive environment, exploring these interactions. Models in standard 3-D CAD format are readily available from the 2018 replica, accompanied by pseudo-code primed for effortless translation into VR simulations.
How will you field-test your improvement idea?
We propose one SoP24 Townhall session to share the main concepts and directions. We also plan one invitation-only VR session for A/B testing the user interactions with our VR App and an IRL workshop at the 15-19 June 2024 ASC 60th anniversary event in Washington DC
Who will be able to judge the quality of your output?
Starting List:
Dr. Paul Pangaro - President of the American Society for Cybernetics
Alex Coulombe - Co-Founder and CEO of Agile Lens, XR Creative Studio
How will you publish and evangelize your improvement idea?
The results will be published on the web in open source.
- A Guidebook for designing a conversation protocol that meets the elements of a “great” conversation.
- Visualizations that capture the evolution of shared context over the conversation timeline
- A VR App called “Conversations about Conversations” will be made available to the public for free via the Meta AppLab and supported on both Quest-2 and Quest-3 VR headsets. If additional funds will be secured, the app will also be deployed on the Apple Vision Pro.
- A series of online meetings open to anyone who has downloaded the VR App.
What is the success vision for your idea?
Imagine a protocol design movement that takes conversation seriously. Could it create a revolution? The Industrial Revolution harnessed physical machines to extend and enhance our muscles. The Information Revolution harnessed virtual machines to extend and enhance our nervous systems. A “Conversation Revolution” would harness the existing infrastructure of physical machines and virtual machines to create a mesh out of “networks of objects” and networks of individuals and organizations.
Such a mesh would enhance coordination and collaboration and create wealth by introducing new efficiencies. It would also expand opportunities to generate new knowledge. Imagine a search engine designed for effective conversation, with all the knowledge on the Web participating. We would no longer be focused on “search,” nor would we be using an “engine.” What should it be called? Who will build it first? — Dubberly and Pangaro, “What is conversation? Can we design for effective conversation?”, ACM Interactions Magazine, Volume XVI.4, July + August 2009 (PDF)
We believe this could be the genesis of the new spatialized social conversation platform.