[PIG] [Weekly Update] Autonomous Realities

[botaoamberhu]

Week 1. May 13 - May 19

What did you accomplish last week?

This week, I participated in CHI 2024 in beautiful Hawaii. I was part of a workshop titled “With or Without Permission: Site-Specific Augmented Reality for Social Justice 1” that aims to facilitate the design, development, and use of place-based Augmented Reality experiences supporting Social Justice efforts.

“Movements for social change are often tied a particular locale. This makes augmented reality (AR), which changes how people perceive their surroundings, a promising technology for social justice. Site-specific AR empowers activists to re-tell the story of a place, with or without permission of its owner. “

In this group of experts, who use the permissionless nature of AR to express social justice, I introduced my idea of autonomous realities to the discussion to assess the need for protocolizing their expression.

What do you plan to do next week?

I intend to conduct a literature review, create a prototype, and prepare a workshop agenda. These steps will help me find a venture to discuss permissionless realities and FingerSync. However, one might question the necessity of these actions.

What (if anything) is blocking your progress?

We need to reach out more experts in this domain.

Fun insight/tidbit, link, or idea?

1. https://ar4socialjustice.org/
2. My random thoughts after discussing with the human-computer interaction community.

"Protocol is a standardized “interaction”.

Week 2. May 20 - May 26.

What did you accomplish last week?

This week, I participated in the Join Datus-Nusas Workshop in Singapore. It was a two-day intense brainstorming session to envision protocols that could help South East Asia flourish. It’s very enjoyable to brainstorm in person with people who are researching protocol science.

Some highlights for me:

1. I proposed to compare the hardness concept in protocol science to material science.

   Why not metaphorically borrow the hardness concept in material science to study hardness in protocol science? Such as Strength, Toughness, Elasticity, Plasticity, Stiffness, Ductility, Malleability, Hardness, Creep, Fatigue, Resilience.

   Venkatesh Rao recommend me to read the book "The New Science of Strong Materials: or Why You Don’t Fall Through the Floor” .

2. I got inspired by flourishing concept in A Phenomenology of Protocols by Janna:

   The purpose of protocols is to encourage or discourage our ability to pursue human flourishing.

3. Brainstorming with Venkatesh Rao about the form of the presentation of Autonoumous Realities:

   We think we could create a series of art projects to reflect the potential protocol-related issues in merging two mixed realities. We want to keep the form as minimal as possible, setting it as “a room with two people with two Vision Pros.” Then, we would create an experience showing the different ways mixed realities merge after two people shake hands.

What do you plan to do next week?

Due to travel, I postponed the literature review for Autonomous Realities to next week.

What (if anything) is blocking your progress?

No.

Fun insight/tidbit, link, or idea?

I drew some illustrations during the workshop to help me think about protocols and cultures.

Week 3. May 27 - June 2.

What did you accomplish last week?

1. Conduct an initial literature review on Autonomous Reality.

   I searched for Human-Computer Interaction literature to understand the research questions and results in social and collaborative computing in mixed reality, envisioning a future with prevalent MR headsets.

   Multi-user collaborative mixed reality is a well-researched area, but it often assumes that all users are running the same MR layer. But in the future, MR wearer will start to encounter each other spontaneously in real-world settings. This situation introduces new complexities in how they interact. Few literature addresses about this merging process.

   The merging process naturally involves the ontology of digital objects and the access control system for these objects. Can you see the objects I see? How do mixed realities merge? What protocols will be necessary to allow users into each other’s realities?

2. Conduct a workshop at the International Conference on Live Coding (ICLC) at New York University’s Shanghai campus. The workshop will feature a multiplayer inter-bodily live-coding mixed reality experience to explore engagement and conflicts in multiplayer mixed reality.

   How does the group form a social protocol by dynamically assigning augmented affordances to individuals through live-coding?

What do you plan to do next week?

Write a workshop draft for Computer-Supported Cooperative Work & Social Computing (CSCW) to recruit more people to discuss this topic.

What (if anything) is blocking your progress?

As I read many references in HCI/MR, I feel confused about defining our research questions for Autonomous Realities. The problem is that HCI focuses on collaborative MR settings. On the surface, it looks related to our research, but it’s too narrow, focusing on how people interact. We want to focus on what kind of protocols help MR merge and persist. There is little literature in HCI that addresses this.

Fun insight/tidbit, link, or idea?

I drew the illustration to describe the asymmetry between wearers and non-wearers.

Week 4. June 3 - June 9.

What did you accomplish last week?

1. I wrote an proposal with research questions list during “merging mixed realities”.

Here is it.

See here:

What do you plan to do next week?

1. Learn new tech from WWDC 2024 by reading the development documents of Vision Pro.
2. Start to play with and develop on Vision Pro.

What (if anything) is blocking your progress?

After listing many research questions about merging mixed reality, I need to refine the research questions specifically for Autonomous Realities.

Fun insight/tidbit, link, or idea?

Week 5. June 10 - June 16.

What did you accomplish last week?

1. Apple released VisionOS 2 at WWDC.

I started developing on Vision Pro with FingerSync.

I finished the initial communication between two Vision Pros using MultipeerConnectivity, the technology behind AirDrop.

1. Prototyping different kinds of gestures for FingerSync

What do you plan to do next week?

Continue development.

What (if anything) is blocking your progress?

I’m still not clear about the research questions and context.

Fun insight/tidbit, link, or idea?

Week 6. June 17 - June 23.

What did you accomplish last week?

1. Fix bugs and learn more about Vision Pro.
2. Continue to develop my first FingerSync scenario.
3. This week, I worked full-time teaching PlayShop at the China Academy of Art.

What do you plan to do next week?

1. Continue development.

What (if anything) is blocking your progress?

The research questions and context are still not clear.

Fun insight/tidbit, link, or idea?

Spatial Persona 2 now supports fist bumps and handshakes.

Week 7. June 24 - June 30.

What did you accomplish last week?

1. Discussion with Yuemin Huang and Fangting.
   1. Yuemin showed me her idea, “Protocolized Realities.” Reality is perceived by how we see it, and it’s totally protocolized.
2. Brainstorm scenarios for merging mixed realities.
   1. When we shake hands, we can steal objects from each other’s realities.
   2. When we shake hands, after I touch an object, it transfers to your reality (I can’t see it), but when you touch the object, it goes back to my reality.
   3. When we shake hands, our virtual time stops (the rain stops).

What do you plan to do next week?

Next week, I will be working full-time on PlayShop at the China Academy of Art.

What (if anything) is blocking your progress?

1. I need to spend more time finding context and methodology for Autonomous Realities. My brainstormed scenario is very random and lacks strong meaning.
2. This week, I worked full-time teaching PlayShop at the China Academy of Art.

Fun insight/tidbit, link, or idea?

1. I continue my thought on material science and protocol science.
   1. The protocol has a very important property called hardness, which is defined by the essay “Atoms, Institutions, Blockchains”. It refers to either the probability that the protocol fails or the cost to create a situation that causes the protocol to fail. I was wondering if we could compare material science to protocol science to describe hardness. In material science, stiffness is defined as a material’s ability to resist deformation under an applied force. Toughness is a measure of a material’s ability to absorb energy and deform plastically without fracturing (failure).
   2. I plan to draw a genealogical diagram to help organize my thoughts on protocol science. I am fascinated by how protocols emerge from disorder to order.

Week 8. July 1 - July 7.

What did you accomplish last week?

I met Anthony Dunne, a pioneer in speculative design.

He proposed a speculative design topic: Celebrating A World of Many Worlds. From a more-than-human perspective, various species possess sensory systems that differ significantly from those of humans. Consequently, the realities they experience are vastly different from what we perceive. He illustrated this concept with artwork that depicted a sonic sensory system, where sound is as tangible as rock.

His concept is similar to my concept of Autonomous Realities: A Reality of Many Entangled Realities.

What do you plan to do next week?

I will start to research on speculative design for Autonomous Realities.

What (if anything) is blocking your progress?

1. This week, I worked full-time teaching PlayShop at the China Academy of Art.

Fun insight/tidbit, link, or idea?

1. I started reading “On the Existence of Digital Objects” by Yuk Hui. I think this will help me understand how to define the protocol for digital objects in mixed reality when merging.

1. I created a protocol stack to explain the hierarchical layers of protocols that form the foundation for actions like buying stocks or tokens.

Week 9. July 8 - July 14.

What did you accomplish last week?

1. Eureka! Following Dunne’s inspiration, I began researching speculative design and discovered a methodology called Ethnographic Experiential Futures (EXF) in Futures Studies. I believe it’s perfect for the Autonomous Realities project, for which I have been unsure about the research questions for a while.
2. My thoughts: First, we map out potential scenarios when merging realities. Then, we create a future-probing artifact: an extension protocol that defines the ontological information of digital objects, such as permission, access control, and appearance, similar to XML descriptive language. Next, we let people experience a series of future scenarios we created based on different configurations of our proposed protocol. Lastly, we gather people’s ideas about these futures ethnographically, and iterate.
3. For example, previous brainstormed idea is a configuration of the extension protocol.
   1. When we shake hands, we can steal objects from each other’s realities.
      We can move the object and transfer ownership at the same time.
   2. When we shake hands, after I touch an object, it transfers to your reality (I can’t see it), but when you touch the object, it goes back to my reality.
      We can define the rules that a digital object can apply to, such as transferring visibility and ownership simultaneously.
   3. When we shake hands, our virtual time stops (the rain stops).
      We can define the rules about mixed reality’s time.

1. I am going to organize the Merging Mixed Realities workshop at Dweb Camp from August 8-11 to call for ideas and discussions.

What do you plan to do next week?

Designing a toy protocol for the ontology of digital objects.

What (if anything) is blocking your progress?

Fun insight/tidbit, link, or idea?

Experiential Futures: A brief outline

Week 10. July 15 - July 21.

What did you accomplish last week?

This is a reading week.

I did a literature review on Metaverse Standards Forum (https://metaverse-standards.org/) to see if there is some standard t hat my extension protocol can based on.

1. 3D Assets and Interoperability
   1. glTF: 3D assets standard for transporting.
   2. OpenUSD: 3D assets standard for authoring.
2. 3D Web:
   1. X3D: X3D (Extensible 3D) is an open standard for representing 3D graphics on the web.
   2. WebXR.
   3. Open AR Cloud.
3. 3D Asset Interoperability
   1. https://www.youtube.com/watch?v=30Soea0MA5s

I start to read more books.

Ian Cheng, Emissary’s Guide To Worlding

Liam Mitchell, Ludopolitics: Videogames against Control

Alexander R. Galloway, Protocol: How Control Exists After Decentralization

What do you plan to do next week?

Go to EdCON and SIGGRAPH 2024 to ask more experts on USD and metaverse.

What (if anything) is blocking your progress?

I realized it’s too late to find the context for my project.

Fun insight/tidbit, link, or idea?

1. Thoughts on Synergetics

   This week, I read a lot about synergetics in complex systems and found the concept fascinating. It’s similar to protocols, where order parameters make complex systems form synergy, transitioning from disorder to order. Examples include bird flocking, phase transitions, and emergent rules. A protocol is a type of synergy. It forms when conditions reduce the system’s entropy and ceases when the order parameters change. Protocols help systems thrive, while ossified protocols prevent them from flourishing. Protocols create a flourishing space for higher-level protocols.

Week 11. July 22 - July 28.

What did you accomplish last week?

1. I am traveling to EDCon in Tokyo, and then I am flying to Denver for SIGGRAPH.
2. I am drafting a speculative protocol to describe the ontology of digital objects when merging.
3. I read more books on complex systems through the lens of protocols.

Scale: The Universal Laws of Growth, Innovation, Sustainability, and the Pace of Life in Organisms, Cities, Economies, and Companies

What do you plan to do next week?

1. Go to EdCON and SIGGRAPH 2024 to ask more experts on USD and metaverse.
2. Finish my draft slides for Autonomous Realities, researcher discussion.

What (if anything) is blocking your progress?

1. I need some clear writing to help organize my project. It’s taking too long in its current confusing state.

Fun insight/tidbit, link, or idea?

1. I fell in love with complex systems. They help explain the emergence and decline of protocols, as well as their power to create flourishing spaces.

[botaoamberhu]