bcchang.com / Art / Synthecology

Research and Development

Synthecology's primary foundation is the CAVE^[1], but the project presented a number of new technical challenges. Some of this work was specifically oriented toward this specific artwork, though much of it was done with the additional goal of advancing the expressive capabilities of immersive virtual environments for artists generally. Synthecology showcased the ATS VRLab's recent research in spatialized audio for immersive environments, as well as powerful and reusable components ranging from particle systems to web interfaces.

Goals

The core goal of Synthecology was to create a multi-user immersive virtual environment for creating collaborative, evolving spatialized soundscapes. Beginning with the CAVE as display system and the Ygdrasil programming language, we identified several key technologies that needed to be developed or integrated into Synthecology:

1. Spatialized audio and real-time synthesis
   Synthecology is essentially an environment for building virtual soundscapes, in which the illusion of spatial location of sounds is crucial. For a large public installation, a multiple-speaker array approach offers advantages over binaural headphone playback. Additional requirements included both sample playback and realtime synthesis, reasonable performance (approx 30 to 50 spatialized sound sources at any one time), usability on reasonably common off-the-shelf hardware, easy configuration for variable size speaker arrays, support for real-time DSP (such as reverb), integration into the Ygdrasil and CAVE system, and reusability.
   While the research on multichannel spatialization is substantial, a review of existing solutions found nothing that met our needs. The solution, building on ongoing research at the ATS VRLab in virtual audio, was to develop an extensible sound server for the CAVE using SuperCollider, controlled using Open Sound Control and with a simple node interface in Ygdrasil and a testing GUI in Max/MSP.
2. Database integration
   Many virtual environments use sound by loading a set of defined sounds at startup, or by defining relationships between sounds and objects or events. In Synthecology, the set of sounds is not predefined but is constantly evolving and growing as new sounds are contributed over the web. Participants sending sounds over the web can provide additional information such as a category or visual association for the sound when it appears in the virtual environment; we needed a way to get all this information from the web into the CAVE. We used a MySQL database to store the sounds and all their metadata, and wrote Ygdrasil nodes for reading from the database, while using the existing CAVERN framework for distributing information between networked CAVE's.
3. Database and soundserver management layer
   To aid in integrating Ygdrasil, SuperCollider, and MySQL, we developed an additional software layer in Python. This centralized much of the most complex control and logic in the system, and improved performance by removing tasks such as time-costly SQL queries from both the SuperCollider and Ygdrasil code.
4. Ygdrasil Dynamics
   There were several important aspects to Synthecology that were difficult to implement using existing capabilities in the Ygdrasil language. Wind was particularly important. Since the primary activities involved planting, growing, arranging, handling, and sharing objects, it was crucial for the objects to exhibit characteristics such as weight and gravity. Additionally, the objects had to change the way they behaved as they grew through different states. To solve these challenges, we developed a rudimentary system for physics simulation, a general-purpose particle system, and a new multiplexor node that can be used to easily build complex state machines in Ygdrasil.
5. Flash front end
   While Synthecology is centered on the idea of group interaction and collaboration, the specialized hardware of the immersive display itself limits the number of participants. As a way of addressing this issue, we created an additional way to enter the Synthecology world using a simplified web interface in Flash. The Flash interface and the primary CAVE world were connected using the MySQL database, raising the intriguing possibility of creating shared virtual worlds with interfaces ranging from the CAVE to cellphones.

For the full paper on the sound server development for Synthecology, see

Footnotes

1. CAVE is used as shorthand here to additionally refer to the larger class of projection-based immersive display systems. The public display system for Synthecology was a single-wall passive-stereo rear-screen projection system with eye and hand tracking, often called a C-Wall or GeoWall. Some of the participants in Synthecology connected using standard desktop PC's with just a DSL internet connection - but the same CAVE software is used on all these systems.