Sonic Moves is a platform which consists of two main components: You create body-interactive compositions in Sonic Moves Creator (macOS, iPadOS) which you and others can play in Sonic Moves Player (iOS, iPadOS). This guide focuses mostly on the usage of Sonic Moves Creator.
Projects
Every Sonic Moves composition is created in a project. The first screen of Sonic Moves Creator shows an overview of existing projects. Next to each existing projects info, an ellipsis button lets us perform further actions, like removing projects or creating templates from them.
Speaking of templates, we can also start a project from a template. To do this, click on the second leftmost tile and select the template we want to use. Sonic Moves Creator comes with a couple of built-in templates which were used to create the tracks that ship with the Sonic Moves Player app.
Home with existing projects and the "New project" and "Templates" tile.
Creating a project.
Creating a composition
To create a new project, simply click/tap the tile with the + Icon. We give our project a name (it can be changed later), and the project workspace will appear.
The Workspace
A blank project. Top: Project info, tempo, time signature and undo/redo. Right: Input, output and interactions info.
Sounds and Interactions
A composition consists of two main parts: Part 1 is the "audio chain". These are all our sounds which will often feed into other sounds, like effects, and eventually to the main output, which is what arrives at the players ears. Part 2 are the interactions. We can make most sound parameters interactive by connecting them to body tracking input.
Compositions in Sonic Moves are different from static songs, as by default they do not have a fixed duration (we can change this though in the song info). This means that they loop indefinitely until the player decides that they want to stop.
Sounds
To add a sound, click on the + icon and select the type of sound to add. There are different types:
Sound selection.
- Audio files: Load your own files like wav, mp3 etc. to use them either as a simple loop or for a grain sampler. Here we can also find a small library of preselected sounds. Finally, we can also create our own library of sounds we want to use again in the future.
- Software instruments: Synths, samplers and drum machine. They play note patterns which we arrange.
- Generators: Different forms of live synthesis: Oscillators, noise, vocal tract simulation.
- Outputs: These are advanced tools to send tracking and MIDI data out of Sonic Moves to other apps.
After we added a sound, it will appear as a "node" on the canvas and we can move it around freely.
When we click/tap a sound node, the left sidebar appears. Here are all the sound parameters. Each sound has it specific set of parameters. An audio file for example has "volume", "direction" and "looping", and more. When a sound parameter can be interactive, there is a button called "Add interaction" below it.
Sound sidebar
Sound Processors
Often we want to change the original sound, essentially "processing" it. To do so, we need a "Processing Bank". A processing bank is another node which sound feeds into. The processing bank itself lets us filter the sound and change some other parameters.
A sound source and a processing bank. In the sidebar, we can access settings of the processing bank, such as filters.
To add a processing bank to a sound, tap the arrow icon on the node and select "Add sound processing". The processing bank node will be added to the canvas.
But a processing bank can do more: When we tap the + icon on the processing bank node, we can add filters, effects and amplitude nodes to the chain. We can have many of these in one processing bank. They are ordered in a list, and each node in the list feeds into the one below it. We can reorder them by dragging them up and down.
Adding a processing bank with effect.
A simple audio chain with a sound source, a processing bank and a reverb effect.
Adding more sounds
We can add more sounds with the big plus button. From there we can select sounds directly from the menu or show the panel with all sounds.
Changing Outputs
By default, every sound feeds into the main output. But we can rewire the chain by tapping the arrrow icon and selecting "Change output". After that we can select a new output.
Spatial Audio
The main output is by default the "center", which means a standard stereo output. Sonic Moves also features up to four spatial audio channels, so we can distribute the sounds in space. This feature should be used with headphones, as the audio will be rendered "binaurally", meaning a simulation of spatial sound through stereo headphones. If the player uses a device with head tracking (like Airpods Pro) the sound location will change as they move their head.
Interactions
Sonic Moves is all about body-interactive sound, so creating interactions is a big part of it. As of now, all interactions are "mappings": An input value, like the velocity of a joint is transformed into the value of a sound parameter, like the volume of an audio file.
Make a parameter interactive
To add an interaction, select a node and find the parameter in the sidebar which should be interactive. In this example, we want to make the volume of an audio file interactive. Click "Add interaction" below "Volume".
Player
This mannequin represents the player. The circles are the body tracking joints, which are the inputs for our interaction. Now we need to select one or more joints that this parameter should respond to. We can choose different input types: Velocities, height, distances and rotation. We select the left hand joint and from the menu select the "Height" parameter.
Mapping input to output
Now we configure how the input should be translated to the sound parameter. The blue and orange circles in the center pane define the minimum and maximum input. On the right, under "1 Output", we define the range into which this input will be translated to the volume. We drag the start and end circles to do this. Below that, we can also define how quickly the parameter changes.
The connection line shows us that the sound and the body are now connected.
Playback
We can play our composition in Sonic Moves Creator by clicking the play button in the bottom right. We can change most sound parameters dynamically during playback. There are a couple of ways that we can use for playback, which we can select in the right sidebar under the "Input" tab.
Static playback
By default, the playback is static. This means that all sounds play as they are configured and no interaction takes place.
Using the Camera
We can use the camera(s) of our device, or even external cameras, for live interaction with the composition.
Using Videos
We can also load videos to use them for tracking in Sonic Moves. This is especially helpful when we want repeatable results. We can load videos from the gallery (iPad) or from files to simulate interaction. The body tracking will work the same with video as it will with live input.
The video selection with a couple of videos that have already been loaded into Sonic Moves Creator. Add videos by using the plus button.
Please note that on macOS, you need to download a little helper app to use the camera or video tracking. The app is called "Sonic Moves Tracking Helper" and does exactly what the name says. You can download it from the App Store here. On iPad, this is not needed.
When video or camera input is selected, we can see the live tracking with a visualization during playback, in the right sidebar.
Using motion capture recordings
Sonic Moves Creator comes with a couple of prerecorded motion captures with some basic movements for testing purposes. They include rotation, jumping, crouching, stretching, and a few more. To select one, tap on the "Motion Capture" menu in the Output node and select "Recordings".
Testing in Sonic Moves Player
The quickest way to test our composition in Sonic Moves Player on iPhone or iPad is to use the "Send to Device" function with Airdrop. Click the "Send to device" button in the top bar, then select Airdrop and choose the device we want to send to. This will only work when Bluetooth is turned on on both devices.
Download Sonic Moves Player
After our iPhone or iPad has received the file, it will show you a pretty empty preview. In the bottom you can tap the option "Sonic Moves". This will import the file into the Sonic Moves Player. Then you can play it.
Publishing
When we are ready to share our composition with others, we want to export it. Click on the export button. This will create a ".smsong" file in the "export" folder of our project. This single file contains everything that is needed to be imported and played in Sonic Moves Player.
Please note that this process differs slightly depending on the platform. On macOS we have to select the location to which the file should be exported. On iPadOS, it will be automatically saved into the "Export" folder of the project.
Each composition has some meta information which we can edit, like title, artist name, artwork and more. We can also include notes to tell people how to play our song.
Editing meta data and exporting.
Sharing/Publishing
Right now there are two ways we can share a composition with others. We can either send them the .smsong file, or we can upload it to a web space and share the url with them. We are also working on other ways to do this in the future.
Tap the tab "My Library" and select the method to import. "Load from Files" will open the files app, and "Open from URL" will prompt you to enter the online location of a .smsong file.
Settings
Using the gear icon in the header we can reveal a couple of settings. From here we can also reference this manual again and join the Sonic Moves Community on Discord, or give feedback. We can also change the colors of the interface in the "Colors" section. And very importantly, we can manage our license.
There is Much More
Now you have seen the main features of Sonic Moves Creator. But there are much more: dynamic instruments, interaction curves, visualizations… We will cover those in future tutorials, but we hope that this page helps you to get started with body-interactive composition. If you have any questions or feedback, please send us a message right away to feedback@sonicmoves.net!
Thank You!