Launching of Sound to Vision Desktop

For the last 3–4 years I have been working on a project called Sound To Vision (SToV), soundtovision.com. Tired of my academic work and looking for a new challenge I decided to leave the research institute which had been my home for the previous nine years and move to London with a dream of developing professional audio-reactive software. Software which facilitates the creation of compelling music visuals

Early Concept for Sound to Vision Web

Unclear what my goal was at the beginning I started to experiment with different ideas. I knew that I wanted to pursue something which challenged me in terms of its engineering complexity but I also wanted to pursue something artistic as well, aspects which I had struggled to reconcile when I was working as a scientist previously. To explore this I started to make videos and experiment with music created by my friends.

[embed]Early visual experiments.

Not long after beginning the project I started to collaborate with the musician and my friend, Jan (AKA Atomise, AKA NOAMEX). We started to perform audiovisual gigs together and I would create the visuals and also the software which we would be use during the performances. I always remembered a quote about Amazon. When Amazon created Amazon Web Services, their wildly successful cloud management system, they developed it because they needed it, but they made it so that it could be used and sold to customers also. This probably slowed down their development a lot, but meant they created a power house of functionality and usability which they later monetised for a great amounts. With SToV I have made sure I have done the same, anything I have created I have made sure anyone else could create using the tools and scripts I have made.

Visuals from a gig in Camberwell

This naturally lead to an interface and over time I have developed the different functionalities. From my experience in Academia I had learnt how powerful web interface design is. Web pages have by far the best interfaces in terms of flexibility, speed and ease of design. I had found that software developed to work in a browser was not only extremely accessible being just a click away but is also probably the best way of presenting complex tools to a wide audience.

An early version of the interface working in a browser

The browser was the perfect fit for creating the software in terms of usability and ease-of-access and off-the-shelf had video playback and audio rendering functionality built-in. What I found however was that not only did users want to showcase their visuals live, but they also wanted to save their projects and also render out videos to use on their social media. Browser based software is not good for this however as it is not generally possible to implement fast video rendering in the browser (with a few specific exceptions), nor is it possible to do extensive file operations. All leading browsers prevent direct manipulation of the file system.

Early reactive visuals from SToV

I thought the solution was to use cloud services, and as you know, at the time I was a big fan of Amazon Web Services, the perfect host for this kind of backend infrastructure. The idea being, that if users wanted to store media assets and their save files for later use they could do so by uploading them into the cloud. Unfortunately however, although I developed a working solution for this, my friend quickly broke the system, I had loving created, by trying to upload a multi gigabyte file onto the server and also insisted on 4K rendering for the output videos. Although I optimised the system extensively I realised that the cloud-based system for high-end rendering was not going to really work. It was just too expensive and too slow to upload anything but small files in this way.

[embed]An early video render form SToV

Around this time I came into contact with Node.js. Node.js is a server-side rather than browser version of Javascript which allows you to do all the file system like things you can’t do in a browser. It’s an easily language to use, but also very powerful. I realised I could create a simple software which would sit alongside the browser version of SToV and allow the user to render their projects into videos locally, rather than having to do it through the cloud. This software opens a bit like Zoom, essentially when you use Zoom you visit a website and then when you are required to start a videocall the browser opens the software automatically for you and you are ready to start the call.

A screenshot of the SToV RenderApp which opens when you need to render out a video

I experimented on this for some time, to get it right, but slowly perfected the concept making it possible to efficiently create 4K 60 FPS rendered videos with just a few clicks. I also realised that it was possible to save projects neatly using zip files. If a project was created it could then be exported with its component assets included using a zip file (or what I call a .stov file). This would allow users to export and import whole projects readily and locally without the need to reload all the media files included in the project manually. Although not perfect, this system allowed the browser version of SToV to function very powerfully despite the shortcomings of the browser. The only drawbacks of this approach was that it was still necessary to manually export and import project files and also the size of videos which could be safely imported was no more than 2 GB. I knew however that there was something about the accessibility of the browser which meant it was a bit too good to ignore. Therefore SToV Browser is the entry level, very accessible version of the software, which I give for free to get people into the SToV eco-system, but which still has some memory and usability limits.

What is SToV Desktop and what need does it meet?

SToV Desktop, Node.js and Electron fully contained software.

SToV Desktop takes all the functionally of the SToV Browser software but adds the convenience of being able to save projects whenever and wherever you want, using media located anywhere in your file system. You can import in seconds all your videos and images and render out videos in one seamless experience. For the interface we use the same design as SToV Browser but we run the whole thing together in a framework called Electron which provides a browser-like experience and which runs standalone. This means you don’t need an internet connection to use it. SToV Desktop can handle 4K videos of very large size and has no memory limitations beyond that of your system, in addition, I have also bundled in some advanced video and image processing algorithms for your enjoyment. So if you have enjoyed SToV browser (or never used it) then I highly recommend you upgrade to SToV Desktop for the complete experience. Further details can be found here: soundtovision software. Essentially it has all the benefits of SToV Browser but none of the drawbacks!