Deep Learning for Sound-to-Light Automation in Stage Lighting Applications

Abstract: 

Stage lighting applications usually require an operator who programs the dependency between sound and light. The industry is working on an improved sound-to-light automation whereas the scientific research in the field is limited. This thesis analyzed how sound-to-light automation can benefit from deep learning.

Therefore, three different data sets were created: A synthetic data set where a light is pulsing with the beat, a trace of a basic real world light show and an extraction of control signals from a video of a professional light show. 

The proposed model consists of a convolutional recurrent neural network (CRNN) that receives a log-Mel spectrogram together with its positive difference as input features. The loss function takes the absolute values of hue and brightness into account, as well as their change compared to the previous sample. Four additional metrics support the evaluation of the performance.

The results showed that learning a direct dependency between low level music features and the lighting output via a CRNN is a promising approach for sound-to-light automation. 
The first two data sets turned out to be more valuable than the third for this task. While the model learned the general structure of the first data set, especially the second revealed limitations that require further research. The relation between sound and light was not sufficient in the third data set.