Audio Analysis and Transcription

Table of Contents

Program IO 

The prototype of this thesis can be separated into three different fields. Automatic music transcription, automatic music generation and MIDI to sheet music transcription. The first two fields utilize different models and datasets provided by Magenta, while an external software deals with the MIDI to sheet music transcription. This graphic display the overall idea of this prototype. The three different components are marked in blue.
In order to achieve user friendliness, building a graphical user interface proves to be mandatory, as it connects all the fields into a coherent program. 

Program Structure

This graphic displays the exact structure of this program. The python scripts are marked in yellow. Signals and Slots system provides a seamless transition between the scripts and external applications. This way, input parameters are all automatically passed on where needed.
In the beginning the user can choose to select a WAV audio or MIDI file from their local explorer. Optionally, they can record a WAV file on their microphone with recording.py. To use the transcription feature transcription.py, a Wave file must be selected. Then, this script will transcribe the Wave audio into a MIDI file and save it in the same directory as the input file.

This program automatically passes on the MIDI file to other features that all automatically generate new sequences. In Python, these scripts are referred to as melody.py, chords.py, improv.py and bach.py.

By the end, MuseScore3 transcribes the produced MIDI to Sheet Music and saves it as a PDF file.

Music Transcription

The main libraries are Magenta and Tensorflow. This script contains machine learning algorithms that detect onsets and frames in raw piano audio. First, the function create_example() reads in the specified WAV file and processes it to the fitting format. The library called librosa, which features powerful music processing functions, handles the procedure. This step is necessary because the input audio needs to have a certain representation before feeding it into the algorithm. After preprocessing the audio input, the main function transcribe(argv, config\_map, data\_fn) detects and transcribes musical notes from the WAV file. The input parameters are preset as the algorithm uses a pre-trained checkpoint to predict onset and frames. The function creates a new MIDI file which is saved in the same folder as the WAV. The filename consists of the original WAV filename with ".MIDI" added at the end. 

Music Generation

1. melody.py
   This feature creates a new melody and adds it on top of the Primer MIDI, creating a continuation. It operates on a fully monophonic basis. This means, that if necessary, the algorithm will transform the Primer MIDI into a single stream of note events. For this, it filters out the highest note in each harmony and discards the rest in order to keep the existing melody. Then, a new monophonic sequence is produced and attached to the existing piece. The sheet music consists of one single voice.
2. chords.py
   This script finds suitable chords to a Primer MIDI which is transformed into a monophonic melody. Hence, it automatically fits a harmony onto an existing musical piece. The chords consist of one base tone and the corresponding triad which are played as semibreves. If the Primer MIDI is short, the algorithm will generate additional notes to reach the appropriate length. Therefore, it also continues the melody. This program renders the backing chords as notes in the output MIDI file. On sheet music, three voices are displayed. The first voice depicts the Primer MIDI, the second voice shows the triad and the third voice contains only the base tone.
3. improv.py
   Another application that continues an existing piece of music. Unlike melody.py it does run on a polyphonic basis. Nonetheless, the algorithm analyses the Primer MIDI and simplifies larger harmonies while maintaining the existing melody. The generated piano sequence is polyphonic and has a distinct sound. It is fast-paced as it mostly contains 16th notes, has large interval jumps and therefore creates a very vivid, rushed listening experience.
4. bach.py
   This script generates a new polyphonic piano sequence based on Bach. The algorithm was trained on the Bach Chorales Dataset. The Primer MIDI is again simplified but still polyphonic. As the name implies, the result musically imitates Bach’s work, as his piano pieces have a very distinct sound. \\

Test Results

Evaluation

• Algorithms succeed in synthetic, clean piano audio
• Note accuracy drops immediately with noise
• Velocity accuracy drops due to changes in dynamics

Survey 

1. From a scale of 1 to 10, with 10 being the best, how good does this piece sound?

2. Do you think an AI created this piece?

3. Would you consider this creative?

 (20 probands were asked)

Evaluation

• Bach -> familiar -> good sounding ?

• Good sounding -> human creation VS bad sounding -> AI creation ?

• Something is being created -> creative