Creating Intelligent Musical Instruments

Dr Charles Martin - The Australian National University

web:     mastodon: @[email protected]

Ngunnawal & Ngambri & Ngarigu Country


Intelligent Musical Instruments become a normal part of musical performance and production.


Assist professional musicians & composers

Engage novice musicians & students

Create new kinds of music!

making intelligent musical predictions

predicting sequences

Interacting with predictions


  • “Experiments in Musical Intelligence” (1987)
  • Neural Networks for recognising musical gestures (1991)
  • LSTM RNNs for generating music (2002)
  • OMax Musical Agent (2006)
  • Wekinator (2009)
  • Google Magenta MelodyRNN (2016)
  • Magenta Studio (Ableton Plugins) (2019)

Performance data is diverse

Music Systems Data
Score / Notation Symbolic Music, Image
Digital Instruments MIDI
Recording & Production Digital Audio
New Musical Interfaces Gestural and Sensor Data
Show Control Video, Audio, Lighting, Control Signals

Interactive RNN Instrument

Physical Intelligent Instrument

  • Generates endless music with a melody RNN.
  • Switchable Dataset.
  • Controls for sampling “temperature”.

Physical Intelligent Instrument



Gesture RNN

  • Predicts 1 of 9 “gestures” for three AI performers.
  • Trained on labelled data from 5 hours of quartet performances.
  • Actual “sounds” are chunks of each gesture played back.


Robojam and Microjam

Robojam Interaction

  • Predicts next touch location in screen (x, y, dt).
  • Trained on ~1500 5s performances.
  • Produces duet “responses” to the user.

Mixture Density Network

IMPS System

IMPS System

  • Opinionated Neural Network for interacting with NIMES.
  • Automatically collects data and trains.
  • “Wekinator” for deep learning?

Three easy steps…

  • Collect some data: IMPS logs interactions automatically to build up a dataset
  • Train an MDRNN: IMPS includes good presets, no need to train for days/weeks
  • Perform! IMPS includes three interaction modes, scope to extend in future!

Embodied Predictive Musical Instrument (EMPI)

Embodied Predictive Musical Instrument (EMPI)

  • Predicts next movement and time, represents physically.
  • Experiments with interaction mappings; mainly focussed on call-response
  • Weird and confusing/fun?

How to build one

  • Brain: Raspberry Pi 3/4
  • Interface: Arduino Pro Mini or similar
  • Amplifier: Adafruit Mono 2.5W (PAM8302)
  • Speaker: scavenged from monitor?
  • Case: custom 3D print
  • Software:


  • Sound: Pure Data (pd) running in headless mode
  • Predictions: IMPS (running on RPi)
  • Interface to MCU: MIDI over USB

Software starts on boot on the RPi, can configure over a network.

Training Data

Human Data Sine Data Square Data Saw Data Noise Data

Generated Data

Human Generation Synth Generation Noise Generation

Using Predictions to Make Music

Emulate or enhance ensemble experience

Engage in call-and-response improvisation

Model a performer’s personal style

Modify/improve performance actions in place

Evaluating Predictive Instruments?

Does the ML model make good predictions?

Is this computationally practical?

Is this useful to musicians?

Try out IMPS or EMPI!