Making Predictive NIMEs with Neural Networks

A model of a predictive interaction interface.

Do you want apply machine learning or AI in creative applications, but don’t know where to start? Do you want to make NIMEs that play themselves? Do you want your computer to compose endless video game soundtracks? Then this is the workshop for you!

Workshop Description

In this workshop, we’ll apply predictive machine learning models to creative data and use them in interactive music applications. We will show you how to create and train neural networks and how to use our new toolkit for interactive musical prediction (IMPS). Join us to define new predictive NIME prototypes and future research directions!

The first half of the workshop will focus on recurrent neural networks (RNNs), deep learning models that can be used to generate sequences and mixture density networks (MDNs) that can creatively predict multivariate data. We will walk through the steps for training creative RNNs using live-coded demonstrations with Python code in Jupyter Notebooks.

The second half of the workshop will focus on the Interactive Musical Prediction System (IMP), an end-to-end solution for adding an MDRNN to a musical interface with communication over OSC. This system allows you to focus on defining new predictive NIMEs that can “fill in” parts of a musical performance or to “dream” new performances and accompaniments.

Session Plan

Getting Started

Welcome to the Creative Prediction workshop at NIME! Thanks for coming along!

This workshop will introduce the basics of deep learning generation of creative sequences (e.g., text, music, videos, movements, etc!). We will cover a bit of the theory behind recurrent neural networks, and mixture density networks, and show you how to construct your own with Python and Keras.

All of the demonstration code for the workshop is contained in Jupyter Notebooks, an open standard for mixing code, text, and visualisations in a document that can be opened in web browser. We will display this code on the screen for you to follow along and see how it works, but for maximum fun, you’ll want to install Jupyter, Python, and Keras on your own computer

Another way to try out the Jupyter Notebooks is with Google Colaboratory, a free-to-use Jupyter notebook environment that has most of the necessary Python libraries pre-installed. It even works on a tablet! If you want to get started quickly without slowing down to get your Python install right, Colab is a great way to go.

Colab has some amazing features:

There are some downsides though:

The notebooks have some sections with a comment like “Use this if on Colab!” to work around some of the limitations.

Feel free to download all examples (and this website) from our GitHub repository if you want :-D

Overview of Deep Learning and Creativity


(30 minutes talk).

Generating Text and Music with RNNs


(30m talk, 60m hack).

Using Mixture Density Networks (MDNs) to predict NIME data with RoboJam


(45m talk, 45m hack).

Making Predictive Musical Interactions with the IMPS system

The IMPS system in use


IMPS information

(30m talk, 90m hack).

Future directions for creative neural networks at NIME and beyond


Let’s define the future of predictive NIMEs… together.

(60m discussion and brainstorming).