Making Predictive NIMEs with Neural Networks
Do you want apply machine learning or AI in creative applications, but don’t know where to start? Do you want to make NIMEs that play themselves? Do you want your computer to compose endless video game soundtracks? Then this is the workshop for you!
Workshop Description
In this workshop, we’ll apply predictive machine learning models to creative data and use them in interactive music applications. We will show you how to create and train neural networks and how to use our new toolkit for interactive musical prediction (IMPS). Join us to define new predictive NIME prototypes and future research directions!
The first half of the workshop will focus on recurrent neural networks (RNNs), deep learning models that can be used to generate sequences and mixture density networks (MDNs) that can creatively predict multivariate data. We will walk through the steps for training creative RNNs using live-coded demonstrations with Python code in Jupyter Notebooks.
The second half of the workshop will focus on the Interactive Musical Prediction System (IMP), an end-to-end solution for adding an MDRNN to a musical interface with communication over OSC. This system allows you to focus on defining new predictive NIMEs that can “fill in” parts of a musical performance or to “dream” new performances and accompaniments.
Session Plan
- 9:00 Meeting, Intro, and Python Setup
- 9:15 Overview of Deep Learning and Creativity
- 9:45 Generating Text and Music with RNNs (practical)
- 10:30 (coffee break)
- 10:45 Hack: inventing Star Trek episode titles
- 11:30 Using Mixture Density Networks (MDNs)
- 13:00 Lunch (90 minutes)
- 14:30 Making Predictive Musical Interactions with the IMPS system
- 16:00 (coffee break)
- 16:15 Future directions for creative neural networks at NIME and beyond
Getting Started
Welcome to the Creative Prediction workshop at NIME! Thanks for coming along!
This workshop will introduce the basics of deep learning generation of creative sequences (e.g., text, music, videos, movements, etc!). We will cover a bit of the theory behind recurrent neural networks, and mixture density networks, and show you how to construct your own with Python and Keras.
All of the demonstration code for the workshop is contained in Jupyter Notebooks, an open standard for mixing code, text, and visualisations in a document that can be opened in web browser. We will display this code on the screen for you to follow along and see how it works, but for maximum fun, you’ll want to install Jupyter, Python, and Keras on your own computer
Another way to try out the Jupyter Notebooks is with Google Colaboratory, a free-to-use Jupyter notebook environment that has most of the necessary Python libraries pre-installed. It even works on a tablet! If you want to get started quickly without slowing down to get your Python install right, Colab is a great way to go.
Colab has some amazing features:
- You can load all the notebooks for this workshop straight from the GitHub repo.
- You can use a GPU for free to train a biggish RNN
There are some downsides though:
- uploading and downloading data is a bit fiddly.
- you can’t use music21’s playback features to hear generated MIDI scores, or visualise them with musescore.
- you can’t run IMPS in Colab (it’s not a jupyter notebook).
The notebooks have some sections with a comment like “Use this if on Colab!” to work around some of the limitations.
Feel free to download all examples (and this website) from our GitHub repository if you want :-D
Overview of Deep Learning and Creativity
(30 minutes talk).
- sequence learning with RNNs and Long Short-Term Memory (LSTM) in Keras and Python
- Exercise: Set up Python, Keras, and Jupyter on your computer
- Exercise: Try out a Google Colab Notebook
Generating Text and Music with RNNs
(30m talk, 60m hack).
- Predicting sequences with LSTM networks
- Generating text character-by-character
- Generating music note-by-note
- Demo: Star Trek Episode Title Generator
- Demo: Zeldic Music Generator
- Exercise: Make your own text generator
- Demo and Tutorial Code
Using Mixture Density Networks (MDNs) to predict NIME data with RoboJam
(45m talk, 45m hack).
- Predicting continuous data from NIMEs
- Gaussian Mixture Models
- Mixture Density RNNs
- Demo: Kanji Generation
- Demo and Tutorial Code
Making Predictive Musical Interactions with the IMPS system
(30m talk, 90m hack).
- The “Interactive Music Prediction System”: IMPS
- Setup and interaction scheme
- Demo: IMPS with Lightpad block and Behringer Xtouch Mini
- Exercise: Set up IMPS with 3D prediction (x, y, t) with Processing and Pd
Future directions for creative neural networks at NIME and beyond
Let’s define the future of predictive NIMEs… together.
- What even is a predictive NIME?
- What NIMEs could benefit from prediction?
- What tools are needed?
- Who can help?
- What resources are necessary?
- What’s going on with this “Creative AI” field anyway?
(60m discussion and brainstorming).