Ngunnawal & Ngambri Country

Creative Machine Learning with ml5.js

What is machine learning (ML)?

How can I use ML in p5?

How can we make art with this?

What is Machine Learning Anyway?

Creating computer programs without explicitly programming them.

Algorithms that learn by example.

Algorithms that learn through experience.

Kind of a big deal ($$$)

Kind of problematic (!!!)

Let’s Solve a Problem

Suppose the boss wants a program where the screen colour changes to red when the mouse moves to certain locations.

(N.B.: the screen is 400 pixels wide)

How would you do it?

Let’s write a configurable algorithm!

if (mouseX > ??) {
    background(255,0,0);
} else {
    background(0,0,0);
}

one decision (red or black background)

one input (mouseX)

What if we had more inputs?

Maybe we could make more complicated decisions?

Likely to get more complicated to configure the algorithm.

Pictures as Inputs

Pictures are 2D arrays of colours! (represented as numbers)

So we had enough ifs and elses then maybe we could make a doggo classifier!

(Photo by Ashleigh Robertson on Unsplash)

Simplifying inputs

One trick we often use is to design a configurable algorithm which can:

• take lots of numbers as inputs
• boil this all down to just one number as output.

The “configuration” would be “choosing how much of each input to listen to”

One example is a “perceptron” (1958)

Fast forward 50 years.

Trick 1: feed the outputs of perceptrons into more perceptrons in a kind of network

Trick 2: tricky algorithms to choose the configuration

Trick 3: big fast computers with lots of data to learn from

By the way, another name for a perceptron is an artificial neuron. So the above is a… neural network…

Some terminology

Model: an instance of a trainable algorithm

Pre-Trained Model: a trainable algorithm which has… already been trained.

Training/Optimising: using training data to make a trained model.

Prediction/Inference: using a trained model to generate an output using unseen data

Classification: an ML task for choosing a “class” (or description) for a piece of data

ml5.js: Friendly Machine Learning

ml5.js is a JavaScript library that provides access to machine learning models in a web browser.

You can load up pre-trained models and start doing prediction right away!

Related to and inspired by p5.js.

Get started

Just need to load it in our index.html:

<script src="https://unpkg.com/[email protected]/dist/ml5.min.js" type="text/javascript"></script>

Open up a p5 web editor sketch with ml5 already loaded.

Classifying Images

Let’s classify some doggos. We’ll use a pretrained model called MobileNet

// load the classifier
classifier = ml5.imageClassifier('MobileNet');
// classify an image
classifier.classify(img, gotResult)

Where does the result go? Need to define a callback function gotResult(error, results).

Classifying Video

We can access a webcam in our sketch:

video = createCapture(VIDEO);
video.size(320, 240);
video.hide();

And we can just ask the classifier to only make predictions from this video stream:

classifier = ml5.imageClassifier('MobileNet', video);
classifier.classify(gotResult); // classify one frame

What is problematic about doggos?

What is MobileNet?

Photo by Alina Grubnyak on Unsplash

What was the training data?

Are there hidden costs?

Artistinal Bespoke Machine Learning

Let’s make our own custom image classifier with Teachable Machine