Sound and Music II

To learn how to use the sound functions in p5js see the p5js sound reference.

Audio Editing | Scene Timers | Sycnhronizing Visuals to Audio

Editing and Labelling Audio with Audacity

Audacity is a pro-quality free audio editor that works on Windows or Mac. There is good online documentation. It allows multiple tracks to be mixed and edited from many different sound file formats. The program also has many built-in effects, such as reverb, delay, normalization, and many others.

A useful feature of the program for our purposes is that labels can be added at precise locations in the audio. Export the labels to a file to load times into Processing, to synchronize elements of your Processing sketches to music. Following is a screenshot of a stereo sound with text labels in Audacity:

Scene-Change Timing in p5js

Sometimes we wish to have events occur in our sketch at precise predetermined times. Following is a method for timing scene changes in p5js with a list of time locations specified in milliseconds and stored in an Array. We use an integer counter int currentScene to keep track of where we are in the sequence of scenes. This counter gets incremented when millis() > times[currentScene]:

// millis() as a timer with an array of scene times
// M. Casey, Dartmouth College, CS2

// initialize a literal int[] array of times in milliseconds
let times = [0, 1000, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 7000, 10000]; 
let currentScene = 0; // scene counter

function setup(){
 createCanvas(800,800); 
 textSize(48); // use the default font, no need to load one.
}

function draw(){
  // is elapsed time in milliseconds greater than time of next scene change?
  if(currentScene < times.length && millis() > times[currentScene]){ // array end-point guard
    background(random(255),random(255),random(255));
    let title = "Scene " + currentScene; // make a title: Scene 0, Scene 1, etc...
    text(title, width/2-textWidth(title)/2, height/2); // display the title, centered
    currentScene++; // update the scene counter, to wait for next scene change
  }  
}

Flow Control for Scene Changes

Let's say we want different functions to draw the different scenes, then we can use the scene counter in a conditional statement such as if (currentScene==0){ drawScene0(); }:


// Flow-control using if( ... ){ ... }
// M. Casey, Dartmouth College, CS2

// initialize a literal int[] array of times in milliseconds
let times = [0, 1000, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 7000, 10000]; 
let currentScene = 0; // scene counter

function setup(){
 createCanvas(800,800); 
 textSize(48); // use the default font, no need to load one.
}


function draw()
{
  // is song.currentTime()*1000 in milliseconds greater than time of next scene change?
  if(currentScene < labels.length && song.currentTime()*1000 >= times[currentScene]){ // array end-point guard
    currentScene++;      // update the scene counter, to wait for next scene
  }
  if(currentScene==0)
    drawScene0();
  if(currentScene==1)
   drawScene1();
  if(currentScene==2)
   drawScene2();
  if(currentScene==3)
   drawScene3();
//  etc ...
}

function drawScene0(){
  // your drawing code here
}

function drawScene1(){
  // your drawing code here
}

function drawScene3(){
  // your drawing code here
}

// etc...

A more efficient way of conditional is the switch statement, which allows us to achieve the same result but with slightly tidier syntax:


// Flow-control using switch( ...) and case: 
// M. Casey, Dartmouth College, CS2

// initialize a literal int[] array of times in milliseconds
let times = [0, 1000, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 7000, 10000]; 
let currentScene = 0; // scene counter

function setup(){
 createCanvas(800,800); 
 textSize(48); // use the default font, no need to load one.
}


function draw()
{
  // is song.currentTime()*1000 in milliseconds greater than time of next scene change?
  if(currentScene < labels.length && song.currentTime()*1000 >= times[currentScene]){ // array end-point guard
    currentScene++;      // update the scene counter, to wait for next scene
  }
    switch(currentScene){    
    case 0:
    drawScene0();
    break;
   case 1:
    drawScene1();
    break;
   case 2:
    drawScene2();
    break;
  case 3:
    drawScene3();
    break;
//  etc ...
  default:
    drawEnd():
 }
}

function drawScene0(){
  // your drawing code here
}

function drawScene1(){
  // your drawing code here
}

function drawScene2(){
  // your drawing code here
}

// etc...

Synchronizing Audio and Animation using Timers

After using File>Import->ImportAudio in Audacity to import your audio file, add labels at signiciant time locations (intro, verse, chorus, etc...) in the audio using cmd-B (Edit->Labels->Add Label at Selection). Times will be added automatically to your labels, and you may not even need to add text to your labels if you don't need. Once you have marked all the events, you can export the times using File->Export->ExportLabels. This will create a text file consisting of [TAB]-separated rows of start-time, end-time, and description, with each label on a separate line:

File labels.txt with rows of tab-separated times and descriptions. Export from Audacity and added to Processing sketch:

  0.000000 0.000000 MOZART K448 Piano Sonata in D Major
  0.976234 0.9762341 EXPOSITION Introduction
  8.267483 8.2674835 (1a) primary theme presentation, tonic prolongation
  15.314673 15.3146739 (1a) continuation ||: I Vb -> ii V I [PAC] :||
  29.164994 29.16499417 (1b,  inverted) descending sequence I vi IV II/V
  39.476467 39.47646723 (1a) Modulation to Dominant I II/V I -> secondary modulation to V of V in Dominant (A Major)
  42.954301 42.95430125 (1c) Transition sequential prolongation of V to [IAC] then [PAC] in Dominant (A Major)
  59.184194 59.18419434 (2a) secondary theme, period, dominant (A Major)
  74.620896 74.62089642 (2a) contrapuntal continuation -> [PAC] Dominant (A Major)
  88.074622 88.07462249 (1a fragment) sequential build, crescendo, tonic pedal (A Major)
  100.613129 100.61312956 (1a fragment) descending sequence, I vi IV ii
  108.041661 108.04166160 (1a) climax Vb I vi Vb -> [PAC]  
  115.165119 115.16511964 (1a) continuation, descending sequence
  131.852621 131.85262173 (1a) coda, tonic pedal prolongation -> [PAC] dominant (A Major)
  146.374104 146.37410481 REPEAT Introduction
  153.634845 153.63484585 (1a) primary theme presentation, tonic prolongation
  160.712542 160.71254289 (1a) continuation ||: I Vb -> ii V I [PAC] :||
  174.623878 174.62387897 (1b,  inverted) descending sequence I vi IV II/V
  188.413185 188.413185105 (1c) Transition sequential prolongation of V to [IAC] then [PAC] in Dominant (A Major)
  204.582063 204.582063114 (2a) secondary theme, period, dominant (A Major)
  220.293331 220.293331122 (2a) contrapuntal continuation -> [PAC] Dominant (A Major)
  233.960609 233.960609129 (1a fragment) sequential build, crescendo, tonic pedal (A Major)
  246.407594 246.407594136 (1a fragment) descending sequence, I vi IV ii
  253.820872 253.820872140 (1a) climax Vb I vi Vb -> [PAC] 
  261.020598 261.020598144 (1a) continuation, descending sequence
  277.799622 277.799622153 (1a) coda, tonic pedal prolongation -> [PAC] dominant (A Major)

To use these labels, load them into Processing as a String[] array using loadStrings("yourFileName.txt"). Then, using a for(...) loop, split each string in the array at the tab characters '\t' using split(labels[i], '\t') to get the time from column [0] and the descripton from column [2]:

// Music timing
// Use audacity to label audio, export labels file
// Import labels file, use in an array to time scenes

// Import the library
// Declare variables corresponding to minim and song classes
let song;
let labels = [];
let times;
let currentScene = 0;
f
function setup()
{
  createCanvas(600, 600);
  labels = loadStrings("labels.txt");   // load a String[] array, one label entry per line
  times = new Array(labels.length);       // initialize an int[] array to hold times in milliseconds

  song = loadSound("Mozart_k448_i.wav");  // load the sound file
  song.play();                                 // start playing, to start the song.currentTime()*1000 timer

  // initialize the times (in milliseconds) from labels file
  for(let i=0; i < labels.length; i++){
    labs = split(labels[i], '\t'); // split the i-th row of labels at tabs '\t'
    times[i] = int( number( labs[0] ) * 1000 ); // grab the 1st column, convert to milliseconds
    print(times[i]);
  }
  
  background(0); // inital background
  textSize(36);  
}

function draw()
{
  // is song.currentTime()*1000 in milliseconds greater than time of next scene change?
  if(currentScene < labels.length && song.currentTime()*1000 >= times[currentScene]){ // array end-point guard
    description = split(labels[currentScene], '\t')[2]; // grab third column [2] of tab-separated list
    background(random(255),random(255),random(255));     // change the background
    text(description, width/2-textWidth(description)/2, height/2);   // display the description
    if (currentScene==0) // at start pause song after title, wait for spacebar press to continue
      song.pause();
    currentScene++;      // update the scene counter, to wait for next scene
  }
}

function keyPressed(){
 if(key==' ') 
   if(song.isPlaying() )
     song.pause();
   else
     song.play();   
}