Tinkering Audio 3

Adding functionality.

Callbacks / Delegates in Unity

We’ve seen one example of delegates in Unity, but it turns out they are everywhere once you start looking for them! When you click a button on a canvas, you need to attach the logic you want to execute. You can use delegates any time you want to run a function and want the user/developer to swap out what function is used. There are a few good ways in which we could use this functionality:

The wave functions from last lab
Attaching functions to events in the scene (eg, when a button is pressed)

Here is a guide on using Delegates in Unity

Refactorting Start

We’ll need to populate this dictionary. As notes don’t change we could do this in start. Our start method is starting to look a bit crowded. We’ll look at refactoring the code (changing the structure without changing the functionality). We’ll do this by breaking it out into functions.

This is what my start method looks like (for reference only, do not copy this!):

public Start() {
    // code from previous session here

    scale = generateScale( frequency );
    notes = new int[]{ 7, 7, 2, 3, 5, 5, 3, 2, 0, 0, -1, 3, -1, 7, 7, 5, 3, 2, 2, 3, 5, 5, 7, 7, 3, 3, 0, -1, 0, -1 };
    durationInSeconds = notes.Length * noteDuration;

    int durationInSamples = (int)( sampleRate * durationInSeconds );
    generatedClip = AudioClip.Create("GeneratedAudio", durationInSamples, 1,
    	sampleRate, true, OnAudioRead, OnAudioSetPosition);
    source = GetComponent<AudioSource>();
}

Tip

your code should look slightly different to this. I’ve been giving you parts of the script step by step, and you may (should) have been adding your own content/features as you go. You will also note my notesToPlay variable is just called notes (hence the bug from last week :) ). You should not just copy and paste the above code it’s there for reference only!

Our start code is begining to look a bit crowded. It’s now responisible for doing an aweful lot! Let’s clean up the code by splitting out the functions into smaller chunks. first let’s figure out what it’s doing.

Seperating out the functionality

My code is doing 3 things:

Populating a list of frequencies (and notes)
Creating a clip (and figuring out the length)
Ensuring we have access to an audio source (for elsewhere in the code)

That last one is just one line, but the other two could be their own functions (and we’ll see why that might be useful in a second).

First our clip code:

private void SetupClip() {
    // TODO move the bits of Start() that are dealing with setting up the clip here
}

Now we’ll looking at the populate notes function:

private void PopulateNotes() {
    // TODO move the bits of Start() that are dealing with the frequencies here
}

Note	you have the option of putting the duration in samples in either function. It’s related to notes, but we generate it purely for setting up the clip. I’ve decided for my code, it makes more sense to go into the SetupClip method.

Then start we just need to call the two functions we just made:

public Start() {
    PopulateNotes();
    SetupClip();
    source = GetComponent<AudioSource>();
}

Using dynamic collections

In this section, we’ll be making use of collections. There is a syntax we’ll be using that you might not be familar with. Let’s talk about that before we continue.

Collections use a slightly unusual syntax: List<ThingThatGoesInTheList>. We’ve already seen one example of this our Unity code. Notice the triangle brackets in the code: <ThingThatGoesInTheList>

Recall that to get the AudioSource component we did this:

AudioSource source = GetComponent<AudioSource>();

This is an example of Generics, these are present in both C# and Java (a similar but different concept, templates, exists in C++). The idea behind generics is to allow a developer to create type-safe code that doesn’t `care' about the type that’s being used internally until it’s used. This allows us to write reusable utility classes. This comes in handy once you start building re-usable utilities that can be used across projects and means you don’t need a seperate list class for every type (ListForInts, ListForBools, ListForClips, etc…).

We’ll be using a few different collections in our code for this next part.

Using dictionaries

So far, we’ve made use of integers to track which nodes are being played. This works, but it makes managing our track data quite tedious - as some of you may have already found if you’ve tried to convert a tune into integers to play. Instead, we could make use of the standardized musical notation we’ve already seen (eg, C4 rather than 3, and A4 rather than 0).

Adapting notes to play

Create a new dictionary at the top of the file - what datatypes are we mapping to and from (eg, "A4" → 440). For dictionaries, the first type (eg, "A4" in my example) is called the key, the second type is called the value. Keys are unique, values don’t have to be (but in our case probably are).

private Dictionary< KeyType, ValueType > noteNames = new Dictionary< KeyType, ValueType >(); // TODO replace KeyType and ValueType with the correct types.

Q: This can replace/use/duplicate data already in an array that is in our code, which one?

Task: Populate noteNames in your code

Note	you could do this by manually adding each note (ie. `noteNames["A4"] = 440.0f;`), but there is a better way of doing this using a for loop.

Altering NotesToPlay

We can now change NotesToPlay/Notes to make use of our new feature:

Change NotesToPlay from a int[] to a String[]
In OnAudioRead we don’t want to be looking up the next note via the Scale array anymore, we want to use our dictionary.

Because I don’t want you to spend too much time recreating the tune, here is the note sequence I used for last week:

// new[] = shorthand for new string[] in this example
notesToPlay = new[] {
	"E4", "B4", "C4", "D4", "C4", "B4", "A4",
	"A4", "C4", "E4", "D4", "C4", "B4",
	"C4", "D4", "E4", "C4", "A4", "A4"
};

Then to play the notes, we’d need to look up the note in the dictionary:

    void OnAudioRead(float[] data) {
    	int count = 0;
	while ( count < data.Length ) {

		int index = (int)( position / ( noteDuration * sampleRate) );
		**ADD TYPE HERE** scaleNote = notesToPlay[index]; // TODO fix this line
		if ( scaleNote != null ) {
			float noteFrequency = noteNames[scaleNote];
			data[count] = TriangleWave( noteFrequency, position );
		} else {
			data[count] = 0.0f;
		}

		position++;
		count++;
	}
   }

Try and play the new track, you might notice a few things:

The note durations seem a little off (some notes that should be held arn’t)
The gaps in the tune are missing
If you’ve made the same mistake I did when implementing this, it sounds nothing like Tetris!

Let’s investigate the last one together (we’ll look at fixes for the first two in a second).

Note Names

If you added your notes by simply adding one letter each time, your tune doesn’t sound right. You may have thought of a nice little shortcut to storing the names (the trick is at the bottom of this section if you’re intrested). But notes don’t work like that.

We can check if the following two notes (the opening note from the Tetris theme) match up, they should do:

Debug.Log(noteNames["E4"]);
Debug.Log(scale[7]);

If you’ve made the same mistake I did, they won’t. Notes don’t incrementing the letter one by one for each note. Recall that from the Notes List from last time.

The full list for our octave is:

A
A# (A sharp), aka Bb (B flat)
B
C
C# (C sharp - not the language :) ), Db (D flat)
D
E
F
F# (also not the language), Gb
G
G# (hey, why isn’t there a G# language?), Ab

When generating our dictionary, we need to know which note corrisponds to which frequency on this list. To work around this, I’ve used the following array in PopulateNotes:

string[] scaleList = new []{"A", "A#", "B", "C", "C#", "D", "E", "F", "F#", "G", "G#" };

This actually still isn’t correct. The correct note names aren’t A4, …, G#4. The point that the notes become the 5 is actually the C, ie, the note after B4 is actually C5. Meaning B4 is before C5, not C4.

I’m not going to fix this in the script, but have a go at fixing this in your code. If you fix it without hard-coding this change, it also makes adding more than one octave really simple.

Tip	the incorrect note names used in the tetris theme above can be generated using: `string noteName = scaleList[i] + "4";`

Note	If they did increase by one letter each note, there is a nice trick we can perform: (char)('A' + 1) == 'B'

Adding Rests

To get rests (notes when no note is being played) working in the integer version from last week, I added a special -1 note that meant, "play frequency 0". I’ve ported that solution over the code above using the special value null. There is slightly nicer way of doing this. Instead, I could have made it so that I added an extra note to the dictionary (eg, NotesToPlay['-'] = 0.0f). Re-add these rests into the tetris example to get it sounding more like tetris again.

Durations

There is one more change I made to the tetris theme. Some notes need to be played for longer than others. The way I fixed this last time was simply to repeat the note twice (ie, the first notes in the integer version is 7, 7, or E4, E4 in the broken notation I’ve used). This works well for some sequences, but if I lots of changes in duration this would quickly get annoying to write. It also means that all of my note lengths are equal to the duration (while in music you can have notes that last less than 1 beat).

For now, if you want to recover the original tetris tune I was using, repeat this note repetition. A better solution would be to store a note and duration rather than just a note by itself. To do this we could use a class, and we’ll revisit this later. It will take fairly extensive code changes which is why it’ll be it’s own section.

Using Lists

So far, we’ve made use of arrays for storing our tune. Arrays are a very versitle data structure, but they are fixed length - You can’t make them longer or shorter once they have been created. If we want an array-like object which can change side, we need to use a List.

To make use of lists, we’re going to create a method that adds a new note to our melody every time the user in the scene presses a button. To make this work, we’ll need to refactor our code slightly:

We need to make notesToPlay a List<string> rather than string[]
At the moment, we create the audio track when the Component starts (using the Start method) - we need to split this out into it’s own method so we can call it when things change.

Let’s make the change to notes to play variable first:

private List<string> notesToPlay = new List<string>();

In our PopulateNotes code (or wherever you are generating your notes), we need some syntax changes to make the code work with a list:

// your notes will probably be a little different depending on how you've addressed the points above.
notesToPlay = new List<string>
{
   "E4", "E4", "B4", "C4", "C4", "D4", "C4", "B4", "A4",
   "A4", "C4", "E4", "D4", "C4", "B4",
   "C4", "D4", "E4", "C4", "A4", null, "A4"
};
durationInSeconds = notesToPlay.Count * noteDuration;

Refactoring note generation

If you’ve put the generating the melody code in the PopulateNotes, this function looks like it could do with some splitting. There are two reasons for this:

If we want to generate new melodies at runtime, we’ll want to rerun the creating of the notes to play
We don’t want to keep repopulating our Dictionary every time we do this

Let’s split this out into two functions:

PopulateNotes will deal with populating the dictionary
GenerateMelody will deal with populating/resetting the notesToPlay code

As the durationInSeconds is no longer fixed (it will depend how many notes are added), I’m going to move this into SetupClip next to durationInSamples.

    private PopulateNotes() {
    	// existing code for populating the dictionary here

	// notesToPlay code now in it's own function
	GenerateMelody();
    }

    public void GenerateMelody()
    {
	    notesToPlay.Clear(); // get rid of existing notes
	    notesToPlay = new List<string>
	    {
		    "E4", "E4", "B4", "C4", "C4", "D4", "C4", "B4", "A4",
		    "A4", "C4", "E4", "D4", "C4", "B4",
		    "C4", "D4", "E4", "C4", "A4", null, "A4"
	    };
    }

Note I’ve also made the GenerateMelody function public. This means it can be called from other scripts in unity.

Adding a random note

Let’s add a random note to notesToPlay. First, we’ll write a utility method that takes an IEnumable and returns a random element from it:

    public T GetRandomElement<T>(IEnumerable<T> elements)
    {
	    var list = elements.ToList();
	    int element = Random.Range(0, list.Count);
	    return list[element];
    }

Note	you might need to add imports to make IEnumerable work, use the lightbulb in Visual Studio to do this for you.

That <T> means 'Whenever you see T in this function, I mean the generic type that is being referred to'. This function therefore return a value based on whatever T happens to be:

a List<string> will return a string
a List<int> will return an int
a List<Anything> will return an Anything
etc…

var means 'figure out the type for me', which is useful to avoid writing out the full type.

Let’s make use of our new method to select a random note from our Dictionary’s keys:

    public void addRandomNote()
    {
	    string randomNote = GetRandomElement(noteNames.Keys);
	    notesToPlay.Add(randomNote);

    }

Tip	IEnumerable is a supertype of list, by using it rather than list, our function will also work with non-List collections that are still enumerable (eg, sets).

We need to do one more thing before this usable. Remember that clip knows how long it is. As a result, we need to re-create the clip every time a button that alters the length of the clip is pressed. Luckily, we already have a function that does this, SetupClip:

    public void addRandomNote()
    {
	    string randomNote = GetRandomElement(noteNames.Keys);
	    notesToPlay.Add(randomNote);
	    SetupClip();
    }

As we will be testing our our new addRandomNote method, we can also comment out the call to GenerateMelody in PopulateNotes.

Try and run the code as is. You may notice a warning in the window, regarding the clip length being zero.

We can fix this by guarding SetupClip so that it does nothing if there are no notes to play.

    private void SetupClip() {
	    if (notesToPlay.Count == 0)
	    {
		    return;
	    }

	    // rest of function here

Adding a new button

Let’s add a new button to our code. This time, I want to show you how you can make use of our audio manager in other game objects. So rather than add the button in code, let’s add it in a new script:

Right click on the Hierachy in Unity, and select UI › Button. This should add a canvas to the scene, along with an Event Manager and the button.

Select the button in the Hierachy, and look at the inspector window. You should see On Click as one of the options. This makes use of our old friend, delegates. Press the + button on the On Click panel.

Next, select drag your AudioManager game object over the words None (object)

click add callback

Finally, where it says, "No Function" select AudioManager › addRandomNote. If you can’t see Add random note, make sure the function is public. Any function we want to use from another script should be marked as Public.

click add object

Note	you can change the text and contents of the button if you wish.

scene with btns

Press the button we’ve just added a few times, then press our play button.

TASK add a new button that resets the audio back to tetris (using GenerateMelody from before).

Tip	you need to re-generate the clip in GenerateMelody

TASK add a new button that clears notesToPlay and add it to a third button.

Accessing other scripts

Note	this section is just here for assignment help

You can use the same approach (creating a public variable in another script) and invoking public methods from your audio manager to add functionality in game events (eg, collisions or player actions).

public class MyOtherBehaviour : MonoBehaviour {
	public AudioManager myManager;

	// or whatever seems like the right place for it...
	public void Start() {
		manager.GenerateMelody();
		manager.playAudio();
	}
}

As an aside, this is why I made you write PlayAudio as a function all the way back in tinkering audio 1. Remember to drag your audio manager in the inspector!

(Advanced) adding classes

Note	this is considered extention work. You need to understand classes, but dealing with them in this way is additional work.

So far we’ve stored audio (note) data in a bunch of ways. First, as floating point values directly. We then abstracted that into integers, so we could more easily build 'higher level' (more abstract) tunes (songs). We abstracted this futher into strings, so we could express those songs using a standardised notation (rather than list index or raw frequencies). Let’s take this one step futher, and start including additional infomation.

There are two aspects of audio we’ve only briefly touched on:

Volume
Duration

Both of these are important, but implementing volume is relatively streightforward based on what you already know. We’re going to look at a better way of implementing durations now.

We can create a class to store note and duration together, you could also store additional data like wave type, volume, etc…:

class Note {
	public string name;
	public float duration;

	public Note(string name)
	{
		this.name = name;
		this.duration = 1.0f;
	}

	public Note(string name, float duration)
	{
		this.name = name;
		this.duration = duration;
	}
}

Note	this is an example of method (constructor) overloading, we can create a note either using just a name (in which case, the duration is 1.0f), or a name and duration.

Once again, notesToPlay needs a change to make use of this:

private List<Note> notesToPlay = new List<>();

To store a note, we’d need to create a new instance of this class:

// this
"C4"

// becomes this...
new Note("C4");

//and this...
"C4", "C4"

// becomes this...
new Note("C4", 2.0f);

Note	changing this will generate errors in other bits of your code. Try and fix these yourselves.

Calculating length

There is a complication to this approach. We can no longer safely assume the length of the track (in noteDurations) is equal to the length of the notes.

In SetupClip, we will need to:

Set a durationInSeconds to be 0.0f
for each note in the NotesToPlay
add note.Duration * noteDuration to durationInSeconds

Duration Calculations

There is another complication, we cannot simply use position to figure out what index into the list to play.

We instead need to keep track of this seperately. The simplist way of doing this is to keep track of the currently playing index. If the current duration (in samples) is less than the note duration in samples, keep playing the note, else select the next note in the list.

There are lots of way do do this, my approach uses 3 variables:

	private int currentNoteIndex;
	private int currentNoteSample;
	private int maxNoteSample;

And a (non-working) hint as to how you might use these:

	if ( currentNoteSample <= maxNoteSample ) {
		currentNoteIndex++;
		currentNoteSample = 0;
		maxNoteSample = NotesToPlay[currentNoteIndex].duration * noteDuration * sampleRate;
	}

You will also need to either calculate, or reset these in OnAudioSetPosition.