soundbank | triggerable Web Audio API audio slots , supporting audio | Audio Utils library

You may read the file into a string and run the following regex:

The default synthesizer, "Microsoft GS Wavetable SW Synth", uses the file gm.dls, which is a system file and should not be replaced. There is no programmatic way to choose another sound font.

Firstly, to be able to add music, it needs to be hosted on a publicly accesible https endpoint, see the docs. So make sure you can access your file even when using a private browsing mode such as incognito on chrome.

Secondly, If you choose to use SSML to play your audio, the audio will become part of the speech response. By doing this, you won't be able to create any custom interruptions or control over the music. The user can only stop the music by stopping your action or saying "Okay Google" again to interrupt your response.

If you would want to allow your users to control the music you send to them, try having a look at the media responses within the actions on google library.

Seems like you use refs in the wrong way.

The useRef hook returns a "ref" object that you should pass directly to the ref prop of the tag. Then access the audio using audio.current in handleKeyPress function. Also, you should wrap the event handler into useCallback hook because the current implementation gives you a new handleKeyPress function on each render cycle.

PS: Why are you trying to remove the event listener in the returned value from your handleKeyPress function? It's not necessary to manage event listeners manually with React unless you added them manually using addEventListener.

You're missing a note_on. For example:

An audio file is basically a list of numbers. Those numbers are read one at a time at a particular speed called the 'sample rate'. The sample rate is otherwise defined as the number of audio samples read every second e.g. if an audio files sample rate is 44100, then there are 44100 samples (or numbers) read every second.

If you are with me so far, the simplest way to lower the pitch of an audio file is to play the file back at a lower sample rate (which is normally fixed in place). In most cases you wont be able to do this, so you need to achieve the same effect by resampling the file i.e adding new samples to the file in between the old samples to make it literally longer. For this you would need to understand interpolation.

The drawback to this technique in either case is that the sound will also play back at a slower speed, as well as at a lower pitch. If it is a problem that the sound has slowed down as well as lowered in pitch as a result of your processing, then you will also have to use a timestretching algorithm to fix the playback speed.

You may also have problems doing this using MP3 files. In this case you may have to uncompress the data in the MP3 file before you can operate on it in such a way that changes the pitch of the file. WAV files are more ideal in audio processing. In any case, you essentially need to turn the file into a list of floating point numbers, and change those numbers to be effectively read back at a slower rate.

Other methods of pitch shifting would probably need to involve the use of ffts, and would be a more complicated affair to say the least.

I am not familiar with nodejs I'm afraid.