spectrogram | OpenBSD sndio Spectrogram | Learning library

I got the code corrected..

A more general solution is to do that yourself. 1D FFTs can be split in smaller ones thanks to the well-known Cooley–Tukey FFT algorithm and multidimentional decomposition. For more information about this strategy, please read The Design and Implementation of FFTW3. You can do the operation in virtually mapped memory so to do that more easily. Some library/package like the FFTW enable you to relatively-easily perform fast in-place FFTs. You may need to write your own Python package or to use Cython so not to allocate additional memory that is not memory mapped.

One alternative solution is to save your data in HDF5 (for example using h5py, and then use out_of_core_fft and then read again the file. But, be aware that this package is a bit old and appear not to be maintained anymore.

I think the error comes on how you are iterating though the wav files.

change this:

Remove all kernel_regularizers, BatchNormalization and dropout layer from Convolution layers which are not required.
Keep kernel_regularizers and Dropout only in Dense layers in your model definition as well as change the number of kernels in Conv2d layer.

and try again training your model using below code:

I have approached problems like this by using a fixed input-size CNN to do a simple classification and then called the CNN multiple times as you scan across your variable length sample (1-5 sec sound bite).

For example, let's say you create a CNN that inputs 0.2s of data, the input size is now fixed. You can compute a {0, 1} label for that 0.2s based on whether the center point of the sample is within an event as you defined in your question. You could try different input sizes using the same method.

Now you ask the CNN to make a prediction at every point in your 1-5 second sample. To start with you pass the CNN the first 0.2s of data, then step forward one or more data points (your step size is a hyper-parameter you can tune). Let's say your step size is 0.1s, your second step would produce a CNN classification using the data from 0.1s to 0.3s in your sample. Continue until you reach the end of your sample. You now have classifications across the sample. In principle you could get a classification at every data point so you have as many predictions as you have data points. A rolling median filter (see pandas) is a great way to smooth out the predictions.

This is a very simple CNN to set up. You also benefit by increasing your training data quite a bit because each sound file is now many training samples. Your resolution for predictions is very granular with this method.

Here's a paper that describes the approach in greater depth (there's also a slightly earlier version on arXiv by the same title if that's pay walled for you), start reading at Section 3 onward:

https://academic.oup.com/mnras/article/476/1/1151/4828364

In that paper we're working with 1D astronomy data, which is structured basically the same as 1D audio data, so the technique will apply. In that paper I'm doing a bit more than just classification, using the same technique I'm localizing zero or more events as well as characterizing those events (I would start with just the classification for your purposes). So you can see that this approach extends quite well. In fact even multiple events that partially overlap each other in time can be identified and extracted effectively.

File doesn't seem to be 1 sec. (different from other files.) First check if "not a sec file" is acceptable and then fix your file. Alternatively change the plotting code to receive any length of files.

You are outputing the linear-layer before the sigmoid. Change the code as following:

Check e.g. this part of the MelGAN code: https://github.com/descriptinc/melgan-neurips/blob/master/mel2wav/modules.py#L26

Specifically, the Audio2Mel module simply uses standard methods to create log-magnitude mel spectrograms like this:

• Compute the STFT by applying the Fourier transform to windows of the input audio,
• Take the magnitude of the resulting complex spectrogram,
• Multiply the magnitude spectrogram by a mel filter matrix. Note that they actually get this matrix from librosa!
• Take the logarithm of the resulting mel spectrogram.

Your confusion might stem from the fact that, usually, authors of Deep Learning papers only mean their mel-to-audio "decoder" when they talk about "vocoders" -- the audio-to-mel part is always more or less the same. I say this might be confusing since, to my understanding, the classical meaning of the term "vocoder" includes both an encoder and a decoder.

Unfortunately, these methods will not always work exactly in the same manner as there are e.g. different methods to create the mel filter matrix, different padding conventions etc.

For example, librosa.stft has a center argument that will pad the audio before applying the STFT, while tensorflow.signal.stft does not have this (it would require manual padding beforehand).

An example for the different methods to create mel filters would be the htk argument in librosa.filters.mel, which switches between the "HTK" method and "Slaney". Again taking Tensorflow as an example, tf.signal.linear_to_mel_weight_matrix does not support this argument and always uses the HTK method. Unfortunately, I am not familiar with torchaudio, so I don't know if you need to be careful there, as well.

Finally, there are of course many parameters such as the STFT window size, hop length, the frequencies covered by the mel filters etc, and changing these relative to what a reference implementation used may impact your results. Since different code repositories likely use slightly different parameters, I suppose the answer to your question "will every method do the operation(to create a mel spectrogram) in the same manner?" is "not really". At the end of the day, you will have to settle for one set of parameters either way...

The direction Mel -> Audio is hard. Not even Mel -> ("normal") spectrogram is well-defined since the conversion to mel spectrum is lossy and cannot be inverted. Finally, converting a spectrogram to audio is difficult since the phase needs to be estimated. You may be familiar with methods like Griffin-Lim (again, librosa has it so you can try it out). These produce noisy, low-quality audio. So the research focuses on improving this process using powerful models.

On the other hand, Audio -> Mel is simple, well-defined and fast. There is no need to define "custom encoders".

Now, a whole different question is whether mel spectrograms are a "good" encoding. Using methods like variational autoencoders, you could perhaps find better (e.g. more compact, less lossy) audio encodings. These would include custom encoders and decoders and you would not get away with standard librosa functions...

The part of the instrument audio that gives its distinctive sound, independently from the pitch played, is called the timbre. The modern approach to get a vector representation, would be to train a neural network. This kind of learned vector representation is often called to create an audio embedding.

An example implementation of this is described in Learning Disentangled Representations Of Timbre And Pitch For Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders (2019).