speechcommand | Code for speech command | Speech library

Since you are talking about the speech recognition and pytorch, I would recommend you to use a well-developed set of tools instead of doing speech-related training tasks from scratch.

A good repo on github is Espnet. It contains some quite recent work on text-to-speech and speech-to-text models as well as ready-to-use scripts to train on popular open-source dataset in different languages. It also includes trained models for you to use directly.

Back to your question, if you want to use pytorch to train your own speech recognition model on your own dataset, I would recommend you to go to this Espnet Librispeech ASR recipe. Although it uses .flac files, some little modifications on data preparation script and change some parameters in the major entry script asr.sh may feed your demand.

Note that, in addition to knowledge on python and torch, espnet needs you to be familiar with shell scripts as well. Their asr.sh script is quite long. This may not be an easy task for people who are more comfort with minimal pytorch codes for one specific model. Espnet is designed to accomodate many models and many datasets. It contains many preprocessing stages, e.g. speech feature extracting, length filtering, token preparation, language model training and so on, which are necessary for good speech recognition models.

If you insist on the repo that you found. You need to write a custom Dataset and Dataloader classes. You can refer to pytorch dataloading tutorial, but this link uses images as an example, if you want an audio example, maybe from some github repos like deepspeech pytorch dataloader