IMS-Toucan | Speech Toolkit of the Speech and Language Technologies Group | Machine Learning library

To install this toolkit, clone it onto the machine you want to use it on (should have at least one GPU if you intend to train models on that machine. For inference, you can get by without GPU). Navigate to the directory you have cloned. We are going to create and activate a conda virtual environment to install the basic requirements into. After creating the environment, the command you need to use to activate the virtual environment is displayed. The commands below show everything you need to do. We use an ensemble of Speechbrain's ECAPA-TDNN and Speechbrain's x-Vector as the speaker conditioning. In the current version of the toolkit no further action should be required. When you are using multispeaker for the first time, it requires an internet connection to (automatically) download the pretrained models though. And finally you need to have espeak-ng installed on your system, because it is used as backend for the phonemizer. If you replace the phonemizer, you don't need it. On most Linux environments it will be installed already, and if it is not, and you have the sufficient rights, you can install it by simply running. You don't need to use pretrained models, but it can speed things up tremendously. Go into the release section and download the aligner model, the HiFiGAN model and the multi-lingual-multi-speaker FastSpeech2 model. Place them in Models/ALigner/aligner.pt, Models/HiFiGAN_combined/best.pt and Models/FastSpeech2_Meta/best.pt.
In the directory called Utility there is a file called file_lists.py. In this file you should write a function that returns a list of all the absolute paths to each of the audio files in your dataset as strings. Then go to the directory TrainingInterfaces/TrainingPipelines. In there, make a copy of any existing pipeline that has HiFiGAN in its name. We will use this as reference and only make the necessary changes to use the new dataset. Import the function you have just written as get_file_list. Now look out for a variable called model_save_dir. This is the default directory that checkpoints will be saved into, unless you specify another one when calling the training script. Change it to whatever you like. Now you need to add your newly created pipeline to the pipeline dictionary in the file run_training_pipeline.py in the top level of the toolkit. In this file, import the run function from the pipeline you just created and give it a speaking name. Now in the pipeline_dict, add your imported function as value and use as key a shorthand that makes sense. And just like that you're done.
In the directory called Utility there is a file called path_to_transcript_dicts.py. In this file you should write a function that returns a dictionary that has all the absolute paths to each of the audio files in your dataset as strings as the keys and the textual transcriptions of the corresponding audios as the values. Then go to the directory TrainingInterfaces/TrainingPipelines. In there, make a copy of any existing pipeline that has FastSpeech 2 in its name. We will use this copy as reference and only make the necessary changes to use the new dataset. Import the function you have just written as build_path_to_transcript_dict. Since the data will be processed a considerable amount, a cache will be built and saved as file for quick and easy restarts. So find the variable cache_dir and adapt it to your needs. The same goes for the variable save_dir, which is where the checkpoints will be saved to. This is a default value, you can overwrite it when calling the pipeline later using a command line argument, in case you want to fine-tune from a checkpoint and thus save into a different directory. In your new pipeline file, look out for the line in which the acoustic_model is loaded. Change the path to the checkpoint of an Aligner model. It can either be the one that is supplied with the toolkit on the release page, or one that you trained yourself. In the example pipelines, the one that we provide is finetuned to the dataset it is applied to before it is used to extract durations. Since we are using text here, we have to make sure that the text processing is adequate for the language. So check in Preprocessing/TextFrontend whether the TextFrontend already has a language ID (e.g. 'en' and 'de') for the language of your dataset. If not, you'll have to implement handling for that, but it should be pretty simple by just doing it analogous to what is there already. Now back in the pipeline, change the lang argument in the creation of the dataset and in the call to the train loop function to the language ID that matches your data. Now navigate to the implementation of the train_loop that is called in the pipeline. In this file, find the function called plot_progress_spec. This function will produce spectrogram plots during training, which is the most important way to monitor the progress of the training. In there, you may need to add an example sentence for the language of the data you are using. It should all be pretty clear from looking at it. Once this is done, we are almost done, now we just need to make it available to the run_training_pipeline.py file in the top level. In said file, import the run function from the pipeline you just created and give it a speaking name. Now in the pipeline_dict, add your imported function as value and use as key a shorthand that makes sense. And that's it.