{"model":{"name":"tacotron2_pyt_ckpt_amp","orgName":"nvidia","description":"## Model Overview\n\nThe Tacotron 2 and WaveGlow model form a text-to-speech system that enables user to synthesise a natural sounding speech from raw transcripts.\n\n## Model Architecture\n\n\nThe Tacotron 2 model is a recurrent sequence-to-sequence model with attention that\npredicts mel-spectrograms from text. The encoder (blue blocks in the figure\nbelow) transforms the whole text into a fixed-size hidden feature\nrepresentation. This feature representation is then consumed by the\nautoregressive decoder (orange blocks) that produces one spectrogram frame at\na time. In our implementation, the autoregressive WaveNet (green block) is\nreplaced by the flow-based generative WaveGlow.\n\n![](https://github.com/NVIDIA/DeepLearningExamples/raw/master/PyTorch/SpeechSynthesis/Tacotron2/img/tacotron2_arch.png \"Tacotron 2 architecture\")\n\n\nFigure 1. Architecture of the Tacotron 2 model. Taken from the\n[Tacotron 2](https://arxiv.org/abs/1712.05884) paper.\n\nThe WaveGlow model is a flow-based generative model that generates audio\nsamples from Gaussian distribution using mel-spectrogram conditioning (Figure\n2). During training, the model learns to transform the dataset distribution\ninto spherical Gaussian distribution through a series of flows. One step of a\nflow consists of an invertible convolution, followed by a modified WaveNet\narchitecture that serves as an affine coupling layer. During inference, the\nnetwork is inverted and audio samples are generated from the Gaussian\ndistribution. Our implementation uses 512 residual channels in the coupling layer.\n\n![](https://github.com/NVIDIA/DeepLearningExamples/raw/master/PyTorch/SpeechSynthesis/Tacotron2/img/waveglow_arch.png \"WaveGlow architecture\")\n\n\nFigure 2. Architecture of the WaveGlow model. Taken from the\n[WaveGlow](https://arxiv.org/abs/1811.00002) paper.\n\n\n    \n## Training\n\nThis model was trained using script available on [NGC](https://ngc.nvidia.com/catalog/resources/nvidia:tacotron_2_and_waveglow_for_pytorch) and in [GitHub repo](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2)\n\n## Dataset\n\nThe following datasets were used to train this model:\n- [LJSpeech-1.1](https://keithito.com/LJ-Speech-Dataset/) - Dataset consisting of 13,100 short audio clips of a single speaker reading passages from 7 non-fiction books. A transcription is provided for each clip. Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.\n\n\n## Performance\n\nPerformance numbers for this model are available in [NGC](https://ngc.nvidia.com/catalog/resources/nvidia:tacotron_2_and_waveglow_for_pytorch/performance)\n\n## References\n- [original paper](https://arxiv.org/abs/1712.05884)\n- [NVIDIA model implementation in NGC](https://ngc.nvidia.com/catalog/resources/nvidia:tacotron_2_and_waveglow_for_pytorch)\n- [NVIDIA model implementation on GitHub](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2)\n\n## License\n\nThis model was trained using open-source software available in [Deep Learning Examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2) repository.\nFor terms of use, please refer to the license of the script and the datasets the model was derived from.","createdDate":"2020-08-21T02:13:24.974Z","updatedDate":"2023-04-04T19:54:32.237Z","isPublic":true,"framework":"PyTorch","modelFormat":"checkpoint","publicDatasetUsed":{},"labels":["TTS","application:model:speech_synthesis","application:model:uscs_text_to_speech","platform:model:pltfm_deep_learning_examples","technology:model:soln_conversational_ai"],"policyLabels":[],"latestVersionIdStr":"19.12.0","latestVersionSizeInBytes":338398023,"latestVersionGpuModel":"V100","shortDescription":"Tacotron2 PyTorch checkpoint trained with AMP","displayName":"Tacotron2 PyTorch checkpoint (AMP)","precision":"AMP","application":"SpeechSynthesis","logo":"https://assets.nvidiagrid.net/ngc/logos/Deep-Learning-Examples.png","publisher":"NVIDIA Deep Learning Examples","isReadOnly":true,"canGuestDownload":true,"hasPlayground":false,"isPlaygroundEnabled":false,"hasSignedVersion":false},"requestStatus":{"statusCode":"SUCCESS","requestId":"d7cd1f64-459253"}}