(2/2) Build an ElevenLabs Clone: PyTorch, Next.js 15, AWS, Inngest, FastAPI, React, Tailwind (2025)

Video Description

Source code: https://github.com/Andreaswt/elevenlabs-clone Part 1: https://www.youtube.com/watch?v=48A-E-xJPMo Discord & More: https://andreastrolle.com Inngest: https://innge.st/yt-andreas-1 Hi 🤙 In this video, you'll build a full-stack ElevenLabs clone with text-to-speech, voice conversion, and audio generation. Some tutorials would just call an API like ElevenLabs', but not us! Instead of external API services, you'll self-host three AI models (StyleTTS2, Seed-VC, and Make-An-Audio) from GitHub, fine-tune them to specific voices, then containerize them with Docker and expose inference endpoints via FastAPI. The AI backend will be built using Python and PyTorch. You'll create a Next.js application where users can use the AI models to generate audio, and also switch between voices and view previously generated audio files, stored in an S3 bucket. The project includes user authentication, a credit system, and an Inngest queue to prevent overloading of the server hosting the AI models. The web application is built on the T3 Stack with Next.js, React, Tailwind, and Auth.js. Follow along for the entire process from development to deployment. Features 🔊 Text-to-speech synthesis with StyleTTS2 🎭 Voice conversion with Seed-VC 🎵 Audio generation from text with Make-An-Audio 🤖 Custom voice fine-tuning capabilities 🐳 Docker containerization of AI models 🚀 FastAPI backend endpoints 🔄 Inngest queue to prevent server overload 📊 User credit management system 💾 AWS S3 for audio file storage 👥 Multiple pre-trained voice models 📱 Responsive Next.js web interface 🔐 User authentication with Auth.js 🎛️ Voice picker 📝 Generated audio history 🎨 Modern UI with Tailwind CSS 💲 Costs + How to follow along for free The total fine-tuning cost for both models is ~5-10 USD. When deploying the endpoint it’s ~1 USD per hour of uptime. S3 is really cheap. IAM roles, users etc are free. Following along for free: -When building the next.js application in part 2 of the video, I create a mock endpoint that means you don't have to host the AI models with EC2 unless you want to learn it. You can just use that mock endpoint throughout the video. -Don't fine-tune the models, but just use the model files (.pth) made by the researchers, as I also do before fine-tuning. -Don't create EC2 instances. They are the main cost driver. -S3 buckets are required for the voice-to-voice feature. If you want this feature, stay within the 5GB free tier. See more under storage and S3 here: https://aws.amazon.com/free -You can of course still follow the video, learn the concepts, and code along. You can also test the docker containers for training locally, without training the model, to learn as much as possible, without the actual fine-tuning. 📖 Chapters 00:00:00 Intro 00:04:08 Authentication 01:10:43 Dashboard beginning 01:18:57 Sidebar 02:10:45 Right sidebar 02:18:51 Voice selector 02:41:28 History 03:35:20 Playbar 04:34:04 Mobile settings 04:45:08 Text-to-speech page 05:10:58 Voice changer page 05:31:27 Text-to-SFX generate page 05:53:35 Text-to-SFX history page 06:06:08 Mock AI api 06:14:11 Queue and Throttling theory 06:22:24 Inngest implementation 06:39:38 Generate server actions 06:48:09 Generation status polling 07:06:36 Toast 07:24:21 Hook rest of pages up to API 07:29:48 Testing with EC2 AI deployment 07:46:16 Closing notes 07:47:02 Exercises for you