How To Scrape Multiple Pages on a Website (BeautifulSoup Pagination)

Video Description

🧠 Don’t miss out! Get FREE access to my Skool community — packed with resources, tools, and support to help you with Data, Machine Learning, and AI Automations! 📈 https://www.skool.com/data-and-ai-automations-4579 Learn how to handle pagination with BeautifulSoup in this Python web scraping tutorial! 🚀 In this video, we’ll show you how to extract data from websites with multiple pages by automating the pagination process. Whether you're scraping product listings, articles, or any paginated content, this guide will help you scrape data efficiently and responsibly. Code: https://ryanandmattdatascience.com/beautifulsoup-pagination/ 🚀 Hire me for Data Work: https://ryanandmattdatascience.com/data-freelancing/ 👨‍💻 Mentorships: https://ryanandmattdatascience.com/mentorship/ 📧 Email: [email protected] 🌐 Website & Blog: https://ryanandmattdatascience.com/ 🖥️ Discord: https://discord.com/invite/F7dxbvHUhg 📚 *Practice SQL & Python Interview Questions: https://stratascratch.com/?via=ryan 📖 *SQL and Python Courses: https://datacamp.pxf.io/XYD7Qg 🍿 WATCH NEXT Python Web Scraping Playlist: https://www.youtube.com/playlist?list=PLcQVY5V2UY4KIcizJ7px2XJ_LVvnTDHYC Learn how to scrape multiple pages on websites using Python in this comprehensive web scraping pagination tutorial. In this video, I walk you through scraping paginated websites, setting up timers to avoid detection, and building both basic and random timers for ethical web scraping. We cover two real examples: Books to Scrape and a hockey stats website with multiple pages of data. I explain how to identify URL patterns across pages, build for loops and while loops to navigate pagination, and use Python libraries like BeautifulSoup, Requests, and Pandas. You'll learn how to extract data from tables, format URLs dynamically, and create functions to make your web scraping code reusable. We also cover important concepts like handling 404 errors, setting delays between requests, and building random timers using Python's random module to make your scraper less detectable. By the end of this tutorial, you'll know exactly how to scrape websites with pagination, whether they have 10 pages or 1,000 pages. All code examples are available on the website linked in the description, and make sure to check out the full Python web scraping series for more tutorials on extracting data from the web. TIMESTAMPS 00:00 Introduction to Web Scraping Pagination 00:46 Understanding Website Pagination Patterns 02:30 Introduction to Timers for Web Scraping 03:50 Setting Up a Basic Timer 05:14 Setting Up a Random Timer 06:56 Using Strings to Format URLs 08:16 Example 1: Books to Scrape Website 13:17 Example 2: Hockey Stats Website Setup 15:17 Building the Scraping Function 18:25 Adding Random Timer to Function 19:41 Debugging and Testing the Code 21:00 Filtering and Exporting Data to CSV 22:07 Recap: Understanding URL Patterns 23:18 Final Tips and Best Practices OTHER SOCIALS: Ryan’s LinkedIn: https://www.linkedin.com/in/ryan-p-nolan/ Matt’s LinkedIn: https://www.linkedin.com/in/matt-payne-ceo/ Twitter/X: https://x.com/RyanMattDS Who is Ryan Ryan is a Data Scientist at a fintech company, where he focuses on fraud prevention in underwriting and risk. Before that, he worked as a Data Analyst at a tax software company. He holds a degree in Electrical Engineering from UCF. Who is Matt Matt is the founder of Width.ai, an AI and Machine Learning agency. Before starting his own company, he was a Machine Learning Engineer at Capital One. *This is an affiliate program. We receive a small portion of the final sale at no extra cost to you.