Scrapy Course – Python Web Scraping for Beginners

The Scrapy Beginners Course will teach you everything you need to learn to start scraping websites at scale using Python Scrapy.

The course covers:
- Creating your first Scrapy spider
- Crawling through websites & scraping data from each page
- Cleaning data with Items & Item Pipelines
- Saving data to CSV files, MySQL & Postgres databases
- Using fake user-agents & headers to avoid getting blocked
- Using proxies to scale up your web scraping without getting banned
- Deploying your scraper to the cloud & scheduling it to run periodically

✏️ Course created by Joe Kearney.

⭐️ Resources ⭐️
Course Resources
- Scrapy Docs: https://docs.scrapy.org/en/latest/
- Course Guide: https://thepythonscrapyplaybook.com/f...
- Course Github: https://github.com/orgs/python-scrapy...
- The Python Scrapy Playbook: https://thepythonscrapyplaybook.com/

Cloud Environments
- Scrapyd: https://github.com/scrapy/scrapyd
- ScrapydWeb: https://github.com/my8100/scrapydweb
- ScrapeOps Monitor & Scheduler: https://scrapeops.io/monitoring-sched...
- Scrapy Cloud: https://www.zyte.com/scrapy-cloud/

Proxies
- Proxy Plan Comparison Tool: https://scrapeops.io/proxy-providers/...
- ScrapeOps Proxy Aggregator: https://scrapeops.io/proxy-api-aggreg...
- Smartproxy: https://smartproxy.com/deals/proxyser...

⭐️ Contents ⭐️
⌨️ (0:00:00) Part 1 - Scrapy & Course Introduction
⌨️ (0:08:22) Part 2 - Setup Virtual Env & Scrapy
⌨️ (0:16:28) Part 3 - Creating a Scrapy Project
⌨️ (0:28:17) Part 4 - Build your First Scrapy Spider
⌨️ (0:55:09) Part 5 - Build Discovery & Extraction Spider
⌨️ (1:20:11) Part 6 - Cleaning Data with Item Pipelines
⌨️ (1:44:19) Part 7 - Saving Data to Files & Databases
⌨️ (2:04:33) Part 8 - Fake User-Agents & Browser Headers
⌨️ (2:40:12) Part 9 - Rotating Proxies & Proxy APIs
⌨️ (3:18:12) Part 10 - Run Spiders in Cloud with Scrapyd
⌨️ (4:03:46) Part 11 - Run Spiders in Cloud with ScrapeOps
⌨️ (4:20:04) Part 12 - Run Spiders in Cloud with Scrapy Cloud
⌨️ (4:30:36) Part 13 - Conclusion & Next Steps

🎉 Thanks to our Champion and Sponsor supporters:
👾 davthecoder
👾 jedi-or-sith
👾 南宮千影
👾 Agustín Kussrow
👾 Nattira Maneerat
👾 Heather Wcislo
👾 Serhiy Kalinets
👾 Justin Hual
👾 Otis Morgan

--

Learn to code for free and get a developer job: https://www.freecodecamp.org

Read hundreds of articles on programming: https://freecodecamp.org/news