The Scrapy Beginners Course will teach you everything you need to learn to start scraping websites at scale using Python Scrapy.
The course covers:
– Creating your first Scrapy spider
– Crawling through websites & scraping data from each page
– Cleaning data with Items & Item Pipelines
– Saving data to CSV files, MySQL & Postgres databases
– Using fake user-agents & headers to avoid getting blocked
– Using proxies to scale up your web scraping without getting banned
– Deploying your scraper to the cloud & scheduling it to run periodically
?? Course created by Joe Kearney.
?? Resources ??
Course Resources
– Scrapy Docs: https://docs.scrapy.org/en/latest/
– Course Guide: https://thepythonscrapyplaybook.com/freecodecamp-beginner-course/
– Course Github: https://github.com/orgs/python-scrapy-playbook/repositories
– The Python Scrapy Playbook: https://thepythonscrapyplaybook.com/
Cloud Environments
– Scrapyd: https://github.com/scrapy/scrapyd
– ScrapydWeb: https://github.com/my8100/scrapydweb
– ScrapeOps Monitor & Scheduler: https://scrapeops.io/monitoring-scheduling/
– Scrapy Cloud: https://www.zyte.com/scrapy-cloud/
Proxies
– Proxy Plan Comparison Tool: https://scrapeops.io/proxy-providers/comparison/free-proxy-providers
– ScrapeOps Proxy Aggregator: https://scrapeops.io/proxy-api-aggregator/
– Smartproxy: https://smartproxy.com/deals/proxyservers/ips
?? Contents ??
?? (0:00:00) Part 1 – Scrapy & Course Introduction
?? (0:08:22) Part 2 – Setup Virtual Env & Scrapy
?? (0:16:28) Part 3 – Creating a Scrapy Project
?? (0:28:17) Part 4 – Build your First Scrapy Spider
?? (0:55:09) Part 5 – Build Discovery & Extraction Spider
?? (1:20:11) Part 6 – Cleaning Data with Item Pipelines
?? (1:44:19) Part 7 – Saving Data to Files & Databases
?? (2:04:33) Part 8 – Fake User-Agents & Browser Headers
?? (2:40:12) Part 9 – Rotating Proxies & Proxy APIs
?? (3:18:12) Part 10 – Run Spiders in Cloud with Scrapyd
?? (4:03:46) Part 11 – Run Spiders in Cloud with ScrapeOps
?? (4:20:04) Part 12 – Run Spiders in Cloud with Scrapy Cloud
?? (4:30:36) Part 13 – Conclusion & Next Steps
? Thanks to our Champion and Sponsor supporters:
? davthecoder
? jedi-or-sith
? ????
? Agustín Kussrow
? Nattira Maneerat
? Heather Wcislo
? Serhiy Kalinets
? Justin Hual
? Otis Morgan
—
Learn to code for free and get a developer job: https://www.freecodecamp.org
Read hundreds of articles on programming: https://freecodecamp.org/news