Modern Web Scraping with Python using Scrapy Splash Selenium

Become an expert in web scraping and web crawling using Python 3, Scrapy, Splash and Selenium 2nd EDITION (2021)

4.24 (3727 reviews)

Udemy

platform

English

language

Programming Languages

What you will learn

Understand the fundamentals of Web Scraping

Scrape websites using Scrapy

Understand Xpath & CSS Selectors

Build a complete Spider from A to Z

Store the extracted Data in MongoDb & SQLite3

Scrape JavaScript websites using Splash & Selenium

Build a CrawlSpider

Understand the Crawling behavior

Build a custom Middleware

Web Scraping best practices

Avoid getting banned while scraping websites

Bypass cloudflare

Scrape APIs

Scrape infinite scroll websites

Working with Cookies

Deploy spiders locally and to the cloud

Run spiders periodically

Prevent storing duplicated data

Build datasets

Login to websites using Scrapy

Download images and files using Scrapy

Why take this course?

Web Scraping nowadays has become one of the hottest topics, there are plenty of paid tools out there in the market that don't show you anything how things are done as you will be always limited to their functionalities as a consumer.

In this course you won't be a consumer anymore, i'll teach you how you can build your own scraping tool ( spider ) using Scrapy.

You will learn:

The fundamentals of Web Scraping
How to build a complete spider
The fundamentals of XPath & CSS Selectors
How to locate content/nodes from the DOM using XPath & CSS
How to store the data in JSON, CSV... and even to an external database(MongoDb & SQLite3)
How to write your own custom Pipeline
Fundamentals of Splash
How to scrape Javascript websites using Scrapy Splash & Selenium
The Crawling behavior
How to build a CrawlSpider
How to avoid getting banned while scraping websites
How to build a custom Middleware
Web Scraping best practices
How to scrape APIs
How to use Request Cookies
How to scrape infinite scroll websites
Host spiders in Heroku for free
Run spiders periodically with a custom script
Prevent storing duplicated data
Deploy Splash to Heroku
Write data to Excel files
Login to websites using Scrapy
Download Files & Images using Scrapy
Use Proxies with Scrapy Spider
Use Crawlera with Scrapy & Splash
Use Proxies with CrawlSpider

What makes this course different from the others, and why you should enroll ?

First, this is the most updated course. You will be using Python 3.7, Scrapy 1.6 and Splash 3.0

You will have an in-depth step by step guide on how to become a professional web scraper.
You will learn how to use Splash & Selenium to scrape JavaScript websites and I can assure you, you won't find any tutorials out there that teaches how to really use Splash like I'll be doing in this course.
You will learn how to host spiders in Heroku as well as Splash(Exclusive).
You will learn how to create a custom script so spiders can run periodically without any intervention from you.

30 days money back guarantee by Udemy

So whether you are a data analyst who wants to add web scraping to his tool set or someone else who wants to learn how to extract unstructured data from unstructured HTML web pages and then store back that data in a structured way to apply some data analysis on it then you are welcome to join this course.

**STUDENTS THOUGHTS ABOUT THIS COURSE **

"I was particularly looking for web scraping using XPATHs and this course is addressing that. It also covers dynamic paging. A proper mix of theory and practical. A must-have for those who wants to do web scraping . GREAT learning experience !!! ". By Hiran Kumar

"90% of what I was searching for!!! Great job!! Clear explanations and great communication with Ahmed". By Raylyson Estanista

"Admed’s Web scraping course is awesome . His approach using Python with scrapy and splash works well with all websites especially those that make heavy use of JavaScript. Ahmed is a gifted educator: expert communicator, passionate, conscientious and accessible to his students. I highly recommend this course and any of Ahmed Rafik’s Udemy courses. ". By Richard Blackmon

"Great course, and a nice introduction to Scrapy (I'm someone with no Python experience whatsoever).". By I S

"Excellent course. Quick and thorough at the same time. Ahmed is incredibly responsive to the students and often replies to questions within minutes! Highest recommendation." By Robert Nolte

"That course is very good and explanation is crystal clear! The instructor is very supportive in case of questions. Highly recommended." By Shubina Ekaterina

"I like the course. Clear explanations and good comunication with Ahmed. All topics is interesting and full of information. I improved my skils in Scrapy. Author update course content by new videos. It's a big bonus) Explained more advance topics I never see in other courses. Thank you, Ahmed. Waiting for new videos)". By Ruslan Romanenko

Screenshots

Modern Web Scraping with Python using Scrapy Splash Selenium - Screenshot_01

Modern Web Scraping with Python using Scrapy Splash Selenium - Screenshot_02

Modern Web Scraping with Python using Scrapy Splash Selenium - Screenshot_03

Modern Web Scraping with Python using Scrapy Splash Selenium - Screenshot_04

Our review

---- ### **Global Course Rating:** 4.21 **Overall Summary:** The course "Web Scraping with Python & Scrapy" has received a high level of satisfaction from its students, reflected in its average rating of 4.21 out of 5. The feedback indicates that the course is well-structured and informative, with Ahmed, the instructor, being commended for his clear explanations and teaching style that caters to both beginners and advanced programmers. However, there are notable concerns regarding the course's currency, as some content has been identified as outdated and in need of an update to reflect the latest versions of Scrapy and other technologies discussed. **Pros:** - Ahmed's teaching style is clear, structured, and poised; he doesn't rush through explanations. - The course is suitable for both PC and Mac users, with considerations for different keyboard shortcuts. - It provides a solid step-by-step guide for basic web scraping. - The instructor sets up environments and creates helper files to aid learning. - Many students find the course to be one of the best Scrapy courses available online. - Ahmed is described as a "NINJA" in web scraping, with deep knowledge that remains relevant even if some content is outdated. **Cons:** - Some tutorials within the course cannot be completed due to the content being out of date. - The author has not provided updates or responded to messages regarding issues with deprecated content. - The pace can sometimes be too fast, especially for non-native English speakers who might benefit more from slower delivery. - The lack of conceptual understanding in some areas, where students are expected to follow instructions without the 'why' behind them. - Some students experienced difficulty due to the use of an old version of Scrapy and would appreciate updates to align with current standards. **Additional Notes:** - It is recommended that Ahmed includes a proper update or explanations for why certain methods are deprecated or should not be used. - The course's title being updated to 2023 but without corresponding content updates has led to some confusion and frustration among students. - Some students have expressed that while the course is good, it could delve deeper into each topic for a more comprehensive understanding. - A few students pointed out that the captions provided were helpful when the video's pace was too fast or the accent was hard to understand. **Conclusion:** Despite some shortcomings related to content currency and responsiveness from the instructor, the course "Web Scraping with Python & Scrapy" is overall well-regarded for its quality of instruction and the depth of knowledge imparted by the instructor, Ahmed. It is recommended as a valuable resource for learning web scraping techniques, with the caveat that students should be prepared to supplement with their own research or updates when necessary.

Charts

Price

Rating

Enrollment distribution

Modern Web Scraping with Python using Scrapy Splash Selenium - Distribution chart