Professional Web Scraping with Java

Learn how to scrape data from any static or dynamic / AJAX web page using Java in a short and concise way.

3.95 (193 reviews)
Udemy
platform
English
language
Web Development
category
instructor
1,488
students
1.5 hours
content
Aug 2018
last update
$49.99
regular price

What you will learn

Have a solid understanding of web scraping with Java

Beeing able to scrape practically any web page (static AND dynamic / AJAX) though you learn the concepts behind web scraping

Download, parse and extract data from websites with Jsoup

Call web APIs in Java with Unirest

Export your data as CSV or JSON

Build web scrapers that stay undetected and do not get blocked or banned

Description

In this short and concise course you will learn everything to get started with web scraping using Java. 

You will learn the concepts behind web scraping that you can apply to practically any web page (static AND dynamic / AJAX).


Course structure

We start with an overview of what web scraping is and what you can do with it. 

Then we explain the difference in scraping static pages vs dynamic / AJAX pages. You learn how to classify a website in one of the two categories and then apply the right concept in order to scrape the data you want.

Now you will learn how to export the scraped data either as CSV or JSON. These are some popular formats that can be used for further processing.

Unfortunately many websites try to block scrapers or sometimes you just do not want to be detected. In the section going undercover you will learn how to stay undetected and avoid getting blocked.

At the end of the course you can download the full source code of all the lectures and we discuss an outlook to some advanced topics (private proxies, cloud deployment, multi threading ...). Those advanced topics are covered in a follow up course I am going to teach.


Why you should take this course

Stop imagining you can scrape data from websites and use the skills for your next web project, you can do it now.

  • Stay ahead of your competition
  • Be more efficient and automate tedious, manual tasks
  • Increase your value by adding web scraping to your skill set


Enroll now!

Content

Course Introduction

Introduction: Course overview and requirements

Scraping static web pages

What is a static web page?
Concept: how to scrape static web pages
Jsoup: the jQuery for Java
Example: scraping the top 10 Google search results

Scraping dynamic / AJAX web pages

What is a dynamic / AJAX web page?
Concept: how to scrape dynamic / AJAX web pages
Making HTTP Requests with Unirest
Example: scraping results from peoplefinders.com
Bonus: Proxycrawl to scrape dynamic pages
Dealing with WebSockets

Exporting your data

Export your data as CSV
Export your data as JSON

Going undercover

How to stay undetected
Bonus: Overview of Proxycrawl
Bonus: Example using Proxycrawl for peoplefinders.com

Conclusion

Conclusion and what's next
Bonus: Mind Map of the course content

Screenshots

Professional Web Scraping with Java - Screenshot_01Professional Web Scraping with Java - Screenshot_02Professional Web Scraping with Java - Screenshot_03Professional Web Scraping with Java - Screenshot_04

Reviews

Mohammad
September 22, 2023
Spend some time with the CSS selectors, It's really difficult to understand how the tags are being picked from the browser.
Javier
August 19, 2023
The course is good to get started with the concepts behind web scraping, and you can actually make your own basic web scraper for static web pages after the first lessons. However, it is also true that most of the practical examples cannot be replicated anymore because the web sites have changed. Some topics you might want to check on your own (since they are missing in the course content, at the moment) are how to add cookies and headers to Unirest requests, as well as scraping dynamic web pages with Selenium, which is, in my experience, a better alternative than HtmlUnit.
Manuel
February 7, 2021
El curso está obsoleto. Los ejemplos que usa el curso, en este momento, no corresponden con los de la actualidad.
Jiri
August 29, 2017
The course contains intelligible explanation of the basic scraping concepts and even simple techniques to overcome obstacles which you might encounter when scraping the web (USER_AGENT, HTTP Headers, ...). Personally I like infrastructure choice as I use the same (Intellij IDEA IDE with Maven project).
Eoin
June 14, 2017
The instructor was excellent and very clear in explaining things, he moves quite quickly through the course which I found a challenge but I am very new to java so that's not his fault. I took half a star off because I feel the source code could be better commented. Overall I am very happy with the course
Sudhakar
May 6, 2017
In the tutorial it was mentioned that browser and program are giving different output, he explained what caused and what needs to be done. But he has not shown that in live program demo. The demo was simple and he could have very well shown how to resolve the differential behavior between browser and code
Armin
March 30, 2017
It's very straightforward and everything is broken down into clear steps. I really recommend if you want to learn the basics.
Shayan
March 26, 2017
I've been trying to learn Jsoup by reading the API and searching for info online and sometimes it can be a confusing however, you're making things very clear and answering a lot of questions that I've had! Thanks!
Kirill
March 3, 2017
Good course for starters. Yet it would be nice to see more practical examples. I couldn't use the example with peoplefinders.com. For some reasons, I have been denied access to the website.
Romek
November 16, 2016
tutorial has clear logic. just like following line. it simply works. the funny thing was disabling js in the browser with udemy in other window... well done :) Patrick. Looking forward for next courses.
Alexander.matsaylo
August 21, 2016
Udemy asked too yearly and that's how I feel at that time. Begining of the course was too generic I could do this with example from the site as well. More interesting things I would expect.
Michael
July 14, 2016
Patrick definitely knows and understands the subject matter and presents it very well. I'm glad he created this video course - this is much better than reading a long, boring technical tutorial.
Arun
May 24, 2016
Really nice tips. I am sure this will help me a lot. You can easily tell they know what they're talking about! Thanks
Peter
May 19, 2016
The course is very easy to follow and I was surprised how quick I was able to scrape the web. Lot of possibilities running through my head now how I can use this knowledge. Thanks a lot for this great course!
Josef
May 18, 2016
Since I've completed this course I created without any problems some tools with the web scraping technology. A really good spoken good structured course which I could defenetly recommend. Thanks for this great course - Go on with it!

Charts

Price

Professional Web Scraping with Java - Price chart

Rating

Professional Web Scraping with Java - Ratings chart

Enrollment distribution

Professional Web Scraping with Java - Distribution chart

Related Topics

756050
udemy ID
2/8/2016
course created date
6/27/2020
course indexed date
Bot
course submited by