The aim of URL clustering is to divide URLs into groups based on string patterns. To reduce throwaway crawls, we need a system that can make a decision using just the URL of the webpage. And wasted crawling is resource inefficient. The issue with this approach though is that it requires that we crawl URLs first in order to classify them. Supervised classification approach to identifying product URLs, both visually and using HTML (features) Our initial approach to tackling this problem was to frame it as a supervised multi-class classification problem, with the HTML and screenshot of each page being sent through a classifier. Challenge #1вЂЉ-вЂЉIdentifying productВ URLs Sample productВ page Supervised Classification? Other pages arenвЂ™t of use to use, except to discover product pages. Since our goal is to capture product details, weвЂ™ll look to isolating just the product pages. Product pages (which describe a specific product and allow you to purchase it).Review pages and forums (customer generated content).Seller information pages (about the business selling the product, in the case of marketplaces).Deal specific pages (discounts and offers).Websites contain all sorts of pages, so at the outset, we need a system to separate the useful pages from the rest.

Challenge #1вЂЉ-вЂЉIdentifying Product URLs Types of Pages Our goal in this project is to extract the entire catalog of an ecommerce site given just its homepage URL (see image above).

The goal of our automated crawling project In this article, IвЂ™d like to describe the system that weвЂ™ve built and the algorithms behind it this work is the subject of a recent patent filing by our team. In the age of machine learning, is there a smarter, more hands-off way of doing crawling? This is a goal that weвЂ™ve been chipping away at for years now, and over time weвЂ™ve made decent progress in doing automated generalizable crawling for a specific domainвЂЉ-вЂЉecommerce. WhatвЂ™s more, website structures change with time, so these systems have to be robust to change. The issue is that while itвЂ™s easy to build a one-off crawler, writing systems that generalize across sites is not easy, since websites usually have distinct unique underlying patterns. Writing crawlers to extract data from websites is a seemingly intractable problem.

Generalized webscraper