Python script: Simple domain crawler

Python Script: Collecting websites with Domain Crawler

This is my second Python script I’d like to publish. Also this script is a really simple one but after some modification / adding features can be used for a lot of different things. What it basically does? Well, what you can do with it is to collect domains (with certain TLD) of websites, based on how these websites are connected to each other:

Therefore the Domain Crawler Python script connects to a website (from a list of websites to crawl) gets all the outgoing links of this website, reduces them on the domain (plus a certain top-level domain you have to specify), puts them in a list and repeats the entire procedure again, and again, and again…

By the way: You can use this Python crawler to find out about Google Analytics usage, getting content or extracting adresses for example.

Download: Get the script (.py in .rar)

If this sounds like something useful to you, check out download link below.

Download: Domain Crawler to collect links for Python

If you feel better without downloading anything, see the script below.

Fast Lane: Domain Crawler for Copy & Paste