Netcraft can offer us information about the servers without even interacting with them, and this is something valuable from a passive information gathering point of view.
We can use the service by visiting
https://sitereport.netcraft.com and entering the target domain.
Some interesting details we can observe from the report are:
||General information about the domain, including the date it was first seen by Netcraft crawlers.|
||Information about the netblock owner, hosting company, nameservers, etc.|
||Latest IPs used, webserver, and target OS.|
We need to pay special attention to the latest IPs used. Sometimes we can spot the actual IP address from the webserver before it was placed behind a load balancer, web application firewall, or IDS, allowing us to connect directly to it if the configuration allows it.
We can check one of the first versions of
facebook.com captured on December 1, 2005, which is interesting, perhaps gives us a sense of nostalgia but is also extremely useful for us as security researchers.
We can also use the tool waybackurls to inspect URLs saved by Wayback Machine and look for specific keywords. Provided we have
Go set up correctly on our host, we can install the tool as follows:
neutron@kali[/kali]$ go get github.com/tomnomnom/waybackurls
To get a list of crawled URLs from a domain with the date it was obtained, we can add the
-dates switch to our command as follows:
neutron@kali[/kali]$ waybackurls -dates https://facebook.com > waybackurls.txt neutron@kali[/kali]$ cat waybackurls.txt 2018-05-20T09:46:07Z http://www.facebook.com./ 2018-05-20T10:07:12Z https://www.facebook.com/ 2018-05-20T10:18:51Z http://www.facebook.com/#!/pages/Welcome-Baby/143392015698061?ref=tsrobots.txt 2018-05-20T10:19:19Z http://www.facebook.com/ 2018-05-20T16:00:13Z http://facebook.com 2018-05-21T22:12:55Z https://www.facebook.com 2018-05-22T15:14:09Z http://www.facebook.com 2018-05-22T17:34:48Z http://www.facebook.com/#!/Syerah?v=info&ref=profile/robots.txt 2018-05-23T11:03:47Z http://www.facebook.com/#!/Bin595 <SNIP>