Spider website 0.1
Spider website can retrieve a page of a site and follow all links recursively to retrieve all the site URLs.
The crawling can be restricted to URLs with a particular extension. It can also avoid accessing pages listed in the site robots.txt file, or pages set with the no index or no follow meta tags.
PHP 5.0 or higher