Robots_txt 1.1

Robots_txt takes the URL of a page and retrieves the robots.txt file of the same site. The robots.txt is parsed and the rules defined in it are looked up, in order to determine if crawling a page is allowed.

Robots_txt also stores the time when a page is crawled to check whether next time another page of the same site is being crawled it is honoring the intended crawl delay and request rate limits.

PHP 5.0 or higher

License type: GPL - GNU General Public License
Date added: 7 years, 6 months 17 days ago | Last updated: 7 years, 6 months 17 days ago

More popular Web Search

This is actually one of the most advanced image hosting script. The imagetize key features: SEO, Ads Ready, Admin

Listing Files

Hot Scripts