robots.spider(tm) will read a robots.txt file and compare it with the pages on the site. Search engines spider the site. If there are no references to a directory or file, it will not be indexed.
robots.txt files should not refer to unreferenced content because it allows other viewers to identify areas of your site that you don't want indexed. The read depth limits the number of pages scanned.