blob: 60180e7b01a555dbfb794d8a66a1f16eb5cc96eb [file] [log] [blame]
RobotFileParser now correctly returns default values for crawl_delay and
request_rate. Initial patch by Peter Wirtz.