| commit | 8047e0e1c620f69cc21f9ca48b24bf2cdd5c3668 | [log] [tgz] |
|---|---|---|
| author | Rémi Lapeyre <remi.lapeyre@henki.fr> | Sun Jun 16 08:48:57 2019 +0200 |
| committer | Tal Einat <taleinat@gmail.com> | Sun Jun 16 09:48:57 2019 +0300 |
| tree | 7b22a0fce0ff2d1170bfbbc41f929bcc8d68fc9f | |
| parent | 3a1d50e7e573efb577714146bed5c03b9c95f466 [diff] |
bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or request rate (GH-11791) Co-Authored-By: Tal Einat <taleinat+github@gmail.com>