| commit | 58a1a76baefc92d9e2392a5dbf65e39e44fb8f55 | [log] [tgz] |
|---|---|---|
| author | Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com> | Sun Jun 16 00:07:54 2019 -0700 |
| committer | GitHub <noreply@github.com> | Sun Jun 16 00:07:54 2019 -0700 |
| tree | 04b3001a1bc09863fcd62a0442d0da2445ad0fba | |
| parent | a8e7ebe2880f4c1d3b91d40b9730bb4032d514d0 [diff] |
bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or request rate (GH-11791) Co-Authored-By: Tal Einat <taleinat+github@gmail.com> (cherry picked from commit 8047e0e1c620f69cc21f9ca48b24bf2cdd5c3668) Co-authored-by: Rémi Lapeyre <remi.lapeyre@henki.fr>