| commit | 73fd46d24e45c34f0fb87261e5471584a7c273df | [log] [tgz] |
|---|---|---|
| author | Jeremy Hylton <jeremy@alum.mit.edu> | Fri Jul 18 20:59:44 2008 +0000 |
| committer | Jeremy Hylton <jeremy@alum.mit.edu> | Fri Jul 18 20:59:44 2008 +0000 |
| tree | cf36eca08149e5fe933a90b71e7b3b3a1521305a | |
| parent | 48577d1944c6b03be12bd7b144eb22db6bd6d296 [diff] |
Bug 3347: robotparser failed because it didn't convert bytes to string. The solution is to convert bytes to text via utf-8. I'm not entirely sure if this is safe, but it looks like robots.txt is expected to be ascii.