[Site Home] [Forum Home] [Articles] [File DB] [News Archives]
Networks/Telecom/Security -> Crazy web robots
(View original topic)
| Red Squirrel - May-13-2005 server time |
It's web robots that crawl the site, but thing is, slurp will crawl multiple pages per second, and stay for hours, so it eats up lot of bandwidth and resources on the server. Slurp is used by yahoo, so we're not loosing much by blocking it.  |
| Pyr-O-Rgasm - May-13-2005 server time |
What? I'm lost.  |
| Red Squirrel - May-13-2005 server time |
| Slurp has been insane lately, I actually had to block it off the site. I was hoping for it to be temporary but does not look like it. I should actually make a robots.txt file and add it in, and hope it follows it, but meanwhile I can just monitor it's crazyness. |
(Showing 50 last posts, newest on top)