Aggressive MSN Search Bots

[Ickie]

For months now I have been fighting high server loads, and you get "the server is busy messages", well I finally figured it out, it was caused by (Aggressive MSN Search Bots)
I finally got the loads way down to below where I have ever seen them. I killed the beast!

Google on the other hand doesn't seem to bother our loads so I left them alone.
Below is a graph of our server loads for the past month and you can see the results. You will also see spikes and these are when I killed another IP owned by M$.

View attachment 89794

 

[Ickie]

For website owners:

for a 2 second delay in bot searches add to robots.txt: (Crawl-delay: 2) or Crawl-delay: 2 , I did both

ban IP range:
157.54.0.0/15
157.56.0.0/14
157.60.0.0/16


​

</STRONG>

 

[Panther_99FS]

Go get 'em Icky!

 

[papab]

Thanks for the Vigilance in keeping the site up and running!:salute:
Rick

 

[misson]

seems not only here ,: http://social.microsoft.com/Forums/...8e964/msn-search-bot-trying-to-attack-my-site

 

[Ickie]

seems to be a waste of time asking MS, heck it took only seconds to block them altogether once I figured it all out.

I only hope I helped someone else who is wonering why their server runs slow or their bandwidth is being ate up by these out of control bots.

 

[Skyhawk_310R]

Microsoft is supposed to work to prevent DDOS attacks from exploiting vulnerabilities in their software. They are not supposed to be the ones initiating the DDOS attacks!

Sounds like MS needs to get control of some of their software engineers else they might be exposing their company to serious legal jeapardy.

Ken

 

[Ickie]

another graph on the 3rd of july you can see the trend, i'll post one weekly
View attachment 89799

 

[Roger]

Good on yer Ickie!

 

[Ickie]

what really burns me, is that our forum software vbulletin has ip ban and I put the ip range in there and msn blew right through there. it didn't even slow them down.

next our forum software has a setting for search engine friendly and I had searches turned off, again they blew right in.

but what really pissed me off is .... our library, we have it set up so you have to be a free member to use it, ... it is tied into our member database, lol, again msn just let anybody download from us.

between this and the massive searches, brought us to our knees daily, it was a real job managing our server loads, not to mention our bandwidth which the charter membership paid for, not them.

I would feel better about this if only they shared their $$$$ they get from advertising on us with us.

View attachment 89801
they are now blocked at the servers front and back doors!!

 

[robert41]

Thanks Ickie.

 

[Ivan]

seems to be a waste of time asking MS, heck it took only seconds to block them altogether once I figured it all out.

I only hope I helped someone else who is wonering why their server runs slow or their bandwidth is being ate up by these out of control bots.

Do they respect ROBOTS.TXT files?

- Ivan.

 

[Ickie]

good question, I don't really know, haha


but they cannot get through my security scripts at the server front and back door, and my graphs prove it.

 

[Dangerousdave26]

Oh so we did put the Big Hammer Down.

Good glad to hear it.