I recently noticed that “competitor research” and “SEO” bots were accounting for almost 60% of the traffic on my webserver. Here were the worst offenders:
trendiction.com (trendictionbot)
User Agent
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20170101 Firefox/67.0"
Network(s)
- 144.76.0.0/16 (Hetzner Online GmbH)
IP(s)
- 144.76.14.34
- 144.76.14.35
- 144.76.14.36
- 144.76.14.37
- 144.76.14.38
- 144.76.14.39
- 144.76.14.42
- 144.76.14.45
- 144.76.14.47
- 144.76.14.51
- 144.76.14.55
moz.com (DotBot)
User Agent
"Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)"
Network(s)
- 216.244.64.0/19 (Wowrack.com)
IP(s)
- 216.244.66.233
ahrefs.com (AhrefsBot)
User Agent
"Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
Network(s)
- 54.38.0.0/16 (OVH Cloud)
- 51.222.0.0/16 (OVH Cloud)
- 54.36.0.0/15 (OVH Cloud)
IP(s)
- over 300 IPs, stopped counting & blocked OVH Cloud
semrush.com (SemrushBot)
User Agent
"Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
Network(s)
- 185.191.171.0/24 (Semrush_Net)
- 85.208.96.0/24 (Semrush_Net)
IP(s)
- over 40 IPs, stopped counting & blocked Semrush_Net