BLOCKING CORPORATE BOTS

Many corporations in the SEO business with high monthly fees use crawler bots to check out various web properties. There is no sale here as this site is content rich and does not need some dog and pony show to show me how to operate. Google is free for webmasters with no strings attached. Bing and Yandex are also free. Other search engines do not appear to have much for webmasters at this point in time.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Crawl-delay: 10

User-agent: SemrushBot
Disallow: /

User-agent: Ahrefs
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: Linkfluence
Disallow: /

Sitemap: https://www.hardcoregames.ca/sitemap.xml
Sitemap: https://www.hardcoregames.ca/news-sitemap.xml

The current robots.txt has blocked 4 known corporate hucksters. This may grow larger as more of them come out of the woodwork. It’s likely that some bots ignore the robits.txt file which will require more security.

Crawl-delay is not a standard for robots.txt but Bing and Yandex both recognize it and act accordingly. Google will eventually fall into line. Crawl-delay of 10 seconds allows search to proceed without impacting the performance for human traffic.

Google has largely crawled the site but they still check to see if posts are modified. Bing is still far from finished indexing the site which at the current rate may take some 3-4 additional months. Yandex has partially indexed the site but their webmaster portal needs an overhaul.

Smaller search engines have not been noticed suggesting they are simply operating meta search. There are at least 20 known smaller search engines.