So far over 20 more web crawlers are now added to the known list. The list now reaches about 175 or so known search engines. Worldwide there seems to be several hundred engines.

Conveniently the bots list is a discrete PHP file so it can be easily edited live. Once it is saved it immediately is available. The goal is to eventually expand the known bots as much as possible. This tool is published on GitHub with GNU license to it will be 100% compatible with WordPress which is also GNU. Technically the work is a fork which is all too common with open source and edits can be merged or separated into a new code base.

Undesirable bots are currently in the robots.txt but additional steps may be needed to block malicious crawlers.

The Petalbot is operated by Huawei for their mobile services and Petal is their search tool. Mobile is really big business in China so websites need to be sure they can format for phones and tablets. The Graphene theme used here is flexible in the extreme. WordPress is relatively machine translate friendly.

In China, almost everyone now has a mobile phone. Apple is the most popular by far. Tablet machines are also incredibly popular as they can be used at home and take up little space. Laptops are growing in popularity as well. Desktop machines however make up only a smaller segment of the market but growth has been strong.

Some that are SEO companies are banished in the robots.txt file as undesirable. Some bots are operated by advertising companies which are welcome.