- Discussions
- 4 649
- Messages
- 12 075
- Solutions
- 81
- J'aime
- 7 620
- Points
- 198
This XenForo 2.0 addon adds additional definitions for bot detection in sessions.
Requirements :
This addon requires PHP 5.4 or higher and only works on XenForo 2.0.x
Usage :
When you look at Current Visitors, you'll see additional robots identified - also look at the "Robots" list on that page
We also add the current robot count to the Members online widget and the Online statistics widget (from the current visitor page). This can be disabled via the options.
Télécharger V2.5.0 : Version 2.6.0 change New bots :
Télécharger V2.6.0 : Version 2.7.0 change : New functionality: tool to test bot detection - paste in a user agent string and the tool will tell you if it detects a bot
Télécharger V2.7.0 : v3.0.0 update (unreleased) :
Major new feature: add generic bot detection
User Agent strings are scanned for the keywords "bot", "crawl", or "spider" - any User Agents not already detected as a bot which contain one of these strings are stored in the cache and made visible through the admin UI, with the option to have this information emailed on a weekly basis.
new bots: AccompanyBot; PostmanRuntime
v3.1.0 update : (took less than 5 minutes after installing it on several of my production forums to identify new bots!!)
new bots: amazonbot; petalbot; slackbot
Télécharger V3.1.0 : Version 3.2.0 big bot update :
Also found another set of false-positives:
Cubot mobile phones added to false positive list (Logicom BOT phones are already on the list)
Télécharger V3.2.0 :
Télécharger V3.3.0 :
Télécharger V3.4.0 :
Télécharger V3.5.0 :
Télécharger V3.6.0 :
Télécharger V3.7.0 : Version 3.8.0 weekly bot updates :
Télécharger V3.8.0 :
Télécharger V3.9.0 :
Télécharger V3.10.0 :
Télécharger V3.11.0 :
Télécharger V3.12.0 :
Télécharger V3.13.0 :
Télécharger V3.14.0 :
Télécharger V3.15.0 :
Télécharger V3.16.0 : Version 3.17.0 weekly bot updates new false positives :
cubot r11
spider v7 build/lmy47i
spider v7 (MyCell Spider v7 from Bangladesh)
Télécharger V3.17.0 :
Télécharger V3.18.0 :
Télécharger V3.19.0 :
Télécharger V3.20.0 : KnownBots v4.0.0 - complete re-write :
KnownBots v4 is a completely new build - bots are no longer hard coded, but updated via API calls and uses the XF code cache to store bot data
This release includes additional sanity checks to prevent bad data returned from the API from breaking the forums.
If any of the data returned by the API is not in the exact format we expect, the entire download is discarded and no changes applied to the forum. An error message will be logged prompting further investigation.
After upgrading to 4.0.1, you should manually force a fetch of new API data by executing the following command from your forum root:
Télécharger V4.0.1 :
Télécharger V5.0.0 :
v6 major rewrite
v6 is another major rewrite of the core functionality of the addon, aimed at improving the submission process for newly detected useragents and performance improvements.
Important for v4 users: with this release, I am deprecating the v1 API - addon versions v4.x and earlier will continue to function for a while but will then start returning 404 error codes once I turn off the v1 API. Anyone still running KnownBots v4.x should upgrade as soon as possible.
Important for v5 users: The v2 API used in addon v5.x for fetching new bots will remain operational, however, I am deprecating the email based submission system in favour of a new API based user agent submission system. After a transition period, the inbound email system will be disabled and any emails sent to the knownbots@hampel.io address will bounce back as undelieverable. Anyone still running KnownBots v5.x should either upgrade, or at least disable the "Email user agents" option in the v5.x addon options.
Important for anyone upgrading to v6: the new submission system in v6 uses an authentication process to ensure only valid submissions occur. After upgrading to v6, to continue submitting new user agents for analysis, you must first configure the authentication system - it is a very simple process - see instructions on the addon page. The options for v6 have changed - you should check them after upgrading.
The new submission system in v6 utilises the XenForo customer validation API to authenticate sites when submitting agents via our new API.
To configure the API, enter the License validation token for your site, found in the XenForo customer interrface. The validation token will be sent to the XenForo customer validation API by the KnownBots system and if valid, a KnownBots API token will be generated and returned back to the requesting forum for subsequent authentication purposes.
With a validated license, the authentication process is automatic. API tokens are regenerated every 28 days and are re-authenticated automatically. Customer details are automatically purged from the KnownBots database after 30 days of inactivity (see privacy details on main addon page). Regenerating your license validation token will automatically cause API revalidation to fail and customer details to be purged - unless you re-configure the addon options with the new license validation token.
Changelog for v6 :
This update contains an important bugfix to handle malformed UTF-8 in user agent strings.
This update will simply ignore any user agents with malformed UTF-8, avoiding errors when trying to send updates via the API. These user agents are invalid and so there is no point undertaking any further analysis - thus they are silently discarded.
This new version also contains an additional CLI tool for importing user agents from a text file, for testing purposes.
Télécharger V6.0.3 : Version 6.0.4 improve XF 2.3 compatibility :
- Improved XF 2.3 compatibility
Télécharger V6.0.4 : Version 6.0.5 bugfix : XF2.3 compatibility for sending new bots via an email attachment.
Télécharger V6.0.5 : Version 6.1.0 - admin permissions update :
- now requires Manage known bots admin permission to see admin tools.
Télécharger V6.1.0 :
Requirements :
This addon requires PHP 5.4 or higher and only works on XenForo 2.0.x
Usage :
When you look at Current Visitors, you'll see additional robots identified - also look at the "Robots" list on that page
http://www.example.com/community/online/?type=robotWe also add the current robot count to the Members online widget and the Online statistics widget (from the current visitor page). This can be disabled via the options.
Télécharger V2.5.0 : Version 2.6.0 change New bots :
- TelegramBot (thanks @alsoGAMER )
- InternetNZ Webscan
- Microsoft WinHttp
- Microsoft Office (Excel / Word / etc)
- AspiegelBot (thanks @VersoBit )
Télécharger V2.6.0 : Version 2.7.0 change : New functionality: tool to test bot detection - paste in a user agent string and the tool will tell you if it detects a bot
Télécharger V2.7.0 : v3.0.0 update (unreleased) :
Major new feature: add generic bot detection
User Agent strings are scanned for the keywords "bot", "crawl", or "spider" - any User Agents not already detected as a bot which contain one of these strings are stored in the cache and made visible through the admin UI, with the option to have this information emailed on a weekly basis.
new bots: AccompanyBot; PostmanRuntime
v3.1.0 update : (took less than 5 minutes after installing it on several of my production forums to identify new bots!!)
new bots: amazonbot; petalbot; slackbot
Télécharger V3.1.0 : Version 3.2.0 big bot update :
- 360spider
- adsbot
- adstxtcrawler
- awariorssbot
- bitlybot
- boardreader
- ccbot
- cincraw
- clickagy intelligence bot
- crawlson
- crsspxlbot
- datagnionbot
- deskyobot
- df bot
- dnsresearchbot
- duckduckgo favicons bot
- eyeotabot
- feedlybot
- foocrawlerbot
- germcrawler
- google adwords instant
- gumgum bot
- hatena
- hetrixtools
- hubpages
- ias_crawler
- internet structure research project bot
- jugendschutzprogramm-crawler
- krzana bot
- lightspeedsystemcrawler
- linespider
- livelapbot
- lufsbot
- moatbot
- netestate ne crawler
- netpeakchekerbot
- netvibes
- nimbostratus bot
- nixstatsbot
- obot
- odklbot
- paperlibot
- pilicanbot
- pleskbot
- politecrawl
- pubmatic crawler bot
- rogerbot
- scooperbot
- screaming frog seo spider
- semantic-visions.com
- semanticbot
- seolizer
- sirdatabot
- surdotlybot
- tapatalk cloudsearch platform
- tpradstxtcrawler
- velenpublicwebcrawler
- yisouspider
- wp.com feedbot
- zoombot
Cubot mobile phones added to false positive list (Logicom BOT phones are already on the list)
Télécharger V3.2.0 :
v3.3.0 updates :
- added a phrase for email subject instead of hardcoding the string
- removed Dalvik from list of bots - it's an Android browser
- fixed missing user agent string for rssingbot
- new Cubot phone false positives
- cubot echo
- cubot_j3
- cubot king kong
- cubot max
- cubot note plus
- cubot_note_s
- cubot_note_s build/mra58k
- cubot_nova
- cubot r9
- cubot_x18_plus
- lots of new bots
- adstxtlab.com
- anderspinkbot
- arquivo-web-crawler
- barkrowler
- botw spider
- buzzbot
- centro ads.txt crawler
- cispa webcrawler
- claritybot
- cognitiveseo.com
- crowdtanglebot
- dingtalkbot-linkservice
- elmer, the thinglink imagebot
- elisabot
- grfzbot
- hubspot crawler
- hubspot url validation check
- icc-crawler
- jobboersebot
- kingbot
- knot group
- lawinsiderbot
- makemoneyteamworkbot
- mastodon
- maxpointcrawler
- mediapartners-google
- mediumbot-metatagfetcher
- metajobbot
- msiecrawler
- neticle crawler
- netseer crawler
- nextcloud server crawler
- ninjbot
- pagething.com
- pagepeeker
- pandalytics
- parsijoobot
- pimeyes.com
- popscreen bot
- pulno
- pulsepoint-ads.txt-crawler
- ravencrawler
- redditbot
- rely bot
- rssmicro.com
- sbooksnet
- scanmine newsspider
- seo-audit-check-bot
- seoclaritycrawl
- sitelockspider
- slack-imgproxy
- slackbot-linkexpanding
- snappreviewbot
- spiderling
- squidbot
- stormcrawler
- superfeedr bot
- superpagesbot
- tmmbot
- trendkite-akashic-crawler
- ucmore crawler app
- vebidoobot
- yellowbrandprotectionbot
- vkrobot
- voilabot
- wiederfreibot
- yacybot
- zumbot
Télécharger V3.3.0 :
update: email subject line now includes addon version
feature: now optionally uses Monolog Logging Service addon for logging info about sent emails
fixed missing mailto: in some email links
new Cubot phone false positives:
cubot h3
cubot_note_s build/lmy47i
weekly new bots:
3dd trunk
3w24bot
auto spider
aylien
bidswitchbot
bnf.fr_bot
bublupbot
checkmarknetwork
chimebot
cloudservermarketspider
curious george
diffbot
everyfeed-spider
ffzbot
finditanswersbot
fyrebot
gigabot
graydon bot
hrankbot
huaweiwebcatbot
hypestat
hyscore
implisensebot
jasper's lil' bot'
jetslide
jpg-newsbot
konturbot
letsearchbot
looid.com crawler
magibot
mauibot
mindupbot
miralinks robot
onalyticabot
online-webceo-bot
queryseekerspider
randomsurfer
rankurbot
researchbot
safednsbot
serptimizerbot
sitecheckerbotcrawler
somdsearchbot
ssblog rsscrawler
sserobots
statdom.ru
synthesio crawler
toutiaospider
turnitinbot
website-audit.be crawler
who.is bot
wikido
wlc pywikibot
x28-job-bot
xyz spider
y!j-asr
yetibot
zombiebot
feature: now optionally uses Monolog Logging Service addon for logging info about sent emails
fixed missing mailto: in some email links
new Cubot phone false positives:
cubot h3
cubot_note_s build/lmy47i
weekly new bots:
3dd trunk
3w24bot
auto spider
aylien
bidswitchbot
bnf.fr_bot
bublupbot
checkmarknetwork
chimebot
cloudservermarketspider
curious george
diffbot
everyfeed-spider
ffzbot
finditanswersbot
fyrebot
gigabot
graydon bot
hrankbot
huaweiwebcatbot
hypestat
hyscore
implisensebot
jasper's lil' bot'
jetslide
jpg-newsbot
konturbot
letsearchbot
looid.com crawler
magibot
mauibot
mindupbot
miralinks robot
onalyticabot
online-webceo-bot
queryseekerspider
randomsurfer
rankurbot
researchbot
safednsbot
serptimizerbot
sitecheckerbotcrawler
somdsearchbot
ssblog rsscrawler
sserobots
statdom.ru
synthesio crawler
toutiaospider
turnitinbot
website-audit.be crawler
who.is bot
wikido
wlc pywikibot
x28-job-bot
xyz spider
y!j-asr
yetibot
zombiebot
Télécharger V3.4.0 :
new Cubot phone false positives:
cubot cheetah 2
weekly new bots:
better uptime bot
btcrawler
crawler_eb_germany_2.0
dle_spider.exe
dyno mapper crawler
flockbrain robot
infoobot
microadbot
nesotebot
reachabilitycheckbot
se ranking gentle bot
seobilitybot
siteguru linkchecker
statonlinerubot
tombot
viulinkcrawler
webgains-bot'
_zbot
cubot cheetah 2
weekly new bots:
better uptime bot
btcrawler
crawler_eb_germany_2.0
dle_spider.exe
dyno mapper crawler
flockbrain robot
infoobot
microadbot
nesotebot
reachabilitycheckbot
se ranking gentle bot
seobilitybot
siteguru linkchecker
statonlinerubot
tombot
viulinkcrawler
webgains-bot'
_zbot
Télécharger V3.6.0 :
new Cubot phone false positives:
cubot dinosaur
new GLX phone false positives:
GLX Spideri
weekly new bots:
ant.com beta
bha2r_bot
browserspybot
c-t bot
cms crawler
ioncrawl
js-crawler
k7mlwcbot
quora-bot
qwarrycrawler
sidetrade indexer bot
webzip
cubot dinosaur
new GLX phone false positives:
GLX Spideri
weekly new bots:
ant.com beta
bha2r_bot
browserspybot
c-t bot
cms crawler
ioncrawl
js-crawler
k7mlwcbot
quora-bot
qwarrycrawler
sidetrade indexer bot
webzip
Télécharger V3.7.0 : Version 3.8.0 weekly bot updates :
weekly new bots:
ag_dm_spider
aihitbot
avocetcrawler
b2b bot
checkbot
em-crawler
feedsearch-crawler
fullstorybot
krzana-rss-bot
mediacloud bot for open academic research
mlbot
niuebot
ocarinabot
prft-bot
qwantbot
rasabot
riverbot
rytebot
sabsimbot
scribbr-citation-bot
seodiver
snapbot
solomonobot
temeliobot-keyword-scrapper
vsusearchspider
your-search-bot
ag_dm_spider
aihitbot
avocetcrawler
b2b bot
checkbot
em-crawler
feedsearch-crawler
fullstorybot
krzana-rss-bot
mediacloud bot for open academic research
mlbot
niuebot
ocarinabot
prft-bot
qwantbot
rasabot
riverbot
rytebot
sabsimbot
scribbr-citation-bot
seodiver
snapbot
solomonobot
temeliobot-keyword-scrapper
vsusearchspider
your-search-bot
Télécharger V3.8.0 :
weekly new bots:
earwigbot
fess
metadata-downloader-bot
page audit bot
plurkbot
r6_commentreader
r6_feedfetcher
sc_bot
tkbot
willie irc bot
earwigbot
fess
metadata-downloader-bot
page audit bot
plurkbot
r6_commentreader
r6_feedfetcher
sc_bot
tkbot
willie irc bot
Télécharger V3.9.0 :
weekly new bots:
aasa-bot
amazon-advertising-ad-standards-bot
crawler/0.0.1
danibot
dragonbot
irlbot
jooblebot
ldspider
linkisbot
mohawk-crawler
newsgator
ottobot
pigafetta
pingdom.com_bot
planckspider
rss bot
speedy spider
summalybot
virusdie crawler
aasa-bot
amazon-advertising-ad-standards-bot
crawler/0.0.1
danibot
dragonbot
irlbot
jooblebot
ldspider
linkisbot
mohawk-crawler
newsgator
ottobot
pigafetta
pingdom.com_bot
planckspider
rss bot
speedy spider
summalybot
virusdie crawler
Télécharger V3.10.0 :
weekly new bots:
girafabot
googebot malware scanning
i-market-bot
intelx.io_bot
ninjabot
pywebcopybot
serpstatbot
supremesearch.net
todoexpertosbot
webmoney megastock robot
wordchampbot
xaldon webspider
yesslebot
girafabot
googebot malware scanning
i-market-bot
intelx.io_bot
ninjabot
pywebcopybot
serpstatbot
supremesearch.net
todoexpertosbot
webmoney megastock robot
wordchampbot
xaldon webspider
yesslebot
Télécharger V3.11.0 :
weekly new bots:
experiancrawluk
impact radius compliance bot
kauaibot
leuchtfeuer crawler
redirectbot
simpleanalyticsbot
skimbot
sogou pic spider
testcrawler
experiancrawluk
impact radius compliance bot
kauaibot
leuchtfeuer crawler
redirectbot
simpleanalyticsbot
skimbot
sogou pic spider
testcrawler
Télécharger V3.12.0 :
new Cubot phone false positives:
cubot a5
cubot j3 pro
weekly new bots:
addsugarspiderbot
annuairefrancais.fr
coibotparser
criteobot
discobot
domainspider-bot
dropboxpreviewbot
dumbot
gnowitnewsbot
hoaxybot
ichiro
lamarkbot
media-bot
nusearch spider
radian6_default_
xovionpagecrawler
zyborg
cubot a5
cubot j3 pro
weekly new bots:
addsugarspiderbot
annuairefrancais.fr
coibotparser
criteobot
discobot
domainspider-bot
dropboxpreviewbot
dumbot
gnowitnewsbot
hoaxybot
ichiro
lamarkbot
media-bot
nusearch spider
radian6_default_
xovionpagecrawler
zyborg
Télécharger V3.13.0 :
weekly new bots:
antbot/1.0
becomebot
castlebot
custom-crawler
feedsearch bot
gulper web bot
istellabot
lmspider
marketingminer bot
my nutch spider
networking4all bot
nfwebcrawler
tombapublicwebcrawler
uptimebot
vuhuvbot
webcrawl.net
webspider 1.0
antbot/1.0
becomebot
castlebot
custom-crawler
feedsearch bot
gulper web bot
istellabot
lmspider
marketingminer bot
my nutch spider
networking4all bot
nfwebcrawler
tombapublicwebcrawler
uptimebot
vuhuvbot
webcrawl.net
webspider 1.0
Télécharger V3.14.0 :
new Dinobot Android TV false positives:
dinobot 4k plus
weekly new bots:
cis455crawler
crystalsemanticsbot
discoverspider
envolk[its]spider
geograph linkcheck bot
gg peekbot
iccrawler
mybot
psbot
suggybot
testbot
dinobot 4k plus
weekly new bots:
cis455crawler
crystalsemanticsbot
discoverspider
envolk[its]spider
geograph linkcheck bot
gg peekbot
iccrawler
mybot
psbot
suggybot
testbot
Télécharger V3.15.0 :
weekly new bots:
boitho.com-dc
msc crawl project radboud university
niocbot
open web analytics bot
quetextbot
rc-crawler
tokenspider
womlpefactory
yeti/1.0
boitho.com-dc
msc crawl project radboud university
niocbot
open web analytics bot
quetextbot
rc-crawler
tokenspider
womlpefactory
yeti/1.0
Télécharger V3.16.0 : Version 3.17.0 weekly bot updates new false positives :
cubot r11
spider v7 build/lmy47i
spider v7 (MyCell Spider v7 from Bangladesh)
Télécharger V3.17.0 :
new false positives:
cubot; j5
baiduboxapp
new bots:
200pleasebot
a8bot
abilogicbot
acoonbot
adform robot
arhpostbot
atomseobot
awariorendererbot
badoobot
bl.uk_ldfc_bot
brobot
charityengine bot
charlotte
cosmos
coveobot
crawlbot/1.0.0
cxensebot
facebot
fandomopengraphbot
freshpingbot
fuelbot
geedobot
getlocalbot
google-safety
gpcsupbot
grub-client
gynxbot
healrworld crawler
hgfalphaxcrawl
hoodle crawler
idmarch automatic
imrbot
jambot
justlocal.nl
kantarsifomediaauditbot
keobsbot
keybasebot
koepabot
lanaibot
landsbokasafn
lapozzbot
linkpulse metacrawler
linksmanager.com_bot
lxrbot
mbot v
moreoverbot
netpeakspiderbot
www.niraiya.com
node/simplecrawler
nu.marginalia.wmsa.edge-crawler
nutchcvs
oer commons bot
omniexplorer_bot
onefuncbot
oozbot
pickybot
piepmatz bot
plukkie
pu_in crawler
punkspider
pwa-crawler
reasonalbot
revuebot
runet-research-crawler
screenerbot crawler
searchenginecrawler
sebot-wa
seekbot
shopwiki
showyoubot
siteauditbot
sitescorebot
spinn3r
squirrobot
ssl-crawler
thinkbot
tsmbot
tweetedtimes bot
ucrawl
umichbot
urlappendbot
verticalleap-sitestatusbot
webgraph
weblinkchecker
websquash.com
wellknownbot
wizenozebot
cubot; j5
baiduboxapp
new bots:
200pleasebot
a8bot
abilogicbot
acoonbot
adform robot
arhpostbot
atomseobot
awariorendererbot
badoobot
bl.uk_ldfc_bot
brobot
charityengine bot
charlotte
cosmos
coveobot
crawlbot/1.0.0
cxensebot
facebot
fandomopengraphbot
freshpingbot
fuelbot
geedobot
getlocalbot
google-safety
gpcsupbot
grub-client
gynxbot
healrworld crawler
hgfalphaxcrawl
hoodle crawler
idmarch automatic
imrbot
jambot
justlocal.nl
kantarsifomediaauditbot
keobsbot
keybasebot
koepabot
lanaibot
landsbokasafn
lapozzbot
linkpulse metacrawler
linksmanager.com_bot
lxrbot
mbot v
moreoverbot
netpeakspiderbot
www.niraiya.com
node/simplecrawler
nu.marginalia.wmsa.edge-crawler
nutchcvs
oer commons bot
omniexplorer_bot
onefuncbot
oozbot
pickybot
piepmatz bot
plukkie
pu_in crawler
punkspider
pwa-crawler
reasonalbot
revuebot
runet-research-crawler
screenerbot crawler
searchenginecrawler
sebot-wa
seekbot
shopwiki
showyoubot
siteauditbot
sitescorebot
spinn3r
squirrobot
ssl-crawler
thinkbot
tsmbot
tweetedtimes bot
ucrawl
umichbot
urlappendbot
verticalleap-sitestatusbot
webgraph
weblinkchecker
websquash.com
wellknownbot
wizenozebot
Télécharger V3.18.0 :
new false positives:
- spider v9 phone
- 80legs.com: 80legs Crawler
- adssellerscrawler: Ads Sellers Crawler
- apesearch crawler: ApeSearch Crawler
- aportcatalogrobot: AportCatalogRobot
- bot dns-cache.fr: bot dns-cache.fr
- crusty broad web crawler: Crusty Broad Web Crawler
- docomo: NTT Docomo Goo Bot
- dubbotbot: DubBot Bot
- electricmonk: DueDil Electric Monk Crawler
- finbot: Finbot
- flok's crawler: Flok's Crawler
- goodbot: GoodBot
- grover: Grover web crawler
- joc web spider: JOC Web Spider
- linkarchiver twitter bot: LinkArchiver Twitter Bot
- linkscrawler: LinksCrawler
- neevabot: Neevabot Search Engine Bot
- newsharecounts.com: Newshare Counts
- newslitbot: Newslit Bot
- opensearch@mpdl: OpenSearch@MPDL
- orbbot: orbbot
- pajbot1: Pajbot
- richaudience brandsafety bot: Rich Audience Brandsafety Bot
- semjibot: SemjiBot
- simplecrawler/0.1: SimpleCrawler
- snap url preview service: Snap URL Preview Service
- spider_bot/3.0: Spider_bot
- spotibobot: Spotibo bot
- synologychatbot: Synology Chat Bot
- terrawizbot: TerrawizBot
- top100.rambler.ru crawler: Top-100 Rambler Crawler
- webshopchecker bot: Webshop Checker Bot
- wi job roboter spider: Web Integration Job Robot
- woobot: WooBot
- woriobot: Zite.com WorioBot
- wp fastest cache preload bot: WP Fastest Cache Preload Bot
- xenforo: XenForo
- your bot: Your Bot
- yunsecuritybot: YunSecurityBot
- yurichevbot: Yurichev Bot
Télécharger V3.19.0 :
new bots discovered in June 2021
- btbot: BT Bot
- catchbot: Catchbot
- comodospider: Comodo SSL Spider
- deepnoc: deepnoc bot (network optimized crawling)
- dispenserbot: Dispenser Dab Solver Checklinks Bot
- epicbot: Epictions EpicBot
- esperanzabot: EsperanzaBot
- fast enterprise crawler: Fast enterprise crawler 6 used by Schibsted
- fleabot: Mercadopar Fleabot
- fuseonbot: Fuseon Link Affinity Bot
- google/bot: google/bot
- greenflare seo crawler: Greenflare SEO Crawler
- gsitecrawler: GSiteCrawler
- holmes: Morfeo Holmes Bot
- infotigerbot: InfoTiger Search Engine Bot
- internet security survey bot: Internet Security Survey Bot
- jetpack-bot: JetPack Bot
- mojoo robot: Mojoo Bot
- nicecrawler: NiceCrawler
- prem.moe crawler: Prem.moe Crawler
- sayindexbot: SayIndex Bot
- sbl-bot: SoftByte Labs Bot
- siteliner: Siteliner Bot
- swjschecketbot: Swjschecketbot
- trade desk ads.txt & sellers.json crawler: Trade Desk ads.txt & sellers.json crawler
- vortex: Marty Anstey Vortex Bot
- www.hlabs.co.ke: hLabs Bot
- zspider: Red Kolibri ZSpider
Télécharger V3.20.0 : KnownBots v4.0.0 - complete re-write :
KnownBots v4 is a completely new build - bots are no longer hard coded, but updated via API calls and uses the XF code cache to store bot data
- raw bot data downloaded from API is stored in internal_data/knownbots.json
- new CLI tool for manually fetching bots from API (Cron task is also provided)
- new CLI tool for manually loading bots from knownbots.json
- new CLI tool for testing user agent matches
This release includes additional sanity checks to prevent bad data returned from the API from breaking the forums.
If any of the data returned by the API is not in the exact format we expect, the entire download is discarded and no changes applied to the forum. An error message will be logged prompting further investigation.
After upgrading to 4.0.1, you should manually force a fetch of new API data by executing the following command from your forum root:
php cmd.php known-bots:fetch -fTélécharger V4.0.1 :
v5.0.0 is a major rewrite of the core functionality of this addon aimed at improving processing speed, bot detection sophistication and greatly enhancing our ability to identify new bots.
Note that the options have changed - so please check the options after upgrading. More information about each option is provided on the main addon page.
Note that the options have changed - so please check the options after upgrading. More information about each option is provided on the main addon page.
- major rewrite - no longer use "bot|spider|crawl" search strings and false-positive lists to identify possible bots, rely instead on search strings supplied by API to identify valid browsers and store them directly in the database rather than the SimpleCache, ready for emailing
- more complete agent reprocessing - check for valid browsers and ignored agents
- change the core userAgentMatchesRobot function to use strpos instead of preg_match, it's much faster and won't fall over with extremely high numbers of bot match strings
- allow BotFetcher to be manually configured to bypass untrusted http agent - used for testing when API source is on a .local domain. Default action remains to use the untrusted http agent to allow for proxying outbound API calls.
- change email cron to daily send
- using new v2 API from KnownBots
- replace generic bots with complex (regex) based searches
- add "Fetch new bots" button to Known Bots List in admin UI
- automatically reprocess user agents after loading new bot data
- new Cli command for reprocessing user agents, including the option to force all user agents to be reprocessed
- improvements to user agent test in admin ui to be more descriptive
- bcc additional email address to keep them private
- bugfix: don't linkify known bot list when no links supplied
v6 is another major rewrite of the core functionality of the addon, aimed at improving the submission process for newly detected useragents and performance improvements.
Important for v4 users: with this release, I am deprecating the v1 API - addon versions v4.x and earlier will continue to function for a while but will then start returning 404 error codes once I turn off the v1 API. Anyone still running KnownBots v4.x should upgrade as soon as possible.
Important for v5 users: The v2 API used in addon v5.x for fetching new bots will remain operational, however, I am deprecating the email based submission system in favour of a new API based user agent submission system. After a transition period, the inbound email system will be disabled and any emails sent to the knownbots@hampel.io address will bounce back as undelieverable. Anyone still running KnownBots v5.x should either upgrade, or at least disable the "Email user agents" option in the v5.x addon options.
Important for anyone upgrading to v6: the new submission system in v6 uses an authentication process to ensure only valid submissions occur. After upgrading to v6, to continue submitting new user agents for analysis, you must first configure the authentication system - it is a very simple process - see instructions on the addon page. The options for v6 have changed - you should check them after upgrading.
The new submission system in v6 utilises the XenForo customer validation API to authenticate sites when submitting agents via our new API.
To configure the API, enter the License validation token for your site, found in the XenForo customer interrface. The validation token will be sent to the XenForo customer validation API by the KnownBots system and if valid, a KnownBots API token will be generated and returned back to the requesting forum for subsequent authentication purposes.
With a validated license, the authentication process is automatic. API tokens are regenerated every 28 days and are re-authenticated automatically. Customer details are automatically purged from the KnownBots database after 30 days of inactivity (see privacy details on main addon page). Regenerating your license validation token will automatically cause API revalidation to fail and customer details to be purged - unless you re-configure the addon options with the new license validation token.
Changelog for v6 :
- new CLI tool known-bots
arse to parse web server log files and display detected bots - new CLI tool known-bots:send to send newly detected user agents to the KnownBots API for analysis
- new CLI tool known-bots:check-token to validate that the API token successfully authenticates - and optionally have the system regenerate a new API token if it has expired
- knownbots@hampel.io email address is deprecated and will be removed soon - emails should no longer be sent to this address
- new configuration option to "Send user agents via API", which requires configuration by entering a XenForo license validation token. New agents are sent directly via api and no longer by email
- the "Email user agents" option remains - but is used only for forum administrators to send themselves emails if they choose. Upgrading to v6 of the addon removes any reference to knownbots@hampel.io from this configuration option.
- addon now uses v3 of the bot fetch API, which includes new functionality
- v2 of the bot fetch API remains operational for sites still using addon v5.x
- v1 of the bot fetch API is now deprecated and will soon stop functioning - sites still using addon v4.x should upgrade as soon as possible
- new functionality for the addon - a list of regex based ignore strings to remove malformed or obfuscated user agents from analysis. This also allows us to ignore user agents containing sql-injection and other forms of attack which typically flood a system with a large number of unique user agents in a short period of time.
- performance enhancement - we no longer do browser or ignored checks for user agents of users who are logged in. We assume that anyone logged in with a valid XenForo user id is using a valid browser. Note that bot detection is still run, just in case. This significantly reduces the amount of processing performed by the addon for valid users.
6.0.2 minor bug fix
Minor bug fix - no need to update unless you are experiencing problems sending bot updates.- when sending bots via email, include the bot list as an attachment rather than in the body
- new CLI tool to send bots via email directly, used for debugging bot sending issues
This update contains an important bugfix to handle malformed UTF-8 in user agent strings.
This update will simply ignore any user agents with malformed UTF-8, avoiding errors when trying to send updates via the API. These user agents are invalid and so there is no point undertaking any further analysis - thus they are silently discarded.
This new version also contains an additional CLI tool for importing user agents from a text file, for testing purposes.
Télécharger V6.0.3 : Version 6.0.4 improve XF 2.3 compatibility :
- Improved XF 2.3 compatibility
Télécharger V6.0.4 : Version 6.0.5 bugfix : XF2.3 compatibility for sending new bots via an email attachment.
Télécharger V6.0.5 : Version 6.1.0 - admin permissions update :
- now requires Manage known bots admin permission to see admin tools.
Télécharger V6.1.0 :
![[OzzModz] New Thread Avatar 2.0.1](https://www.societyfrenchmodding.net/data/attachments/2/2444-f2c490ab93d24827ced30fe9c61ee01c.jpg){kind=small}
Dernière édition par un modérateur:
![[SC] Count Robots 2.0.0](https://www.societyfrenchmodding.net/data/attachments/9/9693-2b1b70cfe0f12235c155364549d7e450.jpg?hash=Kxtwz-DxIj){kind=small}