ridwan.chendarma.com

Checking User Agent (Browser)

March 16, 2007

Time to check for bad user-agent which includes bad browser, bad bot and bad spiders. Some of my top browsers are of course IE, Mozilla, Safari, Opera, Windows Media Player, Netscape. Other than that are within my suspicion as bad user-agent.

Percentage is based on February 2007 sessions count on 1 server.

PASS
Sogou Spider (0.13%) - belongs to a chinese search engine sohu.com inc
Up.Browser (0.03%) - a mobile phone browser known as Openwave.
Konqueror (0.03%) – file manager, web browser and file viewer developed part of KDE Unix-like OS.
Hatena Antenna (0.01%) – webpage bookmark plugin. hatena is now featured as Google desktop gadgets.
Baiduspider+ (<0.01%) – search engine spider belongs to baidu.com
ia_archiver (<0.01%) – Alexa robot crawler
BlackBerry (0.01%) – blackberry mobile browser
Nokia(*) (0.01%) – nokia mobile browser
Vodafone (0.01%) – uk mobil network, mobile browser
SonyEricsson (<0.01%) – SonyEricsson mobile browser
HPiPAQhw6900 (<0.01%) – HP iPAQ mobile browser

UNKNOWN
HTTP Fetcher (<0.01%) – need more info
JAVA (<0.01%) – need more investigation. most of the time this agent is mark for blocking, including python urllib.
WGet (0.01%) – undecided, file retrieval using HTTP, unix-based os.

BAD
Download Ninja (0.04%) – bad.
Internet Ninja (0.02%) – bad.
WebAuto – bad.
WebCopier – bad.

Slowly building user-agent blacklist… more to come.

Filed under: Web