Fortinet 5.0 Patch 6 - page 395

Fortinet 395 FortiWeb 5.0 Patch 6 Administration Guide
Tip: Some attackers use 4XX and 5XX HTTP response codes for
web site reconnaissance when identifying potential targets: to
determine whether a page exists, has login failures, is Not
Implemented, Service Unavailable, etc. Normally, the FortiWeb
appliance records attack logs for 4XX and 5XX response codes,
but HTTP response codes are also commonly innocent, and too
many HTTP response code detections may make it more difficult to
notice other information disclosure logs. To disable response code
violations, disable both the HTTP Return Code 4XX and HTTP
Return Code 5XX options in this rule’s area.
Tip: Because this feature can potentially require the FortiWeb
appliance to rewrite the header and body of every request from a
server, it can decrease performance. To minimize impact, Fortinet
recommends enabling this feature only to help you identify
information disclosure through logging, and until you can
reconfigure the server to omit such sensitive information.
Bad Robot Enable to analyze the User-Agent: HTTP header and block
known content scrapers, spiders looking for vulnerabilities, and
other typically unwanted automated clients.
FortiWeb predefined signatures for many well-known robots, such
as link checkers, search engine indexers, spiders, and web
crawlers for Google, Baidu, and Bing, which you can use to restrict
access by Internet robots such as web crawlers, as well as
malicious automated tools.
Search engines, link checkers, retrievals of entire web sites for a
user’s offline use, and other automated uses of the web
(sometimes called robots, spiders, web crawlers, or automated
user agents) often access web sites at a more rapid rate than
human users. However, it would be unusual for them to request the
same URL within that time frame.
Usually, web crawlers request many different URLs in rapid
sequence. For example, while indexing a web site, a search
engine’s web crawler may rapidly request the web site’s most
popular URLs. If the URLs are web pages, it may also follow the
hyperlinks by requesting all URLs mentioned in those pages. In this
way, the behavior of web crawlers differs from a typical brute force
login attack, which focuses repeatedly on one URL.
Some robots, however, are not well-behaved. You can request that
robots not index and/or follow links, and disallow their access to
specific URLs (see http://www.robotstxt.org/). However,
misbehaving robots frequently ignore the request, and there is no
single standard way to rate-limit robots.
To verify that bad robot detection is being applied, attempt to
download a web page using wget, which is sometimes used for
content scraping.
Setting name Description