Guidelines on Securing Public Web Servers

Disallow: /_borders/ 

Disallow: /_fpclass/ 

Disallow: /_overlay/ 

Disallow: /_private/ 

Disallow: /_themes/ 

To disallow all (compliant) bots from the entire Web site: 

User agent: * 

Disallow: / 

To disallow a specific bot (in this case the Googlebot) from examining a specific Web 

page: 

User agent: GoogleBot 

Disallow: tempindex.htm 

Note that the robots.txt file is available to everyone.  Thus, a Web administrator should not 

specify the names of sensitive files or folders.  If these must be excluded, it is better to use 

password protected pages that cannot be accessed by bots.  Password protection is the only 

reliable way to exclude noncompliant bots.  See Section 7 for more information on Web based 

authentication methods. 

5.3  Using File Integrity Checkers 

A file integrity checker is an application that computes and stores a checksum for every 

guarded file and establishes a database of file checksums.  It allows a system administrator to 

easily recognize changes to critical files, particularly unauthorized changes.  Checksums 

should be recomputed regularly to test the current value against the stored value to identify any 

file modifications.  A file integrity checker capability is often included with host based 

intrusion detection systems (see Section 8.2.2) and is also available separately (see Appendix 

E). 

Although an integrity checker is a useful tool that does not require a high degree of human 

interaction, it needs to be used carefully to ensure that it is effective.  To create the first 

reference database a file integrity checker requires a system that is known to be in a secure 

state.  Otherwise, cryptographic hashes of a compromised system may be created and therefore 

create a false sense of security for the tester.  The reference database should be stored off line 

so that an attacker cannot compromise the system and modify the database to hide tracks of the 

attack.  A file integrity checker can also generate false positive alarms.  Each file update and 

system patch implementation changes the file and will therefore require an update of the 

checksum database.  Thus, keeping the database up to date may be difficult.  However, even if 

the integrity checker is run only once (when the system is first installed), it can still be a useful 

activity for determining which files have been modified in case of a suspected compromise.  

Finally, attackers have demonstrated an ability to modify a file in ways the commonly used 32 

bit cyclic redundancy check (CRC) checksum could not detect.  Therefore, stronger 

checksums are recommended to ensure the integrity of data that is stored in the checksum 

database. 

Integrity checkers should be run nightly on a selection of system files that would be affected 

by a compromise.  Integrity checkers should also be used when a compromise is suspected for 

34