搜索引擎的蜘蛛人（ROBOTS）秘密

User-agent: GoogleBot
Disallow: /cgi-bin
Disallow: /java
Disallow: /images
Disallow: /development
Disallow:   /third
Disallow: /beta
Disallow: /webmaster_logs
Disallow: /virtual
Disallow: /shockwave
Disallow: /TRANSCRIPTS
Disallow:   /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
Disallow:   /alt_index.html

[