搜索引擎的蜘蛛人（ROBOTS）秘密
发布时间：2006-07-13 13:49:44 来源： 网友评论 0 条例子:
# robots, scram
User-agent: *
Disallow: /cgi-bin
Disallow: /TRANSCRIPTS
Disallow: /development
Disallow: /third
Disallow: /beta
Disallow: /java
Disallow: /shockwave
Disallow: /JOBS
Disallow: /pr
Disallow: /Interactive
Disallow: /alt_index.html
Disallow: /webmaster_logs
Disallow: /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
User-agent: Mozilla/3.01 (hotwired-test/0.1)
Disallow: /cgi-bin
Disallow: /TRANSCRIPTS
Disallow: /development
Disallow: /third
Disallow: /beta
Disallow: /java
Disallow: /shockwave
Disallow: /JOBS
Disallow: /pr
Disallow: /Interactive
Disallow: /alt_index.html
Disallow: /webmaster_logs
Disallow: /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
User-agent: Slurp
Disallow: /cgi-bin
Disallow: /TRANSCRIPTS
Disallow: /development
Disallow: /third
Disallow: /beta
Disallow: /java
Disallow: /shockwave
Disallow: /JOBS
Disallow: /pr
Disallow: /Interactive
Disallow: /alt_index.html
Disallow: /webmaster_logs
Disallow: /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
# robots, scram
User-agent: *
Disallow: /cgi-bin
Disallow: /TRANSCRIPTS
Disallow: /development
Disallow: /third
Disallow: /beta
Disallow: /java
Disallow: /shockwave
Disallow: /JOBS
Disallow: /pr
Disallow: /Interactive
Disallow: /alt_index.html
Disallow: /webmaster_logs
Disallow: /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
User-agent: Mozilla/3.01 (hotwired-test/0.1)
Disallow: /cgi-bin
Disallow: /TRANSCRIPTS
Disallow: /development
Disallow: /third
Disallow: /beta
Disallow: /java
Disallow: /shockwave
Disallow: /JOBS
Disallow: /pr
Disallow: /Interactive
Disallow: /alt_index.html
Disallow: /webmaster_logs
Disallow: /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
User-agent: Slurp
Disallow: /cgi-bin
Disallow: /TRANSCRIPTS
Disallow: /development
Disallow: /third
Disallow: /beta
Disallow: /java
Disallow: /shockwave
Disallow: /JOBS
Disallow: /pr
Disallow: /Interactive
Disallow: /alt_index.html
Disallow: /webmaster_logs
Disallow: /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
[
- 推荐阅讯
- 技巧再现:css网页设计技巧
- 巧用背景颜色和间距快速制作出细边线表格
- 不用script仅用css编写无限分级弹出菜单
- 脚本控制三行三列自适应高度DIV布局
- fieldset实现表中表效果
- CSS仿淘宝首页导航条按钮布局效果
- 调整CSS类型的顺序改变链接状态
- CSS中的行为——HTC
- 用CSS打造韩式风格网站
- 锦囊妙计 CSS实现样式布局22招