단축키

Prev이전 문서

Next다음 문서

수정 삭제

단축키

Prev이전 문서

Next다음 문서

수정 삭제
Extra Form

www.google.com


위 구글에서 실제로 작성된 robots.txt 파일규칙입니다.


URL에서 위 주소 뒤에 www.google.com/robots.txt 라고 치면 이렇게 나옵니다.


아래 기준으로 작성하셔서 / 루트에 올리신 후 웹마스터도구에서 정상작동되는지 테스트해보세요



모든 bot을 허용한다

User-agent: *
Allow: /


위 allow: 말고 다른 방법도 가능합니다
예: Disallow:



뒤 / 슬러시는 allow와 disallow에서 반대로 사용되는 개념입니다.

sitemap 사이트맵 xml 파일이 있다면 루트에 올려두고 


Sitemap: http://www.domain.com/sitemap.xml


하단 줄을 추가하시면 됩니다.


User-agent: *
Disallow: /search
Allow: /search/about
Disallow: /sdch
Disallow: /groups
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /relpage/
Disallow: /imgres
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom
Disallow: /advanced_group_search
Disallow: /googlesite
Disallow: /preferences
Disallow: /setprefs
Disallow: /swr
Disallow: /url
Disallow: /default
Disallow: /m?
Disallow: /m/
Allow:    /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /bsd?
Disallow: /linux?
Disallow: /mac?
Disallow: /microsoft?
Disallow: /unclesam?
Disallow: /answers/search?q=
Disallow: /local?
Disallow: /local_url
Disallow: /shihui?
Disallow: /shihui/
Disallow: /froogle?
Disallow: /products?
Disallow: /froogle_
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*q=*
Disallow: /books?*q=*
Disallow: /books?*output=*
Disallow: /books?*pg=*
Disallow: /books?*jtp=*
Disallow: /books?*jscmd=*
Disallow: /books?*buy=*
Disallow: /books?*zoom=*
Allow: /books?*q=related:*
Allow: /books?*q=editions:*
Allow: /books?*q=subject:*
Allow: /books/about
Allow: /booksrightsholders
Allow: /books?*zoom=1*
Allow: /books?*zoom=5*
Disallow: /ebooks/
Disallow: /ebooks?*q=*
Disallow: /ebooks?*output=*
Disallow: /ebooks?*pg=*
Disallow: /ebooks?*jscmd=*
Disallow: /ebooks?*buy=*
Disallow: /ebooks?*zoom=*
Allow: /ebooks?*q=related:*
Allow: /ebooks?*q=editions:*
Allow: /ebooks?*q=subject:*
Allow: /ebooks?*zoom=1*
Allow: /ebooks?*zoom=5*
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Disallow: /citations?*cstart=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Disallow: /complete
Disallow: /s?
Disallow: /sponsoredlinks
Disallow: /videosearch?
Disallow: /videopreview?
Disallow: /videoprograminfo?
Allow: /maps?*output=classic*
Allow: /maps?*file=
Allow: /maps/api/js?
Allow: /maps/d/
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /mapprint?
Disallow: /maps/api/js/
Disallow: /maps/api/staticmap?
Disallow: /maps/api/streetview
Disallow: /mld?
Disallow: /staticmap?
Disallow: /places/
Allow: /places/$
Disallow: /maps/preview
Disallow: /maps/place
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /sms/demo?
Disallow: /katrina?
Disallow: /blogsearch?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /mbd?
Disallow: /extern_js/
Disallow: /xjs/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /trends/beta
Disallow: /notebook/search?
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /browsersync
Disallow: /call
Disallow: /archivesearch?
Disallow: /archivesearch/url
Disallow: /archivesearch/advanced_search
Disallow: /base/reportbadoffer
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /codesearch?
Disallow: /codesearch/feeds/search?
Disallow: /wapsearch?
Disallow: /safebrowsing
Allow: /safebrowsing/diagnostic
Allow: /safebrowsing/report_badware/
Allow: /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/
Disallow: /reviews/search?
Disallow: /orkut/albums
Allow: /jsapi
Disallow: /views?
Disallow: /c/
Disallow: /cbk
Allow: /cbk?output=tile&cb_client=maps_sv
Disallow: /kh
Disallow: /vt
Disallow: /maps/vt
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /translate_a/
Disallow: /translate_c
Disallow: /translate_f
Disallow: /translate_static/
Disallow: /translate_suggestion
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/home
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /adwordsresellers
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /toolkit/
Allow: /toolkit/*.html
Disallow: /globalmarketfinder/
Allow: /globalmarketfinder/*.html
Disallow: /qnasearch?
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /buzz/post
Disallow: /compressiontest/
Disallow: /analytics/reporting/
Disallow: /analytics/admin/
Disallow: /analytics/web/
Disallow: /analytics/feeds/
Disallow: /analytics/settings/
Disallow: /analytics/portal/
Disallow: /analytics/uploads/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /ads/search?
Disallow: /ads/plan/action_plan?
Disallow: /ads/plan/api/
Disallow: /ads/hotels/partners
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /flights/rpc
Disallow: /commercesearch/services/
Disallow: /evaluation/
Disallow: /chrome/browser/mobile/tour
Disallow: /compare/*/apply*
Disallow: /forms/perks/
Disallow: /baraza/*/search
Disallow: /baraza/*/report
Disallow: /shopping/suppliers/search
Disallow: /ct/
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /adwords/proposal
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/reviewer
Disallow: /about/careers/apply/
Disallow: /about/careers/applications/
Disallow: /landing/signout.html
Disallow: /webmasters/sitemaps/ping?
Disallow: /ping?
Allow: /gb/images
Allow: /gb/js
Disallow: /gallery/
Allow: /chromecast/setup$
Allow: /chromecast/setup/$
Disallow: /landing/now/ontap/
Disallow: /dlpage/driveforoffice/thankyou.html
Disallow: /dlpage/driveforoffice/eula.html
Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml
Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml
Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
Sitemap: https://www.google.com/sitemap.xml



네이버에서 사용하는 robots.txt 규칙


User-agent: *

Disallow: /



모든 검색 엔진을 막아버리네요 -_-;;; 하아;;;너무한다





0 0 0 4 0 1 0 0 0 0 0 2 0 0 0 0 0 0 0 0
List of Articles
번호 분류 제목 글쓴이 조회 수 날짜
공지 뉴스 구글 최신 뉴스 file 덕후냥이 1455 2024.12.12
공지 일반 샤오미 BE6500 라우터 실사용 후기 (Wi-Fi 7 + 2.5G 스위치 기능까지 ㄷㄷ) 4 덕후냥이 1020 2025.06.28
공지 🚨(뉴비필독) 전체공지 & 포인트안내 20 무명의덕질 28944 2024.11.04
공지 URL만 붙여넣으면 끝! 임베드 기능 무명의덕질 23054 2025.01.21
10641 일반 3900x에서 573D 업글은 무의미한짓이군요 file 덕후냥이 1001 2025.06.28
10640 일반 수냉쿨러 질문드립니다. 덕후냥이 1031 2025.06.28
10639 일반 CPU 시피유에 좋은 램속도를 알아봅시다 file 덕후냥이 1039 2025.06.28
10638 일반 샤오미 BE6500 라우터 실사용 후기 (Wi-Fi 7 + 2.5G 스위치 기능까지 ㄷㄷ) 4 덕후냥이 1020 2025.06.28
10637 일반 이제 로지텍 마우스는 더 이상 못 사겠네요. 덕후냥이 1008 2025.06.26
10636 일반 Windows 95를 SSD에 설치하면 생기는 일 1 file 덕후냥이 852 2025.06.23
10635 일반 라이젠3600이랑 5600 똥값됐네 덕후냥이 1418 2025.06.21
10634 일반 ipfilter 다운로드 및 사용법 (ip 토렌트 노출 차단) file 덕후냥이 1642 2025.06.20
10633 정보 토렌트 최신 트래커 모음 (2025년 06월) file 덕후냥이 1687 2025.06.20
10632 일반 윈도우10이 곧 지원 종료 예정 (윈도우11 PowerToys 설치해보세) file 덕후냥이 423 2025.06.10
10631 일반 윈도우에서 바로가기 버튼으로 모니터 화면 자동 전환(가로/세로) 방법 덕후냥이 573 2025.06.05
10630 일반 아 진짜 요새 SKT 해킹 뭐시기 때문에 신경 쓰여 죽겠어 ㅠㅠ 2 덕후냥이 301 2025.05.20
10629 일반 저렴한 중국산 외장 SSD의 실체 file 덕후냥이 578 2025.05.17
10628 일반 인텔 애로우레이크의 CPU 내부 다이 모습 file 덕후냥이 650 2025.05.10
10627 일반 스카이프, 오늘 서비스 종료 file 덕후냥이 655 2025.05.10
10626 일반 SKT "보상 시 고객도 입증책임" file 덕후냥이 673 2025.05.10
10625 정보 KISA, 신규 악성코드 위협 공지...SKT 내부 서버에서 ‘BPF도어’ 악성코드 변종 8종 추... 덕후냥이 640 2025.05.10
10624 일반 “SK하이닉스 내부자에 뚫렸다” 中이직 앞두고 1만여건 기술 유출 file 덕후냥이 616 2025.05.10
10623 일반 기가바이트, 12년만에 메인보드에 새로운 기능을 추가 file 덕후냥이 646 2025.05.10
10622 일반 SKT “위약금 면제 시 수백만명 해지… 분위기 휩쓸려 시장 대혼란” file 덕후냥이 633 2025.05.10
Board Pagination Prev 1 2 3 4 5 ... 533 Next
/ 533