id sid tid token lemma pos en-wikipedia-org-3679 1 1 Robots robot NNS en-wikipedia-org-3679 1 2 exclusion exclusion NN en-wikipedia-org-3679 1 3 standard standard JJ en-wikipedia-org-3679 1 4 - - HYPH en-wikipedia-org-3679 1 5 Wikipedia wikipedia NN en-wikipedia-org-3679 1 6 Robots Robots NNP en-wikipedia-org-3679 1 7 exclusion exclusion NN en-wikipedia-org-3679 1 8 standard standard NN en-wikipedia-org-3679 1 9 From from IN en-wikipedia-org-3679 1 10 Wikipedia Wikipedia NNP en-wikipedia-org-3679 1 11 , , , en-wikipedia-org-3679 1 12 the the DT en-wikipedia-org-3679 1 13 free free JJ en-wikipedia-org-3679 1 14 encyclopedia encyclopedia NN en-wikipedia-org-3679 1 15 This this DT en-wikipedia-org-3679 1 16 is be VBZ en-wikipedia-org-3679 1 17 the the DT en-wikipedia-org-3679 1 18 latest late JJS en-wikipedia-org-3679 1 19 accepted accepted JJ en-wikipedia-org-3679 1 20 revision revision NN en-wikipedia-org-3679 1 21 , , , en-wikipedia-org-3679 1 22 reviewed review VBN en-wikipedia-org-3679 1 23 on on IN en-wikipedia-org-3679 1 24 29 29 CD en-wikipedia-org-3679 1 25 November November NNP en-wikipedia-org-3679 1 26 2020 2020 CD en-wikipedia-org-3679 1 27 . . . en-wikipedia-org-3679 2 1 Jump jump VB en-wikipedia-org-3679 2 2 to to IN en-wikipedia-org-3679 2 3 navigation navigation NN en-wikipedia-org-3679 2 4 Jump Jump NNP en-wikipedia-org-3679 2 5 to to TO en-wikipedia-org-3679 2 6 search search VB en-wikipedia-org-3679 2 7 Standard Standard NNP en-wikipedia-org-3679 2 8 used use VBN en-wikipedia-org-3679 2 9 to to TO en-wikipedia-org-3679 2 10 advise advise VB en-wikipedia-org-3679 2 11 web web NN en-wikipedia-org-3679 2 12 crawlers crawler NNS en-wikipedia-org-3679 2 13 and and CC en-wikipedia-org-3679 2 14 scrapers scraper NNS en-wikipedia-org-3679 2 15 not not RB en-wikipedia-org-3679 2 16 to to TO en-wikipedia-org-3679 2 17 index index VB en-wikipedia-org-3679 2 18 a a DT en-wikipedia-org-3679 2 19 web web NN en-wikipedia-org-3679 2 20 page page NN en-wikipedia-org-3679 2 21 or or CC en-wikipedia-org-3679 2 22 site site NN en-wikipedia-org-3679 2 23 The the DT en-wikipedia-org-3679 2 24 robots robot NNS en-wikipedia-org-3679 2 25 exclusion exclusion NN en-wikipedia-org-3679 2 26 standard standard NN en-wikipedia-org-3679 2 27 , , , en-wikipedia-org-3679 2 28 also also RB en-wikipedia-org-3679 2 29 known know VBN en-wikipedia-org-3679 2 30 as as IN en-wikipedia-org-3679 2 31 the the DT en-wikipedia-org-3679 2 32 robots robot NNS en-wikipedia-org-3679 2 33 exclusion exclusion NN en-wikipedia-org-3679 2 34 protocol protocol NN en-wikipedia-org-3679 2 35 or or CC en-wikipedia-org-3679 2 36 simply simply RB en-wikipedia-org-3679 2 37 robots.txt robots.txt CD en-wikipedia-org-3679 2 38 , , , en-wikipedia-org-3679 2 39 is be VBZ en-wikipedia-org-3679 2 40 a a DT en-wikipedia-org-3679 2 41 standard standard NN en-wikipedia-org-3679 2 42 used use VBN en-wikipedia-org-3679 2 43 by by IN en-wikipedia-org-3679 2 44 websites website NNS en-wikipedia-org-3679 2 45 to to TO en-wikipedia-org-3679 2 46 communicate communicate VB en-wikipedia-org-3679 2 47 with with IN en-wikipedia-org-3679 2 48 web web NN en-wikipedia-org-3679 2 49 crawlers crawler NNS en-wikipedia-org-3679 2 50 and and CC en-wikipedia-org-3679 2 51 other other JJ en-wikipedia-org-3679 2 52 web web NN en-wikipedia-org-3679 2 53 robots robot NNS en-wikipedia-org-3679 2 54 . . . en-wikipedia-org-3679 3 1 The the DT en-wikipedia-org-3679 3 2 standard standard JJ en-wikipedia-org-3679 3 3 specifies specify VBZ en-wikipedia-org-3679 3 4 how how WRB en-wikipedia-org-3679 3 5 to to TO en-wikipedia-org-3679 3 6 inform inform VB en-wikipedia-org-3679 3 7 the the DT en-wikipedia-org-3679 3 8 web web NN en-wikipedia-org-3679 3 9 robot robot NN en-wikipedia-org-3679 3 10 about about IN en-wikipedia-org-3679 3 11 which which WDT en-wikipedia-org-3679 3 12 areas area NNS en-wikipedia-org-3679 3 13 of of IN en-wikipedia-org-3679 3 14 the the DT en-wikipedia-org-3679 3 15 website website NN en-wikipedia-org-3679 3 16 should should MD en-wikipedia-org-3679 3 17 not not RB en-wikipedia-org-3679 3 18 be be VB en-wikipedia-org-3679 3 19 processed process VBN en-wikipedia-org-3679 3 20 or or CC en-wikipedia-org-3679 3 21 scanned scan VBN en-wikipedia-org-3679 3 22 . . . en-wikipedia-org-3679 4 1 Robots robot NNS en-wikipedia-org-3679 4 2 are be VBP en-wikipedia-org-3679 4 3 often often RB en-wikipedia-org-3679 4 4 used use VBN en-wikipedia-org-3679 4 5 by by IN en-wikipedia-org-3679 4 6 search search NN en-wikipedia-org-3679 4 7 engines engine NNS en-wikipedia-org-3679 4 8 to to TO en-wikipedia-org-3679 4 9 categorize categorize VB en-wikipedia-org-3679 4 10 websites website NNS en-wikipedia-org-3679 4 11 . . . en-wikipedia-org-3679 5 1 Not not RB en-wikipedia-org-3679 5 2 all all DT en-wikipedia-org-3679 5 3 robots robot NNS en-wikipedia-org-3679 5 4 cooperate cooperate VBP en-wikipedia-org-3679 5 5 with with IN en-wikipedia-org-3679 5 6 the the DT en-wikipedia-org-3679 5 7 standard standard JJ en-wikipedia-org-3679 5 8 ; ; : en-wikipedia-org-3679 5 9 email email NN en-wikipedia-org-3679 5 10 harvesters harvester NNS en-wikipedia-org-3679 5 11 , , , en-wikipedia-org-3679 5 12 spambots spambot NNS en-wikipedia-org-3679 5 13 , , , en-wikipedia-org-3679 5 14 malware malware NNP en-wikipedia-org-3679 5 15 and and CC en-wikipedia-org-3679 5 16 robots robot NNS en-wikipedia-org-3679 5 17 that that WDT en-wikipedia-org-3679 5 18 scan scan VBP en-wikipedia-org-3679 5 19 for for IN en-wikipedia-org-3679 5 20 security security NN en-wikipedia-org-3679 5 21 vulnerabilities vulnerability NNS en-wikipedia-org-3679 5 22 may may MD en-wikipedia-org-3679 5 23 even even RB en-wikipedia-org-3679 5 24 start start VB en-wikipedia-org-3679 5 25 with with IN en-wikipedia-org-3679 5 26 the the DT en-wikipedia-org-3679 5 27 portions portion NNS en-wikipedia-org-3679 5 28 of of IN en-wikipedia-org-3679 5 29 the the DT en-wikipedia-org-3679 5 30 website website NN en-wikipedia-org-3679 5 31 where where WRB en-wikipedia-org-3679 5 32 they -PRON- PRP en-wikipedia-org-3679 5 33 have have VBP en-wikipedia-org-3679 5 34 been be VBN en-wikipedia-org-3679 5 35 told tell VBN en-wikipedia-org-3679 5 36 to to TO en-wikipedia-org-3679 5 37 stay stay VB en-wikipedia-org-3679 5 38 out out RP en-wikipedia-org-3679 5 39 . . . en-wikipedia-org-3679 6 1 The the DT en-wikipedia-org-3679 6 2 standard standard NN en-wikipedia-org-3679 6 3 can can MD en-wikipedia-org-3679 6 4 be be VB en-wikipedia-org-3679 6 5 used use VBN en-wikipedia-org-3679 6 6 in in IN en-wikipedia-org-3679 6 7 conjunction conjunction NN en-wikipedia-org-3679 6 8 with with IN en-wikipedia-org-3679 6 9 Sitemaps Sitemaps NNP en-wikipedia-org-3679 6 10 , , , en-wikipedia-org-3679 6 11 a a DT en-wikipedia-org-3679 6 12 robot robot NN en-wikipedia-org-3679 6 13 inclusion inclusion NN en-wikipedia-org-3679 6 14 standard standard NN en-wikipedia-org-3679 6 15 for for IN en-wikipedia-org-3679 6 16 websites website NNS en-wikipedia-org-3679 6 17 . . . en-wikipedia-org-3679 7 1 Contents content NNS en-wikipedia-org-3679 7 2 1 1 CD en-wikipedia-org-3679 7 3 History history NN en-wikipedia-org-3679 7 4 2 2 CD en-wikipedia-org-3679 7 5 Standard standard NN en-wikipedia-org-3679 7 6 3 3 CD en-wikipedia-org-3679 7 7 Security Security NNP en-wikipedia-org-3679 7 8 4 4 CD en-wikipedia-org-3679 7 9 Alternatives Alternatives NNP en-wikipedia-org-3679 7 10 5 5 CD en-wikipedia-org-3679 7 11 Examples Examples NNP en-wikipedia-org-3679 7 12 6 6 CD en-wikipedia-org-3679 7 13 Nonstandard Nonstandard NNP en-wikipedia-org-3679 7 14 extensions extension NNS en-wikipedia-org-3679 7 15 6.1 6.1 CD en-wikipedia-org-3679 7 16 Crawl crawl NN en-wikipedia-org-3679 7 17 - - HYPH en-wikipedia-org-3679 7 18 delay delay NNP en-wikipedia-org-3679 7 19 directive directive VBP en-wikipedia-org-3679 7 20 6.2 6.2 CD en-wikipedia-org-3679 7 21 Allow Allow NNP en-wikipedia-org-3679 7 22 directive directive NN en-wikipedia-org-3679 7 23 6.3 6.3 CD en-wikipedia-org-3679 7 24 Sitemap Sitemap NNP en-wikipedia-org-3679 7 25 6.4 6.4 CD en-wikipedia-org-3679 7 26 Host host NN en-wikipedia-org-3679 7 27 6.5 6.5 CD en-wikipedia-org-3679 7 28 Universal Universal NNP en-wikipedia-org-3679 7 29 " " '' en-wikipedia-org-3679 7 30 * * NN en-wikipedia-org-3679 7 31 " " '' en-wikipedia-org-3679 7 32 match match VBP en-wikipedia-org-3679 7 33 7 7 CD en-wikipedia-org-3679 7 34 Meta meta JJ en-wikipedia-org-3679 7 35 tags tag NNS en-wikipedia-org-3679 7 36 and and CC en-wikipedia-org-3679 7 37 headers header VBZ en-wikipedia-org-3679 7 38 8 8 CD en-wikipedia-org-3679 7 39 See see VB en-wikipedia-org-3679 7 40 also also RB en-wikipedia-org-3679 7 41 9 9 CD en-wikipedia-org-3679 7 42 References reference NNS en-wikipedia-org-3679 7 43 10 10 CD en-wikipedia-org-3679 7 44 External external JJ en-wikipedia-org-3679 7 45 links link NNS en-wikipedia-org-3679 7 46 History[edit history[edit VBP en-wikipedia-org-3679 7 47 ] ] -RRB- en-wikipedia-org-3679 7 48 The the DT en-wikipedia-org-3679 7 49 standard standard NN en-wikipedia-org-3679 7 50 was be VBD en-wikipedia-org-3679 7 51 proposed propose VBN en-wikipedia-org-3679 7 52 by by IN en-wikipedia-org-3679 7 53 Martijn Martijn NNP en-wikipedia-org-3679 7 54 Koster,[1][2 Koster,[1][2 NNP en-wikipedia-org-3679 7 55 ] ] -RRB- en-wikipedia-org-3679 7 56 when when WRB en-wikipedia-org-3679 7 57 working work VBG en-wikipedia-org-3679 7 58 for for IN en-wikipedia-org-3679 7 59 Nexor[3 nexor[3 NN en-wikipedia-org-3679 7 60 ] ] -RRB- en-wikipedia-org-3679 7 61 in in IN en-wikipedia-org-3679 7 62 February February NNP en-wikipedia-org-3679 7 63 1994[4 1994[4 NNP en-wikipedia-org-3679 7 64 ] ] -RRB- en-wikipedia-org-3679 7 65 on on IN en-wikipedia-org-3679 7 66 the the DT en-wikipedia-org-3679 7 67 www www NN en-wikipedia-org-3679 7 68 - - HYPH en-wikipedia-org-3679 7 69 talk talk NN en-wikipedia-org-3679 7 70 mailing mailing NN en-wikipedia-org-3679 7 71 list list NN en-wikipedia-org-3679 7 72 , , , en-wikipedia-org-3679 7 73 the the DT en-wikipedia-org-3679 7 74 main main JJ en-wikipedia-org-3679 7 75 communication communication NN en-wikipedia-org-3679 7 76 channel channel NN en-wikipedia-org-3679 7 77 for for IN en-wikipedia-org-3679 7 78 WWW www NN en-wikipedia-org-3679 7 79 - - HYPH en-wikipedia-org-3679 7 80 related relate VBN en-wikipedia-org-3679 7 81 activities activity NNS en-wikipedia-org-3679 7 82 at at IN en-wikipedia-org-3679 7 83 the the DT en-wikipedia-org-3679 7 84 time time NN en-wikipedia-org-3679 7 85 . . . en-wikipedia-org-3679 8 1 Charles Charles NNP en-wikipedia-org-3679 8 2 Stross Stross NNP en-wikipedia-org-3679 8 3 claims claim VBZ en-wikipedia-org-3679 8 4 to to TO en-wikipedia-org-3679 8 5 have have VB en-wikipedia-org-3679 8 6 provoked provoke VBN en-wikipedia-org-3679 8 7 Koster Koster NNP en-wikipedia-org-3679 8 8 to to TO en-wikipedia-org-3679 8 9 suggest suggest VB en-wikipedia-org-3679 8 10 robots.txt robots.txt CD en-wikipedia-org-3679 8 11 , , , en-wikipedia-org-3679 8 12 after after IN en-wikipedia-org-3679 8 13 he -PRON- PRP en-wikipedia-org-3679 8 14 wrote write VBD en-wikipedia-org-3679 8 15 a a DT en-wikipedia-org-3679 8 16 badly badly RB en-wikipedia-org-3679 8 17 - - HYPH en-wikipedia-org-3679 8 18 behaved behave VBN en-wikipedia-org-3679 8 19 web web NN en-wikipedia-org-3679 8 20 crawler crawler NN en-wikipedia-org-3679 8 21 that that WDT en-wikipedia-org-3679 8 22 inadvertently inadvertently RB en-wikipedia-org-3679 8 23 caused cause VBD en-wikipedia-org-3679 8 24 a a DT en-wikipedia-org-3679 8 25 denial denial JJ en-wikipedia-org-3679 8 26 - - HYPH en-wikipedia-org-3679 8 27 of of IN en-wikipedia-org-3679 8 28 - - HYPH en-wikipedia-org-3679 8 29 service service NN en-wikipedia-org-3679 8 30 attack attack NN en-wikipedia-org-3679 8 31 on on IN en-wikipedia-org-3679 8 32 Koster Koster NNP en-wikipedia-org-3679 8 33 's 's POS en-wikipedia-org-3679 8 34 server server NN en-wikipedia-org-3679 8 35 . . . en-wikipedia-org-3679 9 1 [ [ -LRB- en-wikipedia-org-3679 9 2 5 5 CD en-wikipedia-org-3679 9 3 ] ] -RRB- en-wikipedia-org-3679 9 4 It -PRON- PRP en-wikipedia-org-3679 9 5 quickly quickly RB en-wikipedia-org-3679 9 6 became become VBD en-wikipedia-org-3679 9 7 a a DT en-wikipedia-org-3679 9 8 de de FW en-wikipedia-org-3679 9 9 facto facto FW en-wikipedia-org-3679 9 10 standard standard NN en-wikipedia-org-3679 9 11 that that IN en-wikipedia-org-3679 9 12 present present JJ en-wikipedia-org-3679 9 13 and and CC en-wikipedia-org-3679 9 14 future future JJ en-wikipedia-org-3679 9 15 web web NN en-wikipedia-org-3679 9 16 crawlers crawler NNS en-wikipedia-org-3679 9 17 were be VBD en-wikipedia-org-3679 9 18 expected expect VBN en-wikipedia-org-3679 9 19 to to TO en-wikipedia-org-3679 9 20 follow follow VB en-wikipedia-org-3679 9 21 ; ; : en-wikipedia-org-3679 9 22 most most RBS en-wikipedia-org-3679 9 23 complied complied JJ en-wikipedia-org-3679 9 24 , , , en-wikipedia-org-3679 9 25 including include VBG en-wikipedia-org-3679 9 26 those those DT en-wikipedia-org-3679 9 27 operated operate VBN en-wikipedia-org-3679 9 28 by by IN en-wikipedia-org-3679 9 29 search search NN en-wikipedia-org-3679 9 30 engines engine NNS en-wikipedia-org-3679 9 31 such such JJ en-wikipedia-org-3679 9 32 as as IN en-wikipedia-org-3679 9 33 WebCrawler WebCrawler NNP en-wikipedia-org-3679 9 34 , , , en-wikipedia-org-3679 9 35 Lycos Lycos NNP en-wikipedia-org-3679 9 36 , , , en-wikipedia-org-3679 9 37 and and CC en-wikipedia-org-3679 9 38 AltaVista AltaVista NNP en-wikipedia-org-3679 9 39 . . . en-wikipedia-org-3679 10 1 [ [ -LRB- en-wikipedia-org-3679 10 2 6 6 CD en-wikipedia-org-3679 10 3 ] ] -RRB- en-wikipedia-org-3679 10 4 On on IN en-wikipedia-org-3679 10 5 July July NNP en-wikipedia-org-3679 10 6 1 1 CD en-wikipedia-org-3679 10 7 , , , en-wikipedia-org-3679 10 8 2019 2019 CD en-wikipedia-org-3679 10 9 Google Google NNP en-wikipedia-org-3679 10 10 announced[7 announced[7 NNP en-wikipedia-org-3679 10 11 ] ] -RRB- en-wikipedia-org-3679 10 12 the the DT en-wikipedia-org-3679 10 13 proposal proposal NN en-wikipedia-org-3679 10 14 of of IN en-wikipedia-org-3679 10 15 the the DT en-wikipedia-org-3679 10 16 Robots Robots NNP en-wikipedia-org-3679 10 17 Exclusion Exclusion NNP en-wikipedia-org-3679 10 18 Protocol Protocol NNP en-wikipedia-org-3679 10 19 as as IN en-wikipedia-org-3679 10 20 an an DT en-wikipedia-org-3679 10 21 official official JJ en-wikipedia-org-3679 10 22 standard standard NN en-wikipedia-org-3679 10 23 under under IN en-wikipedia-org-3679 10 24 Internet Internet NNP en-wikipedia-org-3679 10 25 Engineering Engineering NNP en-wikipedia-org-3679 10 26 Task Task NNP en-wikipedia-org-3679 10 27 Force Force NNP en-wikipedia-org-3679 10 28 . . . en-wikipedia-org-3679 11 1 The the DT en-wikipedia-org-3679 11 2 draft[8 draft[8 NNPS en-wikipedia-org-3679 11 3 ] ] -RRB- en-wikipedia-org-3679 11 4 will will MD en-wikipedia-org-3679 11 5 now now RB en-wikipedia-org-3679 11 6 undergo undergo VB en-wikipedia-org-3679 11 7 acceptance acceptance NN en-wikipedia-org-3679 11 8 process process NN en-wikipedia-org-3679 11 9 . . . en-wikipedia-org-3679 12 1 Standard[edit standard[edit LS en-wikipedia-org-3679 12 2 ] ] -RRB- en-wikipedia-org-3679 12 3 When when WRB en-wikipedia-org-3679 12 4 a a DT en-wikipedia-org-3679 12 5 site site NN en-wikipedia-org-3679 12 6 owner owner NN en-wikipedia-org-3679 12 7 wishes wish VBZ en-wikipedia-org-3679 12 8 to to TO en-wikipedia-org-3679 12 9 give give VB en-wikipedia-org-3679 12 10 instructions instruction NNS en-wikipedia-org-3679 12 11 to to IN en-wikipedia-org-3679 12 12 web web NN en-wikipedia-org-3679 12 13 robots robot NNS en-wikipedia-org-3679 12 14 they -PRON- PRP en-wikipedia-org-3679 12 15 place place VBP en-wikipedia-org-3679 12 16 a a DT en-wikipedia-org-3679 12 17 text text NN en-wikipedia-org-3679 12 18 file file NN en-wikipedia-org-3679 12 19 called call VBD en-wikipedia-org-3679 12 20 robots.txt robots.txt NN en-wikipedia-org-3679 12 21 in in IN en-wikipedia-org-3679 12 22 the the DT en-wikipedia-org-3679 12 23 root root NN en-wikipedia-org-3679 12 24 of of IN en-wikipedia-org-3679 12 25 the the DT en-wikipedia-org-3679 12 26 web web NN en-wikipedia-org-3679 12 27 site site NN en-wikipedia-org-3679 12 28 hierarchy hierarchy NN en-wikipedia-org-3679 12 29 ( ( -LRB- en-wikipedia-org-3679 12 30 e.g. e.g. RB en-wikipedia-org-3679 13 1 https://www.example.com/robots.txt https://www.example.com/robots.txt NNP en-wikipedia-org-3679 13 2 ) ) -RRB- en-wikipedia-org-3679 13 3 . . . en-wikipedia-org-3679 14 1 This this DT en-wikipedia-org-3679 14 2 text text NN en-wikipedia-org-3679 14 3 file file NN en-wikipedia-org-3679 14 4 contains contain VBZ en-wikipedia-org-3679 14 5 the the DT en-wikipedia-org-3679 14 6 instructions instruction NNS en-wikipedia-org-3679 14 7 in in IN en-wikipedia-org-3679 14 8 a a DT en-wikipedia-org-3679 14 9 specific specific JJ en-wikipedia-org-3679 14 10 format format NN en-wikipedia-org-3679 14 11 ( ( -LRB- en-wikipedia-org-3679 14 12 see see VB en-wikipedia-org-3679 14 13 examples example NNS en-wikipedia-org-3679 14 14 below below RB en-wikipedia-org-3679 14 15 ) ) -RRB- en-wikipedia-org-3679 14 16 . . . en-wikipedia-org-3679 15 1 Robots robot NNS en-wikipedia-org-3679 15 2 that that WDT en-wikipedia-org-3679 15 3 choose choose VBP en-wikipedia-org-3679 15 4 to to TO en-wikipedia-org-3679 15 5 follow follow VB en-wikipedia-org-3679 15 6 the the DT en-wikipedia-org-3679 15 7 instructions instruction NNS en-wikipedia-org-3679 15 8 try try VBP en-wikipedia-org-3679 15 9 to to TO en-wikipedia-org-3679 15 10 fetch fetch VB en-wikipedia-org-3679 15 11 this this DT en-wikipedia-org-3679 15 12 file file NN en-wikipedia-org-3679 15 13 and and CC en-wikipedia-org-3679 15 14 read read VB en-wikipedia-org-3679 15 15 the the DT en-wikipedia-org-3679 15 16 instructions instruction NNS en-wikipedia-org-3679 15 17 before before IN en-wikipedia-org-3679 15 18 fetching fetch VBG en-wikipedia-org-3679 15 19 any any DT en-wikipedia-org-3679 15 20 other other JJ en-wikipedia-org-3679 15 21 file file NN en-wikipedia-org-3679 15 22 from from IN en-wikipedia-org-3679 15 23 the the DT en-wikipedia-org-3679 15 24 website website NN en-wikipedia-org-3679 15 25 . . . en-wikipedia-org-3679 16 1 If if IN en-wikipedia-org-3679 16 2 this this DT en-wikipedia-org-3679 16 3 file file NN en-wikipedia-org-3679 16 4 does do VBZ en-wikipedia-org-3679 16 5 n't not RB en-wikipedia-org-3679 16 6 exist exist VB en-wikipedia-org-3679 16 7 , , , en-wikipedia-org-3679 16 8 web web NN en-wikipedia-org-3679 16 9 robots robot NNS en-wikipedia-org-3679 16 10 assume assume VBP en-wikipedia-org-3679 16 11 that that IN en-wikipedia-org-3679 16 12 the the DT en-wikipedia-org-3679 16 13 website website NN en-wikipedia-org-3679 16 14 owner owner NN en-wikipedia-org-3679 16 15 does do VBZ en-wikipedia-org-3679 16 16 not not RB en-wikipedia-org-3679 16 17 wish wish VB en-wikipedia-org-3679 16 18 to to TO en-wikipedia-org-3679 16 19 place place VB en-wikipedia-org-3679 16 20 any any DT en-wikipedia-org-3679 16 21 limitations limitation NNS en-wikipedia-org-3679 16 22 on on IN en-wikipedia-org-3679 16 23 crawling crawl VBG en-wikipedia-org-3679 16 24 the the DT en-wikipedia-org-3679 16 25 entire entire JJ en-wikipedia-org-3679 16 26 site site NN en-wikipedia-org-3679 16 27 . . . en-wikipedia-org-3679 17 1 A a DT en-wikipedia-org-3679 17 2 robots.txt robots.txt CD en-wikipedia-org-3679 17 3 file file NN en-wikipedia-org-3679 17 4 on on IN en-wikipedia-org-3679 17 5 a a DT en-wikipedia-org-3679 17 6 website website NN en-wikipedia-org-3679 17 7 will will MD en-wikipedia-org-3679 17 8 function function VB en-wikipedia-org-3679 17 9 as as IN en-wikipedia-org-3679 17 10 a a DT en-wikipedia-org-3679 17 11 request request NN en-wikipedia-org-3679 17 12 that that WDT en-wikipedia-org-3679 17 13 specified specify VBD en-wikipedia-org-3679 17 14 robots robot NNS en-wikipedia-org-3679 17 15 ignore ignore VBP en-wikipedia-org-3679 17 16 specified specified JJ en-wikipedia-org-3679 17 17 files file NNS en-wikipedia-org-3679 17 18 or or CC en-wikipedia-org-3679 17 19 directories directory NNS en-wikipedia-org-3679 17 20 when when WRB en-wikipedia-org-3679 17 21 crawling crawl VBG en-wikipedia-org-3679 17 22 a a DT en-wikipedia-org-3679 17 23 site site NN en-wikipedia-org-3679 17 24 . . . en-wikipedia-org-3679 18 1 This this DT en-wikipedia-org-3679 18 2 might may MD en-wikipedia-org-3679 18 3 be be VB en-wikipedia-org-3679 18 4 , , , en-wikipedia-org-3679 18 5 for for IN en-wikipedia-org-3679 18 6 example example NN en-wikipedia-org-3679 18 7 , , , en-wikipedia-org-3679 18 8 out out IN en-wikipedia-org-3679 18 9 of of IN en-wikipedia-org-3679 18 10 a a DT en-wikipedia-org-3679 18 11 preference preference NN en-wikipedia-org-3679 18 12 for for IN en-wikipedia-org-3679 18 13 privacy privacy NN en-wikipedia-org-3679 18 14 from from IN en-wikipedia-org-3679 18 15 search search NN en-wikipedia-org-3679 18 16 engine engine NN en-wikipedia-org-3679 18 17 results result NNS en-wikipedia-org-3679 18 18 , , , en-wikipedia-org-3679 18 19 or or CC en-wikipedia-org-3679 18 20 the the DT en-wikipedia-org-3679 18 21 belief belief NN en-wikipedia-org-3679 18 22 that that IN en-wikipedia-org-3679 18 23 the the DT en-wikipedia-org-3679 18 24 content content NN en-wikipedia-org-3679 18 25 of of IN en-wikipedia-org-3679 18 26 the the DT en-wikipedia-org-3679 18 27 selected select VBN en-wikipedia-org-3679 18 28 directories directory NNS en-wikipedia-org-3679 18 29 might may MD en-wikipedia-org-3679 18 30 be be VB en-wikipedia-org-3679 18 31 misleading misleading JJ en-wikipedia-org-3679 18 32 or or CC en-wikipedia-org-3679 18 33 irrelevant irrelevant JJ en-wikipedia-org-3679 18 34 to to IN en-wikipedia-org-3679 18 35 the the DT en-wikipedia-org-3679 18 36 categorization categorization NN en-wikipedia-org-3679 18 37 of of IN en-wikipedia-org-3679 18 38 the the DT en-wikipedia-org-3679 18 39 site site NN en-wikipedia-org-3679 18 40 as as IN en-wikipedia-org-3679 18 41 a a DT en-wikipedia-org-3679 18 42 whole whole NN en-wikipedia-org-3679 18 43 , , , en-wikipedia-org-3679 18 44 or or CC en-wikipedia-org-3679 18 45 out out IN en-wikipedia-org-3679 18 46 of of IN en-wikipedia-org-3679 18 47 a a DT en-wikipedia-org-3679 18 48 desire desire NN en-wikipedia-org-3679 18 49 that that IN en-wikipedia-org-3679 18 50 an an DT en-wikipedia-org-3679 18 51 application application NN en-wikipedia-org-3679 18 52 only only RB en-wikipedia-org-3679 18 53 operates operate VBZ en-wikipedia-org-3679 18 54 on on IN en-wikipedia-org-3679 18 55 certain certain JJ en-wikipedia-org-3679 18 56 data datum NNS en-wikipedia-org-3679 18 57 . . . en-wikipedia-org-3679 19 1 Links link NNS en-wikipedia-org-3679 19 2 to to IN en-wikipedia-org-3679 19 3 pages page NNS en-wikipedia-org-3679 19 4 listed list VBN en-wikipedia-org-3679 19 5 in in IN en-wikipedia-org-3679 19 6 robots.txt robots.txt NN en-wikipedia-org-3679 19 7 can can MD en-wikipedia-org-3679 19 8 still still RB en-wikipedia-org-3679 19 9 appear appear VB en-wikipedia-org-3679 19 10 in in IN en-wikipedia-org-3679 19 11 search search NN en-wikipedia-org-3679 19 12 results result NNS en-wikipedia-org-3679 19 13 if if IN en-wikipedia-org-3679 19 14 they -PRON- PRP en-wikipedia-org-3679 19 15 are be VBP en-wikipedia-org-3679 19 16 linked link VBN en-wikipedia-org-3679 19 17 to to IN en-wikipedia-org-3679 19 18 from from IN en-wikipedia-org-3679 19 19 a a DT en-wikipedia-org-3679 19 20 page page NN en-wikipedia-org-3679 19 21 that that WDT en-wikipedia-org-3679 19 22 is be VBZ en-wikipedia-org-3679 19 23 crawled crawled JJ en-wikipedia-org-3679 19 24 . . . en-wikipedia-org-3679 20 1 [ [ -LRB- en-wikipedia-org-3679 20 2 9 9 CD en-wikipedia-org-3679 20 3 ] ] -RRB- en-wikipedia-org-3679 20 4 A a DT en-wikipedia-org-3679 20 5 robots.txt robots.txt CD en-wikipedia-org-3679 20 6 file file NN en-wikipedia-org-3679 20 7 covers cover VBZ en-wikipedia-org-3679 20 8 one one CD en-wikipedia-org-3679 20 9 origin origin NN en-wikipedia-org-3679 20 10 . . . en-wikipedia-org-3679 21 1 For for IN en-wikipedia-org-3679 21 2 websites website NNS en-wikipedia-org-3679 21 3 with with IN en-wikipedia-org-3679 21 4 multiple multiple JJ en-wikipedia-org-3679 21 5 subdomains subdomain NNS en-wikipedia-org-3679 21 6 , , , en-wikipedia-org-3679 21 7 each each DT en-wikipedia-org-3679 21 8 subdomain subdomain NN en-wikipedia-org-3679 21 9 must must MD en-wikipedia-org-3679 21 10 have have VB en-wikipedia-org-3679 21 11 its -PRON- PRP$ en-wikipedia-org-3679 21 12 own own JJ en-wikipedia-org-3679 21 13 robots.txt robots.txt NN en-wikipedia-org-3679 21 14 file file NN en-wikipedia-org-3679 21 15 . . . en-wikipedia-org-3679 22 1 If if IN en-wikipedia-org-3679 22 2 example.com example.com ADD en-wikipedia-org-3679 22 3 had have VBD en-wikipedia-org-3679 22 4 a a DT en-wikipedia-org-3679 22 5 robots.txt robots.txt NN en-wikipedia-org-3679 22 6 file file NN en-wikipedia-org-3679 22 7 but but CC en-wikipedia-org-3679 22 8 a.example.com a.example.com NNP en-wikipedia-org-3679 22 9 did do VBD en-wikipedia-org-3679 22 10 not not RB en-wikipedia-org-3679 22 11 , , , en-wikipedia-org-3679 22 12 the the DT en-wikipedia-org-3679 22 13 rules rule NNS en-wikipedia-org-3679 22 14 that that WDT en-wikipedia-org-3679 22 15 would would MD en-wikipedia-org-3679 22 16 apply apply VB en-wikipedia-org-3679 22 17 for for IN en-wikipedia-org-3679 22 18 example.com example.com ADD en-wikipedia-org-3679 22 19 would would MD en-wikipedia-org-3679 22 20 not not RB en-wikipedia-org-3679 22 21 apply apply VB en-wikipedia-org-3679 22 22 to to IN en-wikipedia-org-3679 22 23 a.example.com a.example.com ADD en-wikipedia-org-3679 22 24 . . . en-wikipedia-org-3679 23 1 In in IN en-wikipedia-org-3679 23 2 addition addition NN en-wikipedia-org-3679 23 3 , , , en-wikipedia-org-3679 23 4 each each DT en-wikipedia-org-3679 23 5 protocol protocol NN en-wikipedia-org-3679 23 6 and and CC en-wikipedia-org-3679 23 7 port port NN en-wikipedia-org-3679 23 8 needs need VBZ en-wikipedia-org-3679 23 9 its -PRON- PRP$ en-wikipedia-org-3679 23 10 own own JJ en-wikipedia-org-3679 23 11 robots.txt robots.txt NN en-wikipedia-org-3679 23 12 file file NN en-wikipedia-org-3679 23 13 ; ; : en-wikipedia-org-3679 23 14 http://example.com/robots.txt http://example.com/robots.txt NNP en-wikipedia-org-3679 23 15 does do VBZ en-wikipedia-org-3679 23 16 not not RB en-wikipedia-org-3679 23 17 apply apply VB en-wikipedia-org-3679 23 18 to to IN en-wikipedia-org-3679 23 19 pages page NNS en-wikipedia-org-3679 23 20 under under IN en-wikipedia-org-3679 23 21 http://example.com:8080/ http://example.com:8080/ NN en-wikipedia-org-3679 23 22 or or CC en-wikipedia-org-3679 23 23 https://example.com/. https://example.com/. CD en-wikipedia-org-3679 24 1 Some some DT en-wikipedia-org-3679 24 2 major major JJ en-wikipedia-org-3679 24 3 search search NN en-wikipedia-org-3679 24 4 engines engine NNS en-wikipedia-org-3679 24 5 following follow VBG en-wikipedia-org-3679 24 6 this this DT en-wikipedia-org-3679 24 7 standard standard NN en-wikipedia-org-3679 24 8 include include VBP en-wikipedia-org-3679 24 9 Ask,[10 ask,[10 NN en-wikipedia-org-3679 24 10 ] ] -RRB- en-wikipedia-org-3679 24 11 AOL,[11 aol,[11 NN en-wikipedia-org-3679 24 12 ] ] -RRB- en-wikipedia-org-3679 24 13 Baidu,[12 Baidu,[12 NNP en-wikipedia-org-3679 24 14 ] ] -RRB- en-wikipedia-org-3679 24 15 DuckDuckGo,[13 DuckDuckGo,[13 NNP en-wikipedia-org-3679 24 16 ] ] -RRB- en-wikipedia-org-3679 24 17 Google,[14 Google,[14 NNP en-wikipedia-org-3679 24 18 ] ] -RRB- en-wikipedia-org-3679 24 19 Yahoo!,[15 Yahoo!,[15 NNP en-wikipedia-org-3679 24 20 ] ] -RRB- en-wikipedia-org-3679 24 21 and and CC en-wikipedia-org-3679 24 22 Yandex Yandex NNP en-wikipedia-org-3679 24 23 . . . en-wikipedia-org-3679 25 1 [ [ -LRB- en-wikipedia-org-3679 25 2 16 16 CD en-wikipedia-org-3679 25 3 ] ] -RRB- en-wikipedia-org-3679 25 4 Bing[17 bing[17 NN en-wikipedia-org-3679 25 5 ] ] -RRB- en-wikipedia-org-3679 25 6 still still RB en-wikipedia-org-3679 25 7 is be VBZ en-wikipedia-org-3679 25 8 not not RB en-wikipedia-org-3679 25 9 fully[18 fully[18 RB en-wikipedia-org-3679 25 10 ] ] -RRB- en-wikipedia-org-3679 25 11 compatible compatible JJ en-wikipedia-org-3679 25 12 with with IN en-wikipedia-org-3679 25 13 the the DT en-wikipedia-org-3679 25 14 standard[3 standard[3 NNP en-wikipedia-org-3679 25 15 ] ] -RRB- en-wikipedia-org-3679 25 16 as as IN en-wikipedia-org-3679 25 17 it -PRON- PRP en-wikipedia-org-3679 25 18 can can MD en-wikipedia-org-3679 25 19 not not RB en-wikipedia-org-3679 25 20 inherit inherit VB en-wikipedia-org-3679 25 21 settings setting NNS en-wikipedia-org-3679 25 22 from from IN en-wikipedia-org-3679 25 23 the the DT en-wikipedia-org-3679 25 24 wildcard wildcard NN en-wikipedia-org-3679 25 25 ( ( -LRB- en-wikipedia-org-3679 25 26 * * NFP en-wikipedia-org-3679 25 27 ) ) -RRB- en-wikipedia-org-3679 25 28 . . . en-wikipedia-org-3679 26 1 The the DT en-wikipedia-org-3679 26 2 volunteering volunteering NN en-wikipedia-org-3679 26 3 group group NN en-wikipedia-org-3679 26 4 Archive Archive NNP en-wikipedia-org-3679 26 5 Team Team NNP en-wikipedia-org-3679 26 6 explicitly explicitly RB en-wikipedia-org-3679 26 7 ignores ignore VBZ en-wikipedia-org-3679 26 8 robots.txt robots.txt NNS en-wikipedia-org-3679 26 9 for for IN en-wikipedia-org-3679 26 10 the the DT en-wikipedia-org-3679 26 11 most most JJS en-wikipedia-org-3679 26 12 part part NN en-wikipedia-org-3679 26 13 , , , en-wikipedia-org-3679 26 14 viewing view VBG en-wikipedia-org-3679 26 15 it -PRON- PRP en-wikipedia-org-3679 26 16 as as IN en-wikipedia-org-3679 26 17 an an DT en-wikipedia-org-3679 26 18 obsolete obsolete JJ en-wikipedia-org-3679 26 19 standard standard NN en-wikipedia-org-3679 26 20 that that WDT en-wikipedia-org-3679 26 21 hinders hinder NNS en-wikipedia-org-3679 26 22 web web NN en-wikipedia-org-3679 26 23 archival archival NN en-wikipedia-org-3679 26 24 efforts effort NNS en-wikipedia-org-3679 26 25 . . . en-wikipedia-org-3679 27 1 According accord VBG en-wikipedia-org-3679 27 2 to to IN en-wikipedia-org-3679 27 3 project project NN en-wikipedia-org-3679 27 4 leader leader NN en-wikipedia-org-3679 27 5 Jason Jason NNP en-wikipedia-org-3679 27 6 Scott Scott NNP en-wikipedia-org-3679 27 7 , , , en-wikipedia-org-3679 27 8 " " `` en-wikipedia-org-3679 27 9 unchecked unchecked JJ en-wikipedia-org-3679 27 10 , , , en-wikipedia-org-3679 27 11 and and CC en-wikipedia-org-3679 27 12 left leave VBD en-wikipedia-org-3679 27 13 alone alone RB en-wikipedia-org-3679 27 14 , , , en-wikipedia-org-3679 27 15 the the DT en-wikipedia-org-3679 27 16 robots.txt robots.txt NN en-wikipedia-org-3679 27 17 file file NN en-wikipedia-org-3679 27 18 ensures ensure VBZ en-wikipedia-org-3679 27 19 no no DT en-wikipedia-org-3679 27 20 mirroring mirroring NN en-wikipedia-org-3679 27 21 or or CC en-wikipedia-org-3679 27 22 reference reference NN en-wikipedia-org-3679 27 23 for for IN en-wikipedia-org-3679 27 24 items item NNS en-wikipedia-org-3679 27 25 that that WDT en-wikipedia-org-3679 27 26 may may MD en-wikipedia-org-3679 27 27 have have VB en-wikipedia-org-3679 27 28 general general JJ en-wikipedia-org-3679 27 29 use use NN en-wikipedia-org-3679 27 30 and and CC en-wikipedia-org-3679 27 31 meaning meaning NN en-wikipedia-org-3679 27 32 beyond beyond IN en-wikipedia-org-3679 27 33 the the DT en-wikipedia-org-3679 27 34 website website NN en-wikipedia-org-3679 27 35 's 's POS en-wikipedia-org-3679 27 36 context context NN en-wikipedia-org-3679 27 37 . . . en-wikipedia-org-3679 28 1 " " `` en-wikipedia-org-3679 28 2 [ [ -LRB- en-wikipedia-org-3679 28 3 19 19 CD en-wikipedia-org-3679 28 4 ] ] -RRB- en-wikipedia-org-3679 28 5 For for IN en-wikipedia-org-3679 28 6 some some DT en-wikipedia-org-3679 28 7 years year NNS en-wikipedia-org-3679 28 8 , , , en-wikipedia-org-3679 28 9 the the DT en-wikipedia-org-3679 28 10 Internet Internet NNP en-wikipedia-org-3679 28 11 Archive Archive NNP en-wikipedia-org-3679 28 12 did do VBD en-wikipedia-org-3679 28 13 not not RB en-wikipedia-org-3679 28 14 crawl crawl VB en-wikipedia-org-3679 28 15 sites site NNS en-wikipedia-org-3679 28 16 with with IN en-wikipedia-org-3679 28 17 robots.txt robots.txt NN en-wikipedia-org-3679 28 18 , , , en-wikipedia-org-3679 28 19 but but CC en-wikipedia-org-3679 28 20 in in IN en-wikipedia-org-3679 28 21 April April NNP en-wikipedia-org-3679 28 22 2017 2017 CD en-wikipedia-org-3679 28 23 , , , en-wikipedia-org-3679 28 24 it -PRON- PRP en-wikipedia-org-3679 28 25 announced[20 announced[20 VBP en-wikipedia-org-3679 28 26 ] ] -RRB- en-wikipedia-org-3679 28 27 that that IN en-wikipedia-org-3679 28 28 it -PRON- PRP en-wikipedia-org-3679 28 29 would would MD en-wikipedia-org-3679 28 30 no no RB en-wikipedia-org-3679 28 31 longer long RBR en-wikipedia-org-3679 28 32 honour honour VB en-wikipedia-org-3679 28 33 directives directive NNS en-wikipedia-org-3679 28 34 in in IN en-wikipedia-org-3679 28 35 the the DT en-wikipedia-org-3679 28 36 robots.txt robots.txt NN en-wikipedia-org-3679 28 37 files file NNS en-wikipedia-org-3679 28 38 . . . en-wikipedia-org-3679 29 1 “ " `` en-wikipedia-org-3679 29 2 Over over IN en-wikipedia-org-3679 29 3 time time NN en-wikipedia-org-3679 29 4 we -PRON- PRP en-wikipedia-org-3679 29 5 have have VBP en-wikipedia-org-3679 29 6 observed observe VBN en-wikipedia-org-3679 29 7 that that IN en-wikipedia-org-3679 29 8 the the DT en-wikipedia-org-3679 29 9 robots.txt robots.txt NN en-wikipedia-org-3679 29 10 files file NNS en-wikipedia-org-3679 29 11 that that WDT en-wikipedia-org-3679 29 12 are be VBP en-wikipedia-org-3679 29 13 geared gear VBN en-wikipedia-org-3679 29 14 toward toward IN en-wikipedia-org-3679 29 15 search search NN en-wikipedia-org-3679 29 16 engine engine NN en-wikipedia-org-3679 29 17 crawlers crawler NNS en-wikipedia-org-3679 29 18 do do VBP en-wikipedia-org-3679 29 19 not not RB en-wikipedia-org-3679 29 20 necessarily necessarily RB en-wikipedia-org-3679 29 21 serve serve VB en-wikipedia-org-3679 29 22 our -PRON- PRP$ en-wikipedia-org-3679 29 23 archival archival NN en-wikipedia-org-3679 29 24 purposes purpose NNS en-wikipedia-org-3679 29 25 " " '' en-wikipedia-org-3679 29 26 . . . en-wikipedia-org-3679 30 1 [ [ -LRB- en-wikipedia-org-3679 30 2 21 21 CD en-wikipedia-org-3679 30 3 ] ] -RRB- en-wikipedia-org-3679 30 4 This this DT en-wikipedia-org-3679 30 5 was be VBD en-wikipedia-org-3679 30 6 in in IN en-wikipedia-org-3679 30 7 response response NN en-wikipedia-org-3679 30 8 to to IN en-wikipedia-org-3679 30 9 entire entire JJ en-wikipedia-org-3679 30 10 domains domain NNS en-wikipedia-org-3679 30 11 being be VBG en-wikipedia-org-3679 30 12 tagged tag VBN en-wikipedia-org-3679 30 13 with with IN en-wikipedia-org-3679 30 14 robots.txt robots.txt CD en-wikipedia-org-3679 30 15 when when WRB en-wikipedia-org-3679 30 16 the the DT en-wikipedia-org-3679 30 17 content content NN en-wikipedia-org-3679 30 18 became become VBD en-wikipedia-org-3679 30 19 obsolete obsolete JJ en-wikipedia-org-3679 30 20 . . . en-wikipedia-org-3679 31 1 [ [ -LRB- en-wikipedia-org-3679 31 2 21 21 CD en-wikipedia-org-3679 31 3 ] ] -RRB- en-wikipedia-org-3679 31 4 Security[edit security[edit CD en-wikipedia-org-3679 31 5 ] ] -RRB- en-wikipedia-org-3679 31 6 Despite despite IN en-wikipedia-org-3679 31 7 the the DT en-wikipedia-org-3679 31 8 use use NN en-wikipedia-org-3679 31 9 of of IN en-wikipedia-org-3679 31 10 the the DT en-wikipedia-org-3679 31 11 terms term NNS en-wikipedia-org-3679 31 12 " " `` en-wikipedia-org-3679 31 13 allow allow VBP en-wikipedia-org-3679 31 14 " " '' en-wikipedia-org-3679 31 15 and and CC en-wikipedia-org-3679 31 16 " " `` en-wikipedia-org-3679 31 17 disallow disallow JJ en-wikipedia-org-3679 31 18 " " '' en-wikipedia-org-3679 31 19 , , , en-wikipedia-org-3679 31 20 the the DT en-wikipedia-org-3679 31 21 protocol protocol NN en-wikipedia-org-3679 31 22 is be VBZ en-wikipedia-org-3679 31 23 purely purely RB en-wikipedia-org-3679 31 24 advisory[22 advisory[22 XX en-wikipedia-org-3679 31 25 ] ] -RRB- en-wikipedia-org-3679 31 26 and and CC en-wikipedia-org-3679 31 27 relies rely VBZ en-wikipedia-org-3679 31 28 on on IN en-wikipedia-org-3679 31 29 the the DT en-wikipedia-org-3679 31 30 compliance compliance NN en-wikipedia-org-3679 31 31 of of IN en-wikipedia-org-3679 31 32 the the DT en-wikipedia-org-3679 31 33 web web NN en-wikipedia-org-3679 31 34 robot robot NN en-wikipedia-org-3679 31 35 . . . en-wikipedia-org-3679 32 1 Malicious malicious JJ en-wikipedia-org-3679 32 2 web web NN en-wikipedia-org-3679 32 3 robots robot NNS en-wikipedia-org-3679 32 4 are be VBP en-wikipedia-org-3679 32 5 unlikely unlikely JJ en-wikipedia-org-3679 32 6 to to TO en-wikipedia-org-3679 32 7 honor honor VB en-wikipedia-org-3679 32 8 robots.txt robots.txt NN en-wikipedia-org-3679 32 9 ; ; : en-wikipedia-org-3679 32 10 some some DT en-wikipedia-org-3679 32 11 may may MD en-wikipedia-org-3679 32 12 even even RB en-wikipedia-org-3679 32 13 use use VB en-wikipedia-org-3679 32 14 the the DT en-wikipedia-org-3679 32 15 robots.txt robots.txt NN en-wikipedia-org-3679 32 16 as as IN en-wikipedia-org-3679 32 17 a a DT en-wikipedia-org-3679 32 18 guide guide NN en-wikipedia-org-3679 32 19 to to TO en-wikipedia-org-3679 32 20 find find VB en-wikipedia-org-3679 32 21 disallowed disallow VBN en-wikipedia-org-3679 32 22 links link NNS en-wikipedia-org-3679 32 23 and and CC en-wikipedia-org-3679 32 24 go go VB en-wikipedia-org-3679 32 25 straight straight RB en-wikipedia-org-3679 32 26 to to IN en-wikipedia-org-3679 32 27 them -PRON- PRP en-wikipedia-org-3679 32 28 . . . en-wikipedia-org-3679 33 1 While while IN en-wikipedia-org-3679 33 2 this this DT en-wikipedia-org-3679 33 3 is be VBZ en-wikipedia-org-3679 33 4 sometimes sometimes RB en-wikipedia-org-3679 33 5 claimed claim VBN en-wikipedia-org-3679 33 6 to to TO en-wikipedia-org-3679 33 7 be be VB en-wikipedia-org-3679 33 8 a a DT en-wikipedia-org-3679 33 9 security security NN en-wikipedia-org-3679 33 10 risk,[23 risk,[23 CD en-wikipedia-org-3679 33 11 ] ] -RRB- en-wikipedia-org-3679 33 12 this this DT en-wikipedia-org-3679 33 13 sort sort NN en-wikipedia-org-3679 33 14 of of IN en-wikipedia-org-3679 33 15 security security NN en-wikipedia-org-3679 33 16 through through IN en-wikipedia-org-3679 33 17 obscurity obscurity NN en-wikipedia-org-3679 33 18 is be VBZ en-wikipedia-org-3679 33 19 discouraged discourage VBN en-wikipedia-org-3679 33 20 by by IN en-wikipedia-org-3679 33 21 standards standard NNS en-wikipedia-org-3679 33 22 bodies body NNS en-wikipedia-org-3679 33 23 . . . en-wikipedia-org-3679 34 1 The the DT en-wikipedia-org-3679 34 2 National National NNP en-wikipedia-org-3679 34 3 Institute Institute NNP en-wikipedia-org-3679 34 4 of of IN en-wikipedia-org-3679 34 5 Standards Standards NNPS en-wikipedia-org-3679 34 6 and and CC en-wikipedia-org-3679 34 7 Technology Technology NNP en-wikipedia-org-3679 34 8 ( ( -LRB- en-wikipedia-org-3679 34 9 NIST NIST NNP en-wikipedia-org-3679 34 10 ) ) -RRB- en-wikipedia-org-3679 34 11 in in IN en-wikipedia-org-3679 34 12 the the DT en-wikipedia-org-3679 34 13 United United NNP en-wikipedia-org-3679 34 14 States States NNP en-wikipedia-org-3679 34 15 specifically specifically RB en-wikipedia-org-3679 34 16 recommends recommend VBZ en-wikipedia-org-3679 34 17 against against IN en-wikipedia-org-3679 34 18 this this DT en-wikipedia-org-3679 34 19 practice practice NN en-wikipedia-org-3679 34 20 : : : en-wikipedia-org-3679 34 21 " " `` en-wikipedia-org-3679 34 22 System system NN en-wikipedia-org-3679 34 23 security security NN en-wikipedia-org-3679 34 24 should should MD en-wikipedia-org-3679 34 25 not not RB en-wikipedia-org-3679 34 26 depend depend VB en-wikipedia-org-3679 34 27 on on IN en-wikipedia-org-3679 34 28 the the DT en-wikipedia-org-3679 34 29 secrecy secrecy NN en-wikipedia-org-3679 34 30 of of IN en-wikipedia-org-3679 34 31 the the DT en-wikipedia-org-3679 34 32 implementation implementation NN en-wikipedia-org-3679 34 33 or or CC en-wikipedia-org-3679 34 34 its -PRON- PRP$ en-wikipedia-org-3679 34 35 components component NNS en-wikipedia-org-3679 34 36 . . . en-wikipedia-org-3679 35 1 " " `` en-wikipedia-org-3679 35 2 [ [ -LRB- en-wikipedia-org-3679 35 3 24 24 CD en-wikipedia-org-3679 35 4 ] ] -RRB- en-wikipedia-org-3679 35 5 In in IN en-wikipedia-org-3679 35 6 the the DT en-wikipedia-org-3679 35 7 context context NN en-wikipedia-org-3679 35 8 of of IN en-wikipedia-org-3679 35 9 robots.txt robots.txt JJ en-wikipedia-org-3679 35 10 files file NNS en-wikipedia-org-3679 35 11 , , , en-wikipedia-org-3679 35 12 security security NN en-wikipedia-org-3679 35 13 through through IN en-wikipedia-org-3679 35 14 obscurity obscurity NN en-wikipedia-org-3679 35 15 is be VBZ en-wikipedia-org-3679 35 16 not not RB en-wikipedia-org-3679 35 17 recommended recommend VBN en-wikipedia-org-3679 35 18 as as IN en-wikipedia-org-3679 35 19 a a DT en-wikipedia-org-3679 35 20 security security NN en-wikipedia-org-3679 35 21 technique technique NN en-wikipedia-org-3679 35 22 . . . en-wikipedia-org-3679 36 1 [ [ -LRB- en-wikipedia-org-3679 36 2 25 25 CD en-wikipedia-org-3679 36 3 ] ] -RRB- en-wikipedia-org-3679 36 4 Alternatives[edit Alternatives[edit `` en-wikipedia-org-3679 36 5 ] ] -RRB- en-wikipedia-org-3679 36 6 Many many JJ en-wikipedia-org-3679 36 7 robots robot NNS en-wikipedia-org-3679 36 8 also also RB en-wikipedia-org-3679 36 9 pass pass VBP en-wikipedia-org-3679 36 10 a a DT en-wikipedia-org-3679 36 11 special special JJ en-wikipedia-org-3679 36 12 user user NN en-wikipedia-org-3679 36 13 - - HYPH en-wikipedia-org-3679 36 14 agent agent NN en-wikipedia-org-3679 36 15 to to IN en-wikipedia-org-3679 36 16 the the DT en-wikipedia-org-3679 36 17 web web NN en-wikipedia-org-3679 36 18 server server NN en-wikipedia-org-3679 36 19 when when WRB en-wikipedia-org-3679 36 20 fetching fetch VBG en-wikipedia-org-3679 36 21 content content NN en-wikipedia-org-3679 36 22 . . . en-wikipedia-org-3679 37 1 [ [ -LRB- en-wikipedia-org-3679 37 2 26 26 CD en-wikipedia-org-3679 37 3 ] ] -RRB- en-wikipedia-org-3679 37 4 A a DT en-wikipedia-org-3679 37 5 web web NN en-wikipedia-org-3679 37 6 administrator administrator NN en-wikipedia-org-3679 37 7 could could MD en-wikipedia-org-3679 37 8 also also RB en-wikipedia-org-3679 37 9 configure configure VB en-wikipedia-org-3679 37 10 the the DT en-wikipedia-org-3679 37 11 server server NN en-wikipedia-org-3679 37 12 to to TO en-wikipedia-org-3679 37 13 automatically automatically RB en-wikipedia-org-3679 37 14 return return VB en-wikipedia-org-3679 37 15 failure failure NN en-wikipedia-org-3679 37 16 ( ( -LRB- en-wikipedia-org-3679 37 17 or or CC en-wikipedia-org-3679 37 18 pass pass VB en-wikipedia-org-3679 37 19 alternative alternative JJ en-wikipedia-org-3679 37 20 content content NN en-wikipedia-org-3679 37 21 ) ) -RRB- en-wikipedia-org-3679 37 22 when when WRB en-wikipedia-org-3679 37 23 it -PRON- PRP en-wikipedia-org-3679 37 24 detects detect VBZ en-wikipedia-org-3679 37 25 a a DT en-wikipedia-org-3679 37 26 connection connection NN en-wikipedia-org-3679 37 27 using use VBG en-wikipedia-org-3679 37 28 one one CD en-wikipedia-org-3679 37 29 of of IN en-wikipedia-org-3679 37 30 the the DT en-wikipedia-org-3679 37 31 robots robot NNS en-wikipedia-org-3679 37 32 . . . en-wikipedia-org-3679 38 1 [ [ -LRB- en-wikipedia-org-3679 38 2 27][28 27][28 CD en-wikipedia-org-3679 38 3 ] ] -RRB- en-wikipedia-org-3679 38 4 Some some DT en-wikipedia-org-3679 38 5 sites site NNS en-wikipedia-org-3679 38 6 , , , en-wikipedia-org-3679 38 7 such such JJ en-wikipedia-org-3679 38 8 as as IN en-wikipedia-org-3679 38 9 Google Google NNP en-wikipedia-org-3679 38 10 , , , en-wikipedia-org-3679 38 11 host host VB en-wikipedia-org-3679 38 12 a a DT en-wikipedia-org-3679 38 13 humans.txt humans.txt NNP en-wikipedia-org-3679 38 14 file file NN en-wikipedia-org-3679 38 15 that that WDT en-wikipedia-org-3679 38 16 displays display VBZ en-wikipedia-org-3679 38 17 information information NN en-wikipedia-org-3679 38 18 meant mean VBN en-wikipedia-org-3679 38 19 for for IN en-wikipedia-org-3679 38 20 humans human NNS en-wikipedia-org-3679 38 21 to to TO en-wikipedia-org-3679 38 22 read read VB en-wikipedia-org-3679 38 23 . . . en-wikipedia-org-3679 39 1 [ [ -LRB- en-wikipedia-org-3679 39 2 29 29 CD en-wikipedia-org-3679 39 3 ] ] -RRB- en-wikipedia-org-3679 39 4 Some some DT en-wikipedia-org-3679 39 5 sites site NNS en-wikipedia-org-3679 39 6 such such JJ en-wikipedia-org-3679 39 7 as as IN en-wikipedia-org-3679 39 8 GitHub GitHub NNP en-wikipedia-org-3679 39 9 redirect redirect VBP en-wikipedia-org-3679 39 10 humans.txt humans.txt ADD en-wikipedia-org-3679 39 11 to to IN en-wikipedia-org-3679 39 12 an an DT en-wikipedia-org-3679 39 13 about about IN en-wikipedia-org-3679 39 14 page page NN en-wikipedia-org-3679 39 15 . . . en-wikipedia-org-3679 40 1 [ [ -LRB- en-wikipedia-org-3679 40 2 30 30 CD en-wikipedia-org-3679 40 3 ] ] -RRB- en-wikipedia-org-3679 40 4 Previously previously RB en-wikipedia-org-3679 40 5 Google Google NNP en-wikipedia-org-3679 40 6 had have VBD en-wikipedia-org-3679 40 7 a a DT en-wikipedia-org-3679 40 8 joke joke NN en-wikipedia-org-3679 40 9 file file NN en-wikipedia-org-3679 40 10 hosted host VBD en-wikipedia-org-3679 40 11 at at IN en-wikipedia-org-3679 40 12 /killer /killer NN en-wikipedia-org-3679 40 13 - - HYPH en-wikipedia-org-3679 40 14 robots.txt robots.txt NNP en-wikipedia-org-3679 40 15 instructing instruct VBG en-wikipedia-org-3679 40 16 the the DT en-wikipedia-org-3679 40 17 Terminator Terminator NNP en-wikipedia-org-3679 40 18 not not RB en-wikipedia-org-3679 40 19 to to TO en-wikipedia-org-3679 40 20 kill kill VB en-wikipedia-org-3679 40 21 the the DT en-wikipedia-org-3679 40 22 company company NN en-wikipedia-org-3679 40 23 founders founder VBZ en-wikipedia-org-3679 40 24 Larry Larry NNP en-wikipedia-org-3679 40 25 Page Page NNP en-wikipedia-org-3679 40 26 and and CC en-wikipedia-org-3679 40 27 Sergey Sergey NNP en-wikipedia-org-3679 40 28 Brin Brin NNP en-wikipedia-org-3679 40 29 . . . en-wikipedia-org-3679 41 1 [ [ -LRB- en-wikipedia-org-3679 41 2 31][32 31][32 CD en-wikipedia-org-3679 41 3 ] ] -RRB- en-wikipedia-org-3679 41 4 Examples[edit examples[edit LS en-wikipedia-org-3679 41 5 ] ] -RRB- en-wikipedia-org-3679 41 6 This this DT en-wikipedia-org-3679 41 7 example example NN en-wikipedia-org-3679 41 8 tells tell VBZ en-wikipedia-org-3679 41 9 all all DT en-wikipedia-org-3679 41 10 robots robot NNS en-wikipedia-org-3679 41 11 that that IN en-wikipedia-org-3679 41 12 they -PRON- PRP en-wikipedia-org-3679 41 13 can can MD en-wikipedia-org-3679 41 14 visit visit VB en-wikipedia-org-3679 41 15 all all DT en-wikipedia-org-3679 41 16 files file NNS en-wikipedia-org-3679 41 17 because because IN en-wikipedia-org-3679 41 18 the the DT en-wikipedia-org-3679 41 19 wildcard wildcard NN en-wikipedia-org-3679 41 20 * * NFP en-wikipedia-org-3679 41 21 stands stand VBZ en-wikipedia-org-3679 41 22 for for IN en-wikipedia-org-3679 41 23 all all DT en-wikipedia-org-3679 41 24 robots robot NNS en-wikipedia-org-3679 41 25 and and CC en-wikipedia-org-3679 41 26 the the DT en-wikipedia-org-3679 41 27 Disallow Disallow NNP en-wikipedia-org-3679 41 28 directive directive NN en-wikipedia-org-3679 41 29 has have VBZ en-wikipedia-org-3679 41 30 no no DT en-wikipedia-org-3679 41 31 value value NN en-wikipedia-org-3679 41 32 , , , en-wikipedia-org-3679 41 33 meaning mean VBG en-wikipedia-org-3679 41 34 no no DT en-wikipedia-org-3679 41 35 pages page NNS en-wikipedia-org-3679 41 36 are be VBP en-wikipedia-org-3679 41 37 disallowed disallow VBN en-wikipedia-org-3679 41 38 . . . en-wikipedia-org-3679 42 1 User user NN en-wikipedia-org-3679 42 2 - - HYPH en-wikipedia-org-3679 42 3 agent agent NN en-wikipedia-org-3679 42 4 : : : en-wikipedia-org-3679 42 5 * * NFP en-wikipedia-org-3679 42 6 Allow allow VB en-wikipedia-org-3679 42 7 : : : en-wikipedia-org-3679 42 8 / / , en-wikipedia-org-3679 42 9 The the DT en-wikipedia-org-3679 42 10 same same JJ en-wikipedia-org-3679 42 11 result result NN en-wikipedia-org-3679 42 12 can can MD en-wikipedia-org-3679 42 13 be be VB en-wikipedia-org-3679 42 14 accomplished accomplish VBN en-wikipedia-org-3679 42 15 with with IN en-wikipedia-org-3679 42 16 an an DT en-wikipedia-org-3679 42 17 empty empty JJ en-wikipedia-org-3679 42 18 or or CC en-wikipedia-org-3679 42 19 missing missing JJ en-wikipedia-org-3679 42 20 robots.txt robots.txt JJR en-wikipedia-org-3679 42 21 file file NN en-wikipedia-org-3679 42 22 . . . en-wikipedia-org-3679 43 1 This this DT en-wikipedia-org-3679 43 2 example example NN en-wikipedia-org-3679 43 3 tells tell VBZ en-wikipedia-org-3679 43 4 all all DT en-wikipedia-org-3679 43 5 robots robot NNS en-wikipedia-org-3679 43 6 to to TO en-wikipedia-org-3679 43 7 stay stay VB en-wikipedia-org-3679 43 8 out out IN en-wikipedia-org-3679 43 9 of of IN en-wikipedia-org-3679 43 10 a a DT en-wikipedia-org-3679 43 11 website website NN en-wikipedia-org-3679 43 12 : : : en-wikipedia-org-3679 43 13 User user NN en-wikipedia-org-3679 43 14 - - HYPH en-wikipedia-org-3679 43 15 agent agent NN en-wikipedia-org-3679 43 16 : : : en-wikipedia-org-3679 43 17 * * NFP en-wikipedia-org-3679 43 18 Disallow Disallow NNP en-wikipedia-org-3679 43 19 : : : en-wikipedia-org-3679 43 20 / / NFP en-wikipedia-org-3679 43 21 This this DT en-wikipedia-org-3679 43 22 example example NN en-wikipedia-org-3679 43 23 tells tell VBZ en-wikipedia-org-3679 43 24 all all DT en-wikipedia-org-3679 43 25 robots robot NNS en-wikipedia-org-3679 43 26 not not RB en-wikipedia-org-3679 43 27 to to TO en-wikipedia-org-3679 43 28 enter enter VB en-wikipedia-org-3679 43 29 three three CD en-wikipedia-org-3679 43 30 directories directory NNS en-wikipedia-org-3679 43 31 : : : en-wikipedia-org-3679 43 32 User user NN en-wikipedia-org-3679 43 33 - - HYPH en-wikipedia-org-3679 43 34 agent agent NN en-wikipedia-org-3679 43 35 : : : en-wikipedia-org-3679 43 36 * * NFP en-wikipedia-org-3679 43 37 Disallow disallow NN en-wikipedia-org-3679 43 38 : : : en-wikipedia-org-3679 43 39 /cgi /cgi NFP en-wikipedia-org-3679 43 40 - - : en-wikipedia-org-3679 43 41 bin/ bin/ NN en-wikipedia-org-3679 43 42 Disallow Disallow NNP en-wikipedia-org-3679 43 43 : : : en-wikipedia-org-3679 43 44 /tmp/ /tmp/ NFP en-wikipedia-org-3679 43 45 Disallow disallow NN en-wikipedia-org-3679 43 46 : : : en-wikipedia-org-3679 43 47 /junk/ /junk/ . en-wikipedia-org-3679 43 48 This this DT en-wikipedia-org-3679 43 49 example example NN en-wikipedia-org-3679 43 50 tells tell VBZ en-wikipedia-org-3679 43 51 all all DT en-wikipedia-org-3679 43 52 robots robot NNS en-wikipedia-org-3679 43 53 to to TO en-wikipedia-org-3679 43 54 stay stay VB en-wikipedia-org-3679 43 55 away away RB en-wikipedia-org-3679 43 56 from from IN en-wikipedia-org-3679 43 57 one one CD en-wikipedia-org-3679 43 58 specific specific JJ en-wikipedia-org-3679 43 59 file file NN en-wikipedia-org-3679 43 60 : : : en-wikipedia-org-3679 43 61 User user NN en-wikipedia-org-3679 43 62 - - HYPH en-wikipedia-org-3679 43 63 agent agent NN en-wikipedia-org-3679 43 64 : : : en-wikipedia-org-3679 43 65 * * NFP en-wikipedia-org-3679 43 66 Disallow Disallow NNP en-wikipedia-org-3679 43 67 : : : en-wikipedia-org-3679 43 68 /directory /directory , en-wikipedia-org-3679 43 69 / / SYM en-wikipedia-org-3679 43 70 file.html file.html NNP en-wikipedia-org-3679 43 71 All all DT en-wikipedia-org-3679 43 72 other other JJ en-wikipedia-org-3679 43 73 files file NNS en-wikipedia-org-3679 43 74 in in IN en-wikipedia-org-3679 43 75 the the DT en-wikipedia-org-3679 43 76 specified specify VBN en-wikipedia-org-3679 43 77 directory directory NN en-wikipedia-org-3679 43 78 will will MD en-wikipedia-org-3679 43 79 be be VB en-wikipedia-org-3679 43 80 processed process VBN en-wikipedia-org-3679 43 81 . . . en-wikipedia-org-3679 44 1 This this DT en-wikipedia-org-3679 44 2 example example NN en-wikipedia-org-3679 44 3 tells tell VBZ en-wikipedia-org-3679 44 4 a a DT en-wikipedia-org-3679 44 5 specific specific JJ en-wikipedia-org-3679 44 6 robot robot NN en-wikipedia-org-3679 44 7 to to TO en-wikipedia-org-3679 44 8 stay stay VB en-wikipedia-org-3679 44 9 out out IN en-wikipedia-org-3679 44 10 of of IN en-wikipedia-org-3679 44 11 a a DT en-wikipedia-org-3679 44 12 website website NN en-wikipedia-org-3679 44 13 : : : en-wikipedia-org-3679 44 14 User user NN en-wikipedia-org-3679 44 15 - - HYPH en-wikipedia-org-3679 44 16 agent agent NN en-wikipedia-org-3679 44 17 : : : en-wikipedia-org-3679 44 18 BadBot badbot VB en-wikipedia-org-3679 44 19 # # NN en-wikipedia-org-3679 44 20 replace replace VBP en-wikipedia-org-3679 44 21 ' ' `` en-wikipedia-org-3679 44 22 BadBot BadBot NNP en-wikipedia-org-3679 44 23 ' ' '' en-wikipedia-org-3679 44 24 with with IN en-wikipedia-org-3679 44 25 the the DT en-wikipedia-org-3679 44 26 actual actual JJ en-wikipedia-org-3679 44 27 user user NN en-wikipedia-org-3679 44 28 - - HYPH en-wikipedia-org-3679 44 29 agent agent NN en-wikipedia-org-3679 44 30 of of IN en-wikipedia-org-3679 44 31 the the DT en-wikipedia-org-3679 44 32 bot bot NN en-wikipedia-org-3679 44 33 Disallow Disallow NNP en-wikipedia-org-3679 44 34 : : : en-wikipedia-org-3679 44 35 / / NFP en-wikipedia-org-3679 44 36 This this DT en-wikipedia-org-3679 44 37 example example NN en-wikipedia-org-3679 44 38 tells tell VBZ en-wikipedia-org-3679 44 39 two two CD en-wikipedia-org-3679 44 40 specific specific JJ en-wikipedia-org-3679 44 41 robots robot NNS en-wikipedia-org-3679 44 42 not not RB en-wikipedia-org-3679 44 43 to to TO en-wikipedia-org-3679 44 44 enter enter VB en-wikipedia-org-3679 44 45 one one CD en-wikipedia-org-3679 44 46 specific specific JJ en-wikipedia-org-3679 44 47 directory directory NN en-wikipedia-org-3679 44 48 : : : en-wikipedia-org-3679 44 49 User user NN en-wikipedia-org-3679 44 50 - - HYPH en-wikipedia-org-3679 44 51 agent agent NN en-wikipedia-org-3679 44 52 : : : en-wikipedia-org-3679 44 53 BadBot badbot VB en-wikipedia-org-3679 44 54 # # NN en-wikipedia-org-3679 44 55 replace replace VBP en-wikipedia-org-3679 44 56 ' ' `` en-wikipedia-org-3679 44 57 BadBot BadBot NNP en-wikipedia-org-3679 44 58 ' ' '' en-wikipedia-org-3679 44 59 with with IN en-wikipedia-org-3679 44 60 the the DT en-wikipedia-org-3679 44 61 actual actual JJ en-wikipedia-org-3679 44 62 user user NN en-wikipedia-org-3679 44 63 - - HYPH en-wikipedia-org-3679 44 64 agent agent NN en-wikipedia-org-3679 44 65 of of IN en-wikipedia-org-3679 44 66 the the DT en-wikipedia-org-3679 44 67 bot bot NN en-wikipedia-org-3679 44 68 User User NNP en-wikipedia-org-3679 44 69 - - HYPH en-wikipedia-org-3679 44 70 agent agent NN en-wikipedia-org-3679 44 71 : : : en-wikipedia-org-3679 44 72 Googlebot Googlebot NNP en-wikipedia-org-3679 44 73 Disallow Disallow NNP en-wikipedia-org-3679 44 74 : : : en-wikipedia-org-3679 44 75 /private/ /private/ NFP en-wikipedia-org-3679 44 76 Example example NN en-wikipedia-org-3679 44 77 demonstrating demonstrate VBG en-wikipedia-org-3679 44 78 how how WRB en-wikipedia-org-3679 44 79 comments comment NNS en-wikipedia-org-3679 44 80 can can MD en-wikipedia-org-3679 44 81 be be VB en-wikipedia-org-3679 44 82 used use VBN en-wikipedia-org-3679 44 83 : : : en-wikipedia-org-3679 44 84 # # $ en-wikipedia-org-3679 44 85 Comments comment NNS en-wikipedia-org-3679 44 86 appear appear VBP en-wikipedia-org-3679 44 87 after after IN en-wikipedia-org-3679 44 88 the the DT en-wikipedia-org-3679 44 89 " " `` en-wikipedia-org-3679 44 90 # # NNP en-wikipedia-org-3679 44 91 " " '' en-wikipedia-org-3679 44 92 symbol symbol NN en-wikipedia-org-3679 44 93 at at IN en-wikipedia-org-3679 44 94 the the DT en-wikipedia-org-3679 44 95 start start NN en-wikipedia-org-3679 44 96 of of IN en-wikipedia-org-3679 44 97 a a DT en-wikipedia-org-3679 44 98 line line NN en-wikipedia-org-3679 44 99 , , , en-wikipedia-org-3679 44 100 or or CC en-wikipedia-org-3679 44 101 after after IN en-wikipedia-org-3679 44 102 a a DT en-wikipedia-org-3679 44 103 directive directive JJ en-wikipedia-org-3679 44 104 User User NNP en-wikipedia-org-3679 44 105 - - HYPH en-wikipedia-org-3679 44 106 agent agent NN en-wikipedia-org-3679 44 107 : : : en-wikipedia-org-3679 44 108 * * NFP en-wikipedia-org-3679 44 109 # # : en-wikipedia-org-3679 44 110 match match VBP en-wikipedia-org-3679 44 111 all all DT en-wikipedia-org-3679 44 112 bots bot NNS en-wikipedia-org-3679 44 113 Disallow Disallow NNP en-wikipedia-org-3679 44 114 : : : en-wikipedia-org-3679 44 115 / / SYM en-wikipedia-org-3679 44 116 # # $ en-wikipedia-org-3679 44 117 keep keep VB en-wikipedia-org-3679 44 118 them -PRON- PRP en-wikipedia-org-3679 44 119 out out RP en-wikipedia-org-3679 44 120 It -PRON- PRP en-wikipedia-org-3679 44 121 is be VBZ en-wikipedia-org-3679 44 122 also also RB en-wikipedia-org-3679 44 123 possible possible JJ en-wikipedia-org-3679 44 124 to to TO en-wikipedia-org-3679 44 125 list list VB en-wikipedia-org-3679 44 126 multiple multiple JJ en-wikipedia-org-3679 44 127 robots robot NNS en-wikipedia-org-3679 44 128 with with IN en-wikipedia-org-3679 44 129 their -PRON- PRP$ en-wikipedia-org-3679 44 130 own own JJ en-wikipedia-org-3679 44 131 rules rule NNS en-wikipedia-org-3679 44 132 . . . en-wikipedia-org-3679 45 1 The the DT en-wikipedia-org-3679 45 2 actual actual JJ en-wikipedia-org-3679 45 3 robot robot NN en-wikipedia-org-3679 45 4 string string NN en-wikipedia-org-3679 45 5 is be VBZ en-wikipedia-org-3679 45 6 defined define VBN en-wikipedia-org-3679 45 7 by by IN en-wikipedia-org-3679 45 8 the the DT en-wikipedia-org-3679 45 9 crawler crawler NN en-wikipedia-org-3679 45 10 . . . en-wikipedia-org-3679 46 1 A a DT en-wikipedia-org-3679 46 2 few few JJ en-wikipedia-org-3679 46 3 robot robot NN en-wikipedia-org-3679 46 4 operators operator NNS en-wikipedia-org-3679 46 5 , , , en-wikipedia-org-3679 46 6 such such JJ en-wikipedia-org-3679 46 7 as as IN en-wikipedia-org-3679 46 8 Google Google NNP en-wikipedia-org-3679 46 9 , , , en-wikipedia-org-3679 46 10 support support VB en-wikipedia-org-3679 46 11 several several JJ en-wikipedia-org-3679 46 12 user user NN en-wikipedia-org-3679 46 13 - - HYPH en-wikipedia-org-3679 46 14 agent agent NN en-wikipedia-org-3679 46 15 strings string NNS en-wikipedia-org-3679 46 16 that that WDT en-wikipedia-org-3679 46 17 allow allow VBP en-wikipedia-org-3679 46 18 the the DT en-wikipedia-org-3679 46 19 operator operator NN en-wikipedia-org-3679 46 20 to to TO en-wikipedia-org-3679 46 21 deny deny VB en-wikipedia-org-3679 46 22 access access NN en-wikipedia-org-3679 46 23 to to IN en-wikipedia-org-3679 46 24 a a DT en-wikipedia-org-3679 46 25 subset subset NN en-wikipedia-org-3679 46 26 of of IN en-wikipedia-org-3679 46 27 their -PRON- PRP$ en-wikipedia-org-3679 46 28 services service NNS en-wikipedia-org-3679 46 29 by by IN en-wikipedia-org-3679 46 30 using use VBG en-wikipedia-org-3679 46 31 specific specific JJ en-wikipedia-org-3679 46 32 user user NN en-wikipedia-org-3679 46 33 - - HYPH en-wikipedia-org-3679 46 34 agent agent NN en-wikipedia-org-3679 46 35 strings string NNS en-wikipedia-org-3679 46 36 . . . en-wikipedia-org-3679 47 1 [ [ -LRB- en-wikipedia-org-3679 47 2 14 14 CD en-wikipedia-org-3679 47 3 ] ] -RRB- en-wikipedia-org-3679 47 4 Example example NN en-wikipedia-org-3679 47 5 demonstrating demonstrate VBG en-wikipedia-org-3679 47 6 multiple multiple JJ en-wikipedia-org-3679 47 7 user user NN en-wikipedia-org-3679 47 8 - - HYPH en-wikipedia-org-3679 47 9 agents agent NNS en-wikipedia-org-3679 47 10 : : : en-wikipedia-org-3679 47 11 User user NN en-wikipedia-org-3679 47 12 - - HYPH en-wikipedia-org-3679 47 13 agent agent NN en-wikipedia-org-3679 47 14 : : : en-wikipedia-org-3679 47 15 googlebot googlebot NNP en-wikipedia-org-3679 47 16 # # NNP en-wikipedia-org-3679 47 17 all all DT en-wikipedia-org-3679 47 18 Google Google NNP en-wikipedia-org-3679 47 19 services services NNPS en-wikipedia-org-3679 47 20 Disallow Disallow NNP en-wikipedia-org-3679 47 21 : : : en-wikipedia-org-3679 47 22 /private/ /private/ VB en-wikipedia-org-3679 47 23 # # $ en-wikipedia-org-3679 47 24 disallow disallow VBP en-wikipedia-org-3679 47 25 this this DT en-wikipedia-org-3679 47 26 directory directory NN en-wikipedia-org-3679 47 27 User user NN en-wikipedia-org-3679 47 28 - - HYPH en-wikipedia-org-3679 47 29 agent agent NN en-wikipedia-org-3679 47 30 : : : en-wikipedia-org-3679 47 31 googlebot googlebot NNP en-wikipedia-org-3679 47 32 - - HYPH en-wikipedia-org-3679 47 33 news news NNP en-wikipedia-org-3679 47 34 # # NN en-wikipedia-org-3679 47 35 only only RB en-wikipedia-org-3679 47 36 the the DT en-wikipedia-org-3679 47 37 news news NN en-wikipedia-org-3679 47 38 service service NN en-wikipedia-org-3679 47 39 Disallow Disallow NNP en-wikipedia-org-3679 47 40 : : : en-wikipedia-org-3679 47 41 / / SYM en-wikipedia-org-3679 47 42 # # $ en-wikipedia-org-3679 47 43 disallow disallow NN en-wikipedia-org-3679 47 44 everything everything NN en-wikipedia-org-3679 47 45 User User NNP en-wikipedia-org-3679 47 46 - - HYPH en-wikipedia-org-3679 47 47 agent agent NN en-wikipedia-org-3679 47 48 : : : en-wikipedia-org-3679 47 49 * * NFP en-wikipedia-org-3679 47 50 # # VBD en-wikipedia-org-3679 47 51 any any DT en-wikipedia-org-3679 47 52 robot robot NN en-wikipedia-org-3679 47 53 Disallow Disallow NNP en-wikipedia-org-3679 47 54 : : : en-wikipedia-org-3679 47 55 /something/ /something/ NN en-wikipedia-org-3679 47 56 # # $ en-wikipedia-org-3679 47 57 disallow disallow NN en-wikipedia-org-3679 47 58 this this DT en-wikipedia-org-3679 47 59 directory directory NN en-wikipedia-org-3679 47 60 Nonstandard Nonstandard NNP en-wikipedia-org-3679 47 61 extensions[edit extensions[edit , en-wikipedia-org-3679 47 62 ] ] -RRB- en-wikipedia-org-3679 47 63 Crawl crawl NN en-wikipedia-org-3679 47 64 - - HYPH en-wikipedia-org-3679 47 65 delay delay NNP en-wikipedia-org-3679 47 66 directive[edit directive[edit LS en-wikipedia-org-3679 47 67 ] ] -RRB- en-wikipedia-org-3679 47 68 The the DT en-wikipedia-org-3679 47 69 crawl crawl NN en-wikipedia-org-3679 47 70 - - HYPH en-wikipedia-org-3679 47 71 delay delay NN en-wikipedia-org-3679 47 72 value value NN en-wikipedia-org-3679 47 73 is be VBZ en-wikipedia-org-3679 47 74 supported support VBN en-wikipedia-org-3679 47 75 by by IN en-wikipedia-org-3679 47 76 some some DT en-wikipedia-org-3679 47 77 crawlers crawler NNS en-wikipedia-org-3679 47 78 to to TO en-wikipedia-org-3679 47 79 throttle throttle VB en-wikipedia-org-3679 47 80 their -PRON- PRP$ en-wikipedia-org-3679 47 81 visits visit NNS en-wikipedia-org-3679 47 82 to to IN en-wikipedia-org-3679 47 83 the the DT en-wikipedia-org-3679 47 84 host host NN en-wikipedia-org-3679 47 85 . . . en-wikipedia-org-3679 48 1 Since since IN en-wikipedia-org-3679 48 2 this this DT en-wikipedia-org-3679 48 3 value value NN en-wikipedia-org-3679 48 4 is be VBZ en-wikipedia-org-3679 48 5 not not RB en-wikipedia-org-3679 48 6 part part NN en-wikipedia-org-3679 48 7 of of IN en-wikipedia-org-3679 48 8 the the DT en-wikipedia-org-3679 48 9 standard standard NN en-wikipedia-org-3679 48 10 , , , en-wikipedia-org-3679 48 11 its -PRON- PRP$ en-wikipedia-org-3679 48 12 interpretation interpretation NN en-wikipedia-org-3679 48 13 is be VBZ en-wikipedia-org-3679 48 14 dependent dependent JJ en-wikipedia-org-3679 48 15 on on IN en-wikipedia-org-3679 48 16 the the DT en-wikipedia-org-3679 48 17 crawler crawler NN en-wikipedia-org-3679 48 18 reading read VBG en-wikipedia-org-3679 48 19 it -PRON- PRP en-wikipedia-org-3679 48 20 . . . en-wikipedia-org-3679 49 1 It -PRON- PRP en-wikipedia-org-3679 49 2 is be VBZ en-wikipedia-org-3679 49 3 used use VBN en-wikipedia-org-3679 49 4 when when WRB en-wikipedia-org-3679 49 5 the the DT en-wikipedia-org-3679 49 6 multiple multiple JJ en-wikipedia-org-3679 49 7 burst burst NN en-wikipedia-org-3679 49 8 of of IN en-wikipedia-org-3679 49 9 visits visit NNS en-wikipedia-org-3679 49 10 from from IN en-wikipedia-org-3679 49 11 bots bot NNS en-wikipedia-org-3679 49 12 is be VBZ en-wikipedia-org-3679 49 13 slowing slow VBG en-wikipedia-org-3679 49 14 down down RP en-wikipedia-org-3679 49 15 the the DT en-wikipedia-org-3679 49 16 host host NN en-wikipedia-org-3679 49 17 . . . en-wikipedia-org-3679 50 1 Yandex Yandex NNP en-wikipedia-org-3679 50 2 interprets interpret VBZ en-wikipedia-org-3679 50 3 the the DT en-wikipedia-org-3679 50 4 value value NN en-wikipedia-org-3679 50 5 as as IN en-wikipedia-org-3679 50 6 the the DT en-wikipedia-org-3679 50 7 number number NN en-wikipedia-org-3679 50 8 of of IN en-wikipedia-org-3679 50 9 seconds second NNS en-wikipedia-org-3679 50 10 to to TO en-wikipedia-org-3679 50 11 wait wait VB en-wikipedia-org-3679 50 12 between between IN en-wikipedia-org-3679 50 13 subsequent subsequent JJ en-wikipedia-org-3679 50 14 visits visit NNS en-wikipedia-org-3679 50 15 . . . en-wikipedia-org-3679 51 1 [ [ -LRB- en-wikipedia-org-3679 51 2 16 16 CD en-wikipedia-org-3679 51 3 ] ] -RRB- en-wikipedia-org-3679 51 4 Bing bing NN en-wikipedia-org-3679 51 5 defines define VBZ en-wikipedia-org-3679 51 6 crawl crawl NN en-wikipedia-org-3679 51 7 - - HYPH en-wikipedia-org-3679 51 8 delay delay NN en-wikipedia-org-3679 51 9 as as IN en-wikipedia-org-3679 51 10 the the DT en-wikipedia-org-3679 51 11 size size NN en-wikipedia-org-3679 51 12 of of IN en-wikipedia-org-3679 51 13 a a DT en-wikipedia-org-3679 51 14 time time NN en-wikipedia-org-3679 51 15 window window NN en-wikipedia-org-3679 51 16 ( ( -LRB- en-wikipedia-org-3679 51 17 from from IN en-wikipedia-org-3679 51 18 1 1 CD en-wikipedia-org-3679 51 19 to to TO en-wikipedia-org-3679 51 20 30 30 CD en-wikipedia-org-3679 51 21 seconds second NNS en-wikipedia-org-3679 51 22 ) ) -RRB- en-wikipedia-org-3679 51 23 during during IN en-wikipedia-org-3679 51 24 which which WDT en-wikipedia-org-3679 51 25 BingBot BingBot NNP en-wikipedia-org-3679 51 26 will will MD en-wikipedia-org-3679 51 27 access access VB en-wikipedia-org-3679 51 28 a a DT en-wikipedia-org-3679 51 29 web web NN en-wikipedia-org-3679 51 30 site site NN en-wikipedia-org-3679 51 31 only only RB en-wikipedia-org-3679 51 32 once once RB en-wikipedia-org-3679 51 33 . . . en-wikipedia-org-3679 52 1 [ [ -LRB- en-wikipedia-org-3679 52 2 33 33 CD en-wikipedia-org-3679 52 3 ] ] -RRB- en-wikipedia-org-3679 52 4 Google Google NNP en-wikipedia-org-3679 52 5 provides provide VBZ en-wikipedia-org-3679 52 6 an an DT en-wikipedia-org-3679 52 7 interface interface NN en-wikipedia-org-3679 52 8 in in IN en-wikipedia-org-3679 52 9 its -PRON- PRP$ en-wikipedia-org-3679 52 10 search search NN en-wikipedia-org-3679 52 11 console console NN en-wikipedia-org-3679 52 12 for for IN en-wikipedia-org-3679 52 13 webmasters webmaster NNS en-wikipedia-org-3679 52 14 , , , en-wikipedia-org-3679 52 15 to to TO en-wikipedia-org-3679 52 16 control control VB en-wikipedia-org-3679 52 17 the the DT en-wikipedia-org-3679 52 18 GoogleBot GoogleBot NNP en-wikipedia-org-3679 52 19 's 's POS en-wikipedia-org-3679 52 20 subsequent subsequent JJ en-wikipedia-org-3679 52 21 visits visit NNS en-wikipedia-org-3679 52 22 . . . en-wikipedia-org-3679 53 1 [ [ -LRB- en-wikipedia-org-3679 53 2 34 34 CD en-wikipedia-org-3679 53 3 ] ] -RRB- en-wikipedia-org-3679 53 4 User user NN en-wikipedia-org-3679 53 5 - - HYPH en-wikipedia-org-3679 53 6 agent agent NN en-wikipedia-org-3679 53 7 : : : en-wikipedia-org-3679 53 8 bingbot bingbot NNP en-wikipedia-org-3679 53 9 Allow Allow NNP en-wikipedia-org-3679 53 10   _SP en-wikipedia-org-3679 53 11 : : : en-wikipedia-org-3679 53 12 / / SYM en-wikipedia-org-3679 53 13 Crawl Crawl NNP en-wikipedia-org-3679 53 14 - - HYPH en-wikipedia-org-3679 53 15 delay delay NNP en-wikipedia-org-3679 53 16 : : : en-wikipedia-org-3679 53 17 10 10 CD en-wikipedia-org-3679 53 18 Allow allow VBP en-wikipedia-org-3679 53 19 directive[edit directive[edit NFP en-wikipedia-org-3679 53 20 ] ] -RRB- en-wikipedia-org-3679 53 21 Some some DT en-wikipedia-org-3679 53 22 major major JJ en-wikipedia-org-3679 53 23 crawlers crawler NNS en-wikipedia-org-3679 53 24 support support VBP en-wikipedia-org-3679 53 25 an an DT en-wikipedia-org-3679 53 26 Allow Allow NNP en-wikipedia-org-3679 53 27 directive directive NN en-wikipedia-org-3679 53 28 , , , en-wikipedia-org-3679 53 29 which which WDT en-wikipedia-org-3679 53 30 can can MD en-wikipedia-org-3679 53 31 counteract counteract VB en-wikipedia-org-3679 53 32 a a DT en-wikipedia-org-3679 53 33 following follow VBG en-wikipedia-org-3679 53 34 Disallow Disallow NNP en-wikipedia-org-3679 53 35 directive directive NN en-wikipedia-org-3679 53 36 . . . en-wikipedia-org-3679 54 1 [ [ -LRB- en-wikipedia-org-3679 54 2 35][36 35][36 CD en-wikipedia-org-3679 54 3 ] ] -RRB- en-wikipedia-org-3679 54 4 This this DT en-wikipedia-org-3679 54 5 is be VBZ en-wikipedia-org-3679 54 6 useful useful JJ en-wikipedia-org-3679 54 7 when when WRB en-wikipedia-org-3679 54 8 one one PRP en-wikipedia-org-3679 54 9 tells tell VBZ en-wikipedia-org-3679 54 10 robots robot NNS en-wikipedia-org-3679 54 11 to to TO en-wikipedia-org-3679 54 12 avoid avoid VB en-wikipedia-org-3679 54 13 an an DT en-wikipedia-org-3679 54 14 entire entire JJ en-wikipedia-org-3679 54 15 directory directory NN en-wikipedia-org-3679 54 16 but but CC en-wikipedia-org-3679 54 17 still still RB en-wikipedia-org-3679 54 18 wants want VBZ en-wikipedia-org-3679 54 19 some some DT en-wikipedia-org-3679 54 20 HTML html NN en-wikipedia-org-3679 54 21 documents document NNS en-wikipedia-org-3679 54 22 in in IN en-wikipedia-org-3679 54 23 that that DT en-wikipedia-org-3679 54 24 directory directory NN en-wikipedia-org-3679 54 25 crawled crawl VBD en-wikipedia-org-3679 54 26 and and CC en-wikipedia-org-3679 54 27 indexed index VBD en-wikipedia-org-3679 54 28 . . . en-wikipedia-org-3679 55 1 While while IN en-wikipedia-org-3679 55 2 by by IN en-wikipedia-org-3679 55 3 standard standard JJ en-wikipedia-org-3679 55 4 implementation implementation NN en-wikipedia-org-3679 55 5 the the DT en-wikipedia-org-3679 55 6 first first JJ en-wikipedia-org-3679 55 7 matching matching JJ en-wikipedia-org-3679 55 8 robots.txt robots.txt NN en-wikipedia-org-3679 55 9 pattern pattern NN en-wikipedia-org-3679 55 10 always always RB en-wikipedia-org-3679 55 11 wins win VBZ en-wikipedia-org-3679 55 12 , , , en-wikipedia-org-3679 55 13 Google Google NNP en-wikipedia-org-3679 55 14 's 's POS en-wikipedia-org-3679 55 15 implementation implementation NN en-wikipedia-org-3679 55 16 differs differ NNS en-wikipedia-org-3679 55 17 in in IN en-wikipedia-org-3679 55 18 that that IN en-wikipedia-org-3679 55 19 Allow allow JJ en-wikipedia-org-3679 55 20 patterns pattern NNS en-wikipedia-org-3679 55 21 with with IN en-wikipedia-org-3679 55 22 equal equal JJ en-wikipedia-org-3679 55 23 or or CC en-wikipedia-org-3679 55 24 more more JJR en-wikipedia-org-3679 55 25 characters character NNS en-wikipedia-org-3679 55 26 in in IN en-wikipedia-org-3679 55 27 the the DT en-wikipedia-org-3679 55 28 directive directive JJ en-wikipedia-org-3679 55 29 path path NN en-wikipedia-org-3679 55 30 win win VBP en-wikipedia-org-3679 55 31 over over IN en-wikipedia-org-3679 55 32 a a DT en-wikipedia-org-3679 55 33 matching matching JJ en-wikipedia-org-3679 55 34 Disallow Disallow NNP en-wikipedia-org-3679 55 35 pattern pattern NN en-wikipedia-org-3679 55 36 . . . en-wikipedia-org-3679 56 1 [ [ -LRB- en-wikipedia-org-3679 56 2 37 37 CD en-wikipedia-org-3679 56 3 ] ] -RRB- en-wikipedia-org-3679 56 4 Bing bing NN en-wikipedia-org-3679 56 5 uses use VBZ en-wikipedia-org-3679 56 6 either either CC en-wikipedia-org-3679 56 7 the the DT en-wikipedia-org-3679 56 8 Allow allow JJ en-wikipedia-org-3679 56 9 or or CC en-wikipedia-org-3679 56 10 Disallow Disallow NNP en-wikipedia-org-3679 56 11 directive directive NN en-wikipedia-org-3679 56 12 , , , en-wikipedia-org-3679 56 13 whichever whichever WDT en-wikipedia-org-3679 56 14 is be VBZ en-wikipedia-org-3679 56 15 more more RBR en-wikipedia-org-3679 56 16 specific specific JJ en-wikipedia-org-3679 56 17 , , , en-wikipedia-org-3679 56 18 based base VBN en-wikipedia-org-3679 56 19 on on IN en-wikipedia-org-3679 56 20 length length NN en-wikipedia-org-3679 56 21 , , , en-wikipedia-org-3679 56 22 like like IN en-wikipedia-org-3679 56 23 Google Google NNP en-wikipedia-org-3679 56 24 . . . en-wikipedia-org-3679 57 1 [ [ -LRB- en-wikipedia-org-3679 57 2 17 17 CD en-wikipedia-org-3679 57 3 ] ] -RRB- en-wikipedia-org-3679 57 4 In in IN en-wikipedia-org-3679 57 5 order order NN en-wikipedia-org-3679 57 6 to to TO en-wikipedia-org-3679 57 7 be be VB en-wikipedia-org-3679 57 8 compatible compatible JJ en-wikipedia-org-3679 57 9 to to IN en-wikipedia-org-3679 57 10 all all DT en-wikipedia-org-3679 57 11 robots robot NNS en-wikipedia-org-3679 57 12 , , , en-wikipedia-org-3679 57 13 if if IN en-wikipedia-org-3679 57 14 one one PRP en-wikipedia-org-3679 57 15 wants want VBZ en-wikipedia-org-3679 57 16 to to TO en-wikipedia-org-3679 57 17 allow allow VB en-wikipedia-org-3679 57 18 single single JJ en-wikipedia-org-3679 57 19 files file NNS en-wikipedia-org-3679 57 20 inside inside IN en-wikipedia-org-3679 57 21 an an DT en-wikipedia-org-3679 57 22 otherwise otherwise RB en-wikipedia-org-3679 57 23 disallowed disallow VBN en-wikipedia-org-3679 57 24 directory directory NN en-wikipedia-org-3679 57 25 , , , en-wikipedia-org-3679 57 26 it -PRON- PRP en-wikipedia-org-3679 57 27 is be VBZ en-wikipedia-org-3679 57 28 necessary necessary JJ en-wikipedia-org-3679 57 29 to to TO en-wikipedia-org-3679 57 30 place place VB en-wikipedia-org-3679 57 31 the the DT en-wikipedia-org-3679 57 32 Allow Allow NNP en-wikipedia-org-3679 57 33 directive(s directive(s NNP en-wikipedia-org-3679 57 34 ) ) -RRB- en-wikipedia-org-3679 57 35 first first RB en-wikipedia-org-3679 57 36 , , , en-wikipedia-org-3679 57 37 followed follow VBN en-wikipedia-org-3679 57 38 by by IN en-wikipedia-org-3679 57 39 the the DT en-wikipedia-org-3679 57 40 Disallow Disallow NNP en-wikipedia-org-3679 57 41 , , , en-wikipedia-org-3679 57 42 for for IN en-wikipedia-org-3679 57 43 example example NN en-wikipedia-org-3679 57 44 : : : en-wikipedia-org-3679 57 45 Allow allow VB en-wikipedia-org-3679 57 46 : : : en-wikipedia-org-3679 57 47 /directory1 /directory1 CD en-wikipedia-org-3679 57 48 / / SYM en-wikipedia-org-3679 57 49 myfile.html myfile.html NNP en-wikipedia-org-3679 57 50 Disallow Disallow NNP en-wikipedia-org-3679 57 51 : : : en-wikipedia-org-3679 57 52 /directory1/ /directory1/ NFP en-wikipedia-org-3679 57 53 This this DT en-wikipedia-org-3679 57 54 example example NN en-wikipedia-org-3679 57 55 will will MD en-wikipedia-org-3679 57 56 Disallow disallow VB en-wikipedia-org-3679 57 57 anything anything NN en-wikipedia-org-3679 57 58 in in IN en-wikipedia-org-3679 57 59 /directory1/ /directory1/ NN en-wikipedia-org-3679 57 60 except except IN en-wikipedia-org-3679 57 61 /directory1 /directory1 CD en-wikipedia-org-3679 57 62 / / SYM en-wikipedia-org-3679 57 63 myfile.html myfile.html NNS en-wikipedia-org-3679 57 64 , , , en-wikipedia-org-3679 57 65 since since IN en-wikipedia-org-3679 57 66 the the DT en-wikipedia-org-3679 57 67 latter latter NN en-wikipedia-org-3679 57 68 will will MD en-wikipedia-org-3679 57 69 match match VB en-wikipedia-org-3679 57 70 first first RB en-wikipedia-org-3679 57 71 . . . en-wikipedia-org-3679 58 1 The the DT en-wikipedia-org-3679 58 2 order order NN en-wikipedia-org-3679 58 3 is be VBZ en-wikipedia-org-3679 58 4 only only RB en-wikipedia-org-3679 58 5 important important JJ en-wikipedia-org-3679 58 6 to to IN en-wikipedia-org-3679 58 7 robots robot NNS en-wikipedia-org-3679 58 8 that that WDT en-wikipedia-org-3679 58 9 follow follow VBP en-wikipedia-org-3679 58 10 the the DT en-wikipedia-org-3679 58 11 standard standard NN en-wikipedia-org-3679 58 12 ; ; : en-wikipedia-org-3679 58 13 in in IN en-wikipedia-org-3679 58 14 the the DT en-wikipedia-org-3679 58 15 case case NN en-wikipedia-org-3679 58 16 of of IN en-wikipedia-org-3679 58 17 the the DT en-wikipedia-org-3679 58 18 Google Google NNP en-wikipedia-org-3679 58 19 or or CC en-wikipedia-org-3679 58 20 Bing Bing NNP en-wikipedia-org-3679 58 21 bots bot NNS en-wikipedia-org-3679 58 22 , , , en-wikipedia-org-3679 58 23 the the DT en-wikipedia-org-3679 58 24 order order NN en-wikipedia-org-3679 58 25 is be VBZ en-wikipedia-org-3679 58 26 not not RB en-wikipedia-org-3679 58 27 important important JJ en-wikipedia-org-3679 58 28 . . . en-wikipedia-org-3679 59 1 Sitemap[edit sitemap[edit LS en-wikipedia-org-3679 59 2 ] ] -RRB- en-wikipedia-org-3679 59 3 Some some DT en-wikipedia-org-3679 59 4 crawlers crawler NNS en-wikipedia-org-3679 59 5 support support VBP en-wikipedia-org-3679 59 6 a a DT en-wikipedia-org-3679 59 7 Sitemap Sitemap NNP en-wikipedia-org-3679 59 8 directive directive NN en-wikipedia-org-3679 59 9 , , , en-wikipedia-org-3679 59 10 allowing allow VBG en-wikipedia-org-3679 59 11 multiple multiple JJ en-wikipedia-org-3679 59 12 Sitemaps Sitemaps NNP en-wikipedia-org-3679 59 13 in in IN en-wikipedia-org-3679 59 14 the the DT en-wikipedia-org-3679 59 15 same same JJ en-wikipedia-org-3679 59 16 robots.txt robots.txt NN en-wikipedia-org-3679 59 17 in in IN en-wikipedia-org-3679 59 18 the the DT en-wikipedia-org-3679 59 19 form form NN en-wikipedia-org-3679 59 20 Sitemap Sitemap NNP en-wikipedia-org-3679 59 21 : : : en-wikipedia-org-3679 59 22 full full JJ en-wikipedia-org-3679 59 23 - - HYPH en-wikipedia-org-3679 59 24 url:[38 url:[38 NN en-wikipedia-org-3679 59 25 ] ] -RRB- en-wikipedia-org-3679 59 26 Sitemap Sitemap NNP en-wikipedia-org-3679 59 27 : : : en-wikipedia-org-3679 59 28 http://www.example.com/sitemap.xml http://www.example.com/sitemap.xml NNP en-wikipedia-org-3679 59 29 Host[edit host[edit FW en-wikipedia-org-3679 59 30 ] ] -RRB- en-wikipedia-org-3679 59 31 Some some DT en-wikipedia-org-3679 59 32 crawlers crawler NNS en-wikipedia-org-3679 59 33 ( ( -LRB- en-wikipedia-org-3679 59 34 Yandex Yandex NNP en-wikipedia-org-3679 59 35 ) ) -RRB- en-wikipedia-org-3679 59 36 support support VBP en-wikipedia-org-3679 59 37 a a DT en-wikipedia-org-3679 59 38 Host host NN en-wikipedia-org-3679 59 39 directive directive NN en-wikipedia-org-3679 59 40 , , , en-wikipedia-org-3679 59 41 allowing allow VBG en-wikipedia-org-3679 59 42 websites website NNS en-wikipedia-org-3679 59 43 with with IN en-wikipedia-org-3679 59 44 multiple multiple JJ en-wikipedia-org-3679 59 45 mirrors mirror NNS en-wikipedia-org-3679 59 46 to to TO en-wikipedia-org-3679 59 47 specify specify VB en-wikipedia-org-3679 59 48 their -PRON- PRP$ en-wikipedia-org-3679 59 49 preferred prefer VBN en-wikipedia-org-3679 59 50 domain:[39 domain:[39 NNP en-wikipedia-org-3679 59 51 ] ] -RRB- en-wikipedia-org-3679 59 52 Host host NN en-wikipedia-org-3679 59 53 : : : en-wikipedia-org-3679 59 54 hosting.example.com hosting.example.com NNP en-wikipedia-org-3679 59 55 This this DT en-wikipedia-org-3679 59 56 is be VBZ en-wikipedia-org-3679 59 57 not not RB en-wikipedia-org-3679 59 58 supported support VBN en-wikipedia-org-3679 59 59 by by IN en-wikipedia-org-3679 59 60 all all DT en-wikipedia-org-3679 59 61 crawlers crawler NNS en-wikipedia-org-3679 59 62 . . . en-wikipedia-org-3679 60 1 Universal Universal NNP en-wikipedia-org-3679 60 2 " " `` en-wikipedia-org-3679 60 3 * * NN en-wikipedia-org-3679 60 4 " " '' en-wikipedia-org-3679 60 5 match[edit match[edit NNS en-wikipedia-org-3679 60 6 ] ] -RRB- en-wikipedia-org-3679 60 7 The the DT en-wikipedia-org-3679 60 8 Robot Robot NNP en-wikipedia-org-3679 60 9 Exclusion Exclusion NNP en-wikipedia-org-3679 60 10 Standard Standard NNP en-wikipedia-org-3679 60 11 does do VBZ en-wikipedia-org-3679 60 12 not not RB en-wikipedia-org-3679 60 13 mention mention VB en-wikipedia-org-3679 60 14 the the DT en-wikipedia-org-3679 60 15 " " `` en-wikipedia-org-3679 60 16 * * NN en-wikipedia-org-3679 60 17 " " '' en-wikipedia-org-3679 60 18 character character NN en-wikipedia-org-3679 60 19 in in IN en-wikipedia-org-3679 60 20 the the DT en-wikipedia-org-3679 60 21 Disallow Disallow NNP en-wikipedia-org-3679 60 22 : : : en-wikipedia-org-3679 60 23 statement statement NN en-wikipedia-org-3679 60 24 . . . en-wikipedia-org-3679 61 1 [ [ -LRB- en-wikipedia-org-3679 61 2 40 40 CD en-wikipedia-org-3679 61 3 ] ] -RRB- en-wikipedia-org-3679 61 4 Meta meta JJ en-wikipedia-org-3679 61 5 tags tag NNS en-wikipedia-org-3679 61 6 and and CC en-wikipedia-org-3679 61 7 headers[edit headers[edit LS en-wikipedia-org-3679 61 8 ] ] -RRB- en-wikipedia-org-3679 61 9 In in IN en-wikipedia-org-3679 61 10 addition addition NN en-wikipedia-org-3679 61 11 to to IN en-wikipedia-org-3679 61 12 root root NN en-wikipedia-org-3679 61 13 - - HYPH en-wikipedia-org-3679 61 14 level level NN en-wikipedia-org-3679 61 15 robots.txt robots.txt NN en-wikipedia-org-3679 61 16 files file NNS en-wikipedia-org-3679 61 17 , , , en-wikipedia-org-3679 61 18 robots robot NNS en-wikipedia-org-3679 61 19 exclusion exclusion NN en-wikipedia-org-3679 61 20 directives directive NNS en-wikipedia-org-3679 61 21 can can MD en-wikipedia-org-3679 61 22 be be VB en-wikipedia-org-3679 61 23 applied apply VBN en-wikipedia-org-3679 61 24 at at IN en-wikipedia-org-3679 61 25 a a DT en-wikipedia-org-3679 61 26 more more RBR en-wikipedia-org-3679 61 27 granular granular JJ en-wikipedia-org-3679 61 28 level level NN en-wikipedia-org-3679 61 29 through through IN en-wikipedia-org-3679 61 30 the the DT en-wikipedia-org-3679 61 31 use use NN en-wikipedia-org-3679 61 32 of of IN en-wikipedia-org-3679 61 33 Robots robots JJ en-wikipedia-org-3679 61 34 meta meta JJ en-wikipedia-org-3679 61 35 tags tag NNS en-wikipedia-org-3679 61 36 and and CC en-wikipedia-org-3679 61 37 X X NNS en-wikipedia-org-3679 61 38 - - HYPH en-wikipedia-org-3679 61 39 Robots robots JJ en-wikipedia-org-3679 61 40 - - HYPH en-wikipedia-org-3679 61 41 Tag tag NN en-wikipedia-org-3679 61 42 HTTP HTTP NNP en-wikipedia-org-3679 61 43 headers header NNS en-wikipedia-org-3679 61 44 . . . en-wikipedia-org-3679 62 1 The the DT en-wikipedia-org-3679 62 2 robots robot NNS en-wikipedia-org-3679 62 3 meta meta JJ en-wikipedia-org-3679 62 4 tag tag NN en-wikipedia-org-3679 62 5 can can MD en-wikipedia-org-3679 62 6 not not RB en-wikipedia-org-3679 62 7 be be VB en-wikipedia-org-3679 62 8 used use VBN en-wikipedia-org-3679 62 9 for for IN en-wikipedia-org-3679 62 10 non non JJ en-wikipedia-org-3679 62 11 - - JJ en-wikipedia-org-3679 62 12 HTML html JJ en-wikipedia-org-3679 62 13 files file NNS en-wikipedia-org-3679 62 14 such such JJ en-wikipedia-org-3679 62 15 as as IN en-wikipedia-org-3679 62 16 images image NNS en-wikipedia-org-3679 62 17 , , , en-wikipedia-org-3679 62 18 text text NN en-wikipedia-org-3679 62 19 files file NNS en-wikipedia-org-3679 62 20 , , , en-wikipedia-org-3679 62 21 or or CC en-wikipedia-org-3679 62 22 PDF PDF NNP en-wikipedia-org-3679 62 23 documents document NNS en-wikipedia-org-3679 62 24 . . . en-wikipedia-org-3679 63 1 On on IN en-wikipedia-org-3679 63 2 the the DT en-wikipedia-org-3679 63 3 other other JJ en-wikipedia-org-3679 63 4 hand hand NN en-wikipedia-org-3679 63 5 , , , en-wikipedia-org-3679 63 6 the the DT en-wikipedia-org-3679 63 7 X X NNP en-wikipedia-org-3679 63 8 - - HYPH en-wikipedia-org-3679 63 9 Robots robots NN en-wikipedia-org-3679 63 10 - - HYPH en-wikipedia-org-3679 63 11 Tag tag NN en-wikipedia-org-3679 63 12 can can MD en-wikipedia-org-3679 63 13 be be VB en-wikipedia-org-3679 63 14 added add VBN en-wikipedia-org-3679 63 15 to to IN en-wikipedia-org-3679 63 16 non non JJ en-wikipedia-org-3679 63 17 - - JJ en-wikipedia-org-3679 63 18 HTML html JJ en-wikipedia-org-3679 63 19 files file NNS en-wikipedia-org-3679 63 20 by by IN en-wikipedia-org-3679 63 21 using use VBG en-wikipedia-org-3679 63 22 .htaccess .htaccess NN en-wikipedia-org-3679 63 23 and and CC en-wikipedia-org-3679 63 24 httpd.conf httpd.conf JJ en-wikipedia-org-3679 63 25 files file NNS en-wikipedia-org-3679 63 26 . . . en-wikipedia-org-3679 64 1 [ [ -LRB- en-wikipedia-org-3679 64 2 41 41 CD en-wikipedia-org-3679 64 3 ] ] -RRB- en-wikipedia-org-3679 64 4 A A NNP en-wikipedia-org-3679 64 5 " " `` en-wikipedia-org-3679 64 6 noindex noindex NN en-wikipedia-org-3679 64 7 " " '' en-wikipedia-org-3679 64 8 meta meta JJ en-wikipedia-org-3679 64 9 tag tag NN en-wikipedia-org-3679 64 10 < < XX en-wikipedia-org-3679 64 11 meta meta JJ en-wikipedia-org-3679 64 12 name="robots name="robot NNS en-wikipedia-org-3679 64 13 " " `` en-wikipedia-org-3679 64 14 content="noindex content="noindex JJ en-wikipedia-org-3679 64 15 " " '' en-wikipedia-org-3679 64 16 / / , en-wikipedia-org-3679 64 17 > > FW en-wikipedia-org-3679 64 18 A a DT en-wikipedia-org-3679 64 19 " " `` en-wikipedia-org-3679 64 20 noindex noindex NN en-wikipedia-org-3679 64 21 " " '' en-wikipedia-org-3679 64 22 HTTP http JJ en-wikipedia-org-3679 64 23 response response NN en-wikipedia-org-3679 64 24 header header NN en-wikipedia-org-3679 64 25 X X NNS en-wikipedia-org-3679 64 26 - - NNS en-wikipedia-org-3679 64 27 Robots robots JJ en-wikipedia-org-3679 64 28 - - HYPH en-wikipedia-org-3679 64 29 Tag tag NN en-wikipedia-org-3679 64 30 : : : en-wikipedia-org-3679 64 31 noindex noindex NN en-wikipedia-org-3679 64 32 The the DT en-wikipedia-org-3679 64 33 X X NNP en-wikipedia-org-3679 64 34 - - HYPH en-wikipedia-org-3679 64 35 Robots robots NN en-wikipedia-org-3679 64 36 - - HYPH en-wikipedia-org-3679 64 37 Tag tag NN en-wikipedia-org-3679 64 38 is be VBZ en-wikipedia-org-3679 64 39 only only RB en-wikipedia-org-3679 64 40 effective effective JJ en-wikipedia-org-3679 64 41 after after IN en-wikipedia-org-3679 64 42 the the DT en-wikipedia-org-3679 64 43 page page NN en-wikipedia-org-3679 64 44 has have VBZ en-wikipedia-org-3679 64 45 been be VBN en-wikipedia-org-3679 64 46 requested request VBN en-wikipedia-org-3679 64 47 and and CC en-wikipedia-org-3679 64 48 the the DT en-wikipedia-org-3679 64 49 server server NN en-wikipedia-org-3679 64 50 responds respond VBZ en-wikipedia-org-3679 64 51 , , , en-wikipedia-org-3679 64 52 and and CC en-wikipedia-org-3679 64 53 the the DT en-wikipedia-org-3679 64 54 robots robot NNS en-wikipedia-org-3679 64 55 meta meta JJ en-wikipedia-org-3679 64 56 tag tag NN en-wikipedia-org-3679 64 57 is be VBZ en-wikipedia-org-3679 64 58 only only RB en-wikipedia-org-3679 64 59 effective effective JJ en-wikipedia-org-3679 64 60 after after IN en-wikipedia-org-3679 64 61 the the DT en-wikipedia-org-3679 64 62 page page NN en-wikipedia-org-3679 64 63 has have VBZ en-wikipedia-org-3679 64 64 loaded load VBN en-wikipedia-org-3679 64 65 , , , en-wikipedia-org-3679 64 66 whereas whereas IN en-wikipedia-org-3679 64 67 robots.txt robots.txt NN en-wikipedia-org-3679 64 68 is be VBZ en-wikipedia-org-3679 64 69 effective effective JJ en-wikipedia-org-3679 64 70 before before IN en-wikipedia-org-3679 64 71 the the DT en-wikipedia-org-3679 64 72 page page NN en-wikipedia-org-3679 64 73 is be VBZ en-wikipedia-org-3679 64 74 requested request VBN en-wikipedia-org-3679 64 75 . . . en-wikipedia-org-3679 65 1 Thus thus RB en-wikipedia-org-3679 65 2 if if IN en-wikipedia-org-3679 65 3 a a DT en-wikipedia-org-3679 65 4 page page NN en-wikipedia-org-3679 65 5 is be VBZ en-wikipedia-org-3679 65 6 excluded exclude VBN en-wikipedia-org-3679 65 7 by by IN en-wikipedia-org-3679 65 8 a a DT en-wikipedia-org-3679 65 9 robots.txt robots.txt NN en-wikipedia-org-3679 65 10 file file NN en-wikipedia-org-3679 65 11 , , , en-wikipedia-org-3679 65 12 any any DT en-wikipedia-org-3679 65 13 robots robot NNS en-wikipedia-org-3679 65 14 meta meta JJ en-wikipedia-org-3679 65 15 tags tag NNS en-wikipedia-org-3679 65 16 or or CC en-wikipedia-org-3679 65 17 X X NNS en-wikipedia-org-3679 65 18 - - HYPH en-wikipedia-org-3679 65 19 Robots robots JJ en-wikipedia-org-3679 65 20 - - HYPH en-wikipedia-org-3679 65 21 Tag tag NN en-wikipedia-org-3679 65 22 headers header NNS en-wikipedia-org-3679 65 23 are be VBP en-wikipedia-org-3679 65 24 effectively effectively RB en-wikipedia-org-3679 65 25 ignored ignore VBN en-wikipedia-org-3679 65 26 because because IN en-wikipedia-org-3679 65 27 the the DT en-wikipedia-org-3679 65 28 robot robot NN en-wikipedia-org-3679 65 29 will will MD en-wikipedia-org-3679 65 30 not not RB en-wikipedia-org-3679 65 31 see see VB en-wikipedia-org-3679 65 32 them -PRON- PRP en-wikipedia-org-3679 65 33 in in IN en-wikipedia-org-3679 65 34 the the DT en-wikipedia-org-3679 65 35 first first JJ en-wikipedia-org-3679 65 36 place place NN en-wikipedia-org-3679 65 37 . . . en-wikipedia-org-3679 66 1 [ [ -LRB- en-wikipedia-org-3679 66 2 41 41 CD en-wikipedia-org-3679 66 3 ] ] -RRB- en-wikipedia-org-3679 66 4 See see VB en-wikipedia-org-3679 66 5 also[edit also[edit , en-wikipedia-org-3679 66 6 ] ] -RRB- en-wikipedia-org-3679 66 7 ads.txt ads.txt CD en-wikipedia-org-3679 66 8 , , , en-wikipedia-org-3679 66 9 a a DT en-wikipedia-org-3679 66 10 standard standard NN en-wikipedia-org-3679 66 11 for for IN en-wikipedia-org-3679 66 12 listing list VBG en-wikipedia-org-3679 66 13 authorized authorize VBN en-wikipedia-org-3679 66 14 ad ad NN en-wikipedia-org-3679 66 15 sellers seller NNS en-wikipedia-org-3679 66 16 security.txt security.txt NN en-wikipedia-org-3679 66 17 , , , en-wikipedia-org-3679 66 18 a a DT en-wikipedia-org-3679 66 19 file file NN en-wikipedia-org-3679 66 20 to to TO en-wikipedia-org-3679 66 21 describe describe VB en-wikipedia-org-3679 66 22 the the DT en-wikipedia-org-3679 66 23 process process NN en-wikipedia-org-3679 66 24 for for IN en-wikipedia-org-3679 66 25 security security NN en-wikipedia-org-3679 66 26 researchers researcher NNS en-wikipedia-org-3679 66 27 to to TO en-wikipedia-org-3679 66 28 follow follow VB en-wikipedia-org-3679 66 29 in in IN en-wikipedia-org-3679 66 30 order order NN en-wikipedia-org-3679 66 31 to to TO en-wikipedia-org-3679 66 32 report report VB en-wikipedia-org-3679 66 33 security security NN en-wikipedia-org-3679 66 34 vulnerabilities vulnerability NNS en-wikipedia-org-3679 66 35 Automated Automated NNP en-wikipedia-org-3679 66 36 Content Content NNP en-wikipedia-org-3679 66 37 Access Access NNP en-wikipedia-org-3679 66 38 Protocol Protocol NNP en-wikipedia-org-3679 66 39 – – : en-wikipedia-org-3679 66 40 a a DT en-wikipedia-org-3679 66 41 failed fail VBN en-wikipedia-org-3679 66 42 proposal proposal NN en-wikipedia-org-3679 66 43 to to TO en-wikipedia-org-3679 66 44 extend extend VB en-wikipedia-org-3679 66 45 robots.txt robots.txt JJR en-wikipedia-org-3679 66 46 BotSeer BotSeer NNP en-wikipedia-org-3679 66 47 – – : en-wikipedia-org-3679 66 48 now now RB en-wikipedia-org-3679 66 49 inactive inactive JJ en-wikipedia-org-3679 66 50 search search NN en-wikipedia-org-3679 66 51 engine engine NN en-wikipedia-org-3679 66 52 for for IN en-wikipedia-org-3679 66 53 robots.txt robots.txt CD en-wikipedia-org-3679 66 54 files file NNS en-wikipedia-org-3679 66 55 Distributed distribute VBN en-wikipedia-org-3679 66 56 web web NN en-wikipedia-org-3679 66 57 crawling crawl VBG en-wikipedia-org-3679 66 58 Focused focused JJ en-wikipedia-org-3679 66 59 crawler crawler NN en-wikipedia-org-3679 66 60 Internet Internet NNP en-wikipedia-org-3679 66 61 Archive Archive NNP en-wikipedia-org-3679 66 62 National National NNP en-wikipedia-org-3679 66 63 Digital Digital NNP en-wikipedia-org-3679 66 64 Library Library NNP en-wikipedia-org-3679 66 65 Program Program NNP en-wikipedia-org-3679 66 66 ( ( -LRB- en-wikipedia-org-3679 66 67 NDLP NDLP NNP en-wikipedia-org-3679 66 68 ) ) -RRB- en-wikipedia-org-3679 66 69 National National NNP en-wikipedia-org-3679 66 70 Digital Digital NNP en-wikipedia-org-3679 66 71 Information Information NNP en-wikipedia-org-3679 66 72 Infrastructure Infrastructure NNP en-wikipedia-org-3679 66 73 and and CC en-wikipedia-org-3679 66 74 Preservation Preservation NNP en-wikipedia-org-3679 66 75 Program Program NNP en-wikipedia-org-3679 66 76 ( ( -LRB- en-wikipedia-org-3679 66 77 NDIIPP NDIIPP NNP en-wikipedia-org-3679 66 78 ) ) -RRB- en-wikipedia-org-3679 66 79 Nofollow Nofollow NNP en-wikipedia-org-3679 66 80 Perma.cc perma.cc PRP$ en-wikipedia-org-3679 66 81 Meta meta JJ en-wikipedia-org-3679 66 82 elements element NNS en-wikipedia-org-3679 66 83 for for IN en-wikipedia-org-3679 66 84 search search NN en-wikipedia-org-3679 66 85 engines engine NNS en-wikipedia-org-3679 66 86 Sitemaps Sitemaps NNP en-wikipedia-org-3679 66 87 Spider Spider NNP en-wikipedia-org-3679 66 88 trap trap NN en-wikipedia-org-3679 66 89 Web web NN en-wikipedia-org-3679 66 90 archiving archive VBG en-wikipedia-org-3679 66 91 Web web NN en-wikipedia-org-3679 66 92 crawler crawler NNP en-wikipedia-org-3679 66 93 noindex noindex NNP en-wikipedia-org-3679 66 94 References[edit references[edit NN en-wikipedia-org-3679 66 95 ] ] -RRB- en-wikipedia-org-3679 66 96 ^ ^ FW en-wikipedia-org-3679 66 97 " " `` en-wikipedia-org-3679 66 98 Historical Historical NNP en-wikipedia-org-3679 66 99 " " '' en-wikipedia-org-3679 66 100 . . . en-wikipedia-org-3679 67 1 Greenhills.co.uk Greenhills.co.uk NNP en-wikipedia-org-3679 67 2 . . . en-wikipedia-org-3679 68 1 Archived archived JJ en-wikipedia-org-3679 68 2 from from IN en-wikipedia-org-3679 68 3 the the DT en-wikipedia-org-3679 68 4 original original NN en-wikipedia-org-3679 68 5 on on IN en-wikipedia-org-3679 68 6 2017 2017 CD en-wikipedia-org-3679 68 7 - - HYPH en-wikipedia-org-3679 68 8 04 04 CD en-wikipedia-org-3679 68 9 - - HYPH en-wikipedia-org-3679 68 10 03 03 CD en-wikipedia-org-3679 68 11 . . . en-wikipedia-org-3679 69 1 Retrieved retrieve VBN en-wikipedia-org-3679 69 2 2017 2017 CD en-wikipedia-org-3679 69 3 - - HYPH en-wikipedia-org-3679 69 4 03 03 CD en-wikipedia-org-3679 69 5 - - HYPH en-wikipedia-org-3679 69 6 03 03 CD en-wikipedia-org-3679 69 7 . . . en-wikipedia-org-3679 70 1 ^ ^ NNP en-wikipedia-org-3679 70 2 Fielding fielding NN en-wikipedia-org-3679 70 3 , , , en-wikipedia-org-3679 70 4 Roy Roy NNP en-wikipedia-org-3679 70 5 ( ( -LRB- en-wikipedia-org-3679 70 6 1994 1994 CD en-wikipedia-org-3679 70 7 ) ) -RRB- en-wikipedia-org-3679 70 8 . . . en-wikipedia-org-3679 71 1 " " `` en-wikipedia-org-3679 71 2 Maintaining maintain VBG en-wikipedia-org-3679 71 3 Distributed distribute VBN en-wikipedia-org-3679 71 4 Hypertext Hypertext NNP en-wikipedia-org-3679 71 5 Infostructures infostructure NNS en-wikipedia-org-3679 71 6 : : : en-wikipedia-org-3679 71 7 Welcome welcome VBP en-wikipedia-org-3679 71 8 to to IN en-wikipedia-org-3679 71 9 MOMspider MOMspider NNP en-wikipedia-org-3679 71 10 's 's POS en-wikipedia-org-3679 71 11 Web web NN en-wikipedia-org-3679 71 12 " " '' en-wikipedia-org-3679 71 13 ( ( -LRB- en-wikipedia-org-3679 71 14 PostScript PostScript NNP en-wikipedia-org-3679 71 15 ) ) -RRB- en-wikipedia-org-3679 71 16 . . . en-wikipedia-org-3679 72 1 First First NNP en-wikipedia-org-3679 72 2 International International NNP en-wikipedia-org-3679 72 3 Conference Conference NNP en-wikipedia-org-3679 72 4 on on IN en-wikipedia-org-3679 72 5 the the DT en-wikipedia-org-3679 72 6 World World NNP en-wikipedia-org-3679 72 7 Wide Wide NNP en-wikipedia-org-3679 72 8 Web Web NNP en-wikipedia-org-3679 72 9 . . . en-wikipedia-org-3679 73 1 Geneva Geneva NNP en-wikipedia-org-3679 73 2 . . . en-wikipedia-org-3679 74 1 Archived archived JJ en-wikipedia-org-3679 74 2 from from IN en-wikipedia-org-3679 74 3 the the DT en-wikipedia-org-3679 74 4 original original NN en-wikipedia-org-3679 74 5 on on IN en-wikipedia-org-3679 74 6 2013 2013 CD en-wikipedia-org-3679 74 7 - - SYM en-wikipedia-org-3679 74 8 09 09 CD en-wikipedia-org-3679 74 9 - - HYPH en-wikipedia-org-3679 74 10 27 27 CD en-wikipedia-org-3679 74 11 . . . en-wikipedia-org-3679 75 1 Retrieved Retrieved NNP en-wikipedia-org-3679 75 2 September September NNP en-wikipedia-org-3679 75 3 25 25 CD en-wikipedia-org-3679 75 4 , , , en-wikipedia-org-3679 75 5 2013 2013 CD en-wikipedia-org-3679 75 6 . . . en-wikipedia-org-3679 76 1 ^ ^ NNP en-wikipedia-org-3679 76 2 a a DT en-wikipedia-org-3679 76 3 b b NNP en-wikipedia-org-3679 76 4 " " `` en-wikipedia-org-3679 76 5 The the DT en-wikipedia-org-3679 76 6 Web web NN en-wikipedia-org-3679 76 7 Robots Robots NNPS en-wikipedia-org-3679 76 8 Pages Pages NNP en-wikipedia-org-3679 76 9 " " '' en-wikipedia-org-3679 76 10 . . . en-wikipedia-org-3679 77 1 Robotstxt.org Robotstxt.org NNP en-wikipedia-org-3679 77 2 . . . en-wikipedia-org-3679 78 1 1994 1994 CD en-wikipedia-org-3679 78 2 - - SYM en-wikipedia-org-3679 78 3 06 06 CD en-wikipedia-org-3679 78 4 - - HYPH en-wikipedia-org-3679 78 5 30 30 CD en-wikipedia-org-3679 78 6 . . . en-wikipedia-org-3679 79 1 Archived archived JJ en-wikipedia-org-3679 79 2 from from IN en-wikipedia-org-3679 79 3 the the DT en-wikipedia-org-3679 79 4 original original NN en-wikipedia-org-3679 79 5 on on IN en-wikipedia-org-3679 79 6 2014 2014 CD en-wikipedia-org-3679 79 7 - - SYM en-wikipedia-org-3679 79 8 01 01 CD en-wikipedia-org-3679 79 9 - - HYPH en-wikipedia-org-3679 79 10 12 12 CD en-wikipedia-org-3679 79 11 . . . en-wikipedia-org-3679 80 1 Retrieved Retrieved NNP en-wikipedia-org-3679 80 2 2013 2013 CD en-wikipedia-org-3679 80 3 - - SYM en-wikipedia-org-3679 80 4 12 12 CD en-wikipedia-org-3679 80 5 - - SYM en-wikipedia-org-3679 80 6 29 29 CD en-wikipedia-org-3679 80 7 . . . en-wikipedia-org-3679 81 1 ^ ^ NNP en-wikipedia-org-3679 81 2 Koster Koster NNP en-wikipedia-org-3679 81 3 , , , en-wikipedia-org-3679 81 4 Martijn Martijn NNP en-wikipedia-org-3679 81 5 ( ( -LRB- en-wikipedia-org-3679 81 6 25 25 CD en-wikipedia-org-3679 81 7 February February NNP en-wikipedia-org-3679 81 8 1994 1994 CD en-wikipedia-org-3679 81 9 ) ) -RRB- en-wikipedia-org-3679 81 10 . . . en-wikipedia-org-3679 82 1 " " `` en-wikipedia-org-3679 82 2 Important important JJ en-wikipedia-org-3679 82 3 : : : en-wikipedia-org-3679 82 4 Spiders spider NNS en-wikipedia-org-3679 82 5 , , , en-wikipedia-org-3679 82 6 Robots Robots NNPS en-wikipedia-org-3679 82 7 and and CC en-wikipedia-org-3679 82 8 Web Web NNP en-wikipedia-org-3679 82 9 Wanderers Wanderers NNPS en-wikipedia-org-3679 82 10 " " '' en-wikipedia-org-3679 82 11 . . . en-wikipedia-org-3679 83 1 www www NN en-wikipedia-org-3679 83 2 - - HYPH en-wikipedia-org-3679 83 3 talk talk NN en-wikipedia-org-3679 83 4 mailing mailing NN en-wikipedia-org-3679 83 5 list list NN en-wikipedia-org-3679 83 6 . . . en-wikipedia-org-3679 84 1 Archived archived JJ en-wikipedia-org-3679 84 2 from from IN en-wikipedia-org-3679 84 3 the the DT en-wikipedia-org-3679 84 4 original original JJ en-wikipedia-org-3679 84 5 ( ( -LRB- en-wikipedia-org-3679 84 6 Hypermail Hypermail NNP en-wikipedia-org-3679 84 7 archived archived JJ en-wikipedia-org-3679 84 8 message message NN en-wikipedia-org-3679 84 9 ) ) -RRB- en-wikipedia-org-3679 84 10 on on IN en-wikipedia-org-3679 84 11 October October NNP en-wikipedia-org-3679 84 12 29 29 CD en-wikipedia-org-3679 84 13 , , , en-wikipedia-org-3679 84 14 2013 2013 CD en-wikipedia-org-3679 84 15 . . . en-wikipedia-org-3679 85 1 ^ ^ FW en-wikipedia-org-3679 85 2 " " `` en-wikipedia-org-3679 85 3 How how WRB en-wikipedia-org-3679 85 4 I -PRON- PRP en-wikipedia-org-3679 85 5 got get VBD en-wikipedia-org-3679 85 6 here here RB en-wikipedia-org-3679 85 7 in in IN en-wikipedia-org-3679 85 8 the the DT en-wikipedia-org-3679 85 9 end end NN en-wikipedia-org-3679 85 10 , , , en-wikipedia-org-3679 85 11 part part NN en-wikipedia-org-3679 85 12 five five CD en-wikipedia-org-3679 85 13 : : : en-wikipedia-org-3679 85 14 " " `` en-wikipedia-org-3679 85 15 things thing NNS en-wikipedia-org-3679 85 16 can can MD en-wikipedia-org-3679 85 17 only only RB en-wikipedia-org-3679 85 18 get get VB en-wikipedia-org-3679 85 19 better well JJR en-wikipedia-org-3679 85 20 ! ! . en-wikipedia-org-3679 85 21 " " '' en-wikipedia-org-3679 85 22 " " '' en-wikipedia-org-3679 85 23 . . . en-wikipedia-org-3679 86 1 Charlie Charlie NNP en-wikipedia-org-3679 86 2 's 's POS en-wikipedia-org-3679 86 3 Diary Diary NNP en-wikipedia-org-3679 86 4 . . . en-wikipedia-org-3679 87 1 19 19 CD en-wikipedia-org-3679 87 2 June June NNP en-wikipedia-org-3679 87 3 2006 2006 CD en-wikipedia-org-3679 87 4 . . . en-wikipedia-org-3679 88 1 Archived archived JJ en-wikipedia-org-3679 88 2 from from IN en-wikipedia-org-3679 88 3 the the DT en-wikipedia-org-3679 88 4 original original NN en-wikipedia-org-3679 88 5 on on IN en-wikipedia-org-3679 88 6 2013 2013 CD en-wikipedia-org-3679 88 7 - - SYM en-wikipedia-org-3679 88 8 11 11 CD en-wikipedia-org-3679 88 9 - - SYM en-wikipedia-org-3679 88 10 25 25 CD en-wikipedia-org-3679 88 11 . . . en-wikipedia-org-3679 89 1 Retrieved retrieve VBN en-wikipedia-org-3679 89 2 19 19 CD en-wikipedia-org-3679 89 3 April April NNP en-wikipedia-org-3679 89 4 2014 2014 CD en-wikipedia-org-3679 89 5 . . . en-wikipedia-org-3679 90 1 ^ ^ NNP en-wikipedia-org-3679 90 2 Barry Barry NNP en-wikipedia-org-3679 90 3 Schwartz Schwartz NNP en-wikipedia-org-3679 90 4 ( ( -LRB- en-wikipedia-org-3679 90 5 30 30 CD en-wikipedia-org-3679 90 6 June June NNP en-wikipedia-org-3679 90 7 2014 2014 CD en-wikipedia-org-3679 90 8 ) ) -RRB- en-wikipedia-org-3679 90 9 . . . en-wikipedia-org-3679 91 1 " " `` en-wikipedia-org-3679 91 2 Robots.txt robots.txt NN en-wikipedia-org-3679 91 3 Celebrates celebrate NNS en-wikipedia-org-3679 91 4 20 20 CD en-wikipedia-org-3679 91 5 Years year NNS en-wikipedia-org-3679 91 6 Of of IN en-wikipedia-org-3679 91 7 Blocking Blocking NNP en-wikipedia-org-3679 91 8 Search Search NNP en-wikipedia-org-3679 91 9 Engines Engines NNPS en-wikipedia-org-3679 91 10 " " '' en-wikipedia-org-3679 91 11 . . . en-wikipedia-org-3679 92 1 Search Search NNP en-wikipedia-org-3679 92 2 Engine Engine NNP en-wikipedia-org-3679 92 3 Land Land NNP en-wikipedia-org-3679 92 4 . . . en-wikipedia-org-3679 93 1 Archived archived JJ en-wikipedia-org-3679 93 2 from from IN en-wikipedia-org-3679 93 3 the the DT en-wikipedia-org-3679 93 4 original original NN en-wikipedia-org-3679 93 5 on on IN en-wikipedia-org-3679 93 6 2015 2015 CD en-wikipedia-org-3679 93 7 - - SYM en-wikipedia-org-3679 93 8 09 09 CD en-wikipedia-org-3679 93 9 - - HYPH en-wikipedia-org-3679 93 10 07 07 CD en-wikipedia-org-3679 93 11 . . . en-wikipedia-org-3679 94 1 Retrieved retrieve VBN en-wikipedia-org-3679 94 2 2015 2015 CD en-wikipedia-org-3679 94 3 - - SYM en-wikipedia-org-3679 94 4 11 11 CD en-wikipedia-org-3679 94 5 - - SYM en-wikipedia-org-3679 94 6 19 19 CD en-wikipedia-org-3679 94 7 . . . en-wikipedia-org-3679 95 1 ^ ^ FW en-wikipedia-org-3679 95 2 " " `` en-wikipedia-org-3679 95 3 Formalizing formalize VBG en-wikipedia-org-3679 95 4 the the DT en-wikipedia-org-3679 95 5 Robots Robots NNPS en-wikipedia-org-3679 95 6 Exclusion Exclusion NNP en-wikipedia-org-3679 95 7 Protocol Protocol NNP en-wikipedia-org-3679 95 8 Specification Specification NNP en-wikipedia-org-3679 95 9 " " '' en-wikipedia-org-3679 95 10 . . . en-wikipedia-org-3679 96 1 Official Official NNP en-wikipedia-org-3679 96 2 Google Google NNP en-wikipedia-org-3679 96 3 Webmaster Webmaster NNP en-wikipedia-org-3679 96 4 Central Central NNP en-wikipedia-org-3679 96 5 Blog Blog NNP en-wikipedia-org-3679 96 6 . . . en-wikipedia-org-3679 97 1 Retrieved Retrieved NNP en-wikipedia-org-3679 97 2 2019 2019 CD en-wikipedia-org-3679 97 3 - - SYM en-wikipedia-org-3679 97 4 07 07 CD en-wikipedia-org-3679 97 5 - - HYPH en-wikipedia-org-3679 97 6 10 10 CD en-wikipedia-org-3679 97 7 . . . en-wikipedia-org-3679 98 1 ^ ^ NNP en-wikipedia-org-3679 98 2 M. M. NNP en-wikipedia-org-3679 98 3 Koster Koster NNP en-wikipedia-org-3679 98 4 , , , en-wikipedia-org-3679 98 5 Stalworthy Stalworthy NNP en-wikipedia-org-3679 98 6 Computing Computing NNP en-wikipedia-org-3679 98 7 , , , en-wikipedia-org-3679 98 8 Ltd. Ltd. NNP en-wikipedia-org-3679 98 9 , , , en-wikipedia-org-3679 98 10 G. G. NNP en-wikipedia-org-3679 98 11 Illyes Illyes NNP en-wikipedia-org-3679 98 12 , , , en-wikipedia-org-3679 98 13 H. H. NNP en-wikipedia-org-3679 98 14 Zeller Zeller NNP en-wikipedia-org-3679 98 15 , , , en-wikipedia-org-3679 98 16 L. L. NNP en-wikipedia-org-3679 98 17 Harvey Harvey NNP en-wikipedia-org-3679 98 18 , , , en-wikipedia-org-3679 98 19 Google Google NNP en-wikipedia-org-3679 98 20 ( ( -LRB- en-wikipedia-org-3679 98 21 2019 2019 CD en-wikipedia-org-3679 98 22 - - SYM en-wikipedia-org-3679 98 23 07 07 CD en-wikipedia-org-3679 98 24 - - HYPH en-wikipedia-org-3679 98 25 01 01 CD en-wikipedia-org-3679 98 26 ) ) -RRB- en-wikipedia-org-3679 98 27 . . . en-wikipedia-org-3679 99 1 " " `` en-wikipedia-org-3679 99 2 Robots Robots NNP en-wikipedia-org-3679 99 3 Exclusion Exclusion NNP en-wikipedia-org-3679 99 4 Protocol Protocol NNP en-wikipedia-org-3679 99 5 draft draft NN en-wikipedia-org-3679 99 6 - - HYPH en-wikipedia-org-3679 99 7 rep rep NNP en-wikipedia-org-3679 99 8 - - HYPH en-wikipedia-org-3679 99 9 wg wg NNP en-wikipedia-org-3679 99 10 - - HYPH en-wikipedia-org-3679 99 11 topic-00 topic-00 NN en-wikipedia-org-3679 99 12 " " '' en-wikipedia-org-3679 99 13 . . . en-wikipedia-org-3679 100 1 IETF IETF NNP en-wikipedia-org-3679 100 2 Documents Documents NNPS en-wikipedia-org-3679 100 3 . . . en-wikipedia-org-3679 101 1 Retrieved Retrieved NNP en-wikipedia-org-3679 101 2 2019 2019 CD en-wikipedia-org-3679 101 3 - - SYM en-wikipedia-org-3679 101 4 09 09 CD en-wikipedia-org-3679 101 5 - - HYPH en-wikipedia-org-3679 101 6 09.CS1 09.cs1 NN en-wikipedia-org-3679 101 7 maint maint NN en-wikipedia-org-3679 101 8 : : : en-wikipedia-org-3679 101 9 multiple multiple JJ en-wikipedia-org-3679 101 10 names name NNS en-wikipedia-org-3679 101 11 : : : en-wikipedia-org-3679 101 12 authors author NNS en-wikipedia-org-3679 101 13 list list NNP en-wikipedia-org-3679 101 14 ( ( -LRB- en-wikipedia-org-3679 101 15 link link NN en-wikipedia-org-3679 101 16 ) ) -RRB- en-wikipedia-org-3679 101 17 ^ ^ NN en-wikipedia-org-3679 101 18 " " `` en-wikipedia-org-3679 101 19 Uncrawled uncrawled JJ en-wikipedia-org-3679 101 20 URLs url NNS en-wikipedia-org-3679 101 21 in in IN en-wikipedia-org-3679 101 22 search search NN en-wikipedia-org-3679 101 23 results result NNS en-wikipedia-org-3679 101 24 " " '' en-wikipedia-org-3679 101 25 . . . en-wikipedia-org-3679 102 1 YouTube YouTube NNP en-wikipedia-org-3679 102 2 . . . en-wikipedia-org-3679 103 1 Oct Oct NNP en-wikipedia-org-3679 103 2 5 5 CD en-wikipedia-org-3679 103 3 , , , en-wikipedia-org-3679 103 4 2009 2009 CD en-wikipedia-org-3679 103 5 . . . en-wikipedia-org-3679 104 1 Archived archived JJ en-wikipedia-org-3679 104 2 from from IN en-wikipedia-org-3679 104 3 the the DT en-wikipedia-org-3679 104 4 original original NN en-wikipedia-org-3679 104 5 on on IN en-wikipedia-org-3679 104 6 2014 2014 CD en-wikipedia-org-3679 104 7 - - SYM en-wikipedia-org-3679 104 8 01 01 CD en-wikipedia-org-3679 104 9 - - HYPH en-wikipedia-org-3679 104 10 06 06 CD en-wikipedia-org-3679 104 11 . . . en-wikipedia-org-3679 105 1 Retrieved Retrieved NNP en-wikipedia-org-3679 105 2 2013 2013 CD en-wikipedia-org-3679 105 3 - - SYM en-wikipedia-org-3679 105 4 12 12 CD en-wikipedia-org-3679 105 5 - - SYM en-wikipedia-org-3679 105 6 29 29 CD en-wikipedia-org-3679 105 7 . . . en-wikipedia-org-3679 106 1 ^ ^ FW en-wikipedia-org-3679 106 2 " " `` en-wikipedia-org-3679 106 3 About about IN en-wikipedia-org-3679 106 4 Ask.com Ask.com NNP en-wikipedia-org-3679 106 5 : : : en-wikipedia-org-3679 106 6 Webmasters Webmasters NNP en-wikipedia-org-3679 106 7 " " '' en-wikipedia-org-3679 106 8 . . . en-wikipedia-org-3679 107 1 About.ask.com about.ask.com ADD en-wikipedia-org-3679 107 2 . . . en-wikipedia-org-3679 108 1 Retrieved Retrieved NNP en-wikipedia-org-3679 108 2 16 16 CD en-wikipedia-org-3679 108 3 February February NNP en-wikipedia-org-3679 108 4 2013 2013 CD en-wikipedia-org-3679 108 5 . . . en-wikipedia-org-3679 109 1 ^ ^ FW en-wikipedia-org-3679 109 2 " " `` en-wikipedia-org-3679 109 3 About about IN en-wikipedia-org-3679 109 4 AOL AOL NNP en-wikipedia-org-3679 109 5 Search Search NNP en-wikipedia-org-3679 109 6 " " '' en-wikipedia-org-3679 109 7 . . . en-wikipedia-org-3679 110 1 Search.aol.com search.aol.com ADD en-wikipedia-org-3679 110 2 . . . en-wikipedia-org-3679 111 1 Retrieved Retrieved NNP en-wikipedia-org-3679 111 2 16 16 CD en-wikipedia-org-3679 111 3 February February NNP en-wikipedia-org-3679 111 4 2013 2013 CD en-wikipedia-org-3679 111 5 . . . en-wikipedia-org-3679 112 1 ^ ^ FW en-wikipedia-org-3679 112 2 " " `` en-wikipedia-org-3679 112 3 Baiduspider Baiduspider NNP en-wikipedia-org-3679 112 4 " " '' en-wikipedia-org-3679 112 5 . . . en-wikipedia-org-3679 113 1 Baidu.com Baidu.com NNP en-wikipedia-org-3679 113 2 . . . en-wikipedia-org-3679 114 1 Retrieved Retrieved NNP en-wikipedia-org-3679 114 2 16 16 CD en-wikipedia-org-3679 114 3 February February NNP en-wikipedia-org-3679 114 4 2013 2013 CD en-wikipedia-org-3679 114 5 . . . en-wikipedia-org-3679 115 1 ^ ^ FW en-wikipedia-org-3679 115 2 " " `` en-wikipedia-org-3679 115 3 DuckDuckGo DuckDuckGo NNP en-wikipedia-org-3679 115 4 Bot Bot NNP en-wikipedia-org-3679 115 5 " " '' en-wikipedia-org-3679 115 6 . . . en-wikipedia-org-3679 116 1 DuckDuckGo.com duckduckgo.com NN en-wikipedia-org-3679 116 2 . . . en-wikipedia-org-3679 117 1 Retrieved retrieve VBN en-wikipedia-org-3679 117 2 25 25 CD en-wikipedia-org-3679 117 3 April April NNP en-wikipedia-org-3679 117 4 2017 2017 CD en-wikipedia-org-3679 117 5 . . . en-wikipedia-org-3679 118 1 ^ ^ NNP en-wikipedia-org-3679 118 2 a a DT en-wikipedia-org-3679 118 3 b b NN en-wikipedia-org-3679 118 4 " " `` en-wikipedia-org-3679 118 5 Webmasters webmaster NNS en-wikipedia-org-3679 118 6 : : : en-wikipedia-org-3679 118 7 Robots.txt Robots.txt NNP en-wikipedia-org-3679 118 8 Specifications Specifications NNPS en-wikipedia-org-3679 118 9 " " '' en-wikipedia-org-3679 118 10 . . . en-wikipedia-org-3679 119 1 Google Google NNP en-wikipedia-org-3679 119 2 Developers Developers NNP en-wikipedia-org-3679 119 3 . . . en-wikipedia-org-3679 120 1 Archived archived JJ en-wikipedia-org-3679 120 2 from from IN en-wikipedia-org-3679 120 3 the the DT en-wikipedia-org-3679 120 4 original original NN en-wikipedia-org-3679 120 5 on on IN en-wikipedia-org-3679 120 6 2013 2013 CD en-wikipedia-org-3679 120 7 - - SYM en-wikipedia-org-3679 120 8 01 01 CD en-wikipedia-org-3679 120 9 - - HYPH en-wikipedia-org-3679 120 10 15 15 CD en-wikipedia-org-3679 120 11 . . . en-wikipedia-org-3679 121 1 Retrieved Retrieved NNP en-wikipedia-org-3679 121 2 16 16 CD en-wikipedia-org-3679 121 3 February February NNP en-wikipedia-org-3679 121 4 2013 2013 CD en-wikipedia-org-3679 121 5 . . . en-wikipedia-org-3679 122 1 ^ ^ FW en-wikipedia-org-3679 122 2 " " `` en-wikipedia-org-3679 122 3 Submitting submit VBG en-wikipedia-org-3679 122 4 your -PRON- PRP$ en-wikipedia-org-3679 122 5 website website NN en-wikipedia-org-3679 122 6 to to IN en-wikipedia-org-3679 122 7 Yahoo Yahoo NNP en-wikipedia-org-3679 122 8 ! ! . en-wikipedia-org-3679 123 1 Search search NN en-wikipedia-org-3679 123 2 " " '' en-wikipedia-org-3679 123 3 . . . en-wikipedia-org-3679 124 1 Archived archived JJ en-wikipedia-org-3679 124 2 from from IN en-wikipedia-org-3679 124 3 the the DT en-wikipedia-org-3679 124 4 original original NN en-wikipedia-org-3679 124 5 on on IN en-wikipedia-org-3679 124 6 2013 2013 CD en-wikipedia-org-3679 124 7 - - SYM en-wikipedia-org-3679 124 8 01 01 CD en-wikipedia-org-3679 124 9 - - HYPH en-wikipedia-org-3679 124 10 21 21 CD en-wikipedia-org-3679 124 11 . . . en-wikipedia-org-3679 125 1 Retrieved Retrieved NNP en-wikipedia-org-3679 125 2 16 16 CD en-wikipedia-org-3679 125 3 February February NNP en-wikipedia-org-3679 125 4 2013 2013 CD en-wikipedia-org-3679 125 5 . . . en-wikipedia-org-3679 126 1 ^ ^ NNP en-wikipedia-org-3679 126 2 a a DT en-wikipedia-org-3679 126 3 b b NN en-wikipedia-org-3679 126 4 " " `` en-wikipedia-org-3679 126 5 Using use VBG en-wikipedia-org-3679 126 6 robots.txt robots.txt NN en-wikipedia-org-3679 126 7 " " '' en-wikipedia-org-3679 126 8 . . . en-wikipedia-org-3679 127 1 Help.yandex.com Help.yandex.com NNP en-wikipedia-org-3679 127 2 . . . en-wikipedia-org-3679 128 1 Archived archived JJ en-wikipedia-org-3679 128 2 from from IN en-wikipedia-org-3679 128 3 the the DT en-wikipedia-org-3679 128 4 original original NN en-wikipedia-org-3679 128 5 on on IN en-wikipedia-org-3679 128 6 2013 2013 CD en-wikipedia-org-3679 128 7 - - SYM en-wikipedia-org-3679 128 8 01 01 CD en-wikipedia-org-3679 128 9 - - HYPH en-wikipedia-org-3679 128 10 25 25 CD en-wikipedia-org-3679 128 11 . . . en-wikipedia-org-3679 129 1 Retrieved Retrieved NNP en-wikipedia-org-3679 129 2 16 16 CD en-wikipedia-org-3679 129 3 February February NNP en-wikipedia-org-3679 129 4 2013 2013 CD en-wikipedia-org-3679 129 5 . . . en-wikipedia-org-3679 130 1 ^ ^ NNP en-wikipedia-org-3679 130 2 a a DT en-wikipedia-org-3679 130 3 b b NN en-wikipedia-org-3679 130 4 " " `` en-wikipedia-org-3679 130 5 Robots Robots NNP en-wikipedia-org-3679 130 6 Exclusion Exclusion NNP en-wikipedia-org-3679 130 7 Protocol Protocol NNP en-wikipedia-org-3679 130 8 : : : en-wikipedia-org-3679 130 9 joining join VBG en-wikipedia-org-3679 130 10 together together RB en-wikipedia-org-3679 130 11 to to TO en-wikipedia-org-3679 130 12 provide provide VB en-wikipedia-org-3679 130 13 better well JJR en-wikipedia-org-3679 130 14 documentation documentation NN en-wikipedia-org-3679 130 15 " " '' en-wikipedia-org-3679 130 16 . . . en-wikipedia-org-3679 131 1 Blogs.bing.com blogs.bing.com ADD en-wikipedia-org-3679 131 2 . . . en-wikipedia-org-3679 132 1 Archived archived JJ en-wikipedia-org-3679 132 2 from from IN en-wikipedia-org-3679 132 3 the the DT en-wikipedia-org-3679 132 4 original original NN en-wikipedia-org-3679 132 5 on on IN en-wikipedia-org-3679 132 6 2014 2014 CD en-wikipedia-org-3679 132 7 - - SYM en-wikipedia-org-3679 132 8 08 08 CD en-wikipedia-org-3679 132 9 - - SYM en-wikipedia-org-3679 132 10 18 18 CD en-wikipedia-org-3679 132 11 . . . en-wikipedia-org-3679 133 1 Retrieved Retrieved NNP en-wikipedia-org-3679 133 2 16 16 CD en-wikipedia-org-3679 133 3 February February NNP en-wikipedia-org-3679 133 4 2013 2013 CD en-wikipedia-org-3679 133 5 . . . en-wikipedia-org-3679 134 1 ^ ^ FW en-wikipedia-org-3679 134 2 " " `` en-wikipedia-org-3679 134 3 How how WRB en-wikipedia-org-3679 134 4 to to TO en-wikipedia-org-3679 134 5 Create create VB en-wikipedia-org-3679 134 6 a a DT en-wikipedia-org-3679 134 7 Robots.txt Robots.txt NNP en-wikipedia-org-3679 134 8 File File NNP en-wikipedia-org-3679 134 9 - - HYPH en-wikipedia-org-3679 134 10 Bing Bing NNP en-wikipedia-org-3679 134 11 Webmaster Webmaster NNP en-wikipedia-org-3679 134 12 Tools Tools NNPS en-wikipedia-org-3679 134 13 " " '' en-wikipedia-org-3679 134 14 . . . en-wikipedia-org-3679 135 1 www.bing.com www.bing.com ADD en-wikipedia-org-3679 135 2 . . . en-wikipedia-org-3679 136 1 Retrieved Retrieved NNP en-wikipedia-org-3679 136 2 2019 2019 CD en-wikipedia-org-3679 136 3 - - SYM en-wikipedia-org-3679 136 4 02 02 CD en-wikipedia-org-3679 136 5 - - HYPH en-wikipedia-org-3679 136 6 06 06 CD en-wikipedia-org-3679 136 7 . . . en-wikipedia-org-3679 137 1 ^ ^ NNP en-wikipedia-org-3679 137 2 Jason Jason NNP en-wikipedia-org-3679 137 3 Scott Scott NNP en-wikipedia-org-3679 137 4 . . . en-wikipedia-org-3679 138 1 " " `` en-wikipedia-org-3679 138 2 Robots.txt robots.txt NN en-wikipedia-org-3679 138 3 is be VBZ en-wikipedia-org-3679 138 4 a a DT en-wikipedia-org-3679 138 5 suicide suicide NN en-wikipedia-org-3679 138 6 note note NN en-wikipedia-org-3679 138 7 " " '' en-wikipedia-org-3679 138 8 . . . en-wikipedia-org-3679 139 1 Archive Archive NNP en-wikipedia-org-3679 139 2 Team Team NNP en-wikipedia-org-3679 139 3 . . . en-wikipedia-org-3679 140 1 Archived archived JJ en-wikipedia-org-3679 140 2 from from IN en-wikipedia-org-3679 140 3 the the DT en-wikipedia-org-3679 140 4 original original NN en-wikipedia-org-3679 140 5 on on IN en-wikipedia-org-3679 140 6 2017 2017 CD en-wikipedia-org-3679 140 7 - - HYPH en-wikipedia-org-3679 140 8 02 02 CD en-wikipedia-org-3679 140 9 - - HYPH en-wikipedia-org-3679 140 10 18 18 CD en-wikipedia-org-3679 140 11 . . . en-wikipedia-org-3679 141 1 Retrieved retrieve VBN en-wikipedia-org-3679 141 2 18 18 CD en-wikipedia-org-3679 141 3 February February NNP en-wikipedia-org-3679 141 4 2017 2017 CD en-wikipedia-org-3679 141 5 . . . en-wikipedia-org-3679 142 1 ^ ^ FW en-wikipedia-org-3679 142 2 " " `` en-wikipedia-org-3679 142 3 Robots.txt Robots.txt NNP en-wikipedia-org-3679 142 4 meant mean VBD en-wikipedia-org-3679 142 5 for for IN en-wikipedia-org-3679 142 6 search search NN en-wikipedia-org-3679 142 7 engines engine NNS en-wikipedia-org-3679 142 8 do do VBP en-wikipedia-org-3679 142 9 n't not RB en-wikipedia-org-3679 142 10 work work VB en-wikipedia-org-3679 142 11 well well RB en-wikipedia-org-3679 142 12 for for IN en-wikipedia-org-3679 142 13 web web NN en-wikipedia-org-3679 142 14 archives archive NNS en-wikipedia-org-3679 142 15 | | NNP en-wikipedia-org-3679 142 16 Internet Internet NNP en-wikipedia-org-3679 142 17 Archive Archive NNP en-wikipedia-org-3679 142 18 Blogs Blogs NNP en-wikipedia-org-3679 142 19 " " '' en-wikipedia-org-3679 142 20 . . . en-wikipedia-org-3679 143 1 blog.archive.org blog.archive.org NNP en-wikipedia-org-3679 143 2 . . . en-wikipedia-org-3679 144 1 Archived archived JJ en-wikipedia-org-3679 144 2 from from IN en-wikipedia-org-3679 144 3 the the DT en-wikipedia-org-3679 144 4 original original NN en-wikipedia-org-3679 144 5 on on IN en-wikipedia-org-3679 144 6 2018 2018 CD en-wikipedia-org-3679 144 7 - - SYM en-wikipedia-org-3679 144 8 12 12 CD en-wikipedia-org-3679 144 9 - - HYPH en-wikipedia-org-3679 144 10 04 04 CD en-wikipedia-org-3679 144 11 . . . en-wikipedia-org-3679 145 1 Retrieved retrieve VBN en-wikipedia-org-3679 145 2 2018 2018 CD en-wikipedia-org-3679 145 3 - - SYM en-wikipedia-org-3679 145 4 12 12 CD en-wikipedia-org-3679 145 5 - - HYPH en-wikipedia-org-3679 145 6 01 01 CD en-wikipedia-org-3679 145 7 . . . en-wikipedia-org-3679 146 1 ^ ^ NNP en-wikipedia-org-3679 146 2 a a DT en-wikipedia-org-3679 146 3 b b NNP en-wikipedia-org-3679 146 4 Jones Jones NNP en-wikipedia-org-3679 146 5 , , , en-wikipedia-org-3679 146 6 Brad Brad NNP en-wikipedia-org-3679 146 7 ( ( -LRB- en-wikipedia-org-3679 146 8 24 24 CD en-wikipedia-org-3679 146 9 April April NNP en-wikipedia-org-3679 146 10 2017 2017 CD en-wikipedia-org-3679 146 11 ) ) -RRB- en-wikipedia-org-3679 146 12 . . . en-wikipedia-org-3679 147 1 " " `` en-wikipedia-org-3679 147 2 The the DT en-wikipedia-org-3679 147 3 Internet internet NN en-wikipedia-org-3679 147 4 Archive archive NN en-wikipedia-org-3679 147 5 Will Will MD en-wikipedia-org-3679 147 6 Ignore Ignore NNP en-wikipedia-org-3679 147 7 Robots.txt robots.txt NN en-wikipedia-org-3679 147 8 Files file NNS en-wikipedia-org-3679 147 9 to to TO en-wikipedia-org-3679 147 10 Maintain maintain VB en-wikipedia-org-3679 147 11 Accuracy accuracy NN en-wikipedia-org-3679 147 12 " " '' en-wikipedia-org-3679 147 13 . . . en-wikipedia-org-3679 148 1 Digital Digital NNP en-wikipedia-org-3679 148 2 Trends Trends NNPS en-wikipedia-org-3679 148 3 . . . en-wikipedia-org-3679 149 1 Archived archived JJ en-wikipedia-org-3679 149 2 from from IN en-wikipedia-org-3679 149 3 the the DT en-wikipedia-org-3679 149 4 original original NN en-wikipedia-org-3679 149 5 on on IN en-wikipedia-org-3679 149 6 2017 2017 CD en-wikipedia-org-3679 149 7 - - SYM en-wikipedia-org-3679 149 8 05 05 CD en-wikipedia-org-3679 149 9 - - HYPH en-wikipedia-org-3679 149 10 16 16 CD en-wikipedia-org-3679 149 11 . . . en-wikipedia-org-3679 150 1 Retrieved retrieve VBN en-wikipedia-org-3679 150 2 8 8 CD en-wikipedia-org-3679 150 3 May May NNP en-wikipedia-org-3679 150 4 2017 2017 CD en-wikipedia-org-3679 150 5 . . . en-wikipedia-org-3679 151 1 ^ ^ FW en-wikipedia-org-3679 151 2 " " `` en-wikipedia-org-3679 151 3 Block Block NNP en-wikipedia-org-3679 151 4 URLs URLs NNPS en-wikipedia-org-3679 151 5 with with IN en-wikipedia-org-3679 151 6 robots.txt robots.txt NN en-wikipedia-org-3679 151 7 : : : en-wikipedia-org-3679 151 8 Learn learn VB en-wikipedia-org-3679 151 9 about about IN en-wikipedia-org-3679 151 10 robots.txt robots.txt NN en-wikipedia-org-3679 151 11 files file NNS en-wikipedia-org-3679 151 12 " " '' en-wikipedia-org-3679 151 13 . . . en-wikipedia-org-3679 152 1 Archived archived JJ en-wikipedia-org-3679 152 2 from from IN en-wikipedia-org-3679 152 3 the the DT en-wikipedia-org-3679 152 4 original original NN en-wikipedia-org-3679 152 5 on on IN en-wikipedia-org-3679 152 6 2015 2015 CD en-wikipedia-org-3679 152 7 - - SYM en-wikipedia-org-3679 152 8 08 08 CD en-wikipedia-org-3679 152 9 - - HYPH en-wikipedia-org-3679 152 10 14 14 CD en-wikipedia-org-3679 152 11 . . . en-wikipedia-org-3679 153 1 Retrieved retrieve VBN en-wikipedia-org-3679 153 2 2015 2015 CD en-wikipedia-org-3679 153 3 - - SYM en-wikipedia-org-3679 153 4 08 08 CD en-wikipedia-org-3679 153 5 - - SYM en-wikipedia-org-3679 153 6 10 10 CD en-wikipedia-org-3679 153 7 . . . en-wikipedia-org-3679 154 1 ^ ^ FW en-wikipedia-org-3679 154 2 " " `` en-wikipedia-org-3679 154 3 Robots.txt Robots.txt NNP en-wikipedia-org-3679 154 4 tells tell VBZ en-wikipedia-org-3679 154 5 hackers hacker NNS en-wikipedia-org-3679 154 6 the the DT en-wikipedia-org-3679 154 7 places place NNS en-wikipedia-org-3679 154 8 you -PRON- PRP en-wikipedia-org-3679 154 9 do do VBP en-wikipedia-org-3679 154 10 n't not RB en-wikipedia-org-3679 154 11 want want VB en-wikipedia-org-3679 154 12 them -PRON- PRP en-wikipedia-org-3679 154 13 to to TO en-wikipedia-org-3679 154 14 look look VB en-wikipedia-org-3679 154 15 " " '' en-wikipedia-org-3679 154 16 . . . en-wikipedia-org-3679 155 1 The the DT en-wikipedia-org-3679 155 2 Register Register NNP en-wikipedia-org-3679 155 3 . . . en-wikipedia-org-3679 156 1 Archived archived JJ en-wikipedia-org-3679 156 2 from from IN en-wikipedia-org-3679 156 3 the the DT en-wikipedia-org-3679 156 4 original original NN en-wikipedia-org-3679 156 5 on on IN en-wikipedia-org-3679 156 6 2015 2015 CD en-wikipedia-org-3679 156 7 - - SYM en-wikipedia-org-3679 156 8 08 08 CD en-wikipedia-org-3679 156 9 - - SYM en-wikipedia-org-3679 156 10 21 21 CD en-wikipedia-org-3679 156 11 . . . en-wikipedia-org-3679 157 1 Retrieved Retrieved NNP en-wikipedia-org-3679 157 2 August August NNP en-wikipedia-org-3679 157 3 12 12 CD en-wikipedia-org-3679 157 4 , , , en-wikipedia-org-3679 157 5 2015 2015 CD en-wikipedia-org-3679 157 6 . . . en-wikipedia-org-3679 158 1 ^ ^ FW en-wikipedia-org-3679 158 2 " " `` en-wikipedia-org-3679 158 3 Guide Guide NNP en-wikipedia-org-3679 158 4 to to IN en-wikipedia-org-3679 158 5 General General NNP en-wikipedia-org-3679 158 6 Server Server NNP en-wikipedia-org-3679 158 7 Security Security NNP en-wikipedia-org-3679 158 8 " " '' en-wikipedia-org-3679 158 9 ( ( -LRB- en-wikipedia-org-3679 158 10 PDF PDF NNP en-wikipedia-org-3679 158 11 ) ) -RRB- en-wikipedia-org-3679 158 12 . . . en-wikipedia-org-3679 159 1 National National NNP en-wikipedia-org-3679 159 2 Institute Institute NNP en-wikipedia-org-3679 159 3 of of IN en-wikipedia-org-3679 159 4 Standards Standards NNPS en-wikipedia-org-3679 159 5 and and CC en-wikipedia-org-3679 159 6 Technology Technology NNP en-wikipedia-org-3679 159 7 . . . en-wikipedia-org-3679 160 1 July July NNP en-wikipedia-org-3679 160 2 2008 2008 CD en-wikipedia-org-3679 160 3 . . . en-wikipedia-org-3679 161 1 Archived Archived NNP en-wikipedia-org-3679 161 2 ( ( -LRB- en-wikipedia-org-3679 161 3 PDF PDF NNP en-wikipedia-org-3679 161 4 ) ) -RRB- en-wikipedia-org-3679 161 5 from from IN en-wikipedia-org-3679 161 6 the the DT en-wikipedia-org-3679 161 7 original original NN en-wikipedia-org-3679 161 8 on on IN en-wikipedia-org-3679 161 9 2011 2011 CD en-wikipedia-org-3679 161 10 - - SYM en-wikipedia-org-3679 161 11 10 10 CD en-wikipedia-org-3679 161 12 - - HYPH en-wikipedia-org-3679 161 13 08 08 CD en-wikipedia-org-3679 161 14 . . . en-wikipedia-org-3679 162 1 Retrieved Retrieved NNP en-wikipedia-org-3679 162 2 August August NNP en-wikipedia-org-3679 162 3 12 12 CD en-wikipedia-org-3679 162 4 , , , en-wikipedia-org-3679 162 5 2015 2015 CD en-wikipedia-org-3679 162 6 . . . en-wikipedia-org-3679 163 1 ^ ^ NFP en-wikipedia-org-3679 163 2 Sverre Sverre NNP en-wikipedia-org-3679 163 3 H. H. NNP en-wikipedia-org-3679 163 4 Huseby Huseby NNP en-wikipedia-org-3679 163 5 ( ( -LRB- en-wikipedia-org-3679 163 6 2004 2004 CD en-wikipedia-org-3679 163 7 ) ) -RRB- en-wikipedia-org-3679 163 8 . . . en-wikipedia-org-3679 164 1 Innocent innocent JJ en-wikipedia-org-3679 164 2 Code Code NNP en-wikipedia-org-3679 164 3 : : : en-wikipedia-org-3679 164 4 A A NNP en-wikipedia-org-3679 164 5 Security Security NNP en-wikipedia-org-3679 164 6 Wake wake NN en-wikipedia-org-3679 164 7 - - HYPH en-wikipedia-org-3679 164 8 Up up RP en-wikipedia-org-3679 164 9 Call call NN en-wikipedia-org-3679 164 10 for for IN en-wikipedia-org-3679 164 11 Web web NN en-wikipedia-org-3679 164 12 Programmers Programmers NNPS en-wikipedia-org-3679 164 13 . . . en-wikipedia-org-3679 165 1 John John NNP en-wikipedia-org-3679 165 2 Wiley Wiley NNP en-wikipedia-org-3679 165 3 & & CC en-wikipedia-org-3679 165 4 Sons Sons NNP en-wikipedia-org-3679 165 5 . . . en-wikipedia-org-3679 166 1 pp pp NNP en-wikipedia-org-3679 166 2 . . . en-wikipedia-org-3679 167 1 91–92 91–92 CD en-wikipedia-org-3679 167 2 . . . en-wikipedia-org-3679 168 1 ISBN ISBN NNP en-wikipedia-org-3679 168 2   _SP en-wikipedia-org-3679 168 3 9780470857472 9780470857472 CD en-wikipedia-org-3679 168 4 . . . en-wikipedia-org-3679 169 1 Archived archived JJ en-wikipedia-org-3679 169 2 from from IN en-wikipedia-org-3679 169 3 the the DT en-wikipedia-org-3679 169 4 original original NN en-wikipedia-org-3679 169 5 on on IN en-wikipedia-org-3679 169 6 2016 2016 CD en-wikipedia-org-3679 169 7 - - HYPH en-wikipedia-org-3679 169 8 04 04 CD en-wikipedia-org-3679 169 9 - - HYPH en-wikipedia-org-3679 169 10 01 01 CD en-wikipedia-org-3679 169 11 . . . en-wikipedia-org-3679 170 1 Retrieved retrieve VBN en-wikipedia-org-3679 170 2 2015 2015 CD en-wikipedia-org-3679 170 3 - - SYM en-wikipedia-org-3679 170 4 08 08 CD en-wikipedia-org-3679 170 5 - - HYPH en-wikipedia-org-3679 170 6 12 12 CD en-wikipedia-org-3679 170 7 . . . en-wikipedia-org-3679 171 1 ^ ^ FW en-wikipedia-org-3679 171 2 " " `` en-wikipedia-org-3679 171 3 List List NNP en-wikipedia-org-3679 171 4 of of IN en-wikipedia-org-3679 171 5 User User NNP en-wikipedia-org-3679 171 6 - - HYPH en-wikipedia-org-3679 171 7 Agents Agents NNP en-wikipedia-org-3679 171 8 ( ( -LRB- en-wikipedia-org-3679 171 9 Spiders Spiders NNPS en-wikipedia-org-3679 171 10 , , , en-wikipedia-org-3679 171 11 Robots Robots NNPS en-wikipedia-org-3679 171 12 , , , en-wikipedia-org-3679 171 13 Browser Browser NNP en-wikipedia-org-3679 171 14 ) ) -RRB- en-wikipedia-org-3679 171 15 " " '' en-wikipedia-org-3679 171 16 . . . en-wikipedia-org-3679 172 1 User-agents.org user-agents.org UH en-wikipedia-org-3679 172 2 . . . en-wikipedia-org-3679 173 1 Archived archived JJ en-wikipedia-org-3679 173 2 from from IN en-wikipedia-org-3679 173 3 the the DT en-wikipedia-org-3679 173 4 original original NN en-wikipedia-org-3679 173 5 on on IN en-wikipedia-org-3679 173 6 2014 2014 CD en-wikipedia-org-3679 173 7 - - SYM en-wikipedia-org-3679 173 8 01 01 CD en-wikipedia-org-3679 173 9 - - HYPH en-wikipedia-org-3679 173 10 07 07 CD en-wikipedia-org-3679 173 11 . . . en-wikipedia-org-3679 174 1 Retrieved Retrieved NNP en-wikipedia-org-3679 174 2 2013 2013 CD en-wikipedia-org-3679 174 3 - - SYM en-wikipedia-org-3679 174 4 12 12 CD en-wikipedia-org-3679 174 5 - - SYM en-wikipedia-org-3679 174 6 29 29 CD en-wikipedia-org-3679 174 7 . . . en-wikipedia-org-3679 175 1 ^ ^ FW en-wikipedia-org-3679 175 2 " " `` en-wikipedia-org-3679 175 3 Access Access NNP en-wikipedia-org-3679 175 4 Control Control NNP en-wikipedia-org-3679 175 5 - - HYPH en-wikipedia-org-3679 175 6 Apache Apache NNP en-wikipedia-org-3679 175 7 HTTP HTTP NNP en-wikipedia-org-3679 175 8 Server Server NNP en-wikipedia-org-3679 175 9 " " '' en-wikipedia-org-3679 175 10 . . . en-wikipedia-org-3679 176 1 Httpd.apache.org Httpd.apache.org NNP en-wikipedia-org-3679 176 2 . . . en-wikipedia-org-3679 177 1 Archived archived JJ en-wikipedia-org-3679 177 2 from from IN en-wikipedia-org-3679 177 3 the the DT en-wikipedia-org-3679 177 4 original original NN en-wikipedia-org-3679 177 5 on on IN en-wikipedia-org-3679 177 6 2013 2013 CD en-wikipedia-org-3679 177 7 - - SYM en-wikipedia-org-3679 177 8 12 12 CD en-wikipedia-org-3679 177 9 - - SYM en-wikipedia-org-3679 177 10 29 29 CD en-wikipedia-org-3679 177 11 . . . en-wikipedia-org-3679 178 1 Retrieved Retrieved NNP en-wikipedia-org-3679 178 2 2013 2013 CD en-wikipedia-org-3679 178 3 - - SYM en-wikipedia-org-3679 178 4 12 12 CD en-wikipedia-org-3679 178 5 - - SYM en-wikipedia-org-3679 178 6 29 29 CD en-wikipedia-org-3679 178 7 . . . en-wikipedia-org-3679 179 1 ^ ^ FW en-wikipedia-org-3679 179 2 " " `` en-wikipedia-org-3679 179 3 Deny Deny NNP en-wikipedia-org-3679 179 4 Strings Strings NNP en-wikipedia-org-3679 179 5 for for IN en-wikipedia-org-3679 179 6 Filtering filter VBG en-wikipedia-org-3679 179 7 Rules Rules NNPS en-wikipedia-org-3679 179 8   _SP en-wikipedia-org-3679 179 9 : : : en-wikipedia-org-3679 179 10 The the DT en-wikipedia-org-3679 179 11 Official Official NNP en-wikipedia-org-3679 179 12 Microsoft Microsoft NNP en-wikipedia-org-3679 179 13 IIS iis RB en-wikipedia-org-3679 179 14 Site Site NNP en-wikipedia-org-3679 179 15 " " '' en-wikipedia-org-3679 179 16 . . . en-wikipedia-org-3679 180 1 Iis.net Iis.net NNP en-wikipedia-org-3679 180 2 . . . en-wikipedia-org-3679 181 1 2013 2013 CD en-wikipedia-org-3679 181 2 - - SYM en-wikipedia-org-3679 181 3 11 11 CD en-wikipedia-org-3679 181 4 - - HYPH en-wikipedia-org-3679 181 5 06 06 CD en-wikipedia-org-3679 181 6 . . . en-wikipedia-org-3679 182 1 Archived archived JJ en-wikipedia-org-3679 182 2 from from IN en-wikipedia-org-3679 182 3 the the DT en-wikipedia-org-3679 182 4 original original NN en-wikipedia-org-3679 182 5 on on IN en-wikipedia-org-3679 182 6 2014 2014 CD en-wikipedia-org-3679 182 7 - - SYM en-wikipedia-org-3679 182 8 01 01 CD en-wikipedia-org-3679 182 9 - - HYPH en-wikipedia-org-3679 182 10 01 01 CD en-wikipedia-org-3679 182 11 . . . en-wikipedia-org-3679 183 1 Retrieved Retrieved NNP en-wikipedia-org-3679 183 2 2013 2013 CD en-wikipedia-org-3679 183 3 - - SYM en-wikipedia-org-3679 183 4 12 12 CD en-wikipedia-org-3679 183 5 - - SYM en-wikipedia-org-3679 183 6 29 29 CD en-wikipedia-org-3679 183 7 . . . en-wikipedia-org-3679 184 1 ^ ^ FW en-wikipedia-org-3679 184 2 " " `` en-wikipedia-org-3679 184 3 Google Google NNP en-wikipedia-org-3679 184 4 humans.txt humans.txt NNP en-wikipedia-org-3679 184 5 " " '' en-wikipedia-org-3679 184 6 . . . en-wikipedia-org-3679 185 1 Retrieved Retrieved NNP en-wikipedia-org-3679 185 2 October October NNP en-wikipedia-org-3679 185 3 3 3 CD en-wikipedia-org-3679 185 4 , , , en-wikipedia-org-3679 185 5 2019 2019 CD en-wikipedia-org-3679 185 6 . . . en-wikipedia-org-3679 186 1 ^ ^ FW en-wikipedia-org-3679 186 2 " " `` en-wikipedia-org-3679 186 3 Github Github NNP en-wikipedia-org-3679 186 4 humans.txt humans.txt NNP en-wikipedia-org-3679 186 5 " " '' en-wikipedia-org-3679 186 6 . . . en-wikipedia-org-3679 187 1 Retrieved Retrieved NNP en-wikipedia-org-3679 187 2 October October NNP en-wikipedia-org-3679 187 3 3 3 CD en-wikipedia-org-3679 187 4 , , , en-wikipedia-org-3679 187 5 2019 2019 CD en-wikipedia-org-3679 187 6 . . . en-wikipedia-org-3679 188 1 ^ ^ NNP en-wikipedia-org-3679 188 2 Newman Newman NNP en-wikipedia-org-3679 188 3 , , , en-wikipedia-org-3679 188 4 Lily Lily NNP en-wikipedia-org-3679 188 5 Hay Hay NNP en-wikipedia-org-3679 188 6 ( ( -LRB- en-wikipedia-org-3679 188 7 2014 2014 CD en-wikipedia-org-3679 188 8 - - SYM en-wikipedia-org-3679 188 9 07 07 CD en-wikipedia-org-3679 188 10 - - HYPH en-wikipedia-org-3679 188 11 03 03 CD en-wikipedia-org-3679 188 12 ) ) -RRB- en-wikipedia-org-3679 188 13 . . . en-wikipedia-org-3679 189 1 " " `` en-wikipedia-org-3679 189 2 Is be VBZ en-wikipedia-org-3679 189 3 This this DT en-wikipedia-org-3679 189 4 a a DT en-wikipedia-org-3679 189 5 Google Google NNP en-wikipedia-org-3679 189 6 Easter Easter NNP en-wikipedia-org-3679 189 7 Egg Egg NNP en-wikipedia-org-3679 189 8 or or CC en-wikipedia-org-3679 189 9 Proof Proof NNP en-wikipedia-org-3679 189 10 That that IN en-wikipedia-org-3679 189 11 Skynet Skynet NNP en-wikipedia-org-3679 189 12 Is be VBZ en-wikipedia-org-3679 189 13 Actually actually RB en-wikipedia-org-3679 189 14 Plotting plot VBG en-wikipedia-org-3679 189 15 World world NN en-wikipedia-org-3679 189 16 Domination Domination NNP en-wikipedia-org-3679 189 17 ? ? . en-wikipedia-org-3679 189 18 " " '' en-wikipedia-org-3679 189 19 . . . en-wikipedia-org-3679 190 1 Slate Slate NNP en-wikipedia-org-3679 190 2 Magazine Magazine NNP en-wikipedia-org-3679 190 3 . . . en-wikipedia-org-3679 191 1 Retrieved Retrieved NNP en-wikipedia-org-3679 191 2 2019 2019 CD en-wikipedia-org-3679 191 3 - - SYM en-wikipedia-org-3679 191 4 10 10 CD en-wikipedia-org-3679 191 5 - - HYPH en-wikipedia-org-3679 191 6 03 03 CD en-wikipedia-org-3679 191 7 . . . en-wikipedia-org-3679 192 1 ^ ^ NNP en-wikipedia-org-3679 192 2 " " `` en-wikipedia-org-3679 192 3 /killer /killer NN en-wikipedia-org-3679 192 4 - - HYPH en-wikipedia-org-3679 192 5 robots.txt robots.txt NNP en-wikipedia-org-3679 192 6 " " '' en-wikipedia-org-3679 192 7 . . . en-wikipedia-org-3679 193 1 2018 2018 CD en-wikipedia-org-3679 193 2 - - SYM en-wikipedia-org-3679 193 3 01 01 CD en-wikipedia-org-3679 193 4 - - HYPH en-wikipedia-org-3679 193 5 10 10 CD en-wikipedia-org-3679 193 6 . . . en-wikipedia-org-3679 194 1 Archived archived JJ en-wikipedia-org-3679 194 2 from from IN en-wikipedia-org-3679 194 3 the the DT en-wikipedia-org-3679 194 4 original original NN en-wikipedia-org-3679 194 5 on on IN en-wikipedia-org-3679 194 6 2018 2018 CD en-wikipedia-org-3679 194 7 - - SYM en-wikipedia-org-3679 194 8 01 01 CD en-wikipedia-org-3679 194 9 - - HYPH en-wikipedia-org-3679 194 10 10 10 CD en-wikipedia-org-3679 194 11 . . . en-wikipedia-org-3679 195 1 Retrieved retrieve VBN en-wikipedia-org-3679 195 2 2018 2018 CD en-wikipedia-org-3679 195 3 - - SYM en-wikipedia-org-3679 195 4 05 05 CD en-wikipedia-org-3679 195 5 - - HYPH en-wikipedia-org-3679 195 6 25 25 CD en-wikipedia-org-3679 195 7 . . . en-wikipedia-org-3679 196 1 ^ ^ NNP en-wikipedia-org-3679 196 2 " " `` en-wikipedia-org-3679 196 3 To to TO en-wikipedia-org-3679 196 4 crawl crawl NN en-wikipedia-org-3679 196 5 or or CC en-wikipedia-org-3679 196 6 not not RB en-wikipedia-org-3679 196 7 to to TO en-wikipedia-org-3679 196 8 crawl crawl VB en-wikipedia-org-3679 196 9 , , , en-wikipedia-org-3679 196 10 that that RB en-wikipedia-org-3679 196 11 is be VBZ en-wikipedia-org-3679 196 12 BingBot BingBot NNP en-wikipedia-org-3679 196 13 's 's POS en-wikipedia-org-3679 196 14 question question NN en-wikipedia-org-3679 196 15 " " '' en-wikipedia-org-3679 196 16 . . . en-wikipedia-org-3679 197 1 3 3 CD en-wikipedia-org-3679 197 2 May May NNP en-wikipedia-org-3679 197 3 2012 2012 CD en-wikipedia-org-3679 197 4 . . . en-wikipedia-org-3679 198 1 Archived archived JJ en-wikipedia-org-3679 198 2 from from IN en-wikipedia-org-3679 198 3 the the DT en-wikipedia-org-3679 198 4 original original NN en-wikipedia-org-3679 198 5 on on IN en-wikipedia-org-3679 198 6 2016 2016 CD en-wikipedia-org-3679 198 7 - - SYM en-wikipedia-org-3679 198 8 02 02 CD en-wikipedia-org-3679 198 9 - - HYPH en-wikipedia-org-3679 198 10 03 03 CD en-wikipedia-org-3679 198 11 . . . en-wikipedia-org-3679 199 1 Retrieved Retrieved NNP en-wikipedia-org-3679 199 2 9 9 CD en-wikipedia-org-3679 199 3 February February NNP en-wikipedia-org-3679 199 4 2016 2016 CD en-wikipedia-org-3679 199 5 . . . en-wikipedia-org-3679 200 1 ^ ^ FW en-wikipedia-org-3679 200 2 " " `` en-wikipedia-org-3679 200 3 Change Change NNP en-wikipedia-org-3679 200 4 Googlebot Googlebot NNP en-wikipedia-org-3679 200 5 crawl crawl NN en-wikipedia-org-3679 200 6 rate rate NN en-wikipedia-org-3679 200 7 - - HYPH en-wikipedia-org-3679 200 8 Search Search NNP en-wikipedia-org-3679 200 9 Console Console NNP en-wikipedia-org-3679 200 10 Help Help NNP en-wikipedia-org-3679 200 11 " " '' en-wikipedia-org-3679 200 12 . . . en-wikipedia-org-3679 201 1 support.google.com support.google.com NNP en-wikipedia-org-3679 201 2 . . . en-wikipedia-org-3679 202 1 Archived archived JJ en-wikipedia-org-3679 202 2 from from IN en-wikipedia-org-3679 202 3 the the DT en-wikipedia-org-3679 202 4 original original NN en-wikipedia-org-3679 202 5 on on IN en-wikipedia-org-3679 202 6 2018 2018 CD en-wikipedia-org-3679 202 7 - - SYM en-wikipedia-org-3679 202 8 11 11 CD en-wikipedia-org-3679 202 9 - - SYM en-wikipedia-org-3679 202 10 18 18 CD en-wikipedia-org-3679 202 11 . . . en-wikipedia-org-3679 203 1 Retrieved retrieve VBN en-wikipedia-org-3679 203 2 22 22 CD en-wikipedia-org-3679 203 3 October October NNP en-wikipedia-org-3679 203 4 2018 2018 CD en-wikipedia-org-3679 203 5 . . . en-wikipedia-org-3679 204 1 ^ ^ FW en-wikipedia-org-3679 204 2 " " `` en-wikipedia-org-3679 204 3 Webmaster Webmaster NNP en-wikipedia-org-3679 204 4 Help Help NNP en-wikipedia-org-3679 204 5 Center Center NNP en-wikipedia-org-3679 204 6 - - : en-wikipedia-org-3679 204 7 How how WRB en-wikipedia-org-3679 204 8 do do VBP en-wikipedia-org-3679 204 9 I -PRON- PRP en-wikipedia-org-3679 204 10 block block VB en-wikipedia-org-3679 204 11 Googlebot Googlebot NNP en-wikipedia-org-3679 204 12 ? ? . en-wikipedia-org-3679 204 13 " " '' en-wikipedia-org-3679 204 14 . . . en-wikipedia-org-3679 205 1 Archived archived JJ en-wikipedia-org-3679 205 2 from from IN en-wikipedia-org-3679 205 3 the the DT en-wikipedia-org-3679 205 4 original original NN en-wikipedia-org-3679 205 5 on on IN en-wikipedia-org-3679 205 6 2010 2010 CD en-wikipedia-org-3679 205 7 - - SYM en-wikipedia-org-3679 205 8 08 08 CD en-wikipedia-org-3679 205 9 - - HYPH en-wikipedia-org-3679 205 10 01 01 CD en-wikipedia-org-3679 205 11 . . . en-wikipedia-org-3679 206 1 Retrieved retrieve VBN en-wikipedia-org-3679 206 2 2007 2007 CD en-wikipedia-org-3679 206 3 - - SYM en-wikipedia-org-3679 206 4 11 11 CD en-wikipedia-org-3679 206 5 - - SYM en-wikipedia-org-3679 206 6 20 20 CD en-wikipedia-org-3679 206 7 . . . en-wikipedia-org-3679 207 1 ^ ^ FW en-wikipedia-org-3679 207 2 " " `` en-wikipedia-org-3679 207 3 How how WRB en-wikipedia-org-3679 207 4 do do VBP en-wikipedia-org-3679 207 5 I -PRON- PRP en-wikipedia-org-3679 207 6 prevent prevent VB en-wikipedia-org-3679 207 7 my -PRON- PRP$ en-wikipedia-org-3679 207 8 site site NN en-wikipedia-org-3679 207 9 or or CC en-wikipedia-org-3679 207 10 certain certain JJ en-wikipedia-org-3679 207 11 subdirectories subdirectory NNS en-wikipedia-org-3679 207 12 from from IN en-wikipedia-org-3679 207 13 being be VBG en-wikipedia-org-3679 207 14 crawled crawl VBN en-wikipedia-org-3679 207 15 ? ? . en-wikipedia-org-3679 208 1 - - : en-wikipedia-org-3679 208 2 Yahoo Yahoo NNP en-wikipedia-org-3679 208 3 Search Search NNP en-wikipedia-org-3679 208 4 Help Help NNP en-wikipedia-org-3679 208 5 " " '' en-wikipedia-org-3679 208 6 . . . en-wikipedia-org-3679 209 1 Archived archived JJ en-wikipedia-org-3679 209 2 from from IN en-wikipedia-org-3679 209 3 the the DT en-wikipedia-org-3679 209 4 original original NN en-wikipedia-org-3679 209 5 on on IN en-wikipedia-org-3679 209 6 2007 2007 CD en-wikipedia-org-3679 209 7 - - SYM en-wikipedia-org-3679 209 8 10 10 CD en-wikipedia-org-3679 209 9 - - SYM en-wikipedia-org-3679 209 10 13 13 CD en-wikipedia-org-3679 209 11 . . . en-wikipedia-org-3679 210 1 Retrieved retrieve VBN en-wikipedia-org-3679 210 2 2007 2007 CD en-wikipedia-org-3679 210 3 - - SYM en-wikipedia-org-3679 210 4 11 11 CD en-wikipedia-org-3679 210 5 - - SYM en-wikipedia-org-3679 210 6 20 20 CD en-wikipedia-org-3679 210 7 . . . en-wikipedia-org-3679 211 1 ^ ^ FW en-wikipedia-org-3679 211 2 " " `` en-wikipedia-org-3679 211 3 Google Google NNP en-wikipedia-org-3679 211 4 's 's POS en-wikipedia-org-3679 211 5 Hidden Hidden NNP en-wikipedia-org-3679 211 6 Interpretation Interpretation NNP en-wikipedia-org-3679 211 7 of of IN en-wikipedia-org-3679 211 8 Robots.txt Robots.txt NNP en-wikipedia-org-3679 211 9 " " '' en-wikipedia-org-3679 211 10 . . . en-wikipedia-org-3679 212 1 Archived archived JJ en-wikipedia-org-3679 212 2 from from IN en-wikipedia-org-3679 212 3 the the DT en-wikipedia-org-3679 212 4 original original NN en-wikipedia-org-3679 212 5 on on IN en-wikipedia-org-3679 212 6 2010 2010 CD en-wikipedia-org-3679 212 7 - - SYM en-wikipedia-org-3679 212 8 11 11 CD en-wikipedia-org-3679 212 9 - - SYM en-wikipedia-org-3679 212 10 20 20 CD en-wikipedia-org-3679 212 11 . . . en-wikipedia-org-3679 213 1 Retrieved retrieve VBN en-wikipedia-org-3679 213 2 2010 2010 CD en-wikipedia-org-3679 213 3 - - SYM en-wikipedia-org-3679 213 4 11 11 CD en-wikipedia-org-3679 213 5 - - SYM en-wikipedia-org-3679 213 6 15 15 CD en-wikipedia-org-3679 213 7 . . . en-wikipedia-org-3679 214 1 ^ ^ FW en-wikipedia-org-3679 214 2 " " `` en-wikipedia-org-3679 214 3 Yahoo Yahoo NNP en-wikipedia-org-3679 214 4 ! ! . en-wikipedia-org-3679 215 1 Search Search NNP en-wikipedia-org-3679 215 2 Blog Blog NNP en-wikipedia-org-3679 215 3 - - HYPH en-wikipedia-org-3679 215 4 Webmasters Webmasters NNP en-wikipedia-org-3679 215 5 can can MD en-wikipedia-org-3679 215 6 now now RB en-wikipedia-org-3679 215 7 auto auto NN en-wikipedia-org-3679 215 8 - - HYPH en-wikipedia-org-3679 215 9 discover discover NN en-wikipedia-org-3679 215 10 with with IN en-wikipedia-org-3679 215 11 Sitemaps Sitemaps NNP en-wikipedia-org-3679 215 12 " " '' en-wikipedia-org-3679 215 13 . . . en-wikipedia-org-3679 216 1 Archived archived JJ en-wikipedia-org-3679 216 2 from from IN en-wikipedia-org-3679 216 3 the the DT en-wikipedia-org-3679 216 4 original original NN en-wikipedia-org-3679 216 5 on on IN en-wikipedia-org-3679 216 6 2009 2009 CD en-wikipedia-org-3679 216 7 - - HYPH en-wikipedia-org-3679 216 8 03 03 CD en-wikipedia-org-3679 216 9 - - HYPH en-wikipedia-org-3679 216 10 05 05 CD en-wikipedia-org-3679 216 11 . . . en-wikipedia-org-3679 217 1 Retrieved Retrieved NNP en-wikipedia-org-3679 217 2 2009 2009 CD en-wikipedia-org-3679 217 3 - - HYPH en-wikipedia-org-3679 217 4 03 03 CD en-wikipedia-org-3679 217 5 - - HYPH en-wikipedia-org-3679 217 6 23 23 CD en-wikipedia-org-3679 217 7 . . . en-wikipedia-org-3679 218 1 ^ ^ FW en-wikipedia-org-3679 218 2 " " `` en-wikipedia-org-3679 218 3 Yandex Yandex NNP en-wikipedia-org-3679 218 4 - - HYPH en-wikipedia-org-3679 218 5 Using use VBG en-wikipedia-org-3679 218 6 robots.txt robots.txt NN en-wikipedia-org-3679 218 7 " " '' en-wikipedia-org-3679 218 8 . . . en-wikipedia-org-3679 219 1 Archived archived JJ en-wikipedia-org-3679 219 2 from from IN en-wikipedia-org-3679 219 3 the the DT en-wikipedia-org-3679 219 4 original original NN en-wikipedia-org-3679 219 5 on on IN en-wikipedia-org-3679 219 6 2013 2013 CD en-wikipedia-org-3679 219 7 - - SYM en-wikipedia-org-3679 219 8 05 05 CD en-wikipedia-org-3679 219 9 - - HYPH en-wikipedia-org-3679 219 10 09 09 CD en-wikipedia-org-3679 219 11 . . . en-wikipedia-org-3679 220 1 Retrieved Retrieved NNP en-wikipedia-org-3679 220 2 2013 2013 CD en-wikipedia-org-3679 220 3 - - SYM en-wikipedia-org-3679 220 4 05 05 CD en-wikipedia-org-3679 220 5 - - SYM en-wikipedia-org-3679 220 6 13 13 CD en-wikipedia-org-3679 220 7 . . . en-wikipedia-org-3679 221 1 ^ ^ FW en-wikipedia-org-3679 221 2 " " `` en-wikipedia-org-3679 221 3 Robots.txt Robots.txt NNP en-wikipedia-org-3679 221 4 Specifications Specifications NNPS en-wikipedia-org-3679 221 5 " " '' en-wikipedia-org-3679 221 6 . . . en-wikipedia-org-3679 222 1 Google Google NNP en-wikipedia-org-3679 222 2 Developers Developers NNP en-wikipedia-org-3679 222 3 . . . en-wikipedia-org-3679 223 1 Retrieved Retrieved NNP en-wikipedia-org-3679 223 2 February February NNP en-wikipedia-org-3679 223 3 15 15 CD en-wikipedia-org-3679 223 4 , , , en-wikipedia-org-3679 223 5 2020 2020 CD en-wikipedia-org-3679 223 6 . . . en-wikipedia-org-3679 224 1 ^ ^ NNP en-wikipedia-org-3679 224 2 a a DT en-wikipedia-org-3679 224 3 b b NNP en-wikipedia-org-3679 224 4 " " `` en-wikipedia-org-3679 224 5 Robots robot VBZ en-wikipedia-org-3679 224 6 meta meta JJ en-wikipedia-org-3679 224 7 tag tag NN en-wikipedia-org-3679 224 8 and and CC en-wikipedia-org-3679 224 9 X X NNS en-wikipedia-org-3679 224 10 - - HYPH en-wikipedia-org-3679 224 11 Robots robots JJ en-wikipedia-org-3679 224 12 - - HYPH en-wikipedia-org-3679 224 13 Tag tag NN en-wikipedia-org-3679 224 14 HTTP http NN en-wikipedia-org-3679 224 15 header header NN en-wikipedia-org-3679 224 16 specifications specification NNS en-wikipedia-org-3679 224 17 - - : en-wikipedia-org-3679 224 18 Webmasters webmaster NNS en-wikipedia-org-3679 224 19 — — : en-wikipedia-org-3679 224 20 Google Google NNP en-wikipedia-org-3679 224 21 Developers Developers NNP en-wikipedia-org-3679 224 22 " " '' en-wikipedia-org-3679 224 23 . . . en-wikipedia-org-3679 225 1 Archived archived JJ en-wikipedia-org-3679 225 2 from from IN en-wikipedia-org-3679 225 3 the the DT en-wikipedia-org-3679 225 4 original original NN en-wikipedia-org-3679 225 5 on on IN en-wikipedia-org-3679 225 6 2013 2013 CD en-wikipedia-org-3679 225 7 - - SYM en-wikipedia-org-3679 225 8 08 08 CD en-wikipedia-org-3679 225 9 - - HYPH en-wikipedia-org-3679 225 10 08 08 CD en-wikipedia-org-3679 225 11 . . . en-wikipedia-org-3679 226 1 Retrieved Retrieved NNP en-wikipedia-org-3679 226 2 2013 2013 CD en-wikipedia-org-3679 226 3 - - SYM en-wikipedia-org-3679 226 4 08 08 CD en-wikipedia-org-3679 226 5 - - SYM en-wikipedia-org-3679 226 6 17 17 CD en-wikipedia-org-3679 226 7 . . . en-wikipedia-org-3679 227 1 External External NNP en-wikipedia-org-3679 227 2 links[edit links[edit , en-wikipedia-org-3679 227 3 ] ] -RRB- en-wikipedia-org-3679 227 4 Official official JJ en-wikipedia-org-3679 227 5 website website NN en-wikipedia-org-3679 227 6 Robots robot NNS en-wikipedia-org-3679 227 7 Database Database NNP en-wikipedia-org-3679 227 8 ( ( -LRB- en-wikipedia-org-3679 227 9 list list NN en-wikipedia-org-3679 227 10 of of IN en-wikipedia-org-3679 227 11 bot bot NN en-wikipedia-org-3679 227 12 names name NNS en-wikipedia-org-3679 227 13 ) ) -RRB- en-wikipedia-org-3679 227 14 Robots.txt Robots.txt NNP en-wikipedia-org-3679 227 15 checker checker NNP en-wikipedia-org-3679 227 16 tool tool NN en-wikipedia-org-3679 227 17 Microsoft Microsoft NNP en-wikipedia-org-3679 227 18 bingbot bingbot NN en-wikipedia-org-3679 227 19 official official JJ en-wikipedia-org-3679 227 20 documentation documentation NN en-wikipedia-org-3679 227 21 Internet internet NN en-wikipedia-org-3679 227 22 portal portal NN en-wikipedia-org-3679 227 23 Retrieved retrieve VBN en-wikipedia-org-3679 227 24 from from IN en-wikipedia-org-3679 227 25 " " `` en-wikipedia-org-3679 227 26 https://en.wikipedia.org/w/index.php?title=Robots_exclusion_standard&oldid=991385881 https://en.wikipedia.org/w/index.php?title=Robots_exclusion_standard&oldid=991385881 NNP en-wikipedia-org-3679 227 27 " " '' en-wikipedia-org-3679 227 28 Categories category NNS en-wikipedia-org-3679 227 29 : : : en-wikipedia-org-3679 227 30 World world NN en-wikipedia-org-3679 227 31 Wide wide JJ en-wikipedia-org-3679 227 32 Web web NN en-wikipedia-org-3679 227 33 Internet internet NN en-wikipedia-org-3679 227 34 bots bot NNS en-wikipedia-org-3679 227 35 Hidden hide VBN en-wikipedia-org-3679 227 36 categories category NNS en-wikipedia-org-3679 227 37 : : : en-wikipedia-org-3679 227 38 CS1 cs1 NN en-wikipedia-org-3679 227 39 maint maint NN en-wikipedia-org-3679 227 40 : : : en-wikipedia-org-3679 227 41 multiple multiple JJ en-wikipedia-org-3679 227 42 names name NNS en-wikipedia-org-3679 227 43 : : : en-wikipedia-org-3679 227 44 authors author NNS en-wikipedia-org-3679 227 45 list list NN en-wikipedia-org-3679 227 46 Wikipedia Wikipedia NNP en-wikipedia-org-3679 227 47 pending pende VBG en-wikipedia-org-3679 227 48 changes change NNS en-wikipedia-org-3679 227 49 protected protect VBD en-wikipedia-org-3679 227 50 pages page NNS en-wikipedia-org-3679 227 51 Articles article NNS en-wikipedia-org-3679 227 52 with with IN en-wikipedia-org-3679 227 53 short short JJ en-wikipedia-org-3679 227 54 description description NN en-wikipedia-org-3679 227 55 Short short JJ en-wikipedia-org-3679 227 56 description description NN en-wikipedia-org-3679 227 57 matches match VBZ en-wikipedia-org-3679 227 58 Wikidata Wikidata NNP en-wikipedia-org-3679 227 59 Official official JJ en-wikipedia-org-3679 227 60 website website NN en-wikipedia-org-3679 227 61 different different JJ en-wikipedia-org-3679 227 62 in in IN en-wikipedia-org-3679 227 63 Wikidata Wikidata NNP en-wikipedia-org-3679 227 64 and and CC en-wikipedia-org-3679 227 65 Wikipedia Wikipedia NNP en-wikipedia-org-3679 227 66 AC AC NNP en-wikipedia-org-3679 227 67 with with IN en-wikipedia-org-3679 227 68 0 0 CD en-wikipedia-org-3679 227 69 elements element NNS en-wikipedia-org-3679 227 70 Navigation Navigation NNP en-wikipedia-org-3679 227 71 menu menu NN en-wikipedia-org-3679 227 72 Personal personal JJ en-wikipedia-org-3679 227 73 tools tool NNS en-wikipedia-org-3679 227 74 Not not RB en-wikipedia-org-3679 227 75 logged log VBN en-wikipedia-org-3679 227 76 in in IN en-wikipedia-org-3679 227 77 Talk Talk NNP en-wikipedia-org-3679 227 78 Contributions Contributions NNP en-wikipedia-org-3679 227 79 Create Create NNP en-wikipedia-org-3679 227 80 account account NN en-wikipedia-org-3679 227 81 Log Log NNP en-wikipedia-org-3679 227 82 in in IN en-wikipedia-org-3679 227 83 Namespaces Namespaces NNP en-wikipedia-org-3679 227 84 Article Article NNP en-wikipedia-org-3679 227 85 Talk talk NN en-wikipedia-org-3679 227 86 Variants variant VBZ en-wikipedia-org-3679 227 87 Views view NNS en-wikipedia-org-3679 227 88 Read read VBP en-wikipedia-org-3679 227 89 Edit edit NN en-wikipedia-org-3679 227 90 View View NNP en-wikipedia-org-3679 227 91 history history NN en-wikipedia-org-3679 227 92 More More JJR en-wikipedia-org-3679 227 93 Search Search NNP en-wikipedia-org-3679 227 94 Navigation Navigation NNP en-wikipedia-org-3679 227 95 Main Main NNP en-wikipedia-org-3679 227 96 page page NN en-wikipedia-org-3679 227 97 Contents content VBZ en-wikipedia-org-3679 227 98 Current current JJ en-wikipedia-org-3679 227 99 events event NNS en-wikipedia-org-3679 227 100 Random random JJ en-wikipedia-org-3679 227 101 article article NN en-wikipedia-org-3679 227 102 About about IN en-wikipedia-org-3679 227 103 Wikipedia Wikipedia NNP en-wikipedia-org-3679 227 104 Contact Contact NNP en-wikipedia-org-3679 227 105 us -PRON- PRP en-wikipedia-org-3679 227 106 Donate Donate NNP en-wikipedia-org-3679 227 107 Contribute contribute NN en-wikipedia-org-3679 227 108 Help help NN en-wikipedia-org-3679 227 109 Learn learn VB en-wikipedia-org-3679 227 110 to to TO en-wikipedia-org-3679 227 111 edit edit VB en-wikipedia-org-3679 227 112 Community community NN en-wikipedia-org-3679 227 113 portal portal JJ en-wikipedia-org-3679 227 114 Recent recent JJ en-wikipedia-org-3679 227 115 changes change NNS en-wikipedia-org-3679 227 116 Upload upload JJ en-wikipedia-org-3679 227 117 file file NN en-wikipedia-org-3679 227 118 Tools tool NNS en-wikipedia-org-3679 227 119 What what WP en-wikipedia-org-3679 227 120 links link NNS en-wikipedia-org-3679 227 121 here here RB en-wikipedia-org-3679 227 122 Related relate VBN en-wikipedia-org-3679 227 123 changes change VBZ en-wikipedia-org-3679 227 124 Upload upload JJ en-wikipedia-org-3679 227 125 file file VBP en-wikipedia-org-3679 227 126 Special special JJ en-wikipedia-org-3679 227 127 pages page NNS en-wikipedia-org-3679 227 128 Permanent permanent JJ en-wikipedia-org-3679 227 129 link link NN en-wikipedia-org-3679 227 130 Page Page NNP en-wikipedia-org-3679 227 131 information information NN en-wikipedia-org-3679 227 132 Cite cite VB en-wikipedia-org-3679 227 133 this this DT en-wikipedia-org-3679 227 134 page page NN en-wikipedia-org-3679 227 135 Wikidata wikidata NN en-wikipedia-org-3679 227 136 item item NN en-wikipedia-org-3679 227 137 Print print NN en-wikipedia-org-3679 227 138 / / SYM en-wikipedia-org-3679 227 139 export export NN en-wikipedia-org-3679 227 140 Download Download NNP en-wikipedia-org-3679 227 141 as as IN en-wikipedia-org-3679 227 142 PDF PDF NNP en-wikipedia-org-3679 227 143 Printable Printable NNP en-wikipedia-org-3679 227 144 version version NN en-wikipedia-org-3679 227 145 Languages language VBZ en-wikipedia-org-3679 227 146 العربية العربية NNP en-wikipedia-org-3679 227 147 Boarisch Boarisch NNP en-wikipedia-org-3679 227 148 Català Català NNP en-wikipedia-org-3679 227 149 Čeština Čeština NNP en-wikipedia-org-3679 227 150 Dansk Dansk NNP en-wikipedia-org-3679 227 151 Deutsch Deutsch NNP en-wikipedia-org-3679 227 152 Español Español NNP en-wikipedia-org-3679 227 153 فارسی فارسی NNP en-wikipedia-org-3679 227 154 Français Français NNP en-wikipedia-org-3679 227 155 한국어 한국어 NN en-wikipedia-org-3679 227 156 Bahasa Bahasa NNP en-wikipedia-org-3679 227 157 Indonesia Indonesia NNP en-wikipedia-org-3679 227 158 Italiano Italiano NNP en-wikipedia-org-3679 227 159 עברית עברית NN en-wikipedia-org-3679 227 160 Nederlands Nederlands NNP en-wikipedia-org-3679 227 161 日本語 日本語 NNP en-wikipedia-org-3679 227 162 Polski Polski NNP en-wikipedia-org-3679 227 163 Português Português NNP en-wikipedia-org-3679 227 164 Русский Русский NNP en-wikipedia-org-3679 227 165 Simple Simple NNP en-wikipedia-org-3679 227 166 English English NNP en-wikipedia-org-3679 227 167 Српски Српски NNP en-wikipedia-org-3679 227 168 / / SYM en-wikipedia-org-3679 227 169 srpski srpski JJ en-wikipedia-org-3679 227 170 Suomi Suomi NNP en-wikipedia-org-3679 227 171 Svenska Svenska NNP en-wikipedia-org-3679 227 172 Türkçe Türkçe NNP en-wikipedia-org-3679 227 173 Українська Українська NNP en-wikipedia-org-3679 227 174 中文 中文 NNP en-wikipedia-org-3679 227 175 Edit Edit NNP en-wikipedia-org-3679 227 176 links link NNS en-wikipedia-org-3679 227 177 This this DT en-wikipedia-org-3679 227 178 page page NN en-wikipedia-org-3679 227 179 was be VBD en-wikipedia-org-3679 227 180 last last RB en-wikipedia-org-3679 227 181 edited edit VBN en-wikipedia-org-3679 227 182 on on IN en-wikipedia-org-3679 227 183 29 29 CD en-wikipedia-org-3679 227 184 November November NNP en-wikipedia-org-3679 227 185 2020 2020 CD en-wikipedia-org-3679 227 186 , , , en-wikipedia-org-3679 227 187 at at IN en-wikipedia-org-3679 227 188 19:54 19:54 CD en-wikipedia-org-3679 227 189   _SP en-wikipedia-org-3679 227 190 ( ( -LRB- en-wikipedia-org-3679 227 191 UTC UTC NNP en-wikipedia-org-3679 227 192 ) ) -RRB- en-wikipedia-org-3679 227 193 . . . en-wikipedia-org-3679 228 1 Text text NN en-wikipedia-org-3679 228 2 is be VBZ en-wikipedia-org-3679 228 3 available available JJ en-wikipedia-org-3679 228 4 under under IN en-wikipedia-org-3679 228 5 the the DT en-wikipedia-org-3679 228 6 Creative Creative NNP en-wikipedia-org-3679 228 7 Commons Commons NNP en-wikipedia-org-3679 228 8 Attribution Attribution NNP en-wikipedia-org-3679 228 9 - - HYPH en-wikipedia-org-3679 228 10 ShareAlike ShareAlike NNP en-wikipedia-org-3679 228 11 License License NNP en-wikipedia-org-3679 228 12 ; ; , en-wikipedia-org-3679 228 13 additional additional JJ en-wikipedia-org-3679 228 14 terms term NNS en-wikipedia-org-3679 228 15 may may MD en-wikipedia-org-3679 228 16 apply apply VB en-wikipedia-org-3679 228 17 . . . en-wikipedia-org-3679 229 1 By by IN en-wikipedia-org-3679 229 2 using use VBG en-wikipedia-org-3679 229 3 this this DT en-wikipedia-org-3679 229 4 site site NN en-wikipedia-org-3679 229 5 , , , en-wikipedia-org-3679 229 6 you -PRON- PRP en-wikipedia-org-3679 229 7 agree agree VBP en-wikipedia-org-3679 229 8 to to IN en-wikipedia-org-3679 229 9 the the DT en-wikipedia-org-3679 229 10 Terms term NNS en-wikipedia-org-3679 229 11 of of IN en-wikipedia-org-3679 229 12 Use Use NNP en-wikipedia-org-3679 229 13 and and CC en-wikipedia-org-3679 229 14 Privacy Privacy NNP en-wikipedia-org-3679 229 15 Policy Policy NNP en-wikipedia-org-3679 229 16 . . . en-wikipedia-org-3679 230 1 Wikipedia Wikipedia NNP en-wikipedia-org-3679 230 2 ® ® NNPS en-wikipedia-org-3679 230 3 is be VBZ en-wikipedia-org-3679 230 4 a a DT en-wikipedia-org-3679 230 5 registered registered JJ en-wikipedia-org-3679 230 6 trademark trademark NN en-wikipedia-org-3679 230 7 of of IN en-wikipedia-org-3679 230 8 the the DT en-wikipedia-org-3679 230 9 Wikimedia Wikimedia NNP en-wikipedia-org-3679 230 10 Foundation Foundation NNP en-wikipedia-org-3679 230 11 , , , en-wikipedia-org-3679 230 12 Inc. Inc. NNP en-wikipedia-org-3679 230 13 , , , en-wikipedia-org-3679 230 14 a a DT en-wikipedia-org-3679 230 15 non non JJ en-wikipedia-org-3679 230 16 - - JJ en-wikipedia-org-3679 230 17 profit profit JJ en-wikipedia-org-3679 230 18 organization organization NN en-wikipedia-org-3679 230 19 . . . en-wikipedia-org-3679 231 1 Privacy privacy NN en-wikipedia-org-3679 231 2 policy policy NN en-wikipedia-org-3679 231 3 About about IN en-wikipedia-org-3679 231 4 Wikipedia Wikipedia NNP en-wikipedia-org-3679 231 5 Disclaimers Disclaimers NNPS en-wikipedia-org-3679 231 6 Contact Contact NNP en-wikipedia-org-3679 231 7 Wikipedia Wikipedia NNP en-wikipedia-org-3679 231 8 Mobile Mobile NNP en-wikipedia-org-3679 231 9 view view NN en-wikipedia-org-3679 231 10 Developers Developers NNP en-wikipedia-org-3679 231 11 Statistics Statistics NNP en-wikipedia-org-3679 231 12 Cookie Cookie NNP en-wikipedia-org-3679 231 13 statement statement NN