id	sid	tid	token	lemma	pos
en-wikipedia-org-3679	1	1	Robots	robot	NNS
en-wikipedia-org-3679	1	2	exclusion	exclusion	NN
en-wikipedia-org-3679	1	3	standard	standard	JJ
en-wikipedia-org-3679	1	4	-	-	HYPH
en-wikipedia-org-3679	1	5	Wikipedia	wikipedia	NN
en-wikipedia-org-3679	1	6	Robots	Robots	NNP
en-wikipedia-org-3679	1	7	exclusion	exclusion	NN
en-wikipedia-org-3679	1	8	standard	standard	NN
en-wikipedia-org-3679	1	9	From	from	IN
en-wikipedia-org-3679	1	10	Wikipedia	Wikipedia	NNP
en-wikipedia-org-3679	1	11	,	,	,
en-wikipedia-org-3679	1	12	the	the	DT
en-wikipedia-org-3679	1	13	free	free	JJ
en-wikipedia-org-3679	1	14	encyclopedia	encyclopedia	NN
en-wikipedia-org-3679	1	15	This	this	DT
en-wikipedia-org-3679	1	16	is	be	VBZ
en-wikipedia-org-3679	1	17	the	the	DT
en-wikipedia-org-3679	1	18	latest	late	JJS
en-wikipedia-org-3679	1	19	accepted	accepted	JJ
en-wikipedia-org-3679	1	20	revision	revision	NN
en-wikipedia-org-3679	1	21	,	,	,
en-wikipedia-org-3679	1	22	reviewed	review	VBN
en-wikipedia-org-3679	1	23	on	on	IN
en-wikipedia-org-3679	1	24	29	29	CD
en-wikipedia-org-3679	1	25	November	November	NNP
en-wikipedia-org-3679	1	26	2020	2020	CD
en-wikipedia-org-3679	1	27	.	.	.
en-wikipedia-org-3679	2	1	Jump	jump	VB
en-wikipedia-org-3679	2	2	to	to	IN
en-wikipedia-org-3679	2	3	navigation	navigation	NN
en-wikipedia-org-3679	2	4	Jump	Jump	NNP
en-wikipedia-org-3679	2	5	to	to	TO
en-wikipedia-org-3679	2	6	search	search	VB
en-wikipedia-org-3679	2	7	Standard	Standard	NNP
en-wikipedia-org-3679	2	8	used	use	VBN
en-wikipedia-org-3679	2	9	to	to	TO
en-wikipedia-org-3679	2	10	advise	advise	VB
en-wikipedia-org-3679	2	11	web	web	NN
en-wikipedia-org-3679	2	12	crawlers	crawler	NNS
en-wikipedia-org-3679	2	13	and	and	CC
en-wikipedia-org-3679	2	14	scrapers	scraper	NNS
en-wikipedia-org-3679	2	15	not	not	RB
en-wikipedia-org-3679	2	16	to	to	TO
en-wikipedia-org-3679	2	17	index	index	VB
en-wikipedia-org-3679	2	18	a	a	DT
en-wikipedia-org-3679	2	19	web	web	NN
en-wikipedia-org-3679	2	20	page	page	NN
en-wikipedia-org-3679	2	21	or	or	CC
en-wikipedia-org-3679	2	22	site	site	NN
en-wikipedia-org-3679	2	23	The	the	DT
en-wikipedia-org-3679	2	24	robots	robot	NNS
en-wikipedia-org-3679	2	25	exclusion	exclusion	NN
en-wikipedia-org-3679	2	26	standard	standard	NN
en-wikipedia-org-3679	2	27	,	,	,
en-wikipedia-org-3679	2	28	also	also	RB
en-wikipedia-org-3679	2	29	known	know	VBN
en-wikipedia-org-3679	2	30	as	as	IN
en-wikipedia-org-3679	2	31	the	the	DT
en-wikipedia-org-3679	2	32	robots	robot	NNS
en-wikipedia-org-3679	2	33	exclusion	exclusion	NN
en-wikipedia-org-3679	2	34	protocol	protocol	NN
en-wikipedia-org-3679	2	35	or	or	CC
en-wikipedia-org-3679	2	36	simply	simply	RB
en-wikipedia-org-3679	2	37	robots.txt	robots.txt	CD
en-wikipedia-org-3679	2	38	,	,	,
en-wikipedia-org-3679	2	39	is	be	VBZ
en-wikipedia-org-3679	2	40	a	a	DT
en-wikipedia-org-3679	2	41	standard	standard	NN
en-wikipedia-org-3679	2	42	used	use	VBN
en-wikipedia-org-3679	2	43	by	by	IN
en-wikipedia-org-3679	2	44	websites	website	NNS
en-wikipedia-org-3679	2	45	to	to	TO
en-wikipedia-org-3679	2	46	communicate	communicate	VB
en-wikipedia-org-3679	2	47	with	with	IN
en-wikipedia-org-3679	2	48	web	web	NN
en-wikipedia-org-3679	2	49	crawlers	crawler	NNS
en-wikipedia-org-3679	2	50	and	and	CC
en-wikipedia-org-3679	2	51	other	other	JJ
en-wikipedia-org-3679	2	52	web	web	NN
en-wikipedia-org-3679	2	53	robots	robot	NNS
en-wikipedia-org-3679	2	54	.	.	.
en-wikipedia-org-3679	3	1	The	the	DT
en-wikipedia-org-3679	3	2	standard	standard	JJ
en-wikipedia-org-3679	3	3	specifies	specify	VBZ
en-wikipedia-org-3679	3	4	how	how	WRB
en-wikipedia-org-3679	3	5	to	to	TO
en-wikipedia-org-3679	3	6	inform	inform	VB
en-wikipedia-org-3679	3	7	the	the	DT
en-wikipedia-org-3679	3	8	web	web	NN
en-wikipedia-org-3679	3	9	robot	robot	NN
en-wikipedia-org-3679	3	10	about	about	IN
en-wikipedia-org-3679	3	11	which	which	WDT
en-wikipedia-org-3679	3	12	areas	area	NNS
en-wikipedia-org-3679	3	13	of	of	IN
en-wikipedia-org-3679	3	14	the	the	DT
en-wikipedia-org-3679	3	15	website	website	NN
en-wikipedia-org-3679	3	16	should	should	MD
en-wikipedia-org-3679	3	17	not	not	RB
en-wikipedia-org-3679	3	18	be	be	VB
en-wikipedia-org-3679	3	19	processed	process	VBN
en-wikipedia-org-3679	3	20	or	or	CC
en-wikipedia-org-3679	3	21	scanned	scan	VBN
en-wikipedia-org-3679	3	22	.	.	.
en-wikipedia-org-3679	4	1	Robots	robot	NNS
en-wikipedia-org-3679	4	2	are	be	VBP
en-wikipedia-org-3679	4	3	often	often	RB
en-wikipedia-org-3679	4	4	used	use	VBN
en-wikipedia-org-3679	4	5	by	by	IN
en-wikipedia-org-3679	4	6	search	search	NN
en-wikipedia-org-3679	4	7	engines	engine	NNS
en-wikipedia-org-3679	4	8	to	to	TO
en-wikipedia-org-3679	4	9	categorize	categorize	VB
en-wikipedia-org-3679	4	10	websites	website	NNS
en-wikipedia-org-3679	4	11	.	.	.
en-wikipedia-org-3679	5	1	Not	not	RB
en-wikipedia-org-3679	5	2	all	all	DT
en-wikipedia-org-3679	5	3	robots	robot	NNS
en-wikipedia-org-3679	5	4	cooperate	cooperate	VBP
en-wikipedia-org-3679	5	5	with	with	IN
en-wikipedia-org-3679	5	6	the	the	DT
en-wikipedia-org-3679	5	7	standard	standard	JJ
en-wikipedia-org-3679	5	8	;	;	:
en-wikipedia-org-3679	5	9	email	email	NN
en-wikipedia-org-3679	5	10	harvesters	harvester	NNS
en-wikipedia-org-3679	5	11	,	,	,
en-wikipedia-org-3679	5	12	spambots	spambot	NNS
en-wikipedia-org-3679	5	13	,	,	,
en-wikipedia-org-3679	5	14	malware	malware	NNP
en-wikipedia-org-3679	5	15	and	and	CC
en-wikipedia-org-3679	5	16	robots	robot	NNS
en-wikipedia-org-3679	5	17	that	that	WDT
en-wikipedia-org-3679	5	18	scan	scan	VBP
en-wikipedia-org-3679	5	19	for	for	IN
en-wikipedia-org-3679	5	20	security	security	NN
en-wikipedia-org-3679	5	21	vulnerabilities	vulnerability	NNS
en-wikipedia-org-3679	5	22	may	may	MD
en-wikipedia-org-3679	5	23	even	even	RB
en-wikipedia-org-3679	5	24	start	start	VB
en-wikipedia-org-3679	5	25	with	with	IN
en-wikipedia-org-3679	5	26	the	the	DT
en-wikipedia-org-3679	5	27	portions	portion	NNS
en-wikipedia-org-3679	5	28	of	of	IN
en-wikipedia-org-3679	5	29	the	the	DT
en-wikipedia-org-3679	5	30	website	website	NN
en-wikipedia-org-3679	5	31	where	where	WRB
en-wikipedia-org-3679	5	32	they	-PRON-	PRP
en-wikipedia-org-3679	5	33	have	have	VBP
en-wikipedia-org-3679	5	34	been	be	VBN
en-wikipedia-org-3679	5	35	told	tell	VBN
en-wikipedia-org-3679	5	36	to	to	TO
en-wikipedia-org-3679	5	37	stay	stay	VB
en-wikipedia-org-3679	5	38	out	out	RP
en-wikipedia-org-3679	5	39	.	.	.
en-wikipedia-org-3679	6	1	The	the	DT
en-wikipedia-org-3679	6	2	standard	standard	NN
en-wikipedia-org-3679	6	3	can	can	MD
en-wikipedia-org-3679	6	4	be	be	VB
en-wikipedia-org-3679	6	5	used	use	VBN
en-wikipedia-org-3679	6	6	in	in	IN
en-wikipedia-org-3679	6	7	conjunction	conjunction	NN
en-wikipedia-org-3679	6	8	with	with	IN
en-wikipedia-org-3679	6	9	Sitemaps	Sitemaps	NNP
en-wikipedia-org-3679	6	10	,	,	,
en-wikipedia-org-3679	6	11	a	a	DT
en-wikipedia-org-3679	6	12	robot	robot	NN
en-wikipedia-org-3679	6	13	inclusion	inclusion	NN
en-wikipedia-org-3679	6	14	standard	standard	NN
en-wikipedia-org-3679	6	15	for	for	IN
en-wikipedia-org-3679	6	16	websites	website	NNS
en-wikipedia-org-3679	6	17	.	.	.
en-wikipedia-org-3679	7	1	Contents	content	NNS
en-wikipedia-org-3679	7	2	1	1	CD
en-wikipedia-org-3679	7	3	History	history	NN
en-wikipedia-org-3679	7	4	2	2	CD
en-wikipedia-org-3679	7	5	Standard	standard	NN
en-wikipedia-org-3679	7	6	3	3	CD
en-wikipedia-org-3679	7	7	Security	Security	NNP
en-wikipedia-org-3679	7	8	4	4	CD
en-wikipedia-org-3679	7	9	Alternatives	Alternatives	NNP
en-wikipedia-org-3679	7	10	5	5	CD
en-wikipedia-org-3679	7	11	Examples	Examples	NNP
en-wikipedia-org-3679	7	12	6	6	CD
en-wikipedia-org-3679	7	13	Nonstandard	Nonstandard	NNP
en-wikipedia-org-3679	7	14	extensions	extension	NNS
en-wikipedia-org-3679	7	15	6.1	6.1	CD
en-wikipedia-org-3679	7	16	Crawl	crawl	NN
en-wikipedia-org-3679	7	17	-	-	HYPH
en-wikipedia-org-3679	7	18	delay	delay	NNP
en-wikipedia-org-3679	7	19	directive	directive	VBP
en-wikipedia-org-3679	7	20	6.2	6.2	CD
en-wikipedia-org-3679	7	21	Allow	Allow	NNP
en-wikipedia-org-3679	7	22	directive	directive	NN
en-wikipedia-org-3679	7	23	6.3	6.3	CD
en-wikipedia-org-3679	7	24	Sitemap	Sitemap	NNP
en-wikipedia-org-3679	7	25	6.4	6.4	CD
en-wikipedia-org-3679	7	26	Host	host	NN
en-wikipedia-org-3679	7	27	6.5	6.5	CD
en-wikipedia-org-3679	7	28	Universal	Universal	NNP
en-wikipedia-org-3679	7	29	"	"	''
en-wikipedia-org-3679	7	30	*	*	NN
en-wikipedia-org-3679	7	31	"	"	''
en-wikipedia-org-3679	7	32	match	match	VBP
en-wikipedia-org-3679	7	33	7	7	CD
en-wikipedia-org-3679	7	34	Meta	meta	JJ
en-wikipedia-org-3679	7	35	tags	tag	NNS
en-wikipedia-org-3679	7	36	and	and	CC
en-wikipedia-org-3679	7	37	headers	header	VBZ
en-wikipedia-org-3679	7	38	8	8	CD
en-wikipedia-org-3679	7	39	See	see	VB
en-wikipedia-org-3679	7	40	also	also	RB
en-wikipedia-org-3679	7	41	9	9	CD
en-wikipedia-org-3679	7	42	References	reference	NNS
en-wikipedia-org-3679	7	43	10	10	CD
en-wikipedia-org-3679	7	44	External	external	JJ
en-wikipedia-org-3679	7	45	links	link	NNS
en-wikipedia-org-3679	7	46	History[edit	history[edit	VBP
en-wikipedia-org-3679	7	47	]	]	-RRB-
en-wikipedia-org-3679	7	48	The	the	DT
en-wikipedia-org-3679	7	49	standard	standard	NN
en-wikipedia-org-3679	7	50	was	be	VBD
en-wikipedia-org-3679	7	51	proposed	propose	VBN
en-wikipedia-org-3679	7	52	by	by	IN
en-wikipedia-org-3679	7	53	Martijn	Martijn	NNP
en-wikipedia-org-3679	7	54	Koster,[1][2	Koster,[1][2	NNP
en-wikipedia-org-3679	7	55	]	]	-RRB-
en-wikipedia-org-3679	7	56	when	when	WRB
en-wikipedia-org-3679	7	57	working	work	VBG
en-wikipedia-org-3679	7	58	for	for	IN
en-wikipedia-org-3679	7	59	Nexor[3	nexor[3	NN
en-wikipedia-org-3679	7	60	]	]	-RRB-
en-wikipedia-org-3679	7	61	in	in	IN
en-wikipedia-org-3679	7	62	February	February	NNP
en-wikipedia-org-3679	7	63	1994[4	1994[4	NNP
en-wikipedia-org-3679	7	64	]	]	-RRB-
en-wikipedia-org-3679	7	65	on	on	IN
en-wikipedia-org-3679	7	66	the	the	DT
en-wikipedia-org-3679	7	67	www	www	NN
en-wikipedia-org-3679	7	68	-	-	HYPH
en-wikipedia-org-3679	7	69	talk	talk	NN
en-wikipedia-org-3679	7	70	mailing	mailing	NN
en-wikipedia-org-3679	7	71	list	list	NN
en-wikipedia-org-3679	7	72	,	,	,
en-wikipedia-org-3679	7	73	the	the	DT
en-wikipedia-org-3679	7	74	main	main	JJ
en-wikipedia-org-3679	7	75	communication	communication	NN
en-wikipedia-org-3679	7	76	channel	channel	NN
en-wikipedia-org-3679	7	77	for	for	IN
en-wikipedia-org-3679	7	78	WWW	www	NN
en-wikipedia-org-3679	7	79	-	-	HYPH
en-wikipedia-org-3679	7	80	related	relate	VBN
en-wikipedia-org-3679	7	81	activities	activity	NNS
en-wikipedia-org-3679	7	82	at	at	IN
en-wikipedia-org-3679	7	83	the	the	DT
en-wikipedia-org-3679	7	84	time	time	NN
en-wikipedia-org-3679	7	85	.	.	.
en-wikipedia-org-3679	8	1	Charles	Charles	NNP
en-wikipedia-org-3679	8	2	Stross	Stross	NNP
en-wikipedia-org-3679	8	3	claims	claim	VBZ
en-wikipedia-org-3679	8	4	to	to	TO
en-wikipedia-org-3679	8	5	have	have	VB
en-wikipedia-org-3679	8	6	provoked	provoke	VBN
en-wikipedia-org-3679	8	7	Koster	Koster	NNP
en-wikipedia-org-3679	8	8	to	to	TO
en-wikipedia-org-3679	8	9	suggest	suggest	VB
en-wikipedia-org-3679	8	10	robots.txt	robots.txt	CD
en-wikipedia-org-3679	8	11	,	,	,
en-wikipedia-org-3679	8	12	after	after	IN
en-wikipedia-org-3679	8	13	he	-PRON-	PRP
en-wikipedia-org-3679	8	14	wrote	write	VBD
en-wikipedia-org-3679	8	15	a	a	DT
en-wikipedia-org-3679	8	16	badly	badly	RB
en-wikipedia-org-3679	8	17	-	-	HYPH
en-wikipedia-org-3679	8	18	behaved	behave	VBN
en-wikipedia-org-3679	8	19	web	web	NN
en-wikipedia-org-3679	8	20	crawler	crawler	NN
en-wikipedia-org-3679	8	21	that	that	WDT
en-wikipedia-org-3679	8	22	inadvertently	inadvertently	RB
en-wikipedia-org-3679	8	23	caused	cause	VBD
en-wikipedia-org-3679	8	24	a	a	DT
en-wikipedia-org-3679	8	25	denial	denial	JJ
en-wikipedia-org-3679	8	26	-	-	HYPH
en-wikipedia-org-3679	8	27	of	of	IN
en-wikipedia-org-3679	8	28	-	-	HYPH
en-wikipedia-org-3679	8	29	service	service	NN
en-wikipedia-org-3679	8	30	attack	attack	NN
en-wikipedia-org-3679	8	31	on	on	IN
en-wikipedia-org-3679	8	32	Koster	Koster	NNP
en-wikipedia-org-3679	8	33	's	's	POS
en-wikipedia-org-3679	8	34	server	server	NN
en-wikipedia-org-3679	8	35	.	.	.
en-wikipedia-org-3679	9	1	[	[	-LRB-
en-wikipedia-org-3679	9	2	5	5	CD
en-wikipedia-org-3679	9	3	]	]	-RRB-
en-wikipedia-org-3679	9	4	It	-PRON-	PRP
en-wikipedia-org-3679	9	5	quickly	quickly	RB
en-wikipedia-org-3679	9	6	became	become	VBD
en-wikipedia-org-3679	9	7	a	a	DT
en-wikipedia-org-3679	9	8	de	de	FW
en-wikipedia-org-3679	9	9	facto	facto	FW
en-wikipedia-org-3679	9	10	standard	standard	NN
en-wikipedia-org-3679	9	11	that	that	IN
en-wikipedia-org-3679	9	12	present	present	JJ
en-wikipedia-org-3679	9	13	and	and	CC
en-wikipedia-org-3679	9	14	future	future	JJ
en-wikipedia-org-3679	9	15	web	web	NN
en-wikipedia-org-3679	9	16	crawlers	crawler	NNS
en-wikipedia-org-3679	9	17	were	be	VBD
en-wikipedia-org-3679	9	18	expected	expect	VBN
en-wikipedia-org-3679	9	19	to	to	TO
en-wikipedia-org-3679	9	20	follow	follow	VB
en-wikipedia-org-3679	9	21	;	;	:
en-wikipedia-org-3679	9	22	most	most	RBS
en-wikipedia-org-3679	9	23	complied	complied	JJ
en-wikipedia-org-3679	9	24	,	,	,
en-wikipedia-org-3679	9	25	including	include	VBG
en-wikipedia-org-3679	9	26	those	those	DT
en-wikipedia-org-3679	9	27	operated	operate	VBN
en-wikipedia-org-3679	9	28	by	by	IN
en-wikipedia-org-3679	9	29	search	search	NN
en-wikipedia-org-3679	9	30	engines	engine	NNS
en-wikipedia-org-3679	9	31	such	such	JJ
en-wikipedia-org-3679	9	32	as	as	IN
en-wikipedia-org-3679	9	33	WebCrawler	WebCrawler	NNP
en-wikipedia-org-3679	9	34	,	,	,
en-wikipedia-org-3679	9	35	Lycos	Lycos	NNP
en-wikipedia-org-3679	9	36	,	,	,
en-wikipedia-org-3679	9	37	and	and	CC
en-wikipedia-org-3679	9	38	AltaVista	AltaVista	NNP
en-wikipedia-org-3679	9	39	.	.	.
en-wikipedia-org-3679	10	1	[	[	-LRB-
en-wikipedia-org-3679	10	2	6	6	CD
en-wikipedia-org-3679	10	3	]	]	-RRB-
en-wikipedia-org-3679	10	4	On	on	IN
en-wikipedia-org-3679	10	5	July	July	NNP
en-wikipedia-org-3679	10	6	1	1	CD
en-wikipedia-org-3679	10	7	,	,	,
en-wikipedia-org-3679	10	8	2019	2019	CD
en-wikipedia-org-3679	10	9	Google	Google	NNP
en-wikipedia-org-3679	10	10	announced[7	announced[7	NNP
en-wikipedia-org-3679	10	11	]	]	-RRB-
en-wikipedia-org-3679	10	12	the	the	DT
en-wikipedia-org-3679	10	13	proposal	proposal	NN
en-wikipedia-org-3679	10	14	of	of	IN
en-wikipedia-org-3679	10	15	the	the	DT
en-wikipedia-org-3679	10	16	Robots	Robots	NNP
en-wikipedia-org-3679	10	17	Exclusion	Exclusion	NNP
en-wikipedia-org-3679	10	18	Protocol	Protocol	NNP
en-wikipedia-org-3679	10	19	as	as	IN
en-wikipedia-org-3679	10	20	an	an	DT
en-wikipedia-org-3679	10	21	official	official	JJ
en-wikipedia-org-3679	10	22	standard	standard	NN
en-wikipedia-org-3679	10	23	under	under	IN
en-wikipedia-org-3679	10	24	Internet	Internet	NNP
en-wikipedia-org-3679	10	25	Engineering	Engineering	NNP
en-wikipedia-org-3679	10	26	Task	Task	NNP
en-wikipedia-org-3679	10	27	Force	Force	NNP
en-wikipedia-org-3679	10	28	.	.	.
en-wikipedia-org-3679	11	1	The	the	DT
en-wikipedia-org-3679	11	2	draft[8	draft[8	NNPS
en-wikipedia-org-3679	11	3	]	]	-RRB-
en-wikipedia-org-3679	11	4	will	will	MD
en-wikipedia-org-3679	11	5	now	now	RB
en-wikipedia-org-3679	11	6	undergo	undergo	VB
en-wikipedia-org-3679	11	7	acceptance	acceptance	NN
en-wikipedia-org-3679	11	8	process	process	NN
en-wikipedia-org-3679	11	9	.	.	.
en-wikipedia-org-3679	12	1	Standard[edit	standard[edit	LS
en-wikipedia-org-3679	12	2	]	]	-RRB-
en-wikipedia-org-3679	12	3	When	when	WRB
en-wikipedia-org-3679	12	4	a	a	DT
en-wikipedia-org-3679	12	5	site	site	NN
en-wikipedia-org-3679	12	6	owner	owner	NN
en-wikipedia-org-3679	12	7	wishes	wish	VBZ
en-wikipedia-org-3679	12	8	to	to	TO
en-wikipedia-org-3679	12	9	give	give	VB
en-wikipedia-org-3679	12	10	instructions	instruction	NNS
en-wikipedia-org-3679	12	11	to	to	IN
en-wikipedia-org-3679	12	12	web	web	NN
en-wikipedia-org-3679	12	13	robots	robot	NNS
en-wikipedia-org-3679	12	14	they	-PRON-	PRP
en-wikipedia-org-3679	12	15	place	place	VBP
en-wikipedia-org-3679	12	16	a	a	DT
en-wikipedia-org-3679	12	17	text	text	NN
en-wikipedia-org-3679	12	18	file	file	NN
en-wikipedia-org-3679	12	19	called	call	VBD
en-wikipedia-org-3679	12	20	robots.txt	robots.txt	NN
en-wikipedia-org-3679	12	21	in	in	IN
en-wikipedia-org-3679	12	22	the	the	DT
en-wikipedia-org-3679	12	23	root	root	NN
en-wikipedia-org-3679	12	24	of	of	IN
en-wikipedia-org-3679	12	25	the	the	DT
en-wikipedia-org-3679	12	26	web	web	NN
en-wikipedia-org-3679	12	27	site	site	NN
en-wikipedia-org-3679	12	28	hierarchy	hierarchy	NN
en-wikipedia-org-3679	12	29	(	(	-LRB-
en-wikipedia-org-3679	12	30	e.g.	e.g.	RB
en-wikipedia-org-3679	13	1	https://www.example.com/robots.txt	https://www.example.com/robots.txt	NNP
en-wikipedia-org-3679	13	2	)	)	-RRB-
en-wikipedia-org-3679	13	3	.	.	.
en-wikipedia-org-3679	14	1	This	this	DT
en-wikipedia-org-3679	14	2	text	text	NN
en-wikipedia-org-3679	14	3	file	file	NN
en-wikipedia-org-3679	14	4	contains	contain	VBZ
en-wikipedia-org-3679	14	5	the	the	DT
en-wikipedia-org-3679	14	6	instructions	instruction	NNS
en-wikipedia-org-3679	14	7	in	in	IN
en-wikipedia-org-3679	14	8	a	a	DT
en-wikipedia-org-3679	14	9	specific	specific	JJ
en-wikipedia-org-3679	14	10	format	format	NN
en-wikipedia-org-3679	14	11	(	(	-LRB-
en-wikipedia-org-3679	14	12	see	see	VB
en-wikipedia-org-3679	14	13	examples	example	NNS
en-wikipedia-org-3679	14	14	below	below	RB
en-wikipedia-org-3679	14	15	)	)	-RRB-
en-wikipedia-org-3679	14	16	.	.	.
en-wikipedia-org-3679	15	1	Robots	robot	NNS
en-wikipedia-org-3679	15	2	that	that	WDT
en-wikipedia-org-3679	15	3	choose	choose	VBP
en-wikipedia-org-3679	15	4	to	to	TO
en-wikipedia-org-3679	15	5	follow	follow	VB
en-wikipedia-org-3679	15	6	the	the	DT
en-wikipedia-org-3679	15	7	instructions	instruction	NNS
en-wikipedia-org-3679	15	8	try	try	VBP
en-wikipedia-org-3679	15	9	to	to	TO
en-wikipedia-org-3679	15	10	fetch	fetch	VB
en-wikipedia-org-3679	15	11	this	this	DT
en-wikipedia-org-3679	15	12	file	file	NN
en-wikipedia-org-3679	15	13	and	and	CC
en-wikipedia-org-3679	15	14	read	read	VB
en-wikipedia-org-3679	15	15	the	the	DT
en-wikipedia-org-3679	15	16	instructions	instruction	NNS
en-wikipedia-org-3679	15	17	before	before	IN
en-wikipedia-org-3679	15	18	fetching	fetch	VBG
en-wikipedia-org-3679	15	19	any	any	DT
en-wikipedia-org-3679	15	20	other	other	JJ
en-wikipedia-org-3679	15	21	file	file	NN
en-wikipedia-org-3679	15	22	from	from	IN
en-wikipedia-org-3679	15	23	the	the	DT
en-wikipedia-org-3679	15	24	website	website	NN
en-wikipedia-org-3679	15	25	.	.	.
en-wikipedia-org-3679	16	1	If	if	IN
en-wikipedia-org-3679	16	2	this	this	DT
en-wikipedia-org-3679	16	3	file	file	NN
en-wikipedia-org-3679	16	4	does	do	VBZ
en-wikipedia-org-3679	16	5	n't	not	RB
en-wikipedia-org-3679	16	6	exist	exist	VB
en-wikipedia-org-3679	16	7	,	,	,
en-wikipedia-org-3679	16	8	web	web	NN
en-wikipedia-org-3679	16	9	robots	robot	NNS
en-wikipedia-org-3679	16	10	assume	assume	VBP
en-wikipedia-org-3679	16	11	that	that	IN
en-wikipedia-org-3679	16	12	the	the	DT
en-wikipedia-org-3679	16	13	website	website	NN
en-wikipedia-org-3679	16	14	owner	owner	NN
en-wikipedia-org-3679	16	15	does	do	VBZ
en-wikipedia-org-3679	16	16	not	not	RB
en-wikipedia-org-3679	16	17	wish	wish	VB
en-wikipedia-org-3679	16	18	to	to	TO
en-wikipedia-org-3679	16	19	place	place	VB
en-wikipedia-org-3679	16	20	any	any	DT
en-wikipedia-org-3679	16	21	limitations	limitation	NNS
en-wikipedia-org-3679	16	22	on	on	IN
en-wikipedia-org-3679	16	23	crawling	crawl	VBG
en-wikipedia-org-3679	16	24	the	the	DT
en-wikipedia-org-3679	16	25	entire	entire	JJ
en-wikipedia-org-3679	16	26	site	site	NN
en-wikipedia-org-3679	16	27	.	.	.
en-wikipedia-org-3679	17	1	A	a	DT
en-wikipedia-org-3679	17	2	robots.txt	robots.txt	CD
en-wikipedia-org-3679	17	3	file	file	NN
en-wikipedia-org-3679	17	4	on	on	IN
en-wikipedia-org-3679	17	5	a	a	DT
en-wikipedia-org-3679	17	6	website	website	NN
en-wikipedia-org-3679	17	7	will	will	MD
en-wikipedia-org-3679	17	8	function	function	VB
en-wikipedia-org-3679	17	9	as	as	IN
en-wikipedia-org-3679	17	10	a	a	DT
en-wikipedia-org-3679	17	11	request	request	NN
en-wikipedia-org-3679	17	12	that	that	WDT
en-wikipedia-org-3679	17	13	specified	specify	VBD
en-wikipedia-org-3679	17	14	robots	robot	NNS
en-wikipedia-org-3679	17	15	ignore	ignore	VBP
en-wikipedia-org-3679	17	16	specified	specified	JJ
en-wikipedia-org-3679	17	17	files	file	NNS
en-wikipedia-org-3679	17	18	or	or	CC
en-wikipedia-org-3679	17	19	directories	directory	NNS
en-wikipedia-org-3679	17	20	when	when	WRB
en-wikipedia-org-3679	17	21	crawling	crawl	VBG
en-wikipedia-org-3679	17	22	a	a	DT
en-wikipedia-org-3679	17	23	site	site	NN
en-wikipedia-org-3679	17	24	.	.	.
en-wikipedia-org-3679	18	1	This	this	DT
en-wikipedia-org-3679	18	2	might	may	MD
en-wikipedia-org-3679	18	3	be	be	VB
en-wikipedia-org-3679	18	4	,	,	,
en-wikipedia-org-3679	18	5	for	for	IN
en-wikipedia-org-3679	18	6	example	example	NN
en-wikipedia-org-3679	18	7	,	,	,
en-wikipedia-org-3679	18	8	out	out	IN
en-wikipedia-org-3679	18	9	of	of	IN
en-wikipedia-org-3679	18	10	a	a	DT
en-wikipedia-org-3679	18	11	preference	preference	NN
en-wikipedia-org-3679	18	12	for	for	IN
en-wikipedia-org-3679	18	13	privacy	privacy	NN
en-wikipedia-org-3679	18	14	from	from	IN
en-wikipedia-org-3679	18	15	search	search	NN
en-wikipedia-org-3679	18	16	engine	engine	NN
en-wikipedia-org-3679	18	17	results	result	NNS
en-wikipedia-org-3679	18	18	,	,	,
en-wikipedia-org-3679	18	19	or	or	CC
en-wikipedia-org-3679	18	20	the	the	DT
en-wikipedia-org-3679	18	21	belief	belief	NN
en-wikipedia-org-3679	18	22	that	that	IN
en-wikipedia-org-3679	18	23	the	the	DT
en-wikipedia-org-3679	18	24	content	content	NN
en-wikipedia-org-3679	18	25	of	of	IN
en-wikipedia-org-3679	18	26	the	the	DT
en-wikipedia-org-3679	18	27	selected	select	VBN
en-wikipedia-org-3679	18	28	directories	directory	NNS
en-wikipedia-org-3679	18	29	might	may	MD
en-wikipedia-org-3679	18	30	be	be	VB
en-wikipedia-org-3679	18	31	misleading	misleading	JJ
en-wikipedia-org-3679	18	32	or	or	CC
en-wikipedia-org-3679	18	33	irrelevant	irrelevant	JJ
en-wikipedia-org-3679	18	34	to	to	IN
en-wikipedia-org-3679	18	35	the	the	DT
en-wikipedia-org-3679	18	36	categorization	categorization	NN
en-wikipedia-org-3679	18	37	of	of	IN
en-wikipedia-org-3679	18	38	the	the	DT
en-wikipedia-org-3679	18	39	site	site	NN
en-wikipedia-org-3679	18	40	as	as	IN
en-wikipedia-org-3679	18	41	a	a	DT
en-wikipedia-org-3679	18	42	whole	whole	NN
en-wikipedia-org-3679	18	43	,	,	,
en-wikipedia-org-3679	18	44	or	or	CC
en-wikipedia-org-3679	18	45	out	out	IN
en-wikipedia-org-3679	18	46	of	of	IN
en-wikipedia-org-3679	18	47	a	a	DT
en-wikipedia-org-3679	18	48	desire	desire	NN
en-wikipedia-org-3679	18	49	that	that	IN
en-wikipedia-org-3679	18	50	an	an	DT
en-wikipedia-org-3679	18	51	application	application	NN
en-wikipedia-org-3679	18	52	only	only	RB
en-wikipedia-org-3679	18	53	operates	operate	VBZ
en-wikipedia-org-3679	18	54	on	on	IN
en-wikipedia-org-3679	18	55	certain	certain	JJ
en-wikipedia-org-3679	18	56	data	datum	NNS
en-wikipedia-org-3679	18	57	.	.	.
en-wikipedia-org-3679	19	1	Links	link	NNS
en-wikipedia-org-3679	19	2	to	to	IN
en-wikipedia-org-3679	19	3	pages	page	NNS
en-wikipedia-org-3679	19	4	listed	list	VBN
en-wikipedia-org-3679	19	5	in	in	IN
en-wikipedia-org-3679	19	6	robots.txt	robots.txt	NN
en-wikipedia-org-3679	19	7	can	can	MD
en-wikipedia-org-3679	19	8	still	still	RB
en-wikipedia-org-3679	19	9	appear	appear	VB
en-wikipedia-org-3679	19	10	in	in	IN
en-wikipedia-org-3679	19	11	search	search	NN
en-wikipedia-org-3679	19	12	results	result	NNS
en-wikipedia-org-3679	19	13	if	if	IN
en-wikipedia-org-3679	19	14	they	-PRON-	PRP
en-wikipedia-org-3679	19	15	are	be	VBP
en-wikipedia-org-3679	19	16	linked	link	VBN
en-wikipedia-org-3679	19	17	to	to	IN
en-wikipedia-org-3679	19	18	from	from	IN
en-wikipedia-org-3679	19	19	a	a	DT
en-wikipedia-org-3679	19	20	page	page	NN
en-wikipedia-org-3679	19	21	that	that	WDT
en-wikipedia-org-3679	19	22	is	be	VBZ
en-wikipedia-org-3679	19	23	crawled	crawled	JJ
en-wikipedia-org-3679	19	24	.	.	.
en-wikipedia-org-3679	20	1	[	[	-LRB-
en-wikipedia-org-3679	20	2	9	9	CD
en-wikipedia-org-3679	20	3	]	]	-RRB-
en-wikipedia-org-3679	20	4	A	a	DT
en-wikipedia-org-3679	20	5	robots.txt	robots.txt	CD
en-wikipedia-org-3679	20	6	file	file	NN
en-wikipedia-org-3679	20	7	covers	cover	VBZ
en-wikipedia-org-3679	20	8	one	one	CD
en-wikipedia-org-3679	20	9	origin	origin	NN
en-wikipedia-org-3679	20	10	.	.	.
en-wikipedia-org-3679	21	1	For	for	IN
en-wikipedia-org-3679	21	2	websites	website	NNS
en-wikipedia-org-3679	21	3	with	with	IN
en-wikipedia-org-3679	21	4	multiple	multiple	JJ
en-wikipedia-org-3679	21	5	subdomains	subdomain	NNS
en-wikipedia-org-3679	21	6	,	,	,
en-wikipedia-org-3679	21	7	each	each	DT
en-wikipedia-org-3679	21	8	subdomain	subdomain	NN
en-wikipedia-org-3679	21	9	must	must	MD
en-wikipedia-org-3679	21	10	have	have	VB
en-wikipedia-org-3679	21	11	its	-PRON-	PRP$
en-wikipedia-org-3679	21	12	own	own	JJ
en-wikipedia-org-3679	21	13	robots.txt	robots.txt	NN
en-wikipedia-org-3679	21	14	file	file	NN
en-wikipedia-org-3679	21	15	.	.	.
en-wikipedia-org-3679	22	1	If	if	IN
en-wikipedia-org-3679	22	2	example.com	example.com	ADD
en-wikipedia-org-3679	22	3	had	have	VBD
en-wikipedia-org-3679	22	4	a	a	DT
en-wikipedia-org-3679	22	5	robots.txt	robots.txt	NN
en-wikipedia-org-3679	22	6	file	file	NN
en-wikipedia-org-3679	22	7	but	but	CC
en-wikipedia-org-3679	22	8	a.example.com	a.example.com	NNP
en-wikipedia-org-3679	22	9	did	do	VBD
en-wikipedia-org-3679	22	10	not	not	RB
en-wikipedia-org-3679	22	11	,	,	,
en-wikipedia-org-3679	22	12	the	the	DT
en-wikipedia-org-3679	22	13	rules	rule	NNS
en-wikipedia-org-3679	22	14	that	that	WDT
en-wikipedia-org-3679	22	15	would	would	MD
en-wikipedia-org-3679	22	16	apply	apply	VB
en-wikipedia-org-3679	22	17	for	for	IN
en-wikipedia-org-3679	22	18	example.com	example.com	ADD
en-wikipedia-org-3679	22	19	would	would	MD
en-wikipedia-org-3679	22	20	not	not	RB
en-wikipedia-org-3679	22	21	apply	apply	VB
en-wikipedia-org-3679	22	22	to	to	IN
en-wikipedia-org-3679	22	23	a.example.com	a.example.com	ADD
en-wikipedia-org-3679	22	24	.	.	.
en-wikipedia-org-3679	23	1	In	in	IN
en-wikipedia-org-3679	23	2	addition	addition	NN
en-wikipedia-org-3679	23	3	,	,	,
en-wikipedia-org-3679	23	4	each	each	DT
en-wikipedia-org-3679	23	5	protocol	protocol	NN
en-wikipedia-org-3679	23	6	and	and	CC
en-wikipedia-org-3679	23	7	port	port	NN
en-wikipedia-org-3679	23	8	needs	need	VBZ
en-wikipedia-org-3679	23	9	its	-PRON-	PRP$
en-wikipedia-org-3679	23	10	own	own	JJ
en-wikipedia-org-3679	23	11	robots.txt	robots.txt	NN
en-wikipedia-org-3679	23	12	file	file	NN
en-wikipedia-org-3679	23	13	;	;	:
en-wikipedia-org-3679	23	14	http://example.com/robots.txt	http://example.com/robots.txt	NNP
en-wikipedia-org-3679	23	15	does	do	VBZ
en-wikipedia-org-3679	23	16	not	not	RB
en-wikipedia-org-3679	23	17	apply	apply	VB
en-wikipedia-org-3679	23	18	to	to	IN
en-wikipedia-org-3679	23	19	pages	page	NNS
en-wikipedia-org-3679	23	20	under	under	IN
en-wikipedia-org-3679	23	21	http://example.com:8080/	http://example.com:8080/	NN
en-wikipedia-org-3679	23	22	or	or	CC
en-wikipedia-org-3679	23	23	https://example.com/.	https://example.com/.	CD
en-wikipedia-org-3679	24	1	Some	some	DT
en-wikipedia-org-3679	24	2	major	major	JJ
en-wikipedia-org-3679	24	3	search	search	NN
en-wikipedia-org-3679	24	4	engines	engine	NNS
en-wikipedia-org-3679	24	5	following	follow	VBG
en-wikipedia-org-3679	24	6	this	this	DT
en-wikipedia-org-3679	24	7	standard	standard	NN
en-wikipedia-org-3679	24	8	include	include	VBP
en-wikipedia-org-3679	24	9	Ask,[10	ask,[10	NN
en-wikipedia-org-3679	24	10	]	]	-RRB-
en-wikipedia-org-3679	24	11	AOL,[11	aol,[11	NN
en-wikipedia-org-3679	24	12	]	]	-RRB-
en-wikipedia-org-3679	24	13	Baidu,[12	Baidu,[12	NNP
en-wikipedia-org-3679	24	14	]	]	-RRB-
en-wikipedia-org-3679	24	15	DuckDuckGo,[13	DuckDuckGo,[13	NNP
en-wikipedia-org-3679	24	16	]	]	-RRB-
en-wikipedia-org-3679	24	17	Google,[14	Google,[14	NNP
en-wikipedia-org-3679	24	18	]	]	-RRB-
en-wikipedia-org-3679	24	19	Yahoo!,[15	Yahoo!,[15	NNP
en-wikipedia-org-3679	24	20	]	]	-RRB-
en-wikipedia-org-3679	24	21	and	and	CC
en-wikipedia-org-3679	24	22	Yandex	Yandex	NNP
en-wikipedia-org-3679	24	23	.	.	.
en-wikipedia-org-3679	25	1	[	[	-LRB-
en-wikipedia-org-3679	25	2	16	16	CD
en-wikipedia-org-3679	25	3	]	]	-RRB-
en-wikipedia-org-3679	25	4	Bing[17	bing[17	NN
en-wikipedia-org-3679	25	5	]	]	-RRB-
en-wikipedia-org-3679	25	6	still	still	RB
en-wikipedia-org-3679	25	7	is	be	VBZ
en-wikipedia-org-3679	25	8	not	not	RB
en-wikipedia-org-3679	25	9	fully[18	fully[18	RB
en-wikipedia-org-3679	25	10	]	]	-RRB-
en-wikipedia-org-3679	25	11	compatible	compatible	JJ
en-wikipedia-org-3679	25	12	with	with	IN
en-wikipedia-org-3679	25	13	the	the	DT
en-wikipedia-org-3679	25	14	standard[3	standard[3	NNP
en-wikipedia-org-3679	25	15	]	]	-RRB-
en-wikipedia-org-3679	25	16	as	as	IN
en-wikipedia-org-3679	25	17	it	-PRON-	PRP
en-wikipedia-org-3679	25	18	can	can	MD
en-wikipedia-org-3679	25	19	not	not	RB
en-wikipedia-org-3679	25	20	inherit	inherit	VB
en-wikipedia-org-3679	25	21	settings	setting	NNS
en-wikipedia-org-3679	25	22	from	from	IN
en-wikipedia-org-3679	25	23	the	the	DT
en-wikipedia-org-3679	25	24	wildcard	wildcard	NN
en-wikipedia-org-3679	25	25	(	(	-LRB-
en-wikipedia-org-3679	25	26	*	*	NFP
en-wikipedia-org-3679	25	27	)	)	-RRB-
en-wikipedia-org-3679	25	28	.	.	.
en-wikipedia-org-3679	26	1	The	the	DT
en-wikipedia-org-3679	26	2	volunteering	volunteering	NN
en-wikipedia-org-3679	26	3	group	group	NN
en-wikipedia-org-3679	26	4	Archive	Archive	NNP
en-wikipedia-org-3679	26	5	Team	Team	NNP
en-wikipedia-org-3679	26	6	explicitly	explicitly	RB
en-wikipedia-org-3679	26	7	ignores	ignore	VBZ
en-wikipedia-org-3679	26	8	robots.txt	robots.txt	NNS
en-wikipedia-org-3679	26	9	for	for	IN
en-wikipedia-org-3679	26	10	the	the	DT
en-wikipedia-org-3679	26	11	most	most	JJS
en-wikipedia-org-3679	26	12	part	part	NN
en-wikipedia-org-3679	26	13	,	,	,
en-wikipedia-org-3679	26	14	viewing	view	VBG
en-wikipedia-org-3679	26	15	it	-PRON-	PRP
en-wikipedia-org-3679	26	16	as	as	IN
en-wikipedia-org-3679	26	17	an	an	DT
en-wikipedia-org-3679	26	18	obsolete	obsolete	JJ
en-wikipedia-org-3679	26	19	standard	standard	NN
en-wikipedia-org-3679	26	20	that	that	WDT
en-wikipedia-org-3679	26	21	hinders	hinder	NNS
en-wikipedia-org-3679	26	22	web	web	NN
en-wikipedia-org-3679	26	23	archival	archival	NN
en-wikipedia-org-3679	26	24	efforts	effort	NNS
en-wikipedia-org-3679	26	25	.	.	.
en-wikipedia-org-3679	27	1	According	accord	VBG
en-wikipedia-org-3679	27	2	to	to	IN
en-wikipedia-org-3679	27	3	project	project	NN
en-wikipedia-org-3679	27	4	leader	leader	NN
en-wikipedia-org-3679	27	5	Jason	Jason	NNP
en-wikipedia-org-3679	27	6	Scott	Scott	NNP
en-wikipedia-org-3679	27	7	,	,	,
en-wikipedia-org-3679	27	8	"	"	``
en-wikipedia-org-3679	27	9	unchecked	unchecked	JJ
en-wikipedia-org-3679	27	10	,	,	,
en-wikipedia-org-3679	27	11	and	and	CC
en-wikipedia-org-3679	27	12	left	leave	VBD
en-wikipedia-org-3679	27	13	alone	alone	RB
en-wikipedia-org-3679	27	14	,	,	,
en-wikipedia-org-3679	27	15	the	the	DT
en-wikipedia-org-3679	27	16	robots.txt	robots.txt	NN
en-wikipedia-org-3679	27	17	file	file	NN
en-wikipedia-org-3679	27	18	ensures	ensure	VBZ
en-wikipedia-org-3679	27	19	no	no	DT
en-wikipedia-org-3679	27	20	mirroring	mirroring	NN
en-wikipedia-org-3679	27	21	or	or	CC
en-wikipedia-org-3679	27	22	reference	reference	NN
en-wikipedia-org-3679	27	23	for	for	IN
en-wikipedia-org-3679	27	24	items	item	NNS
en-wikipedia-org-3679	27	25	that	that	WDT
en-wikipedia-org-3679	27	26	may	may	MD
en-wikipedia-org-3679	27	27	have	have	VB
en-wikipedia-org-3679	27	28	general	general	JJ
en-wikipedia-org-3679	27	29	use	use	NN
en-wikipedia-org-3679	27	30	and	and	CC
en-wikipedia-org-3679	27	31	meaning	meaning	NN
en-wikipedia-org-3679	27	32	beyond	beyond	IN
en-wikipedia-org-3679	27	33	the	the	DT
en-wikipedia-org-3679	27	34	website	website	NN
en-wikipedia-org-3679	27	35	's	's	POS
en-wikipedia-org-3679	27	36	context	context	NN
en-wikipedia-org-3679	27	37	.	.	.
en-wikipedia-org-3679	28	1	"	"	``
en-wikipedia-org-3679	28	2	[	[	-LRB-
en-wikipedia-org-3679	28	3	19	19	CD
en-wikipedia-org-3679	28	4	]	]	-RRB-
en-wikipedia-org-3679	28	5	For	for	IN
en-wikipedia-org-3679	28	6	some	some	DT
en-wikipedia-org-3679	28	7	years	year	NNS
en-wikipedia-org-3679	28	8	,	,	,
en-wikipedia-org-3679	28	9	the	the	DT
en-wikipedia-org-3679	28	10	Internet	Internet	NNP
en-wikipedia-org-3679	28	11	Archive	Archive	NNP
en-wikipedia-org-3679	28	12	did	do	VBD
en-wikipedia-org-3679	28	13	not	not	RB
en-wikipedia-org-3679	28	14	crawl	crawl	VB
en-wikipedia-org-3679	28	15	sites	site	NNS
en-wikipedia-org-3679	28	16	with	with	IN
en-wikipedia-org-3679	28	17	robots.txt	robots.txt	NN
en-wikipedia-org-3679	28	18	,	,	,
en-wikipedia-org-3679	28	19	but	but	CC
en-wikipedia-org-3679	28	20	in	in	IN
en-wikipedia-org-3679	28	21	April	April	NNP
en-wikipedia-org-3679	28	22	2017	2017	CD
en-wikipedia-org-3679	28	23	,	,	,
en-wikipedia-org-3679	28	24	it	-PRON-	PRP
en-wikipedia-org-3679	28	25	announced[20	announced[20	VBP
en-wikipedia-org-3679	28	26	]	]	-RRB-
en-wikipedia-org-3679	28	27	that	that	IN
en-wikipedia-org-3679	28	28	it	-PRON-	PRP
en-wikipedia-org-3679	28	29	would	would	MD
en-wikipedia-org-3679	28	30	no	no	RB
en-wikipedia-org-3679	28	31	longer	long	RBR
en-wikipedia-org-3679	28	32	honour	honour	VB
en-wikipedia-org-3679	28	33	directives	directive	NNS
en-wikipedia-org-3679	28	34	in	in	IN
en-wikipedia-org-3679	28	35	the	the	DT
en-wikipedia-org-3679	28	36	robots.txt	robots.txt	NN
en-wikipedia-org-3679	28	37	files	file	NNS
en-wikipedia-org-3679	28	38	.	.	.
en-wikipedia-org-3679	29	1	“	"	``
en-wikipedia-org-3679	29	2	Over	over	IN
en-wikipedia-org-3679	29	3	time	time	NN
en-wikipedia-org-3679	29	4	we	-PRON-	PRP
en-wikipedia-org-3679	29	5	have	have	VBP
en-wikipedia-org-3679	29	6	observed	observe	VBN
en-wikipedia-org-3679	29	7	that	that	IN
en-wikipedia-org-3679	29	8	the	the	DT
en-wikipedia-org-3679	29	9	robots.txt	robots.txt	NN
en-wikipedia-org-3679	29	10	files	file	NNS
en-wikipedia-org-3679	29	11	that	that	WDT
en-wikipedia-org-3679	29	12	are	be	VBP
en-wikipedia-org-3679	29	13	geared	gear	VBN
en-wikipedia-org-3679	29	14	toward	toward	IN
en-wikipedia-org-3679	29	15	search	search	NN
en-wikipedia-org-3679	29	16	engine	engine	NN
en-wikipedia-org-3679	29	17	crawlers	crawler	NNS
en-wikipedia-org-3679	29	18	do	do	VBP
en-wikipedia-org-3679	29	19	not	not	RB
en-wikipedia-org-3679	29	20	necessarily	necessarily	RB
en-wikipedia-org-3679	29	21	serve	serve	VB
en-wikipedia-org-3679	29	22	our	-PRON-	PRP$
en-wikipedia-org-3679	29	23	archival	archival	NN
en-wikipedia-org-3679	29	24	purposes	purpose	NNS
en-wikipedia-org-3679	29	25	"	"	''
en-wikipedia-org-3679	29	26	.	.	.
en-wikipedia-org-3679	30	1	[	[	-LRB-
en-wikipedia-org-3679	30	2	21	21	CD
en-wikipedia-org-3679	30	3	]	]	-RRB-
en-wikipedia-org-3679	30	4	This	this	DT
en-wikipedia-org-3679	30	5	was	be	VBD
en-wikipedia-org-3679	30	6	in	in	IN
en-wikipedia-org-3679	30	7	response	response	NN
en-wikipedia-org-3679	30	8	to	to	IN
en-wikipedia-org-3679	30	9	entire	entire	JJ
en-wikipedia-org-3679	30	10	domains	domain	NNS
en-wikipedia-org-3679	30	11	being	be	VBG
en-wikipedia-org-3679	30	12	tagged	tag	VBN
en-wikipedia-org-3679	30	13	with	with	IN
en-wikipedia-org-3679	30	14	robots.txt	robots.txt	CD
en-wikipedia-org-3679	30	15	when	when	WRB
en-wikipedia-org-3679	30	16	the	the	DT
en-wikipedia-org-3679	30	17	content	content	NN
en-wikipedia-org-3679	30	18	became	become	VBD
en-wikipedia-org-3679	30	19	obsolete	obsolete	JJ
en-wikipedia-org-3679	30	20	.	.	.
en-wikipedia-org-3679	31	1	[	[	-LRB-
en-wikipedia-org-3679	31	2	21	21	CD
en-wikipedia-org-3679	31	3	]	]	-RRB-
en-wikipedia-org-3679	31	4	Security[edit	security[edit	CD
en-wikipedia-org-3679	31	5	]	]	-RRB-
en-wikipedia-org-3679	31	6	Despite	despite	IN
en-wikipedia-org-3679	31	7	the	the	DT
en-wikipedia-org-3679	31	8	use	use	NN
en-wikipedia-org-3679	31	9	of	of	IN
en-wikipedia-org-3679	31	10	the	the	DT
en-wikipedia-org-3679	31	11	terms	term	NNS
en-wikipedia-org-3679	31	12	"	"	``
en-wikipedia-org-3679	31	13	allow	allow	VBP
en-wikipedia-org-3679	31	14	"	"	''
en-wikipedia-org-3679	31	15	and	and	CC
en-wikipedia-org-3679	31	16	"	"	``
en-wikipedia-org-3679	31	17	disallow	disallow	JJ
en-wikipedia-org-3679	31	18	"	"	''
en-wikipedia-org-3679	31	19	,	,	,
en-wikipedia-org-3679	31	20	the	the	DT
en-wikipedia-org-3679	31	21	protocol	protocol	NN
en-wikipedia-org-3679	31	22	is	be	VBZ
en-wikipedia-org-3679	31	23	purely	purely	RB
en-wikipedia-org-3679	31	24	advisory[22	advisory[22	XX
en-wikipedia-org-3679	31	25	]	]	-RRB-
en-wikipedia-org-3679	31	26	and	and	CC
en-wikipedia-org-3679	31	27	relies	rely	VBZ
en-wikipedia-org-3679	31	28	on	on	IN
en-wikipedia-org-3679	31	29	the	the	DT
en-wikipedia-org-3679	31	30	compliance	compliance	NN
en-wikipedia-org-3679	31	31	of	of	IN
en-wikipedia-org-3679	31	32	the	the	DT
en-wikipedia-org-3679	31	33	web	web	NN
en-wikipedia-org-3679	31	34	robot	robot	NN
en-wikipedia-org-3679	31	35	.	.	.
en-wikipedia-org-3679	32	1	Malicious	malicious	JJ
en-wikipedia-org-3679	32	2	web	web	NN
en-wikipedia-org-3679	32	3	robots	robot	NNS
en-wikipedia-org-3679	32	4	are	be	VBP
en-wikipedia-org-3679	32	5	unlikely	unlikely	JJ
en-wikipedia-org-3679	32	6	to	to	TO
en-wikipedia-org-3679	32	7	honor	honor	VB
en-wikipedia-org-3679	32	8	robots.txt	robots.txt	NN
en-wikipedia-org-3679	32	9	;	;	:
en-wikipedia-org-3679	32	10	some	some	DT
en-wikipedia-org-3679	32	11	may	may	MD
en-wikipedia-org-3679	32	12	even	even	RB
en-wikipedia-org-3679	32	13	use	use	VB
en-wikipedia-org-3679	32	14	the	the	DT
en-wikipedia-org-3679	32	15	robots.txt	robots.txt	NN
en-wikipedia-org-3679	32	16	as	as	IN
en-wikipedia-org-3679	32	17	a	a	DT
en-wikipedia-org-3679	32	18	guide	guide	NN
en-wikipedia-org-3679	32	19	to	to	TO
en-wikipedia-org-3679	32	20	find	find	VB
en-wikipedia-org-3679	32	21	disallowed	disallow	VBN
en-wikipedia-org-3679	32	22	links	link	NNS
en-wikipedia-org-3679	32	23	and	and	CC
en-wikipedia-org-3679	32	24	go	go	VB
en-wikipedia-org-3679	32	25	straight	straight	RB
en-wikipedia-org-3679	32	26	to	to	IN
en-wikipedia-org-3679	32	27	them	-PRON-	PRP
en-wikipedia-org-3679	32	28	.	.	.
en-wikipedia-org-3679	33	1	While	while	IN
en-wikipedia-org-3679	33	2	this	this	DT
en-wikipedia-org-3679	33	3	is	be	VBZ
en-wikipedia-org-3679	33	4	sometimes	sometimes	RB
en-wikipedia-org-3679	33	5	claimed	claim	VBN
en-wikipedia-org-3679	33	6	to	to	TO
en-wikipedia-org-3679	33	7	be	be	VB
en-wikipedia-org-3679	33	8	a	a	DT
en-wikipedia-org-3679	33	9	security	security	NN
en-wikipedia-org-3679	33	10	risk,[23	risk,[23	CD
en-wikipedia-org-3679	33	11	]	]	-RRB-
en-wikipedia-org-3679	33	12	this	this	DT
en-wikipedia-org-3679	33	13	sort	sort	NN
en-wikipedia-org-3679	33	14	of	of	IN
en-wikipedia-org-3679	33	15	security	security	NN
en-wikipedia-org-3679	33	16	through	through	IN
en-wikipedia-org-3679	33	17	obscurity	obscurity	NN
en-wikipedia-org-3679	33	18	is	be	VBZ
en-wikipedia-org-3679	33	19	discouraged	discourage	VBN
en-wikipedia-org-3679	33	20	by	by	IN
en-wikipedia-org-3679	33	21	standards	standard	NNS
en-wikipedia-org-3679	33	22	bodies	body	NNS
en-wikipedia-org-3679	33	23	.	.	.
en-wikipedia-org-3679	34	1	The	the	DT
en-wikipedia-org-3679	34	2	National	National	NNP
en-wikipedia-org-3679	34	3	Institute	Institute	NNP
en-wikipedia-org-3679	34	4	of	of	IN
en-wikipedia-org-3679	34	5	Standards	Standards	NNPS
en-wikipedia-org-3679	34	6	and	and	CC
en-wikipedia-org-3679	34	7	Technology	Technology	NNP
en-wikipedia-org-3679	34	8	(	(	-LRB-
en-wikipedia-org-3679	34	9	NIST	NIST	NNP
en-wikipedia-org-3679	34	10	)	)	-RRB-
en-wikipedia-org-3679	34	11	in	in	IN
en-wikipedia-org-3679	34	12	the	the	DT
en-wikipedia-org-3679	34	13	United	United	NNP
en-wikipedia-org-3679	34	14	States	States	NNP
en-wikipedia-org-3679	34	15	specifically	specifically	RB
en-wikipedia-org-3679	34	16	recommends	recommend	VBZ
en-wikipedia-org-3679	34	17	against	against	IN
en-wikipedia-org-3679	34	18	this	this	DT
en-wikipedia-org-3679	34	19	practice	practice	NN
en-wikipedia-org-3679	34	20	:	:	:
en-wikipedia-org-3679	34	21	"	"	``
en-wikipedia-org-3679	34	22	System	system	NN
en-wikipedia-org-3679	34	23	security	security	NN
en-wikipedia-org-3679	34	24	should	should	MD
en-wikipedia-org-3679	34	25	not	not	RB
en-wikipedia-org-3679	34	26	depend	depend	VB
en-wikipedia-org-3679	34	27	on	on	IN
en-wikipedia-org-3679	34	28	the	the	DT
en-wikipedia-org-3679	34	29	secrecy	secrecy	NN
en-wikipedia-org-3679	34	30	of	of	IN
en-wikipedia-org-3679	34	31	the	the	DT
en-wikipedia-org-3679	34	32	implementation	implementation	NN
en-wikipedia-org-3679	34	33	or	or	CC
en-wikipedia-org-3679	34	34	its	-PRON-	PRP$
en-wikipedia-org-3679	34	35	components	component	NNS
en-wikipedia-org-3679	34	36	.	.	.
en-wikipedia-org-3679	35	1	"	"	``
en-wikipedia-org-3679	35	2	[	[	-LRB-
en-wikipedia-org-3679	35	3	24	24	CD
en-wikipedia-org-3679	35	4	]	]	-RRB-
en-wikipedia-org-3679	35	5	In	in	IN
en-wikipedia-org-3679	35	6	the	the	DT
en-wikipedia-org-3679	35	7	context	context	NN
en-wikipedia-org-3679	35	8	of	of	IN
en-wikipedia-org-3679	35	9	robots.txt	robots.txt	JJ
en-wikipedia-org-3679	35	10	files	file	NNS
en-wikipedia-org-3679	35	11	,	,	,
en-wikipedia-org-3679	35	12	security	security	NN
en-wikipedia-org-3679	35	13	through	through	IN
en-wikipedia-org-3679	35	14	obscurity	obscurity	NN
en-wikipedia-org-3679	35	15	is	be	VBZ
en-wikipedia-org-3679	35	16	not	not	RB
en-wikipedia-org-3679	35	17	recommended	recommend	VBN
en-wikipedia-org-3679	35	18	as	as	IN
en-wikipedia-org-3679	35	19	a	a	DT
en-wikipedia-org-3679	35	20	security	security	NN
en-wikipedia-org-3679	35	21	technique	technique	NN
en-wikipedia-org-3679	35	22	.	.	.
en-wikipedia-org-3679	36	1	[	[	-LRB-
en-wikipedia-org-3679	36	2	25	25	CD
en-wikipedia-org-3679	36	3	]	]	-RRB-
en-wikipedia-org-3679	36	4	Alternatives[edit	Alternatives[edit	``
en-wikipedia-org-3679	36	5	]	]	-RRB-
en-wikipedia-org-3679	36	6	Many	many	JJ
en-wikipedia-org-3679	36	7	robots	robot	NNS
en-wikipedia-org-3679	36	8	also	also	RB
en-wikipedia-org-3679	36	9	pass	pass	VBP
en-wikipedia-org-3679	36	10	a	a	DT
en-wikipedia-org-3679	36	11	special	special	JJ
en-wikipedia-org-3679	36	12	user	user	NN
en-wikipedia-org-3679	36	13	-	-	HYPH
en-wikipedia-org-3679	36	14	agent	agent	NN
en-wikipedia-org-3679	36	15	to	to	IN
en-wikipedia-org-3679	36	16	the	the	DT
en-wikipedia-org-3679	36	17	web	web	NN
en-wikipedia-org-3679	36	18	server	server	NN
en-wikipedia-org-3679	36	19	when	when	WRB
en-wikipedia-org-3679	36	20	fetching	fetch	VBG
en-wikipedia-org-3679	36	21	content	content	NN
en-wikipedia-org-3679	36	22	.	.	.
en-wikipedia-org-3679	37	1	[	[	-LRB-
en-wikipedia-org-3679	37	2	26	26	CD
en-wikipedia-org-3679	37	3	]	]	-RRB-
en-wikipedia-org-3679	37	4	A	a	DT
en-wikipedia-org-3679	37	5	web	web	NN
en-wikipedia-org-3679	37	6	administrator	administrator	NN
en-wikipedia-org-3679	37	7	could	could	MD
en-wikipedia-org-3679	37	8	also	also	RB
en-wikipedia-org-3679	37	9	configure	configure	VB
en-wikipedia-org-3679	37	10	the	the	DT
en-wikipedia-org-3679	37	11	server	server	NN
en-wikipedia-org-3679	37	12	to	to	TO
en-wikipedia-org-3679	37	13	automatically	automatically	RB
en-wikipedia-org-3679	37	14	return	return	VB
en-wikipedia-org-3679	37	15	failure	failure	NN
en-wikipedia-org-3679	37	16	(	(	-LRB-
en-wikipedia-org-3679	37	17	or	or	CC
en-wikipedia-org-3679	37	18	pass	pass	VB
en-wikipedia-org-3679	37	19	alternative	alternative	JJ
en-wikipedia-org-3679	37	20	content	content	NN
en-wikipedia-org-3679	37	21	)	)	-RRB-
en-wikipedia-org-3679	37	22	when	when	WRB
en-wikipedia-org-3679	37	23	it	-PRON-	PRP
en-wikipedia-org-3679	37	24	detects	detect	VBZ
en-wikipedia-org-3679	37	25	a	a	DT
en-wikipedia-org-3679	37	26	connection	connection	NN
en-wikipedia-org-3679	37	27	using	use	VBG
en-wikipedia-org-3679	37	28	one	one	CD
en-wikipedia-org-3679	37	29	of	of	IN
en-wikipedia-org-3679	37	30	the	the	DT
en-wikipedia-org-3679	37	31	robots	robot	NNS
en-wikipedia-org-3679	37	32	.	.	.
en-wikipedia-org-3679	38	1	[	[	-LRB-
en-wikipedia-org-3679	38	2	27][28	27][28	CD
en-wikipedia-org-3679	38	3	]	]	-RRB-
en-wikipedia-org-3679	38	4	Some	some	DT
en-wikipedia-org-3679	38	5	sites	site	NNS
en-wikipedia-org-3679	38	6	,	,	,
en-wikipedia-org-3679	38	7	such	such	JJ
en-wikipedia-org-3679	38	8	as	as	IN
en-wikipedia-org-3679	38	9	Google	Google	NNP
en-wikipedia-org-3679	38	10	,	,	,
en-wikipedia-org-3679	38	11	host	host	VB
en-wikipedia-org-3679	38	12	a	a	DT
en-wikipedia-org-3679	38	13	humans.txt	humans.txt	NNP
en-wikipedia-org-3679	38	14	file	file	NN
en-wikipedia-org-3679	38	15	that	that	WDT
en-wikipedia-org-3679	38	16	displays	display	VBZ
en-wikipedia-org-3679	38	17	information	information	NN
en-wikipedia-org-3679	38	18	meant	mean	VBN
en-wikipedia-org-3679	38	19	for	for	IN
en-wikipedia-org-3679	38	20	humans	human	NNS
en-wikipedia-org-3679	38	21	to	to	TO
en-wikipedia-org-3679	38	22	read	read	VB
en-wikipedia-org-3679	38	23	.	.	.
en-wikipedia-org-3679	39	1	[	[	-LRB-
en-wikipedia-org-3679	39	2	29	29	CD
en-wikipedia-org-3679	39	3	]	]	-RRB-
en-wikipedia-org-3679	39	4	Some	some	DT
en-wikipedia-org-3679	39	5	sites	site	NNS
en-wikipedia-org-3679	39	6	such	such	JJ
en-wikipedia-org-3679	39	7	as	as	IN
en-wikipedia-org-3679	39	8	GitHub	GitHub	NNP
en-wikipedia-org-3679	39	9	redirect	redirect	VBP
en-wikipedia-org-3679	39	10	humans.txt	humans.txt	ADD
en-wikipedia-org-3679	39	11	to	to	IN
en-wikipedia-org-3679	39	12	an	an	DT
en-wikipedia-org-3679	39	13	about	about	IN
en-wikipedia-org-3679	39	14	page	page	NN
en-wikipedia-org-3679	39	15	.	.	.
en-wikipedia-org-3679	40	1	[	[	-LRB-
en-wikipedia-org-3679	40	2	30	30	CD
en-wikipedia-org-3679	40	3	]	]	-RRB-
en-wikipedia-org-3679	40	4	Previously	previously	RB
en-wikipedia-org-3679	40	5	Google	Google	NNP
en-wikipedia-org-3679	40	6	had	have	VBD
en-wikipedia-org-3679	40	7	a	a	DT
en-wikipedia-org-3679	40	8	joke	joke	NN
en-wikipedia-org-3679	40	9	file	file	NN
en-wikipedia-org-3679	40	10	hosted	host	VBD
en-wikipedia-org-3679	40	11	at	at	IN
en-wikipedia-org-3679	40	12	/killer	/killer	NN
en-wikipedia-org-3679	40	13	-	-	HYPH
en-wikipedia-org-3679	40	14	robots.txt	robots.txt	NNP
en-wikipedia-org-3679	40	15	instructing	instruct	VBG
en-wikipedia-org-3679	40	16	the	the	DT
en-wikipedia-org-3679	40	17	Terminator	Terminator	NNP
en-wikipedia-org-3679	40	18	not	not	RB
en-wikipedia-org-3679	40	19	to	to	TO
en-wikipedia-org-3679	40	20	kill	kill	VB
en-wikipedia-org-3679	40	21	the	the	DT
en-wikipedia-org-3679	40	22	company	company	NN
en-wikipedia-org-3679	40	23	founders	founder	VBZ
en-wikipedia-org-3679	40	24	Larry	Larry	NNP
en-wikipedia-org-3679	40	25	Page	Page	NNP
en-wikipedia-org-3679	40	26	and	and	CC
en-wikipedia-org-3679	40	27	Sergey	Sergey	NNP
en-wikipedia-org-3679	40	28	Brin	Brin	NNP
en-wikipedia-org-3679	40	29	.	.	.
en-wikipedia-org-3679	41	1	[	[	-LRB-
en-wikipedia-org-3679	41	2	31][32	31][32	CD
en-wikipedia-org-3679	41	3	]	]	-RRB-
en-wikipedia-org-3679	41	4	Examples[edit	examples[edit	LS
en-wikipedia-org-3679	41	5	]	]	-RRB-
en-wikipedia-org-3679	41	6	This	this	DT
en-wikipedia-org-3679	41	7	example	example	NN
en-wikipedia-org-3679	41	8	tells	tell	VBZ
en-wikipedia-org-3679	41	9	all	all	DT
en-wikipedia-org-3679	41	10	robots	robot	NNS
en-wikipedia-org-3679	41	11	that	that	IN
en-wikipedia-org-3679	41	12	they	-PRON-	PRP
en-wikipedia-org-3679	41	13	can	can	MD
en-wikipedia-org-3679	41	14	visit	visit	VB
en-wikipedia-org-3679	41	15	all	all	DT
en-wikipedia-org-3679	41	16	files	file	NNS
en-wikipedia-org-3679	41	17	because	because	IN
en-wikipedia-org-3679	41	18	the	the	DT
en-wikipedia-org-3679	41	19	wildcard	wildcard	NN
en-wikipedia-org-3679	41	20	*	*	NFP
en-wikipedia-org-3679	41	21	stands	stand	VBZ
en-wikipedia-org-3679	41	22	for	for	IN
en-wikipedia-org-3679	41	23	all	all	DT
en-wikipedia-org-3679	41	24	robots	robot	NNS
en-wikipedia-org-3679	41	25	and	and	CC
en-wikipedia-org-3679	41	26	the	the	DT
en-wikipedia-org-3679	41	27	Disallow	Disallow	NNP
en-wikipedia-org-3679	41	28	directive	directive	NN
en-wikipedia-org-3679	41	29	has	have	VBZ
en-wikipedia-org-3679	41	30	no	no	DT
en-wikipedia-org-3679	41	31	value	value	NN
en-wikipedia-org-3679	41	32	,	,	,
en-wikipedia-org-3679	41	33	meaning	mean	VBG
en-wikipedia-org-3679	41	34	no	no	DT
en-wikipedia-org-3679	41	35	pages	page	NNS
en-wikipedia-org-3679	41	36	are	be	VBP
en-wikipedia-org-3679	41	37	disallowed	disallow	VBN
en-wikipedia-org-3679	41	38	.	.	.
en-wikipedia-org-3679	42	1	User	user	NN
en-wikipedia-org-3679	42	2	-	-	HYPH
en-wikipedia-org-3679	42	3	agent	agent	NN
en-wikipedia-org-3679	42	4	:	:	:
en-wikipedia-org-3679	42	5	*	*	NFP
en-wikipedia-org-3679	42	6	Allow	allow	VB
en-wikipedia-org-3679	42	7	:	:	:
en-wikipedia-org-3679	42	8	/	/	,
en-wikipedia-org-3679	42	9	The	the	DT
en-wikipedia-org-3679	42	10	same	same	JJ
en-wikipedia-org-3679	42	11	result	result	NN
en-wikipedia-org-3679	42	12	can	can	MD
en-wikipedia-org-3679	42	13	be	be	VB
en-wikipedia-org-3679	42	14	accomplished	accomplish	VBN
en-wikipedia-org-3679	42	15	with	with	IN
en-wikipedia-org-3679	42	16	an	an	DT
en-wikipedia-org-3679	42	17	empty	empty	JJ
en-wikipedia-org-3679	42	18	or	or	CC
en-wikipedia-org-3679	42	19	missing	missing	JJ
en-wikipedia-org-3679	42	20	robots.txt	robots.txt	JJR
en-wikipedia-org-3679	42	21	file	file	NN
en-wikipedia-org-3679	42	22	.	.	.
en-wikipedia-org-3679	43	1	This	this	DT
en-wikipedia-org-3679	43	2	example	example	NN
en-wikipedia-org-3679	43	3	tells	tell	VBZ
en-wikipedia-org-3679	43	4	all	all	DT
en-wikipedia-org-3679	43	5	robots	robot	NNS
en-wikipedia-org-3679	43	6	to	to	TO
en-wikipedia-org-3679	43	7	stay	stay	VB
en-wikipedia-org-3679	43	8	out	out	IN
en-wikipedia-org-3679	43	9	of	of	IN
en-wikipedia-org-3679	43	10	a	a	DT
en-wikipedia-org-3679	43	11	website	website	NN
en-wikipedia-org-3679	43	12	:	:	:
en-wikipedia-org-3679	43	13	User	user	NN
en-wikipedia-org-3679	43	14	-	-	HYPH
en-wikipedia-org-3679	43	15	agent	agent	NN
en-wikipedia-org-3679	43	16	:	:	:
en-wikipedia-org-3679	43	17	*	*	NFP
en-wikipedia-org-3679	43	18	Disallow	Disallow	NNP
en-wikipedia-org-3679	43	19	:	:	:
en-wikipedia-org-3679	43	20	/	/	NFP
en-wikipedia-org-3679	43	21	This	this	DT
en-wikipedia-org-3679	43	22	example	example	NN
en-wikipedia-org-3679	43	23	tells	tell	VBZ
en-wikipedia-org-3679	43	24	all	all	DT
en-wikipedia-org-3679	43	25	robots	robot	NNS
en-wikipedia-org-3679	43	26	not	not	RB
en-wikipedia-org-3679	43	27	to	to	TO
en-wikipedia-org-3679	43	28	enter	enter	VB
en-wikipedia-org-3679	43	29	three	three	CD
en-wikipedia-org-3679	43	30	directories	directory	NNS
en-wikipedia-org-3679	43	31	:	:	:
en-wikipedia-org-3679	43	32	User	user	NN
en-wikipedia-org-3679	43	33	-	-	HYPH
en-wikipedia-org-3679	43	34	agent	agent	NN
en-wikipedia-org-3679	43	35	:	:	:
en-wikipedia-org-3679	43	36	*	*	NFP
en-wikipedia-org-3679	43	37	Disallow	disallow	NN
en-wikipedia-org-3679	43	38	:	:	:
en-wikipedia-org-3679	43	39	/cgi	/cgi	NFP
en-wikipedia-org-3679	43	40	-	-	:
en-wikipedia-org-3679	43	41	bin/	bin/	NN
en-wikipedia-org-3679	43	42	Disallow	Disallow	NNP
en-wikipedia-org-3679	43	43	:	:	:
en-wikipedia-org-3679	43	44	/tmp/	/tmp/	NFP
en-wikipedia-org-3679	43	45	Disallow	disallow	NN
en-wikipedia-org-3679	43	46	:	:	:
en-wikipedia-org-3679	43	47	/junk/	/junk/	.
en-wikipedia-org-3679	43	48	This	this	DT
en-wikipedia-org-3679	43	49	example	example	NN
en-wikipedia-org-3679	43	50	tells	tell	VBZ
en-wikipedia-org-3679	43	51	all	all	DT
en-wikipedia-org-3679	43	52	robots	robot	NNS
en-wikipedia-org-3679	43	53	to	to	TO
en-wikipedia-org-3679	43	54	stay	stay	VB
en-wikipedia-org-3679	43	55	away	away	RB
en-wikipedia-org-3679	43	56	from	from	IN
en-wikipedia-org-3679	43	57	one	one	CD
en-wikipedia-org-3679	43	58	specific	specific	JJ
en-wikipedia-org-3679	43	59	file	file	NN
en-wikipedia-org-3679	43	60	:	:	:
en-wikipedia-org-3679	43	61	User	user	NN
en-wikipedia-org-3679	43	62	-	-	HYPH
en-wikipedia-org-3679	43	63	agent	agent	NN
en-wikipedia-org-3679	43	64	:	:	:
en-wikipedia-org-3679	43	65	*	*	NFP
en-wikipedia-org-3679	43	66	Disallow	Disallow	NNP
en-wikipedia-org-3679	43	67	:	:	:
en-wikipedia-org-3679	43	68	/directory	/directory	,
en-wikipedia-org-3679	43	69	/	/	SYM
en-wikipedia-org-3679	43	70	file.html	file.html	NNP
en-wikipedia-org-3679	43	71	All	all	DT
en-wikipedia-org-3679	43	72	other	other	JJ
en-wikipedia-org-3679	43	73	files	file	NNS
en-wikipedia-org-3679	43	74	in	in	IN
en-wikipedia-org-3679	43	75	the	the	DT
en-wikipedia-org-3679	43	76	specified	specify	VBN
en-wikipedia-org-3679	43	77	directory	directory	NN
en-wikipedia-org-3679	43	78	will	will	MD
en-wikipedia-org-3679	43	79	be	be	VB
en-wikipedia-org-3679	43	80	processed	process	VBN
en-wikipedia-org-3679	43	81	.	.	.
en-wikipedia-org-3679	44	1	This	this	DT
en-wikipedia-org-3679	44	2	example	example	NN
en-wikipedia-org-3679	44	3	tells	tell	VBZ
en-wikipedia-org-3679	44	4	a	a	DT
en-wikipedia-org-3679	44	5	specific	specific	JJ
en-wikipedia-org-3679	44	6	robot	robot	NN
en-wikipedia-org-3679	44	7	to	to	TO
en-wikipedia-org-3679	44	8	stay	stay	VB
en-wikipedia-org-3679	44	9	out	out	IN
en-wikipedia-org-3679	44	10	of	of	IN
en-wikipedia-org-3679	44	11	a	a	DT
en-wikipedia-org-3679	44	12	website	website	NN
en-wikipedia-org-3679	44	13	:	:	:
en-wikipedia-org-3679	44	14	User	user	NN
en-wikipedia-org-3679	44	15	-	-	HYPH
en-wikipedia-org-3679	44	16	agent	agent	NN
en-wikipedia-org-3679	44	17	:	:	:
en-wikipedia-org-3679	44	18	BadBot	badbot	VB
en-wikipedia-org-3679	44	19	#	#	NN
en-wikipedia-org-3679	44	20	replace	replace	VBP
en-wikipedia-org-3679	44	21	'	'	``
en-wikipedia-org-3679	44	22	BadBot	BadBot	NNP
en-wikipedia-org-3679	44	23	'	'	''
en-wikipedia-org-3679	44	24	with	with	IN
en-wikipedia-org-3679	44	25	the	the	DT
en-wikipedia-org-3679	44	26	actual	actual	JJ
en-wikipedia-org-3679	44	27	user	user	NN
en-wikipedia-org-3679	44	28	-	-	HYPH
en-wikipedia-org-3679	44	29	agent	agent	NN
en-wikipedia-org-3679	44	30	of	of	IN
en-wikipedia-org-3679	44	31	the	the	DT
en-wikipedia-org-3679	44	32	bot	bot	NN
en-wikipedia-org-3679	44	33	Disallow	Disallow	NNP
en-wikipedia-org-3679	44	34	:	:	:
en-wikipedia-org-3679	44	35	/	/	NFP
en-wikipedia-org-3679	44	36	This	this	DT
en-wikipedia-org-3679	44	37	example	example	NN
en-wikipedia-org-3679	44	38	tells	tell	VBZ
en-wikipedia-org-3679	44	39	two	two	CD
en-wikipedia-org-3679	44	40	specific	specific	JJ
en-wikipedia-org-3679	44	41	robots	robot	NNS
en-wikipedia-org-3679	44	42	not	not	RB
en-wikipedia-org-3679	44	43	to	to	TO
en-wikipedia-org-3679	44	44	enter	enter	VB
en-wikipedia-org-3679	44	45	one	one	CD
en-wikipedia-org-3679	44	46	specific	specific	JJ
en-wikipedia-org-3679	44	47	directory	directory	NN
en-wikipedia-org-3679	44	48	:	:	:
en-wikipedia-org-3679	44	49	User	user	NN
en-wikipedia-org-3679	44	50	-	-	HYPH
en-wikipedia-org-3679	44	51	agent	agent	NN
en-wikipedia-org-3679	44	52	:	:	:
en-wikipedia-org-3679	44	53	BadBot	badbot	VB
en-wikipedia-org-3679	44	54	#	#	NN
en-wikipedia-org-3679	44	55	replace	replace	VBP
en-wikipedia-org-3679	44	56	'	'	``
en-wikipedia-org-3679	44	57	BadBot	BadBot	NNP
en-wikipedia-org-3679	44	58	'	'	''
en-wikipedia-org-3679	44	59	with	with	IN
en-wikipedia-org-3679	44	60	the	the	DT
en-wikipedia-org-3679	44	61	actual	actual	JJ
en-wikipedia-org-3679	44	62	user	user	NN
en-wikipedia-org-3679	44	63	-	-	HYPH
en-wikipedia-org-3679	44	64	agent	agent	NN
en-wikipedia-org-3679	44	65	of	of	IN
en-wikipedia-org-3679	44	66	the	the	DT
en-wikipedia-org-3679	44	67	bot	bot	NN
en-wikipedia-org-3679	44	68	User	User	NNP
en-wikipedia-org-3679	44	69	-	-	HYPH
en-wikipedia-org-3679	44	70	agent	agent	NN
en-wikipedia-org-3679	44	71	:	:	:
en-wikipedia-org-3679	44	72	Googlebot	Googlebot	NNP
en-wikipedia-org-3679	44	73	Disallow	Disallow	NNP
en-wikipedia-org-3679	44	74	:	:	:
en-wikipedia-org-3679	44	75	/private/	/private/	NFP
en-wikipedia-org-3679	44	76	Example	example	NN
en-wikipedia-org-3679	44	77	demonstrating	demonstrate	VBG
en-wikipedia-org-3679	44	78	how	how	WRB
en-wikipedia-org-3679	44	79	comments	comment	NNS
en-wikipedia-org-3679	44	80	can	can	MD
en-wikipedia-org-3679	44	81	be	be	VB
en-wikipedia-org-3679	44	82	used	use	VBN
en-wikipedia-org-3679	44	83	:	:	:
en-wikipedia-org-3679	44	84	#	#	$
en-wikipedia-org-3679	44	85	Comments	comment	NNS
en-wikipedia-org-3679	44	86	appear	appear	VBP
en-wikipedia-org-3679	44	87	after	after	IN
en-wikipedia-org-3679	44	88	the	the	DT
en-wikipedia-org-3679	44	89	"	"	``
en-wikipedia-org-3679	44	90	#	#	NNP
en-wikipedia-org-3679	44	91	"	"	''
en-wikipedia-org-3679	44	92	symbol	symbol	NN
en-wikipedia-org-3679	44	93	at	at	IN
en-wikipedia-org-3679	44	94	the	the	DT
en-wikipedia-org-3679	44	95	start	start	NN
en-wikipedia-org-3679	44	96	of	of	IN
en-wikipedia-org-3679	44	97	a	a	DT
en-wikipedia-org-3679	44	98	line	line	NN
en-wikipedia-org-3679	44	99	,	,	,
en-wikipedia-org-3679	44	100	or	or	CC
en-wikipedia-org-3679	44	101	after	after	IN
en-wikipedia-org-3679	44	102	a	a	DT
en-wikipedia-org-3679	44	103	directive	directive	JJ
en-wikipedia-org-3679	44	104	User	User	NNP
en-wikipedia-org-3679	44	105	-	-	HYPH
en-wikipedia-org-3679	44	106	agent	agent	NN
en-wikipedia-org-3679	44	107	:	:	:
en-wikipedia-org-3679	44	108	*	*	NFP
en-wikipedia-org-3679	44	109	#	#	:
en-wikipedia-org-3679	44	110	match	match	VBP
en-wikipedia-org-3679	44	111	all	all	DT
en-wikipedia-org-3679	44	112	bots	bot	NNS
en-wikipedia-org-3679	44	113	Disallow	Disallow	NNP
en-wikipedia-org-3679	44	114	:	:	:
en-wikipedia-org-3679	44	115	/	/	SYM
en-wikipedia-org-3679	44	116	#	#	$
en-wikipedia-org-3679	44	117	keep	keep	VB
en-wikipedia-org-3679	44	118	them	-PRON-	PRP
en-wikipedia-org-3679	44	119	out	out	RP
en-wikipedia-org-3679	44	120	It	-PRON-	PRP
en-wikipedia-org-3679	44	121	is	be	VBZ
en-wikipedia-org-3679	44	122	also	also	RB
en-wikipedia-org-3679	44	123	possible	possible	JJ
en-wikipedia-org-3679	44	124	to	to	TO
en-wikipedia-org-3679	44	125	list	list	VB
en-wikipedia-org-3679	44	126	multiple	multiple	JJ
en-wikipedia-org-3679	44	127	robots	robot	NNS
en-wikipedia-org-3679	44	128	with	with	IN
en-wikipedia-org-3679	44	129	their	-PRON-	PRP$
en-wikipedia-org-3679	44	130	own	own	JJ
en-wikipedia-org-3679	44	131	rules	rule	NNS
en-wikipedia-org-3679	44	132	.	.	.
en-wikipedia-org-3679	45	1	The	the	DT
en-wikipedia-org-3679	45	2	actual	actual	JJ
en-wikipedia-org-3679	45	3	robot	robot	NN
en-wikipedia-org-3679	45	4	string	string	NN
en-wikipedia-org-3679	45	5	is	be	VBZ
en-wikipedia-org-3679	45	6	defined	define	VBN
en-wikipedia-org-3679	45	7	by	by	IN
en-wikipedia-org-3679	45	8	the	the	DT
en-wikipedia-org-3679	45	9	crawler	crawler	NN
en-wikipedia-org-3679	45	10	.	.	.
en-wikipedia-org-3679	46	1	A	a	DT
en-wikipedia-org-3679	46	2	few	few	JJ
en-wikipedia-org-3679	46	3	robot	robot	NN
en-wikipedia-org-3679	46	4	operators	operator	NNS
en-wikipedia-org-3679	46	5	,	,	,
en-wikipedia-org-3679	46	6	such	such	JJ
en-wikipedia-org-3679	46	7	as	as	IN
en-wikipedia-org-3679	46	8	Google	Google	NNP
en-wikipedia-org-3679	46	9	,	,	,
en-wikipedia-org-3679	46	10	support	support	VB
en-wikipedia-org-3679	46	11	several	several	JJ
en-wikipedia-org-3679	46	12	user	user	NN
en-wikipedia-org-3679	46	13	-	-	HYPH
en-wikipedia-org-3679	46	14	agent	agent	NN
en-wikipedia-org-3679	46	15	strings	string	NNS
en-wikipedia-org-3679	46	16	that	that	WDT
en-wikipedia-org-3679	46	17	allow	allow	VBP
en-wikipedia-org-3679	46	18	the	the	DT
en-wikipedia-org-3679	46	19	operator	operator	NN
en-wikipedia-org-3679	46	20	to	to	TO
en-wikipedia-org-3679	46	21	deny	deny	VB
en-wikipedia-org-3679	46	22	access	access	NN
en-wikipedia-org-3679	46	23	to	to	IN
en-wikipedia-org-3679	46	24	a	a	DT
en-wikipedia-org-3679	46	25	subset	subset	NN
en-wikipedia-org-3679	46	26	of	of	IN
en-wikipedia-org-3679	46	27	their	-PRON-	PRP$
en-wikipedia-org-3679	46	28	services	service	NNS
en-wikipedia-org-3679	46	29	by	by	IN
en-wikipedia-org-3679	46	30	using	use	VBG
en-wikipedia-org-3679	46	31	specific	specific	JJ
en-wikipedia-org-3679	46	32	user	user	NN
en-wikipedia-org-3679	46	33	-	-	HYPH
en-wikipedia-org-3679	46	34	agent	agent	NN
en-wikipedia-org-3679	46	35	strings	string	NNS
en-wikipedia-org-3679	46	36	.	.	.
en-wikipedia-org-3679	47	1	[	[	-LRB-
en-wikipedia-org-3679	47	2	14	14	CD
en-wikipedia-org-3679	47	3	]	]	-RRB-
en-wikipedia-org-3679	47	4	Example	example	NN
en-wikipedia-org-3679	47	5	demonstrating	demonstrate	VBG
en-wikipedia-org-3679	47	6	multiple	multiple	JJ
en-wikipedia-org-3679	47	7	user	user	NN
en-wikipedia-org-3679	47	8	-	-	HYPH
en-wikipedia-org-3679	47	9	agents	agent	NNS
en-wikipedia-org-3679	47	10	:	:	:
en-wikipedia-org-3679	47	11	User	user	NN
en-wikipedia-org-3679	47	12	-	-	HYPH
en-wikipedia-org-3679	47	13	agent	agent	NN
en-wikipedia-org-3679	47	14	:	:	:
en-wikipedia-org-3679	47	15	googlebot	googlebot	NNP
en-wikipedia-org-3679	47	16	#	#	NNP
en-wikipedia-org-3679	47	17	all	all	DT
en-wikipedia-org-3679	47	18	Google	Google	NNP
en-wikipedia-org-3679	47	19	services	services	NNPS
en-wikipedia-org-3679	47	20	Disallow	Disallow	NNP
en-wikipedia-org-3679	47	21	:	:	:
en-wikipedia-org-3679	47	22	/private/	/private/	VB
en-wikipedia-org-3679	47	23	#	#	$
en-wikipedia-org-3679	47	24	disallow	disallow	VBP
en-wikipedia-org-3679	47	25	this	this	DT
en-wikipedia-org-3679	47	26	directory	directory	NN
en-wikipedia-org-3679	47	27	User	user	NN
en-wikipedia-org-3679	47	28	-	-	HYPH
en-wikipedia-org-3679	47	29	agent	agent	NN
en-wikipedia-org-3679	47	30	:	:	:
en-wikipedia-org-3679	47	31	googlebot	googlebot	NNP
en-wikipedia-org-3679	47	32	-	-	HYPH
en-wikipedia-org-3679	47	33	news	news	NNP
en-wikipedia-org-3679	47	34	#	#	NN
en-wikipedia-org-3679	47	35	only	only	RB
en-wikipedia-org-3679	47	36	the	the	DT
en-wikipedia-org-3679	47	37	news	news	NN
en-wikipedia-org-3679	47	38	service	service	NN
en-wikipedia-org-3679	47	39	Disallow	Disallow	NNP
en-wikipedia-org-3679	47	40	:	:	:
en-wikipedia-org-3679	47	41	/	/	SYM
en-wikipedia-org-3679	47	42	#	#	$
en-wikipedia-org-3679	47	43	disallow	disallow	NN
en-wikipedia-org-3679	47	44	everything	everything	NN
en-wikipedia-org-3679	47	45	User	User	NNP
en-wikipedia-org-3679	47	46	-	-	HYPH
en-wikipedia-org-3679	47	47	agent	agent	NN
en-wikipedia-org-3679	47	48	:	:	:
en-wikipedia-org-3679	47	49	*	*	NFP
en-wikipedia-org-3679	47	50	#	#	VBD
en-wikipedia-org-3679	47	51	any	any	DT
en-wikipedia-org-3679	47	52	robot	robot	NN
en-wikipedia-org-3679	47	53	Disallow	Disallow	NNP
en-wikipedia-org-3679	47	54	:	:	:
en-wikipedia-org-3679	47	55	/something/	/something/	NN
en-wikipedia-org-3679	47	56	#	#	$
en-wikipedia-org-3679	47	57	disallow	disallow	NN
en-wikipedia-org-3679	47	58	this	this	DT
en-wikipedia-org-3679	47	59	directory	directory	NN
en-wikipedia-org-3679	47	60	Nonstandard	Nonstandard	NNP
en-wikipedia-org-3679	47	61	extensions[edit	extensions[edit	,
en-wikipedia-org-3679	47	62	]	]	-RRB-
en-wikipedia-org-3679	47	63	Crawl	crawl	NN
en-wikipedia-org-3679	47	64	-	-	HYPH
en-wikipedia-org-3679	47	65	delay	delay	NNP
en-wikipedia-org-3679	47	66	directive[edit	directive[edit	LS
en-wikipedia-org-3679	47	67	]	]	-RRB-
en-wikipedia-org-3679	47	68	The	the	DT
en-wikipedia-org-3679	47	69	crawl	crawl	NN
en-wikipedia-org-3679	47	70	-	-	HYPH
en-wikipedia-org-3679	47	71	delay	delay	NN
en-wikipedia-org-3679	47	72	value	value	NN
en-wikipedia-org-3679	47	73	is	be	VBZ
en-wikipedia-org-3679	47	74	supported	support	VBN
en-wikipedia-org-3679	47	75	by	by	IN
en-wikipedia-org-3679	47	76	some	some	DT
en-wikipedia-org-3679	47	77	crawlers	crawler	NNS
en-wikipedia-org-3679	47	78	to	to	TO
en-wikipedia-org-3679	47	79	throttle	throttle	VB
en-wikipedia-org-3679	47	80	their	-PRON-	PRP$
en-wikipedia-org-3679	47	81	visits	visit	NNS
en-wikipedia-org-3679	47	82	to	to	IN
en-wikipedia-org-3679	47	83	the	the	DT
en-wikipedia-org-3679	47	84	host	host	NN
en-wikipedia-org-3679	47	85	.	.	.
en-wikipedia-org-3679	48	1	Since	since	IN
en-wikipedia-org-3679	48	2	this	this	DT
en-wikipedia-org-3679	48	3	value	value	NN
en-wikipedia-org-3679	48	4	is	be	VBZ
en-wikipedia-org-3679	48	5	not	not	RB
en-wikipedia-org-3679	48	6	part	part	NN
en-wikipedia-org-3679	48	7	of	of	IN
en-wikipedia-org-3679	48	8	the	the	DT
en-wikipedia-org-3679	48	9	standard	standard	NN
en-wikipedia-org-3679	48	10	,	,	,
en-wikipedia-org-3679	48	11	its	-PRON-	PRP$
en-wikipedia-org-3679	48	12	interpretation	interpretation	NN
en-wikipedia-org-3679	48	13	is	be	VBZ
en-wikipedia-org-3679	48	14	dependent	dependent	JJ
en-wikipedia-org-3679	48	15	on	on	IN
en-wikipedia-org-3679	48	16	the	the	DT
en-wikipedia-org-3679	48	17	crawler	crawler	NN
en-wikipedia-org-3679	48	18	reading	read	VBG
en-wikipedia-org-3679	48	19	it	-PRON-	PRP
en-wikipedia-org-3679	48	20	.	.	.
en-wikipedia-org-3679	49	1	It	-PRON-	PRP
en-wikipedia-org-3679	49	2	is	be	VBZ
en-wikipedia-org-3679	49	3	used	use	VBN
en-wikipedia-org-3679	49	4	when	when	WRB
en-wikipedia-org-3679	49	5	the	the	DT
en-wikipedia-org-3679	49	6	multiple	multiple	JJ
en-wikipedia-org-3679	49	7	burst	burst	NN
en-wikipedia-org-3679	49	8	of	of	IN
en-wikipedia-org-3679	49	9	visits	visit	NNS
en-wikipedia-org-3679	49	10	from	from	IN
en-wikipedia-org-3679	49	11	bots	bot	NNS
en-wikipedia-org-3679	49	12	is	be	VBZ
en-wikipedia-org-3679	49	13	slowing	slow	VBG
en-wikipedia-org-3679	49	14	down	down	RP
en-wikipedia-org-3679	49	15	the	the	DT
en-wikipedia-org-3679	49	16	host	host	NN
en-wikipedia-org-3679	49	17	.	.	.
en-wikipedia-org-3679	50	1	Yandex	Yandex	NNP
en-wikipedia-org-3679	50	2	interprets	interpret	VBZ
en-wikipedia-org-3679	50	3	the	the	DT
en-wikipedia-org-3679	50	4	value	value	NN
en-wikipedia-org-3679	50	5	as	as	IN
en-wikipedia-org-3679	50	6	the	the	DT
en-wikipedia-org-3679	50	7	number	number	NN
en-wikipedia-org-3679	50	8	of	of	IN
en-wikipedia-org-3679	50	9	seconds	second	NNS
en-wikipedia-org-3679	50	10	to	to	TO
en-wikipedia-org-3679	50	11	wait	wait	VB
en-wikipedia-org-3679	50	12	between	between	IN
en-wikipedia-org-3679	50	13	subsequent	subsequent	JJ
en-wikipedia-org-3679	50	14	visits	visit	NNS
en-wikipedia-org-3679	50	15	.	.	.
en-wikipedia-org-3679	51	1	[	[	-LRB-
en-wikipedia-org-3679	51	2	16	16	CD
en-wikipedia-org-3679	51	3	]	]	-RRB-
en-wikipedia-org-3679	51	4	Bing	bing	NN
en-wikipedia-org-3679	51	5	defines	define	VBZ
en-wikipedia-org-3679	51	6	crawl	crawl	NN
en-wikipedia-org-3679	51	7	-	-	HYPH
en-wikipedia-org-3679	51	8	delay	delay	NN
en-wikipedia-org-3679	51	9	as	as	IN
en-wikipedia-org-3679	51	10	the	the	DT
en-wikipedia-org-3679	51	11	size	size	NN
en-wikipedia-org-3679	51	12	of	of	IN
en-wikipedia-org-3679	51	13	a	a	DT
en-wikipedia-org-3679	51	14	time	time	NN
en-wikipedia-org-3679	51	15	window	window	NN
en-wikipedia-org-3679	51	16	(	(	-LRB-
en-wikipedia-org-3679	51	17	from	from	IN
en-wikipedia-org-3679	51	18	1	1	CD
en-wikipedia-org-3679	51	19	to	to	TO
en-wikipedia-org-3679	51	20	30	30	CD
en-wikipedia-org-3679	51	21	seconds	second	NNS
en-wikipedia-org-3679	51	22	)	)	-RRB-
en-wikipedia-org-3679	51	23	during	during	IN
en-wikipedia-org-3679	51	24	which	which	WDT
en-wikipedia-org-3679	51	25	BingBot	BingBot	NNP
en-wikipedia-org-3679	51	26	will	will	MD
en-wikipedia-org-3679	51	27	access	access	VB
en-wikipedia-org-3679	51	28	a	a	DT
en-wikipedia-org-3679	51	29	web	web	NN
en-wikipedia-org-3679	51	30	site	site	NN
en-wikipedia-org-3679	51	31	only	only	RB
en-wikipedia-org-3679	51	32	once	once	RB
en-wikipedia-org-3679	51	33	.	.	.
en-wikipedia-org-3679	52	1	[	[	-LRB-
en-wikipedia-org-3679	52	2	33	33	CD
en-wikipedia-org-3679	52	3	]	]	-RRB-
en-wikipedia-org-3679	52	4	Google	Google	NNP
en-wikipedia-org-3679	52	5	provides	provide	VBZ
en-wikipedia-org-3679	52	6	an	an	DT
en-wikipedia-org-3679	52	7	interface	interface	NN
en-wikipedia-org-3679	52	8	in	in	IN
en-wikipedia-org-3679	52	9	its	-PRON-	PRP$
en-wikipedia-org-3679	52	10	search	search	NN
en-wikipedia-org-3679	52	11	console	console	NN
en-wikipedia-org-3679	52	12	for	for	IN
en-wikipedia-org-3679	52	13	webmasters	webmaster	NNS
en-wikipedia-org-3679	52	14	,	,	,
en-wikipedia-org-3679	52	15	to	to	TO
en-wikipedia-org-3679	52	16	control	control	VB
en-wikipedia-org-3679	52	17	the	the	DT
en-wikipedia-org-3679	52	18	GoogleBot	GoogleBot	NNP
en-wikipedia-org-3679	52	19	's	's	POS
en-wikipedia-org-3679	52	20	subsequent	subsequent	JJ
en-wikipedia-org-3679	52	21	visits	visit	NNS
en-wikipedia-org-3679	52	22	.	.	.
en-wikipedia-org-3679	53	1	[	[	-LRB-
en-wikipedia-org-3679	53	2	34	34	CD
en-wikipedia-org-3679	53	3	]	]	-RRB-
en-wikipedia-org-3679	53	4	User	user	NN
en-wikipedia-org-3679	53	5	-	-	HYPH
en-wikipedia-org-3679	53	6	agent	agent	NN
en-wikipedia-org-3679	53	7	:	:	:
en-wikipedia-org-3679	53	8	bingbot	bingbot	NNP
en-wikipedia-org-3679	53	9	Allow	Allow	NNP
en-wikipedia-org-3679	53	10	 	  	_SP
en-wikipedia-org-3679	53	11	:	:	:
en-wikipedia-org-3679	53	12	/	/	SYM
en-wikipedia-org-3679	53	13	Crawl	Crawl	NNP
en-wikipedia-org-3679	53	14	-	-	HYPH
en-wikipedia-org-3679	53	15	delay	delay	NNP
en-wikipedia-org-3679	53	16	:	:	:
en-wikipedia-org-3679	53	17	10	10	CD
en-wikipedia-org-3679	53	18	Allow	allow	VBP
en-wikipedia-org-3679	53	19	directive[edit	directive[edit	NFP
en-wikipedia-org-3679	53	20	]	]	-RRB-
en-wikipedia-org-3679	53	21	Some	some	DT
en-wikipedia-org-3679	53	22	major	major	JJ
en-wikipedia-org-3679	53	23	crawlers	crawler	NNS
en-wikipedia-org-3679	53	24	support	support	VBP
en-wikipedia-org-3679	53	25	an	an	DT
en-wikipedia-org-3679	53	26	Allow	Allow	NNP
en-wikipedia-org-3679	53	27	directive	directive	NN
en-wikipedia-org-3679	53	28	,	,	,
en-wikipedia-org-3679	53	29	which	which	WDT
en-wikipedia-org-3679	53	30	can	can	MD
en-wikipedia-org-3679	53	31	counteract	counteract	VB
en-wikipedia-org-3679	53	32	a	a	DT
en-wikipedia-org-3679	53	33	following	follow	VBG
en-wikipedia-org-3679	53	34	Disallow	Disallow	NNP
en-wikipedia-org-3679	53	35	directive	directive	NN
en-wikipedia-org-3679	53	36	.	.	.
en-wikipedia-org-3679	54	1	[	[	-LRB-
en-wikipedia-org-3679	54	2	35][36	35][36	CD
en-wikipedia-org-3679	54	3	]	]	-RRB-
en-wikipedia-org-3679	54	4	This	this	DT
en-wikipedia-org-3679	54	5	is	be	VBZ
en-wikipedia-org-3679	54	6	useful	useful	JJ
en-wikipedia-org-3679	54	7	when	when	WRB
en-wikipedia-org-3679	54	8	one	one	PRP
en-wikipedia-org-3679	54	9	tells	tell	VBZ
en-wikipedia-org-3679	54	10	robots	robot	NNS
en-wikipedia-org-3679	54	11	to	to	TO
en-wikipedia-org-3679	54	12	avoid	avoid	VB
en-wikipedia-org-3679	54	13	an	an	DT
en-wikipedia-org-3679	54	14	entire	entire	JJ
en-wikipedia-org-3679	54	15	directory	directory	NN
en-wikipedia-org-3679	54	16	but	but	CC
en-wikipedia-org-3679	54	17	still	still	RB
en-wikipedia-org-3679	54	18	wants	want	VBZ
en-wikipedia-org-3679	54	19	some	some	DT
en-wikipedia-org-3679	54	20	HTML	html	NN
en-wikipedia-org-3679	54	21	documents	document	NNS
en-wikipedia-org-3679	54	22	in	in	IN
en-wikipedia-org-3679	54	23	that	that	DT
en-wikipedia-org-3679	54	24	directory	directory	NN
en-wikipedia-org-3679	54	25	crawled	crawl	VBD
en-wikipedia-org-3679	54	26	and	and	CC
en-wikipedia-org-3679	54	27	indexed	index	VBD
en-wikipedia-org-3679	54	28	.	.	.
en-wikipedia-org-3679	55	1	While	while	IN
en-wikipedia-org-3679	55	2	by	by	IN
en-wikipedia-org-3679	55	3	standard	standard	JJ
en-wikipedia-org-3679	55	4	implementation	implementation	NN
en-wikipedia-org-3679	55	5	the	the	DT
en-wikipedia-org-3679	55	6	first	first	JJ
en-wikipedia-org-3679	55	7	matching	matching	JJ
en-wikipedia-org-3679	55	8	robots.txt	robots.txt	NN
en-wikipedia-org-3679	55	9	pattern	pattern	NN
en-wikipedia-org-3679	55	10	always	always	RB
en-wikipedia-org-3679	55	11	wins	win	VBZ
en-wikipedia-org-3679	55	12	,	,	,
en-wikipedia-org-3679	55	13	Google	Google	NNP
en-wikipedia-org-3679	55	14	's	's	POS
en-wikipedia-org-3679	55	15	implementation	implementation	NN
en-wikipedia-org-3679	55	16	differs	differ	NNS
en-wikipedia-org-3679	55	17	in	in	IN
en-wikipedia-org-3679	55	18	that	that	IN
en-wikipedia-org-3679	55	19	Allow	allow	JJ
en-wikipedia-org-3679	55	20	patterns	pattern	NNS
en-wikipedia-org-3679	55	21	with	with	IN
en-wikipedia-org-3679	55	22	equal	equal	JJ
en-wikipedia-org-3679	55	23	or	or	CC
en-wikipedia-org-3679	55	24	more	more	JJR
en-wikipedia-org-3679	55	25	characters	character	NNS
en-wikipedia-org-3679	55	26	in	in	IN
en-wikipedia-org-3679	55	27	the	the	DT
en-wikipedia-org-3679	55	28	directive	directive	JJ
en-wikipedia-org-3679	55	29	path	path	NN
en-wikipedia-org-3679	55	30	win	win	VBP
en-wikipedia-org-3679	55	31	over	over	IN
en-wikipedia-org-3679	55	32	a	a	DT
en-wikipedia-org-3679	55	33	matching	matching	JJ
en-wikipedia-org-3679	55	34	Disallow	Disallow	NNP
en-wikipedia-org-3679	55	35	pattern	pattern	NN
en-wikipedia-org-3679	55	36	.	.	.
en-wikipedia-org-3679	56	1	[	[	-LRB-
en-wikipedia-org-3679	56	2	37	37	CD
en-wikipedia-org-3679	56	3	]	]	-RRB-
en-wikipedia-org-3679	56	4	Bing	bing	NN
en-wikipedia-org-3679	56	5	uses	use	VBZ
en-wikipedia-org-3679	56	6	either	either	CC
en-wikipedia-org-3679	56	7	the	the	DT
en-wikipedia-org-3679	56	8	Allow	allow	JJ
en-wikipedia-org-3679	56	9	or	or	CC
en-wikipedia-org-3679	56	10	Disallow	Disallow	NNP
en-wikipedia-org-3679	56	11	directive	directive	NN
en-wikipedia-org-3679	56	12	,	,	,
en-wikipedia-org-3679	56	13	whichever	whichever	WDT
en-wikipedia-org-3679	56	14	is	be	VBZ
en-wikipedia-org-3679	56	15	more	more	RBR
en-wikipedia-org-3679	56	16	specific	specific	JJ
en-wikipedia-org-3679	56	17	,	,	,
en-wikipedia-org-3679	56	18	based	base	VBN
en-wikipedia-org-3679	56	19	on	on	IN
en-wikipedia-org-3679	56	20	length	length	NN
en-wikipedia-org-3679	56	21	,	,	,
en-wikipedia-org-3679	56	22	like	like	IN
en-wikipedia-org-3679	56	23	Google	Google	NNP
en-wikipedia-org-3679	56	24	.	.	.
en-wikipedia-org-3679	57	1	[	[	-LRB-
en-wikipedia-org-3679	57	2	17	17	CD
en-wikipedia-org-3679	57	3	]	]	-RRB-
en-wikipedia-org-3679	57	4	In	in	IN
en-wikipedia-org-3679	57	5	order	order	NN
en-wikipedia-org-3679	57	6	to	to	TO
en-wikipedia-org-3679	57	7	be	be	VB
en-wikipedia-org-3679	57	8	compatible	compatible	JJ
en-wikipedia-org-3679	57	9	to	to	IN
en-wikipedia-org-3679	57	10	all	all	DT
en-wikipedia-org-3679	57	11	robots	robot	NNS
en-wikipedia-org-3679	57	12	,	,	,
en-wikipedia-org-3679	57	13	if	if	IN
en-wikipedia-org-3679	57	14	one	one	PRP
en-wikipedia-org-3679	57	15	wants	want	VBZ
en-wikipedia-org-3679	57	16	to	to	TO
en-wikipedia-org-3679	57	17	allow	allow	VB
en-wikipedia-org-3679	57	18	single	single	JJ
en-wikipedia-org-3679	57	19	files	file	NNS
en-wikipedia-org-3679	57	20	inside	inside	IN
en-wikipedia-org-3679	57	21	an	an	DT
en-wikipedia-org-3679	57	22	otherwise	otherwise	RB
en-wikipedia-org-3679	57	23	disallowed	disallow	VBN
en-wikipedia-org-3679	57	24	directory	directory	NN
en-wikipedia-org-3679	57	25	,	,	,
en-wikipedia-org-3679	57	26	it	-PRON-	PRP
en-wikipedia-org-3679	57	27	is	be	VBZ
en-wikipedia-org-3679	57	28	necessary	necessary	JJ
en-wikipedia-org-3679	57	29	to	to	TO
en-wikipedia-org-3679	57	30	place	place	VB
en-wikipedia-org-3679	57	31	the	the	DT
en-wikipedia-org-3679	57	32	Allow	Allow	NNP
en-wikipedia-org-3679	57	33	directive(s	directive(s	NNP
en-wikipedia-org-3679	57	34	)	)	-RRB-
en-wikipedia-org-3679	57	35	first	first	RB
en-wikipedia-org-3679	57	36	,	,	,
en-wikipedia-org-3679	57	37	followed	follow	VBN
en-wikipedia-org-3679	57	38	by	by	IN
en-wikipedia-org-3679	57	39	the	the	DT
en-wikipedia-org-3679	57	40	Disallow	Disallow	NNP
en-wikipedia-org-3679	57	41	,	,	,
en-wikipedia-org-3679	57	42	for	for	IN
en-wikipedia-org-3679	57	43	example	example	NN
en-wikipedia-org-3679	57	44	:	:	:
en-wikipedia-org-3679	57	45	Allow	allow	VB
en-wikipedia-org-3679	57	46	:	:	:
en-wikipedia-org-3679	57	47	/directory1	/directory1	CD
en-wikipedia-org-3679	57	48	/	/	SYM
en-wikipedia-org-3679	57	49	myfile.html	myfile.html	NNP
en-wikipedia-org-3679	57	50	Disallow	Disallow	NNP
en-wikipedia-org-3679	57	51	:	:	:
en-wikipedia-org-3679	57	52	/directory1/	/directory1/	NFP
en-wikipedia-org-3679	57	53	This	this	DT
en-wikipedia-org-3679	57	54	example	example	NN
en-wikipedia-org-3679	57	55	will	will	MD
en-wikipedia-org-3679	57	56	Disallow	disallow	VB
en-wikipedia-org-3679	57	57	anything	anything	NN
en-wikipedia-org-3679	57	58	in	in	IN
en-wikipedia-org-3679	57	59	/directory1/	/directory1/	NN
en-wikipedia-org-3679	57	60	except	except	IN
en-wikipedia-org-3679	57	61	/directory1	/directory1	CD
en-wikipedia-org-3679	57	62	/	/	SYM
en-wikipedia-org-3679	57	63	myfile.html	myfile.html	NNS
en-wikipedia-org-3679	57	64	,	,	,
en-wikipedia-org-3679	57	65	since	since	IN
en-wikipedia-org-3679	57	66	the	the	DT
en-wikipedia-org-3679	57	67	latter	latter	NN
en-wikipedia-org-3679	57	68	will	will	MD
en-wikipedia-org-3679	57	69	match	match	VB
en-wikipedia-org-3679	57	70	first	first	RB
en-wikipedia-org-3679	57	71	.	.	.
en-wikipedia-org-3679	58	1	The	the	DT
en-wikipedia-org-3679	58	2	order	order	NN
en-wikipedia-org-3679	58	3	is	be	VBZ
en-wikipedia-org-3679	58	4	only	only	RB
en-wikipedia-org-3679	58	5	important	important	JJ
en-wikipedia-org-3679	58	6	to	to	IN
en-wikipedia-org-3679	58	7	robots	robot	NNS
en-wikipedia-org-3679	58	8	that	that	WDT
en-wikipedia-org-3679	58	9	follow	follow	VBP
en-wikipedia-org-3679	58	10	the	the	DT
en-wikipedia-org-3679	58	11	standard	standard	NN
en-wikipedia-org-3679	58	12	;	;	:
en-wikipedia-org-3679	58	13	in	in	IN
en-wikipedia-org-3679	58	14	the	the	DT
en-wikipedia-org-3679	58	15	case	case	NN
en-wikipedia-org-3679	58	16	of	of	IN
en-wikipedia-org-3679	58	17	the	the	DT
en-wikipedia-org-3679	58	18	Google	Google	NNP
en-wikipedia-org-3679	58	19	or	or	CC
en-wikipedia-org-3679	58	20	Bing	Bing	NNP
en-wikipedia-org-3679	58	21	bots	bot	NNS
en-wikipedia-org-3679	58	22	,	,	,
en-wikipedia-org-3679	58	23	the	the	DT
en-wikipedia-org-3679	58	24	order	order	NN
en-wikipedia-org-3679	58	25	is	be	VBZ
en-wikipedia-org-3679	58	26	not	not	RB
en-wikipedia-org-3679	58	27	important	important	JJ
en-wikipedia-org-3679	58	28	.	.	.
en-wikipedia-org-3679	59	1	Sitemap[edit	sitemap[edit	LS
en-wikipedia-org-3679	59	2	]	]	-RRB-
en-wikipedia-org-3679	59	3	Some	some	DT
en-wikipedia-org-3679	59	4	crawlers	crawler	NNS
en-wikipedia-org-3679	59	5	support	support	VBP
en-wikipedia-org-3679	59	6	a	a	DT
en-wikipedia-org-3679	59	7	Sitemap	Sitemap	NNP
en-wikipedia-org-3679	59	8	directive	directive	NN
en-wikipedia-org-3679	59	9	,	,	,
en-wikipedia-org-3679	59	10	allowing	allow	VBG
en-wikipedia-org-3679	59	11	multiple	multiple	JJ
en-wikipedia-org-3679	59	12	Sitemaps	Sitemaps	NNP
en-wikipedia-org-3679	59	13	in	in	IN
en-wikipedia-org-3679	59	14	the	the	DT
en-wikipedia-org-3679	59	15	same	same	JJ
en-wikipedia-org-3679	59	16	robots.txt	robots.txt	NN
en-wikipedia-org-3679	59	17	in	in	IN
en-wikipedia-org-3679	59	18	the	the	DT
en-wikipedia-org-3679	59	19	form	form	NN
en-wikipedia-org-3679	59	20	Sitemap	Sitemap	NNP
en-wikipedia-org-3679	59	21	:	:	:
en-wikipedia-org-3679	59	22	full	full	JJ
en-wikipedia-org-3679	59	23	-	-	HYPH
en-wikipedia-org-3679	59	24	url:[38	url:[38	NN
en-wikipedia-org-3679	59	25	]	]	-RRB-
en-wikipedia-org-3679	59	26	Sitemap	Sitemap	NNP
en-wikipedia-org-3679	59	27	:	:	:
en-wikipedia-org-3679	59	28	http://www.example.com/sitemap.xml	http://www.example.com/sitemap.xml	NNP
en-wikipedia-org-3679	59	29	Host[edit	host[edit	FW
en-wikipedia-org-3679	59	30	]	]	-RRB-
en-wikipedia-org-3679	59	31	Some	some	DT
en-wikipedia-org-3679	59	32	crawlers	crawler	NNS
en-wikipedia-org-3679	59	33	(	(	-LRB-
en-wikipedia-org-3679	59	34	Yandex	Yandex	NNP
en-wikipedia-org-3679	59	35	)	)	-RRB-
en-wikipedia-org-3679	59	36	support	support	VBP
en-wikipedia-org-3679	59	37	a	a	DT
en-wikipedia-org-3679	59	38	Host	host	NN
en-wikipedia-org-3679	59	39	directive	directive	NN
en-wikipedia-org-3679	59	40	,	,	,
en-wikipedia-org-3679	59	41	allowing	allow	VBG
en-wikipedia-org-3679	59	42	websites	website	NNS
en-wikipedia-org-3679	59	43	with	with	IN
en-wikipedia-org-3679	59	44	multiple	multiple	JJ
en-wikipedia-org-3679	59	45	mirrors	mirror	NNS
en-wikipedia-org-3679	59	46	to	to	TO
en-wikipedia-org-3679	59	47	specify	specify	VB
en-wikipedia-org-3679	59	48	their	-PRON-	PRP$
en-wikipedia-org-3679	59	49	preferred	prefer	VBN
en-wikipedia-org-3679	59	50	domain:[39	domain:[39	NNP
en-wikipedia-org-3679	59	51	]	]	-RRB-
en-wikipedia-org-3679	59	52	Host	host	NN
en-wikipedia-org-3679	59	53	:	:	:
en-wikipedia-org-3679	59	54	hosting.example.com	hosting.example.com	NNP
en-wikipedia-org-3679	59	55	This	this	DT
en-wikipedia-org-3679	59	56	is	be	VBZ
en-wikipedia-org-3679	59	57	not	not	RB
en-wikipedia-org-3679	59	58	supported	support	VBN
en-wikipedia-org-3679	59	59	by	by	IN
en-wikipedia-org-3679	59	60	all	all	DT
en-wikipedia-org-3679	59	61	crawlers	crawler	NNS
en-wikipedia-org-3679	59	62	.	.	.
en-wikipedia-org-3679	60	1	Universal	Universal	NNP
en-wikipedia-org-3679	60	2	"	"	``
en-wikipedia-org-3679	60	3	*	*	NN
en-wikipedia-org-3679	60	4	"	"	''
en-wikipedia-org-3679	60	5	match[edit	match[edit	NNS
en-wikipedia-org-3679	60	6	]	]	-RRB-
en-wikipedia-org-3679	60	7	The	the	DT
en-wikipedia-org-3679	60	8	Robot	Robot	NNP
en-wikipedia-org-3679	60	9	Exclusion	Exclusion	NNP
en-wikipedia-org-3679	60	10	Standard	Standard	NNP
en-wikipedia-org-3679	60	11	does	do	VBZ
en-wikipedia-org-3679	60	12	not	not	RB
en-wikipedia-org-3679	60	13	mention	mention	VB
en-wikipedia-org-3679	60	14	the	the	DT
en-wikipedia-org-3679	60	15	"	"	``
en-wikipedia-org-3679	60	16	*	*	NN
en-wikipedia-org-3679	60	17	"	"	''
en-wikipedia-org-3679	60	18	character	character	NN
en-wikipedia-org-3679	60	19	in	in	IN
en-wikipedia-org-3679	60	20	the	the	DT
en-wikipedia-org-3679	60	21	Disallow	Disallow	NNP
en-wikipedia-org-3679	60	22	:	:	:
en-wikipedia-org-3679	60	23	statement	statement	NN
en-wikipedia-org-3679	60	24	.	.	.
en-wikipedia-org-3679	61	1	[	[	-LRB-
en-wikipedia-org-3679	61	2	40	40	CD
en-wikipedia-org-3679	61	3	]	]	-RRB-
en-wikipedia-org-3679	61	4	Meta	meta	JJ
en-wikipedia-org-3679	61	5	tags	tag	NNS
en-wikipedia-org-3679	61	6	and	and	CC
en-wikipedia-org-3679	61	7	headers[edit	headers[edit	LS
en-wikipedia-org-3679	61	8	]	]	-RRB-
en-wikipedia-org-3679	61	9	In	in	IN
en-wikipedia-org-3679	61	10	addition	addition	NN
en-wikipedia-org-3679	61	11	to	to	IN
en-wikipedia-org-3679	61	12	root	root	NN
en-wikipedia-org-3679	61	13	-	-	HYPH
en-wikipedia-org-3679	61	14	level	level	NN
en-wikipedia-org-3679	61	15	robots.txt	robots.txt	NN
en-wikipedia-org-3679	61	16	files	file	NNS
en-wikipedia-org-3679	61	17	,	,	,
en-wikipedia-org-3679	61	18	robots	robot	NNS
en-wikipedia-org-3679	61	19	exclusion	exclusion	NN
en-wikipedia-org-3679	61	20	directives	directive	NNS
en-wikipedia-org-3679	61	21	can	can	MD
en-wikipedia-org-3679	61	22	be	be	VB
en-wikipedia-org-3679	61	23	applied	apply	VBN
en-wikipedia-org-3679	61	24	at	at	IN
en-wikipedia-org-3679	61	25	a	a	DT
en-wikipedia-org-3679	61	26	more	more	RBR
en-wikipedia-org-3679	61	27	granular	granular	JJ
en-wikipedia-org-3679	61	28	level	level	NN
en-wikipedia-org-3679	61	29	through	through	IN
en-wikipedia-org-3679	61	30	the	the	DT
en-wikipedia-org-3679	61	31	use	use	NN
en-wikipedia-org-3679	61	32	of	of	IN
en-wikipedia-org-3679	61	33	Robots	robots	JJ
en-wikipedia-org-3679	61	34	meta	meta	JJ
en-wikipedia-org-3679	61	35	tags	tag	NNS
en-wikipedia-org-3679	61	36	and	and	CC
en-wikipedia-org-3679	61	37	X	X	NNS
en-wikipedia-org-3679	61	38	-	-	HYPH
en-wikipedia-org-3679	61	39	Robots	robots	JJ
en-wikipedia-org-3679	61	40	-	-	HYPH
en-wikipedia-org-3679	61	41	Tag	tag	NN
en-wikipedia-org-3679	61	42	HTTP	HTTP	NNP
en-wikipedia-org-3679	61	43	headers	header	NNS
en-wikipedia-org-3679	61	44	.	.	.
en-wikipedia-org-3679	62	1	The	the	DT
en-wikipedia-org-3679	62	2	robots	robot	NNS
en-wikipedia-org-3679	62	3	meta	meta	JJ
en-wikipedia-org-3679	62	4	tag	tag	NN
en-wikipedia-org-3679	62	5	can	can	MD
en-wikipedia-org-3679	62	6	not	not	RB
en-wikipedia-org-3679	62	7	be	be	VB
en-wikipedia-org-3679	62	8	used	use	VBN
en-wikipedia-org-3679	62	9	for	for	IN
en-wikipedia-org-3679	62	10	non	non	JJ
en-wikipedia-org-3679	62	11	-	-	JJ
en-wikipedia-org-3679	62	12	HTML	html	JJ
en-wikipedia-org-3679	62	13	files	file	NNS
en-wikipedia-org-3679	62	14	such	such	JJ
en-wikipedia-org-3679	62	15	as	as	IN
en-wikipedia-org-3679	62	16	images	image	NNS
en-wikipedia-org-3679	62	17	,	,	,
en-wikipedia-org-3679	62	18	text	text	NN
en-wikipedia-org-3679	62	19	files	file	NNS
en-wikipedia-org-3679	62	20	,	,	,
en-wikipedia-org-3679	62	21	or	or	CC
en-wikipedia-org-3679	62	22	PDF	PDF	NNP
en-wikipedia-org-3679	62	23	documents	document	NNS
en-wikipedia-org-3679	62	24	.	.	.
en-wikipedia-org-3679	63	1	On	on	IN
en-wikipedia-org-3679	63	2	the	the	DT
en-wikipedia-org-3679	63	3	other	other	JJ
en-wikipedia-org-3679	63	4	hand	hand	NN
en-wikipedia-org-3679	63	5	,	,	,
en-wikipedia-org-3679	63	6	the	the	DT
en-wikipedia-org-3679	63	7	X	X	NNP
en-wikipedia-org-3679	63	8	-	-	HYPH
en-wikipedia-org-3679	63	9	Robots	robots	NN
en-wikipedia-org-3679	63	10	-	-	HYPH
en-wikipedia-org-3679	63	11	Tag	tag	NN
en-wikipedia-org-3679	63	12	can	can	MD
en-wikipedia-org-3679	63	13	be	be	VB
en-wikipedia-org-3679	63	14	added	add	VBN
en-wikipedia-org-3679	63	15	to	to	IN
en-wikipedia-org-3679	63	16	non	non	JJ
en-wikipedia-org-3679	63	17	-	-	JJ
en-wikipedia-org-3679	63	18	HTML	html	JJ
en-wikipedia-org-3679	63	19	files	file	NNS
en-wikipedia-org-3679	63	20	by	by	IN
en-wikipedia-org-3679	63	21	using	use	VBG
en-wikipedia-org-3679	63	22	.htaccess	.htaccess	NN
en-wikipedia-org-3679	63	23	and	and	CC
en-wikipedia-org-3679	63	24	httpd.conf	httpd.conf	JJ
en-wikipedia-org-3679	63	25	files	file	NNS
en-wikipedia-org-3679	63	26	.	.	.
en-wikipedia-org-3679	64	1	[	[	-LRB-
en-wikipedia-org-3679	64	2	41	41	CD
en-wikipedia-org-3679	64	3	]	]	-RRB-
en-wikipedia-org-3679	64	4	A	A	NNP
en-wikipedia-org-3679	64	5	"	"	``
en-wikipedia-org-3679	64	6	noindex	noindex	NN
en-wikipedia-org-3679	64	7	"	"	''
en-wikipedia-org-3679	64	8	meta	meta	JJ
en-wikipedia-org-3679	64	9	tag	tag	NN
en-wikipedia-org-3679	64	10	<	<	XX
en-wikipedia-org-3679	64	11	meta	meta	JJ
en-wikipedia-org-3679	64	12	name="robots	name="robot	NNS
en-wikipedia-org-3679	64	13	"	"	``
en-wikipedia-org-3679	64	14	content="noindex	content="noindex	JJ
en-wikipedia-org-3679	64	15	"	"	''
en-wikipedia-org-3679	64	16	/	/	,
en-wikipedia-org-3679	64	17	>	>	FW
en-wikipedia-org-3679	64	18	A	a	DT
en-wikipedia-org-3679	64	19	"	"	``
en-wikipedia-org-3679	64	20	noindex	noindex	NN
en-wikipedia-org-3679	64	21	"	"	''
en-wikipedia-org-3679	64	22	HTTP	http	JJ
en-wikipedia-org-3679	64	23	response	response	NN
en-wikipedia-org-3679	64	24	header	header	NN
en-wikipedia-org-3679	64	25	X	X	NNS
en-wikipedia-org-3679	64	26	-	-	NNS
en-wikipedia-org-3679	64	27	Robots	robots	JJ
en-wikipedia-org-3679	64	28	-	-	HYPH
en-wikipedia-org-3679	64	29	Tag	tag	NN
en-wikipedia-org-3679	64	30	:	:	:
en-wikipedia-org-3679	64	31	noindex	noindex	NN
en-wikipedia-org-3679	64	32	The	the	DT
en-wikipedia-org-3679	64	33	X	X	NNP
en-wikipedia-org-3679	64	34	-	-	HYPH
en-wikipedia-org-3679	64	35	Robots	robots	NN
en-wikipedia-org-3679	64	36	-	-	HYPH
en-wikipedia-org-3679	64	37	Tag	tag	NN
en-wikipedia-org-3679	64	38	is	be	VBZ
en-wikipedia-org-3679	64	39	only	only	RB
en-wikipedia-org-3679	64	40	effective	effective	JJ
en-wikipedia-org-3679	64	41	after	after	IN
en-wikipedia-org-3679	64	42	the	the	DT
en-wikipedia-org-3679	64	43	page	page	NN
en-wikipedia-org-3679	64	44	has	have	VBZ
en-wikipedia-org-3679	64	45	been	be	VBN
en-wikipedia-org-3679	64	46	requested	request	VBN
en-wikipedia-org-3679	64	47	and	and	CC
en-wikipedia-org-3679	64	48	the	the	DT
en-wikipedia-org-3679	64	49	server	server	NN
en-wikipedia-org-3679	64	50	responds	respond	VBZ
en-wikipedia-org-3679	64	51	,	,	,
en-wikipedia-org-3679	64	52	and	and	CC
en-wikipedia-org-3679	64	53	the	the	DT
en-wikipedia-org-3679	64	54	robots	robot	NNS
en-wikipedia-org-3679	64	55	meta	meta	JJ
en-wikipedia-org-3679	64	56	tag	tag	NN
en-wikipedia-org-3679	64	57	is	be	VBZ
en-wikipedia-org-3679	64	58	only	only	RB
en-wikipedia-org-3679	64	59	effective	effective	JJ
en-wikipedia-org-3679	64	60	after	after	IN
en-wikipedia-org-3679	64	61	the	the	DT
en-wikipedia-org-3679	64	62	page	page	NN
en-wikipedia-org-3679	64	63	has	have	VBZ
en-wikipedia-org-3679	64	64	loaded	load	VBN
en-wikipedia-org-3679	64	65	,	,	,
en-wikipedia-org-3679	64	66	whereas	whereas	IN
en-wikipedia-org-3679	64	67	robots.txt	robots.txt	NN
en-wikipedia-org-3679	64	68	is	be	VBZ
en-wikipedia-org-3679	64	69	effective	effective	JJ
en-wikipedia-org-3679	64	70	before	before	IN
en-wikipedia-org-3679	64	71	the	the	DT
en-wikipedia-org-3679	64	72	page	page	NN
en-wikipedia-org-3679	64	73	is	be	VBZ
en-wikipedia-org-3679	64	74	requested	request	VBN
en-wikipedia-org-3679	64	75	.	.	.
en-wikipedia-org-3679	65	1	Thus	thus	RB
en-wikipedia-org-3679	65	2	if	if	IN
en-wikipedia-org-3679	65	3	a	a	DT
en-wikipedia-org-3679	65	4	page	page	NN
en-wikipedia-org-3679	65	5	is	be	VBZ
en-wikipedia-org-3679	65	6	excluded	exclude	VBN
en-wikipedia-org-3679	65	7	by	by	IN
en-wikipedia-org-3679	65	8	a	a	DT
en-wikipedia-org-3679	65	9	robots.txt	robots.txt	NN
en-wikipedia-org-3679	65	10	file	file	NN
en-wikipedia-org-3679	65	11	,	,	,
en-wikipedia-org-3679	65	12	any	any	DT
en-wikipedia-org-3679	65	13	robots	robot	NNS
en-wikipedia-org-3679	65	14	meta	meta	JJ
en-wikipedia-org-3679	65	15	tags	tag	NNS
en-wikipedia-org-3679	65	16	or	or	CC
en-wikipedia-org-3679	65	17	X	X	NNS
en-wikipedia-org-3679	65	18	-	-	HYPH
en-wikipedia-org-3679	65	19	Robots	robots	JJ
en-wikipedia-org-3679	65	20	-	-	HYPH
en-wikipedia-org-3679	65	21	Tag	tag	NN
en-wikipedia-org-3679	65	22	headers	header	NNS
en-wikipedia-org-3679	65	23	are	be	VBP
en-wikipedia-org-3679	65	24	effectively	effectively	RB
en-wikipedia-org-3679	65	25	ignored	ignore	VBN
en-wikipedia-org-3679	65	26	because	because	IN
en-wikipedia-org-3679	65	27	the	the	DT
en-wikipedia-org-3679	65	28	robot	robot	NN
en-wikipedia-org-3679	65	29	will	will	MD
en-wikipedia-org-3679	65	30	not	not	RB
en-wikipedia-org-3679	65	31	see	see	VB
en-wikipedia-org-3679	65	32	them	-PRON-	PRP
en-wikipedia-org-3679	65	33	in	in	IN
en-wikipedia-org-3679	65	34	the	the	DT
en-wikipedia-org-3679	65	35	first	first	JJ
en-wikipedia-org-3679	65	36	place	place	NN
en-wikipedia-org-3679	65	37	.	.	.
en-wikipedia-org-3679	66	1	[	[	-LRB-
en-wikipedia-org-3679	66	2	41	41	CD
en-wikipedia-org-3679	66	3	]	]	-RRB-
en-wikipedia-org-3679	66	4	See	see	VB
en-wikipedia-org-3679	66	5	also[edit	also[edit	,
en-wikipedia-org-3679	66	6	]	]	-RRB-
en-wikipedia-org-3679	66	7	ads.txt	ads.txt	CD
en-wikipedia-org-3679	66	8	,	,	,
en-wikipedia-org-3679	66	9	a	a	DT
en-wikipedia-org-3679	66	10	standard	standard	NN
en-wikipedia-org-3679	66	11	for	for	IN
en-wikipedia-org-3679	66	12	listing	list	VBG
en-wikipedia-org-3679	66	13	authorized	authorize	VBN
en-wikipedia-org-3679	66	14	ad	ad	NN
en-wikipedia-org-3679	66	15	sellers	seller	NNS
en-wikipedia-org-3679	66	16	security.txt	security.txt	NN
en-wikipedia-org-3679	66	17	,	,	,
en-wikipedia-org-3679	66	18	a	a	DT
en-wikipedia-org-3679	66	19	file	file	NN
en-wikipedia-org-3679	66	20	to	to	TO
en-wikipedia-org-3679	66	21	describe	describe	VB
en-wikipedia-org-3679	66	22	the	the	DT
en-wikipedia-org-3679	66	23	process	process	NN
en-wikipedia-org-3679	66	24	for	for	IN
en-wikipedia-org-3679	66	25	security	security	NN
en-wikipedia-org-3679	66	26	researchers	researcher	NNS
en-wikipedia-org-3679	66	27	to	to	TO
en-wikipedia-org-3679	66	28	follow	follow	VB
en-wikipedia-org-3679	66	29	in	in	IN
en-wikipedia-org-3679	66	30	order	order	NN
en-wikipedia-org-3679	66	31	to	to	TO
en-wikipedia-org-3679	66	32	report	report	VB
en-wikipedia-org-3679	66	33	security	security	NN
en-wikipedia-org-3679	66	34	vulnerabilities	vulnerability	NNS
en-wikipedia-org-3679	66	35	Automated	Automated	NNP
en-wikipedia-org-3679	66	36	Content	Content	NNP
en-wikipedia-org-3679	66	37	Access	Access	NNP
en-wikipedia-org-3679	66	38	Protocol	Protocol	NNP
en-wikipedia-org-3679	66	39	–	–	:
en-wikipedia-org-3679	66	40	a	a	DT
en-wikipedia-org-3679	66	41	failed	fail	VBN
en-wikipedia-org-3679	66	42	proposal	proposal	NN
en-wikipedia-org-3679	66	43	to	to	TO
en-wikipedia-org-3679	66	44	extend	extend	VB
en-wikipedia-org-3679	66	45	robots.txt	robots.txt	JJR
en-wikipedia-org-3679	66	46	BotSeer	BotSeer	NNP
en-wikipedia-org-3679	66	47	–	–	:
en-wikipedia-org-3679	66	48	now	now	RB
en-wikipedia-org-3679	66	49	inactive	inactive	JJ
en-wikipedia-org-3679	66	50	search	search	NN
en-wikipedia-org-3679	66	51	engine	engine	NN
en-wikipedia-org-3679	66	52	for	for	IN
en-wikipedia-org-3679	66	53	robots.txt	robots.txt	CD
en-wikipedia-org-3679	66	54	files	file	NNS
en-wikipedia-org-3679	66	55	Distributed	distribute	VBN
en-wikipedia-org-3679	66	56	web	web	NN
en-wikipedia-org-3679	66	57	crawling	crawl	VBG
en-wikipedia-org-3679	66	58	Focused	focused	JJ
en-wikipedia-org-3679	66	59	crawler	crawler	NN
en-wikipedia-org-3679	66	60	Internet	Internet	NNP
en-wikipedia-org-3679	66	61	Archive	Archive	NNP
en-wikipedia-org-3679	66	62	National	National	NNP
en-wikipedia-org-3679	66	63	Digital	Digital	NNP
en-wikipedia-org-3679	66	64	Library	Library	NNP
en-wikipedia-org-3679	66	65	Program	Program	NNP
en-wikipedia-org-3679	66	66	(	(	-LRB-
en-wikipedia-org-3679	66	67	NDLP	NDLP	NNP
en-wikipedia-org-3679	66	68	)	)	-RRB-
en-wikipedia-org-3679	66	69	National	National	NNP
en-wikipedia-org-3679	66	70	Digital	Digital	NNP
en-wikipedia-org-3679	66	71	Information	Information	NNP
en-wikipedia-org-3679	66	72	Infrastructure	Infrastructure	NNP
en-wikipedia-org-3679	66	73	and	and	CC
en-wikipedia-org-3679	66	74	Preservation	Preservation	NNP
en-wikipedia-org-3679	66	75	Program	Program	NNP
en-wikipedia-org-3679	66	76	(	(	-LRB-
en-wikipedia-org-3679	66	77	NDIIPP	NDIIPP	NNP
en-wikipedia-org-3679	66	78	)	)	-RRB-
en-wikipedia-org-3679	66	79	Nofollow	Nofollow	NNP
en-wikipedia-org-3679	66	80	Perma.cc	perma.cc	PRP$
en-wikipedia-org-3679	66	81	Meta	meta	JJ
en-wikipedia-org-3679	66	82	elements	element	NNS
en-wikipedia-org-3679	66	83	for	for	IN
en-wikipedia-org-3679	66	84	search	search	NN
en-wikipedia-org-3679	66	85	engines	engine	NNS
en-wikipedia-org-3679	66	86	Sitemaps	Sitemaps	NNP
en-wikipedia-org-3679	66	87	Spider	Spider	NNP
en-wikipedia-org-3679	66	88	trap	trap	NN
en-wikipedia-org-3679	66	89	Web	web	NN
en-wikipedia-org-3679	66	90	archiving	archive	VBG
en-wikipedia-org-3679	66	91	Web	web	NN
en-wikipedia-org-3679	66	92	crawler	crawler	NNP
en-wikipedia-org-3679	66	93	noindex	noindex	NNP
en-wikipedia-org-3679	66	94	References[edit	references[edit	NN
en-wikipedia-org-3679	66	95	]	]	-RRB-
en-wikipedia-org-3679	66	96	^	^	FW
en-wikipedia-org-3679	66	97	"	"	``
en-wikipedia-org-3679	66	98	Historical	Historical	NNP
en-wikipedia-org-3679	66	99	"	"	''
en-wikipedia-org-3679	66	100	.	.	.
en-wikipedia-org-3679	67	1	Greenhills.co.uk	Greenhills.co.uk	NNP
en-wikipedia-org-3679	67	2	.	.	.
en-wikipedia-org-3679	68	1	Archived	archived	JJ
en-wikipedia-org-3679	68	2	from	from	IN
en-wikipedia-org-3679	68	3	the	the	DT
en-wikipedia-org-3679	68	4	original	original	NN
en-wikipedia-org-3679	68	5	on	on	IN
en-wikipedia-org-3679	68	6	2017	2017	CD
en-wikipedia-org-3679	68	7	-	-	HYPH
en-wikipedia-org-3679	68	8	04	04	CD
en-wikipedia-org-3679	68	9	-	-	HYPH
en-wikipedia-org-3679	68	10	03	03	CD
en-wikipedia-org-3679	68	11	.	.	.
en-wikipedia-org-3679	69	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	69	2	2017	2017	CD
en-wikipedia-org-3679	69	3	-	-	HYPH
en-wikipedia-org-3679	69	4	03	03	CD
en-wikipedia-org-3679	69	5	-	-	HYPH
en-wikipedia-org-3679	69	6	03	03	CD
en-wikipedia-org-3679	69	7	.	.	.
en-wikipedia-org-3679	70	1	^	^	NNP
en-wikipedia-org-3679	70	2	Fielding	fielding	NN
en-wikipedia-org-3679	70	3	,	,	,
en-wikipedia-org-3679	70	4	Roy	Roy	NNP
en-wikipedia-org-3679	70	5	(	(	-LRB-
en-wikipedia-org-3679	70	6	1994	1994	CD
en-wikipedia-org-3679	70	7	)	)	-RRB-
en-wikipedia-org-3679	70	8	.	.	.
en-wikipedia-org-3679	71	1	"	"	``
en-wikipedia-org-3679	71	2	Maintaining	maintain	VBG
en-wikipedia-org-3679	71	3	Distributed	distribute	VBN
en-wikipedia-org-3679	71	4	Hypertext	Hypertext	NNP
en-wikipedia-org-3679	71	5	Infostructures	infostructure	NNS
en-wikipedia-org-3679	71	6	:	:	:
en-wikipedia-org-3679	71	7	Welcome	welcome	VBP
en-wikipedia-org-3679	71	8	to	to	IN
en-wikipedia-org-3679	71	9	MOMspider	MOMspider	NNP
en-wikipedia-org-3679	71	10	's	's	POS
en-wikipedia-org-3679	71	11	Web	web	NN
en-wikipedia-org-3679	71	12	"	"	''
en-wikipedia-org-3679	71	13	(	(	-LRB-
en-wikipedia-org-3679	71	14	PostScript	PostScript	NNP
en-wikipedia-org-3679	71	15	)	)	-RRB-
en-wikipedia-org-3679	71	16	.	.	.
en-wikipedia-org-3679	72	1	First	First	NNP
en-wikipedia-org-3679	72	2	International	International	NNP
en-wikipedia-org-3679	72	3	Conference	Conference	NNP
en-wikipedia-org-3679	72	4	on	on	IN
en-wikipedia-org-3679	72	5	the	the	DT
en-wikipedia-org-3679	72	6	World	World	NNP
en-wikipedia-org-3679	72	7	Wide	Wide	NNP
en-wikipedia-org-3679	72	8	Web	Web	NNP
en-wikipedia-org-3679	72	9	.	.	.
en-wikipedia-org-3679	73	1	Geneva	Geneva	NNP
en-wikipedia-org-3679	73	2	.	.	.
en-wikipedia-org-3679	74	1	Archived	archived	JJ
en-wikipedia-org-3679	74	2	from	from	IN
en-wikipedia-org-3679	74	3	the	the	DT
en-wikipedia-org-3679	74	4	original	original	NN
en-wikipedia-org-3679	74	5	on	on	IN
en-wikipedia-org-3679	74	6	2013	2013	CD
en-wikipedia-org-3679	74	7	-	-	SYM
en-wikipedia-org-3679	74	8	09	09	CD
en-wikipedia-org-3679	74	9	-	-	HYPH
en-wikipedia-org-3679	74	10	27	27	CD
en-wikipedia-org-3679	74	11	.	.	.
en-wikipedia-org-3679	75	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	75	2	September	September	NNP
en-wikipedia-org-3679	75	3	25	25	CD
en-wikipedia-org-3679	75	4	,	,	,
en-wikipedia-org-3679	75	5	2013	2013	CD
en-wikipedia-org-3679	75	6	.	.	.
en-wikipedia-org-3679	76	1	^	^	NNP
en-wikipedia-org-3679	76	2	a	a	DT
en-wikipedia-org-3679	76	3	b	b	NNP
en-wikipedia-org-3679	76	4	"	"	``
en-wikipedia-org-3679	76	5	The	the	DT
en-wikipedia-org-3679	76	6	Web	web	NN
en-wikipedia-org-3679	76	7	Robots	Robots	NNPS
en-wikipedia-org-3679	76	8	Pages	Pages	NNP
en-wikipedia-org-3679	76	9	"	"	''
en-wikipedia-org-3679	76	10	.	.	.
en-wikipedia-org-3679	77	1	Robotstxt.org	Robotstxt.org	NNP
en-wikipedia-org-3679	77	2	.	.	.
en-wikipedia-org-3679	78	1	1994	1994	CD
en-wikipedia-org-3679	78	2	-	-	SYM
en-wikipedia-org-3679	78	3	06	06	CD
en-wikipedia-org-3679	78	4	-	-	HYPH
en-wikipedia-org-3679	78	5	30	30	CD
en-wikipedia-org-3679	78	6	.	.	.
en-wikipedia-org-3679	79	1	Archived	archived	JJ
en-wikipedia-org-3679	79	2	from	from	IN
en-wikipedia-org-3679	79	3	the	the	DT
en-wikipedia-org-3679	79	4	original	original	NN
en-wikipedia-org-3679	79	5	on	on	IN
en-wikipedia-org-3679	79	6	2014	2014	CD
en-wikipedia-org-3679	79	7	-	-	SYM
en-wikipedia-org-3679	79	8	01	01	CD
en-wikipedia-org-3679	79	9	-	-	HYPH
en-wikipedia-org-3679	79	10	12	12	CD
en-wikipedia-org-3679	79	11	.	.	.
en-wikipedia-org-3679	80	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	80	2	2013	2013	CD
en-wikipedia-org-3679	80	3	-	-	SYM
en-wikipedia-org-3679	80	4	12	12	CD
en-wikipedia-org-3679	80	5	-	-	SYM
en-wikipedia-org-3679	80	6	29	29	CD
en-wikipedia-org-3679	80	7	.	.	.
en-wikipedia-org-3679	81	1	^	^	NNP
en-wikipedia-org-3679	81	2	Koster	Koster	NNP
en-wikipedia-org-3679	81	3	,	,	,
en-wikipedia-org-3679	81	4	Martijn	Martijn	NNP
en-wikipedia-org-3679	81	5	(	(	-LRB-
en-wikipedia-org-3679	81	6	25	25	CD
en-wikipedia-org-3679	81	7	February	February	NNP
en-wikipedia-org-3679	81	8	1994	1994	CD
en-wikipedia-org-3679	81	9	)	)	-RRB-
en-wikipedia-org-3679	81	10	.	.	.
en-wikipedia-org-3679	82	1	"	"	``
en-wikipedia-org-3679	82	2	Important	important	JJ
en-wikipedia-org-3679	82	3	:	:	:
en-wikipedia-org-3679	82	4	Spiders	spider	NNS
en-wikipedia-org-3679	82	5	,	,	,
en-wikipedia-org-3679	82	6	Robots	Robots	NNPS
en-wikipedia-org-3679	82	7	and	and	CC
en-wikipedia-org-3679	82	8	Web	Web	NNP
en-wikipedia-org-3679	82	9	Wanderers	Wanderers	NNPS
en-wikipedia-org-3679	82	10	"	"	''
en-wikipedia-org-3679	82	11	.	.	.
en-wikipedia-org-3679	83	1	www	www	NN
en-wikipedia-org-3679	83	2	-	-	HYPH
en-wikipedia-org-3679	83	3	talk	talk	NN
en-wikipedia-org-3679	83	4	mailing	mailing	NN
en-wikipedia-org-3679	83	5	list	list	NN
en-wikipedia-org-3679	83	6	.	.	.
en-wikipedia-org-3679	84	1	Archived	archived	JJ
en-wikipedia-org-3679	84	2	from	from	IN
en-wikipedia-org-3679	84	3	the	the	DT
en-wikipedia-org-3679	84	4	original	original	JJ
en-wikipedia-org-3679	84	5	(	(	-LRB-
en-wikipedia-org-3679	84	6	Hypermail	Hypermail	NNP
en-wikipedia-org-3679	84	7	archived	archived	JJ
en-wikipedia-org-3679	84	8	message	message	NN
en-wikipedia-org-3679	84	9	)	)	-RRB-
en-wikipedia-org-3679	84	10	on	on	IN
en-wikipedia-org-3679	84	11	October	October	NNP
en-wikipedia-org-3679	84	12	29	29	CD
en-wikipedia-org-3679	84	13	,	,	,
en-wikipedia-org-3679	84	14	2013	2013	CD
en-wikipedia-org-3679	84	15	.	.	.
en-wikipedia-org-3679	85	1	^	^	FW
en-wikipedia-org-3679	85	2	"	"	``
en-wikipedia-org-3679	85	3	How	how	WRB
en-wikipedia-org-3679	85	4	I	-PRON-	PRP
en-wikipedia-org-3679	85	5	got	get	VBD
en-wikipedia-org-3679	85	6	here	here	RB
en-wikipedia-org-3679	85	7	in	in	IN
en-wikipedia-org-3679	85	8	the	the	DT
en-wikipedia-org-3679	85	9	end	end	NN
en-wikipedia-org-3679	85	10	,	,	,
en-wikipedia-org-3679	85	11	part	part	NN
en-wikipedia-org-3679	85	12	five	five	CD
en-wikipedia-org-3679	85	13	:	:	:
en-wikipedia-org-3679	85	14	"	"	``
en-wikipedia-org-3679	85	15	things	thing	NNS
en-wikipedia-org-3679	85	16	can	can	MD
en-wikipedia-org-3679	85	17	only	only	RB
en-wikipedia-org-3679	85	18	get	get	VB
en-wikipedia-org-3679	85	19	better	well	JJR
en-wikipedia-org-3679	85	20	!	!	.
en-wikipedia-org-3679	85	21	"	"	''
en-wikipedia-org-3679	85	22	"	"	''
en-wikipedia-org-3679	85	23	.	.	.
en-wikipedia-org-3679	86	1	Charlie	Charlie	NNP
en-wikipedia-org-3679	86	2	's	's	POS
en-wikipedia-org-3679	86	3	Diary	Diary	NNP
en-wikipedia-org-3679	86	4	.	.	.
en-wikipedia-org-3679	87	1	19	19	CD
en-wikipedia-org-3679	87	2	June	June	NNP
en-wikipedia-org-3679	87	3	2006	2006	CD
en-wikipedia-org-3679	87	4	.	.	.
en-wikipedia-org-3679	88	1	Archived	archived	JJ
en-wikipedia-org-3679	88	2	from	from	IN
en-wikipedia-org-3679	88	3	the	the	DT
en-wikipedia-org-3679	88	4	original	original	NN
en-wikipedia-org-3679	88	5	on	on	IN
en-wikipedia-org-3679	88	6	2013	2013	CD
en-wikipedia-org-3679	88	7	-	-	SYM
en-wikipedia-org-3679	88	8	11	11	CD
en-wikipedia-org-3679	88	9	-	-	SYM
en-wikipedia-org-3679	88	10	25	25	CD
en-wikipedia-org-3679	88	11	.	.	.
en-wikipedia-org-3679	89	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	89	2	19	19	CD
en-wikipedia-org-3679	89	3	April	April	NNP
en-wikipedia-org-3679	89	4	2014	2014	CD
en-wikipedia-org-3679	89	5	.	.	.
en-wikipedia-org-3679	90	1	^	^	NNP
en-wikipedia-org-3679	90	2	Barry	Barry	NNP
en-wikipedia-org-3679	90	3	Schwartz	Schwartz	NNP
en-wikipedia-org-3679	90	4	(	(	-LRB-
en-wikipedia-org-3679	90	5	30	30	CD
en-wikipedia-org-3679	90	6	June	June	NNP
en-wikipedia-org-3679	90	7	2014	2014	CD
en-wikipedia-org-3679	90	8	)	)	-RRB-
en-wikipedia-org-3679	90	9	.	.	.
en-wikipedia-org-3679	91	1	"	"	``
en-wikipedia-org-3679	91	2	Robots.txt	robots.txt	NN
en-wikipedia-org-3679	91	3	Celebrates	celebrate	NNS
en-wikipedia-org-3679	91	4	20	20	CD
en-wikipedia-org-3679	91	5	Years	year	NNS
en-wikipedia-org-3679	91	6	Of	of	IN
en-wikipedia-org-3679	91	7	Blocking	Blocking	NNP
en-wikipedia-org-3679	91	8	Search	Search	NNP
en-wikipedia-org-3679	91	9	Engines	Engines	NNPS
en-wikipedia-org-3679	91	10	"	"	''
en-wikipedia-org-3679	91	11	.	.	.
en-wikipedia-org-3679	92	1	Search	Search	NNP
en-wikipedia-org-3679	92	2	Engine	Engine	NNP
en-wikipedia-org-3679	92	3	Land	Land	NNP
en-wikipedia-org-3679	92	4	.	.	.
en-wikipedia-org-3679	93	1	Archived	archived	JJ
en-wikipedia-org-3679	93	2	from	from	IN
en-wikipedia-org-3679	93	3	the	the	DT
en-wikipedia-org-3679	93	4	original	original	NN
en-wikipedia-org-3679	93	5	on	on	IN
en-wikipedia-org-3679	93	6	2015	2015	CD
en-wikipedia-org-3679	93	7	-	-	SYM
en-wikipedia-org-3679	93	8	09	09	CD
en-wikipedia-org-3679	93	9	-	-	HYPH
en-wikipedia-org-3679	93	10	07	07	CD
en-wikipedia-org-3679	93	11	.	.	.
en-wikipedia-org-3679	94	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	94	2	2015	2015	CD
en-wikipedia-org-3679	94	3	-	-	SYM
en-wikipedia-org-3679	94	4	11	11	CD
en-wikipedia-org-3679	94	5	-	-	SYM
en-wikipedia-org-3679	94	6	19	19	CD
en-wikipedia-org-3679	94	7	.	.	.
en-wikipedia-org-3679	95	1	^	^	FW
en-wikipedia-org-3679	95	2	"	"	``
en-wikipedia-org-3679	95	3	Formalizing	formalize	VBG
en-wikipedia-org-3679	95	4	the	the	DT
en-wikipedia-org-3679	95	5	Robots	Robots	NNPS
en-wikipedia-org-3679	95	6	Exclusion	Exclusion	NNP
en-wikipedia-org-3679	95	7	Protocol	Protocol	NNP
en-wikipedia-org-3679	95	8	Specification	Specification	NNP
en-wikipedia-org-3679	95	9	"	"	''
en-wikipedia-org-3679	95	10	.	.	.
en-wikipedia-org-3679	96	1	Official	Official	NNP
en-wikipedia-org-3679	96	2	Google	Google	NNP
en-wikipedia-org-3679	96	3	Webmaster	Webmaster	NNP
en-wikipedia-org-3679	96	4	Central	Central	NNP
en-wikipedia-org-3679	96	5	Blog	Blog	NNP
en-wikipedia-org-3679	96	6	.	.	.
en-wikipedia-org-3679	97	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	97	2	2019	2019	CD
en-wikipedia-org-3679	97	3	-	-	SYM
en-wikipedia-org-3679	97	4	07	07	CD
en-wikipedia-org-3679	97	5	-	-	HYPH
en-wikipedia-org-3679	97	6	10	10	CD
en-wikipedia-org-3679	97	7	.	.	.
en-wikipedia-org-3679	98	1	^	^	NNP
en-wikipedia-org-3679	98	2	M.	M.	NNP
en-wikipedia-org-3679	98	3	Koster	Koster	NNP
en-wikipedia-org-3679	98	4	,	,	,
en-wikipedia-org-3679	98	5	Stalworthy	Stalworthy	NNP
en-wikipedia-org-3679	98	6	Computing	Computing	NNP
en-wikipedia-org-3679	98	7	,	,	,
en-wikipedia-org-3679	98	8	Ltd.	Ltd.	NNP
en-wikipedia-org-3679	98	9	,	,	,
en-wikipedia-org-3679	98	10	G.	G.	NNP
en-wikipedia-org-3679	98	11	Illyes	Illyes	NNP
en-wikipedia-org-3679	98	12	,	,	,
en-wikipedia-org-3679	98	13	H.	H.	NNP
en-wikipedia-org-3679	98	14	Zeller	Zeller	NNP
en-wikipedia-org-3679	98	15	,	,	,
en-wikipedia-org-3679	98	16	L.	L.	NNP
en-wikipedia-org-3679	98	17	Harvey	Harvey	NNP
en-wikipedia-org-3679	98	18	,	,	,
en-wikipedia-org-3679	98	19	Google	Google	NNP
en-wikipedia-org-3679	98	20	(	(	-LRB-
en-wikipedia-org-3679	98	21	2019	2019	CD
en-wikipedia-org-3679	98	22	-	-	SYM
en-wikipedia-org-3679	98	23	07	07	CD
en-wikipedia-org-3679	98	24	-	-	HYPH
en-wikipedia-org-3679	98	25	01	01	CD
en-wikipedia-org-3679	98	26	)	)	-RRB-
en-wikipedia-org-3679	98	27	.	.	.
en-wikipedia-org-3679	99	1	"	"	``
en-wikipedia-org-3679	99	2	Robots	Robots	NNP
en-wikipedia-org-3679	99	3	Exclusion	Exclusion	NNP
en-wikipedia-org-3679	99	4	Protocol	Protocol	NNP
en-wikipedia-org-3679	99	5	draft	draft	NN
en-wikipedia-org-3679	99	6	-	-	HYPH
en-wikipedia-org-3679	99	7	rep	rep	NNP
en-wikipedia-org-3679	99	8	-	-	HYPH
en-wikipedia-org-3679	99	9	wg	wg	NNP
en-wikipedia-org-3679	99	10	-	-	HYPH
en-wikipedia-org-3679	99	11	topic-00	topic-00	NN
en-wikipedia-org-3679	99	12	"	"	''
en-wikipedia-org-3679	99	13	.	.	.
en-wikipedia-org-3679	100	1	IETF	IETF	NNP
en-wikipedia-org-3679	100	2	Documents	Documents	NNPS
en-wikipedia-org-3679	100	3	.	.	.
en-wikipedia-org-3679	101	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	101	2	2019	2019	CD
en-wikipedia-org-3679	101	3	-	-	SYM
en-wikipedia-org-3679	101	4	09	09	CD
en-wikipedia-org-3679	101	5	-	-	HYPH
en-wikipedia-org-3679	101	6	09.CS1	09.cs1	NN
en-wikipedia-org-3679	101	7	maint	maint	NN
en-wikipedia-org-3679	101	8	:	:	:
en-wikipedia-org-3679	101	9	multiple	multiple	JJ
en-wikipedia-org-3679	101	10	names	name	NNS
en-wikipedia-org-3679	101	11	:	:	:
en-wikipedia-org-3679	101	12	authors	author	NNS
en-wikipedia-org-3679	101	13	list	list	NNP
en-wikipedia-org-3679	101	14	(	(	-LRB-
en-wikipedia-org-3679	101	15	link	link	NN
en-wikipedia-org-3679	101	16	)	)	-RRB-
en-wikipedia-org-3679	101	17	^	^	NN
en-wikipedia-org-3679	101	18	"	"	``
en-wikipedia-org-3679	101	19	Uncrawled	uncrawled	JJ
en-wikipedia-org-3679	101	20	URLs	url	NNS
en-wikipedia-org-3679	101	21	in	in	IN
en-wikipedia-org-3679	101	22	search	search	NN
en-wikipedia-org-3679	101	23	results	result	NNS
en-wikipedia-org-3679	101	24	"	"	''
en-wikipedia-org-3679	101	25	.	.	.
en-wikipedia-org-3679	102	1	YouTube	YouTube	NNP
en-wikipedia-org-3679	102	2	.	.	.
en-wikipedia-org-3679	103	1	Oct	Oct	NNP
en-wikipedia-org-3679	103	2	5	5	CD
en-wikipedia-org-3679	103	3	,	,	,
en-wikipedia-org-3679	103	4	2009	2009	CD
en-wikipedia-org-3679	103	5	.	.	.
en-wikipedia-org-3679	104	1	Archived	archived	JJ
en-wikipedia-org-3679	104	2	from	from	IN
en-wikipedia-org-3679	104	3	the	the	DT
en-wikipedia-org-3679	104	4	original	original	NN
en-wikipedia-org-3679	104	5	on	on	IN
en-wikipedia-org-3679	104	6	2014	2014	CD
en-wikipedia-org-3679	104	7	-	-	SYM
en-wikipedia-org-3679	104	8	01	01	CD
en-wikipedia-org-3679	104	9	-	-	HYPH
en-wikipedia-org-3679	104	10	06	06	CD
en-wikipedia-org-3679	104	11	.	.	.
en-wikipedia-org-3679	105	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	105	2	2013	2013	CD
en-wikipedia-org-3679	105	3	-	-	SYM
en-wikipedia-org-3679	105	4	12	12	CD
en-wikipedia-org-3679	105	5	-	-	SYM
en-wikipedia-org-3679	105	6	29	29	CD
en-wikipedia-org-3679	105	7	.	.	.
en-wikipedia-org-3679	106	1	^	^	FW
en-wikipedia-org-3679	106	2	"	"	``
en-wikipedia-org-3679	106	3	About	about	IN
en-wikipedia-org-3679	106	4	Ask.com	Ask.com	NNP
en-wikipedia-org-3679	106	5	:	:	:
en-wikipedia-org-3679	106	6	Webmasters	Webmasters	NNP
en-wikipedia-org-3679	106	7	"	"	''
en-wikipedia-org-3679	106	8	.	.	.
en-wikipedia-org-3679	107	1	About.ask.com	about.ask.com	ADD
en-wikipedia-org-3679	107	2	.	.	.
en-wikipedia-org-3679	108	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	108	2	16	16	CD
en-wikipedia-org-3679	108	3	February	February	NNP
en-wikipedia-org-3679	108	4	2013	2013	CD
en-wikipedia-org-3679	108	5	.	.	.
en-wikipedia-org-3679	109	1	^	^	FW
en-wikipedia-org-3679	109	2	"	"	``
en-wikipedia-org-3679	109	3	About	about	IN
en-wikipedia-org-3679	109	4	AOL	AOL	NNP
en-wikipedia-org-3679	109	5	Search	Search	NNP
en-wikipedia-org-3679	109	6	"	"	''
en-wikipedia-org-3679	109	7	.	.	.
en-wikipedia-org-3679	110	1	Search.aol.com	search.aol.com	ADD
en-wikipedia-org-3679	110	2	.	.	.
en-wikipedia-org-3679	111	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	111	2	16	16	CD
en-wikipedia-org-3679	111	3	February	February	NNP
en-wikipedia-org-3679	111	4	2013	2013	CD
en-wikipedia-org-3679	111	5	.	.	.
en-wikipedia-org-3679	112	1	^	^	FW
en-wikipedia-org-3679	112	2	"	"	``
en-wikipedia-org-3679	112	3	Baiduspider	Baiduspider	NNP
en-wikipedia-org-3679	112	4	"	"	''
en-wikipedia-org-3679	112	5	.	.	.
en-wikipedia-org-3679	113	1	Baidu.com	Baidu.com	NNP
en-wikipedia-org-3679	113	2	.	.	.
en-wikipedia-org-3679	114	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	114	2	16	16	CD
en-wikipedia-org-3679	114	3	February	February	NNP
en-wikipedia-org-3679	114	4	2013	2013	CD
en-wikipedia-org-3679	114	5	.	.	.
en-wikipedia-org-3679	115	1	^	^	FW
en-wikipedia-org-3679	115	2	"	"	``
en-wikipedia-org-3679	115	3	DuckDuckGo	DuckDuckGo	NNP
en-wikipedia-org-3679	115	4	Bot	Bot	NNP
en-wikipedia-org-3679	115	5	"	"	''
en-wikipedia-org-3679	115	6	.	.	.
en-wikipedia-org-3679	116	1	DuckDuckGo.com	duckduckgo.com	NN
en-wikipedia-org-3679	116	2	.	.	.
en-wikipedia-org-3679	117	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	117	2	25	25	CD
en-wikipedia-org-3679	117	3	April	April	NNP
en-wikipedia-org-3679	117	4	2017	2017	CD
en-wikipedia-org-3679	117	5	.	.	.
en-wikipedia-org-3679	118	1	^	^	NNP
en-wikipedia-org-3679	118	2	a	a	DT
en-wikipedia-org-3679	118	3	b	b	NN
en-wikipedia-org-3679	118	4	"	"	``
en-wikipedia-org-3679	118	5	Webmasters	webmaster	NNS
en-wikipedia-org-3679	118	6	:	:	:
en-wikipedia-org-3679	118	7	Robots.txt	Robots.txt	NNP
en-wikipedia-org-3679	118	8	Specifications	Specifications	NNPS
en-wikipedia-org-3679	118	9	"	"	''
en-wikipedia-org-3679	118	10	.	.	.
en-wikipedia-org-3679	119	1	Google	Google	NNP
en-wikipedia-org-3679	119	2	Developers	Developers	NNP
en-wikipedia-org-3679	119	3	.	.	.
en-wikipedia-org-3679	120	1	Archived	archived	JJ
en-wikipedia-org-3679	120	2	from	from	IN
en-wikipedia-org-3679	120	3	the	the	DT
en-wikipedia-org-3679	120	4	original	original	NN
en-wikipedia-org-3679	120	5	on	on	IN
en-wikipedia-org-3679	120	6	2013	2013	CD
en-wikipedia-org-3679	120	7	-	-	SYM
en-wikipedia-org-3679	120	8	01	01	CD
en-wikipedia-org-3679	120	9	-	-	HYPH
en-wikipedia-org-3679	120	10	15	15	CD
en-wikipedia-org-3679	120	11	.	.	.
en-wikipedia-org-3679	121	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	121	2	16	16	CD
en-wikipedia-org-3679	121	3	February	February	NNP
en-wikipedia-org-3679	121	4	2013	2013	CD
en-wikipedia-org-3679	121	5	.	.	.
en-wikipedia-org-3679	122	1	^	^	FW
en-wikipedia-org-3679	122	2	"	"	``
en-wikipedia-org-3679	122	3	Submitting	submit	VBG
en-wikipedia-org-3679	122	4	your	-PRON-	PRP$
en-wikipedia-org-3679	122	5	website	website	NN
en-wikipedia-org-3679	122	6	to	to	IN
en-wikipedia-org-3679	122	7	Yahoo	Yahoo	NNP
en-wikipedia-org-3679	122	8	!	!	.
en-wikipedia-org-3679	123	1	Search	search	NN
en-wikipedia-org-3679	123	2	"	"	''
en-wikipedia-org-3679	123	3	.	.	.
en-wikipedia-org-3679	124	1	Archived	archived	JJ
en-wikipedia-org-3679	124	2	from	from	IN
en-wikipedia-org-3679	124	3	the	the	DT
en-wikipedia-org-3679	124	4	original	original	NN
en-wikipedia-org-3679	124	5	on	on	IN
en-wikipedia-org-3679	124	6	2013	2013	CD
en-wikipedia-org-3679	124	7	-	-	SYM
en-wikipedia-org-3679	124	8	01	01	CD
en-wikipedia-org-3679	124	9	-	-	HYPH
en-wikipedia-org-3679	124	10	21	21	CD
en-wikipedia-org-3679	124	11	.	.	.
en-wikipedia-org-3679	125	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	125	2	16	16	CD
en-wikipedia-org-3679	125	3	February	February	NNP
en-wikipedia-org-3679	125	4	2013	2013	CD
en-wikipedia-org-3679	125	5	.	.	.
en-wikipedia-org-3679	126	1	^	^	NNP
en-wikipedia-org-3679	126	2	a	a	DT
en-wikipedia-org-3679	126	3	b	b	NN
en-wikipedia-org-3679	126	4	"	"	``
en-wikipedia-org-3679	126	5	Using	use	VBG
en-wikipedia-org-3679	126	6	robots.txt	robots.txt	NN
en-wikipedia-org-3679	126	7	"	"	''
en-wikipedia-org-3679	126	8	.	.	.
en-wikipedia-org-3679	127	1	Help.yandex.com	Help.yandex.com	NNP
en-wikipedia-org-3679	127	2	.	.	.
en-wikipedia-org-3679	128	1	Archived	archived	JJ
en-wikipedia-org-3679	128	2	from	from	IN
en-wikipedia-org-3679	128	3	the	the	DT
en-wikipedia-org-3679	128	4	original	original	NN
en-wikipedia-org-3679	128	5	on	on	IN
en-wikipedia-org-3679	128	6	2013	2013	CD
en-wikipedia-org-3679	128	7	-	-	SYM
en-wikipedia-org-3679	128	8	01	01	CD
en-wikipedia-org-3679	128	9	-	-	HYPH
en-wikipedia-org-3679	128	10	25	25	CD
en-wikipedia-org-3679	128	11	.	.	.
en-wikipedia-org-3679	129	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	129	2	16	16	CD
en-wikipedia-org-3679	129	3	February	February	NNP
en-wikipedia-org-3679	129	4	2013	2013	CD
en-wikipedia-org-3679	129	5	.	.	.
en-wikipedia-org-3679	130	1	^	^	NNP
en-wikipedia-org-3679	130	2	a	a	DT
en-wikipedia-org-3679	130	3	b	b	NN
en-wikipedia-org-3679	130	4	"	"	``
en-wikipedia-org-3679	130	5	Robots	Robots	NNP
en-wikipedia-org-3679	130	6	Exclusion	Exclusion	NNP
en-wikipedia-org-3679	130	7	Protocol	Protocol	NNP
en-wikipedia-org-3679	130	8	:	:	:
en-wikipedia-org-3679	130	9	joining	join	VBG
en-wikipedia-org-3679	130	10	together	together	RB
en-wikipedia-org-3679	130	11	to	to	TO
en-wikipedia-org-3679	130	12	provide	provide	VB
en-wikipedia-org-3679	130	13	better	well	JJR
en-wikipedia-org-3679	130	14	documentation	documentation	NN
en-wikipedia-org-3679	130	15	"	"	''
en-wikipedia-org-3679	130	16	.	.	.
en-wikipedia-org-3679	131	1	Blogs.bing.com	blogs.bing.com	ADD
en-wikipedia-org-3679	131	2	.	.	.
en-wikipedia-org-3679	132	1	Archived	archived	JJ
en-wikipedia-org-3679	132	2	from	from	IN
en-wikipedia-org-3679	132	3	the	the	DT
en-wikipedia-org-3679	132	4	original	original	NN
en-wikipedia-org-3679	132	5	on	on	IN
en-wikipedia-org-3679	132	6	2014	2014	CD
en-wikipedia-org-3679	132	7	-	-	SYM
en-wikipedia-org-3679	132	8	08	08	CD
en-wikipedia-org-3679	132	9	-	-	SYM
en-wikipedia-org-3679	132	10	18	18	CD
en-wikipedia-org-3679	132	11	.	.	.
en-wikipedia-org-3679	133	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	133	2	16	16	CD
en-wikipedia-org-3679	133	3	February	February	NNP
en-wikipedia-org-3679	133	4	2013	2013	CD
en-wikipedia-org-3679	133	5	.	.	.
en-wikipedia-org-3679	134	1	^	^	FW
en-wikipedia-org-3679	134	2	"	"	``
en-wikipedia-org-3679	134	3	How	how	WRB
en-wikipedia-org-3679	134	4	to	to	TO
en-wikipedia-org-3679	134	5	Create	create	VB
en-wikipedia-org-3679	134	6	a	a	DT
en-wikipedia-org-3679	134	7	Robots.txt	Robots.txt	NNP
en-wikipedia-org-3679	134	8	File	File	NNP
en-wikipedia-org-3679	134	9	-	-	HYPH
en-wikipedia-org-3679	134	10	Bing	Bing	NNP
en-wikipedia-org-3679	134	11	Webmaster	Webmaster	NNP
en-wikipedia-org-3679	134	12	Tools	Tools	NNPS
en-wikipedia-org-3679	134	13	"	"	''
en-wikipedia-org-3679	134	14	.	.	.
en-wikipedia-org-3679	135	1	www.bing.com	www.bing.com	ADD
en-wikipedia-org-3679	135	2	.	.	.
en-wikipedia-org-3679	136	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	136	2	2019	2019	CD
en-wikipedia-org-3679	136	3	-	-	SYM
en-wikipedia-org-3679	136	4	02	02	CD
en-wikipedia-org-3679	136	5	-	-	HYPH
en-wikipedia-org-3679	136	6	06	06	CD
en-wikipedia-org-3679	136	7	.	.	.
en-wikipedia-org-3679	137	1	^	^	NNP
en-wikipedia-org-3679	137	2	Jason	Jason	NNP
en-wikipedia-org-3679	137	3	Scott	Scott	NNP
en-wikipedia-org-3679	137	4	.	.	.
en-wikipedia-org-3679	138	1	"	"	``
en-wikipedia-org-3679	138	2	Robots.txt	robots.txt	NN
en-wikipedia-org-3679	138	3	is	be	VBZ
en-wikipedia-org-3679	138	4	a	a	DT
en-wikipedia-org-3679	138	5	suicide	suicide	NN
en-wikipedia-org-3679	138	6	note	note	NN
en-wikipedia-org-3679	138	7	"	"	''
en-wikipedia-org-3679	138	8	.	.	.
en-wikipedia-org-3679	139	1	Archive	Archive	NNP
en-wikipedia-org-3679	139	2	Team	Team	NNP
en-wikipedia-org-3679	139	3	.	.	.
en-wikipedia-org-3679	140	1	Archived	archived	JJ
en-wikipedia-org-3679	140	2	from	from	IN
en-wikipedia-org-3679	140	3	the	the	DT
en-wikipedia-org-3679	140	4	original	original	NN
en-wikipedia-org-3679	140	5	on	on	IN
en-wikipedia-org-3679	140	6	2017	2017	CD
en-wikipedia-org-3679	140	7	-	-	HYPH
en-wikipedia-org-3679	140	8	02	02	CD
en-wikipedia-org-3679	140	9	-	-	HYPH
en-wikipedia-org-3679	140	10	18	18	CD
en-wikipedia-org-3679	140	11	.	.	.
en-wikipedia-org-3679	141	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	141	2	18	18	CD
en-wikipedia-org-3679	141	3	February	February	NNP
en-wikipedia-org-3679	141	4	2017	2017	CD
en-wikipedia-org-3679	141	5	.	.	.
en-wikipedia-org-3679	142	1	^	^	FW
en-wikipedia-org-3679	142	2	"	"	``
en-wikipedia-org-3679	142	3	Robots.txt	Robots.txt	NNP
en-wikipedia-org-3679	142	4	meant	mean	VBD
en-wikipedia-org-3679	142	5	for	for	IN
en-wikipedia-org-3679	142	6	search	search	NN
en-wikipedia-org-3679	142	7	engines	engine	NNS
en-wikipedia-org-3679	142	8	do	do	VBP
en-wikipedia-org-3679	142	9	n't	not	RB
en-wikipedia-org-3679	142	10	work	work	VB
en-wikipedia-org-3679	142	11	well	well	RB
en-wikipedia-org-3679	142	12	for	for	IN
en-wikipedia-org-3679	142	13	web	web	NN
en-wikipedia-org-3679	142	14	archives	archive	NNS
en-wikipedia-org-3679	142	15	|	|	NNP
en-wikipedia-org-3679	142	16	Internet	Internet	NNP
en-wikipedia-org-3679	142	17	Archive	Archive	NNP
en-wikipedia-org-3679	142	18	Blogs	Blogs	NNP
en-wikipedia-org-3679	142	19	"	"	''
en-wikipedia-org-3679	142	20	.	.	.
en-wikipedia-org-3679	143	1	blog.archive.org	blog.archive.org	NNP
en-wikipedia-org-3679	143	2	.	.	.
en-wikipedia-org-3679	144	1	Archived	archived	JJ
en-wikipedia-org-3679	144	2	from	from	IN
en-wikipedia-org-3679	144	3	the	the	DT
en-wikipedia-org-3679	144	4	original	original	NN
en-wikipedia-org-3679	144	5	on	on	IN
en-wikipedia-org-3679	144	6	2018	2018	CD
en-wikipedia-org-3679	144	7	-	-	SYM
en-wikipedia-org-3679	144	8	12	12	CD
en-wikipedia-org-3679	144	9	-	-	HYPH
en-wikipedia-org-3679	144	10	04	04	CD
en-wikipedia-org-3679	144	11	.	.	.
en-wikipedia-org-3679	145	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	145	2	2018	2018	CD
en-wikipedia-org-3679	145	3	-	-	SYM
en-wikipedia-org-3679	145	4	12	12	CD
en-wikipedia-org-3679	145	5	-	-	HYPH
en-wikipedia-org-3679	145	6	01	01	CD
en-wikipedia-org-3679	145	7	.	.	.
en-wikipedia-org-3679	146	1	^	^	NNP
en-wikipedia-org-3679	146	2	a	a	DT
en-wikipedia-org-3679	146	3	b	b	NNP
en-wikipedia-org-3679	146	4	Jones	Jones	NNP
en-wikipedia-org-3679	146	5	,	,	,
en-wikipedia-org-3679	146	6	Brad	Brad	NNP
en-wikipedia-org-3679	146	7	(	(	-LRB-
en-wikipedia-org-3679	146	8	24	24	CD
en-wikipedia-org-3679	146	9	April	April	NNP
en-wikipedia-org-3679	146	10	2017	2017	CD
en-wikipedia-org-3679	146	11	)	)	-RRB-
en-wikipedia-org-3679	146	12	.	.	.
en-wikipedia-org-3679	147	1	"	"	``
en-wikipedia-org-3679	147	2	The	the	DT
en-wikipedia-org-3679	147	3	Internet	internet	NN
en-wikipedia-org-3679	147	4	Archive	archive	NN
en-wikipedia-org-3679	147	5	Will	Will	MD
en-wikipedia-org-3679	147	6	Ignore	Ignore	NNP
en-wikipedia-org-3679	147	7	Robots.txt	robots.txt	NN
en-wikipedia-org-3679	147	8	Files	file	NNS
en-wikipedia-org-3679	147	9	to	to	TO
en-wikipedia-org-3679	147	10	Maintain	maintain	VB
en-wikipedia-org-3679	147	11	Accuracy	accuracy	NN
en-wikipedia-org-3679	147	12	"	"	''
en-wikipedia-org-3679	147	13	.	.	.
en-wikipedia-org-3679	148	1	Digital	Digital	NNP
en-wikipedia-org-3679	148	2	Trends	Trends	NNPS
en-wikipedia-org-3679	148	3	.	.	.
en-wikipedia-org-3679	149	1	Archived	archived	JJ
en-wikipedia-org-3679	149	2	from	from	IN
en-wikipedia-org-3679	149	3	the	the	DT
en-wikipedia-org-3679	149	4	original	original	NN
en-wikipedia-org-3679	149	5	on	on	IN
en-wikipedia-org-3679	149	6	2017	2017	CD
en-wikipedia-org-3679	149	7	-	-	SYM
en-wikipedia-org-3679	149	8	05	05	CD
en-wikipedia-org-3679	149	9	-	-	HYPH
en-wikipedia-org-3679	149	10	16	16	CD
en-wikipedia-org-3679	149	11	.	.	.
en-wikipedia-org-3679	150	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	150	2	8	8	CD
en-wikipedia-org-3679	150	3	May	May	NNP
en-wikipedia-org-3679	150	4	2017	2017	CD
en-wikipedia-org-3679	150	5	.	.	.
en-wikipedia-org-3679	151	1	^	^	FW
en-wikipedia-org-3679	151	2	"	"	``
en-wikipedia-org-3679	151	3	Block	Block	NNP
en-wikipedia-org-3679	151	4	URLs	URLs	NNPS
en-wikipedia-org-3679	151	5	with	with	IN
en-wikipedia-org-3679	151	6	robots.txt	robots.txt	NN
en-wikipedia-org-3679	151	7	:	:	:
en-wikipedia-org-3679	151	8	Learn	learn	VB
en-wikipedia-org-3679	151	9	about	about	IN
en-wikipedia-org-3679	151	10	robots.txt	robots.txt	NN
en-wikipedia-org-3679	151	11	files	file	NNS
en-wikipedia-org-3679	151	12	"	"	''
en-wikipedia-org-3679	151	13	.	.	.
en-wikipedia-org-3679	152	1	Archived	archived	JJ
en-wikipedia-org-3679	152	2	from	from	IN
en-wikipedia-org-3679	152	3	the	the	DT
en-wikipedia-org-3679	152	4	original	original	NN
en-wikipedia-org-3679	152	5	on	on	IN
en-wikipedia-org-3679	152	6	2015	2015	CD
en-wikipedia-org-3679	152	7	-	-	SYM
en-wikipedia-org-3679	152	8	08	08	CD
en-wikipedia-org-3679	152	9	-	-	HYPH
en-wikipedia-org-3679	152	10	14	14	CD
en-wikipedia-org-3679	152	11	.	.	.
en-wikipedia-org-3679	153	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	153	2	2015	2015	CD
en-wikipedia-org-3679	153	3	-	-	SYM
en-wikipedia-org-3679	153	4	08	08	CD
en-wikipedia-org-3679	153	5	-	-	SYM
en-wikipedia-org-3679	153	6	10	10	CD
en-wikipedia-org-3679	153	7	.	.	.
en-wikipedia-org-3679	154	1	^	^	FW
en-wikipedia-org-3679	154	2	"	"	``
en-wikipedia-org-3679	154	3	Robots.txt	Robots.txt	NNP
en-wikipedia-org-3679	154	4	tells	tell	VBZ
en-wikipedia-org-3679	154	5	hackers	hacker	NNS
en-wikipedia-org-3679	154	6	the	the	DT
en-wikipedia-org-3679	154	7	places	place	NNS
en-wikipedia-org-3679	154	8	you	-PRON-	PRP
en-wikipedia-org-3679	154	9	do	do	VBP
en-wikipedia-org-3679	154	10	n't	not	RB
en-wikipedia-org-3679	154	11	want	want	VB
en-wikipedia-org-3679	154	12	them	-PRON-	PRP
en-wikipedia-org-3679	154	13	to	to	TO
en-wikipedia-org-3679	154	14	look	look	VB
en-wikipedia-org-3679	154	15	"	"	''
en-wikipedia-org-3679	154	16	.	.	.
en-wikipedia-org-3679	155	1	The	the	DT
en-wikipedia-org-3679	155	2	Register	Register	NNP
en-wikipedia-org-3679	155	3	.	.	.
en-wikipedia-org-3679	156	1	Archived	archived	JJ
en-wikipedia-org-3679	156	2	from	from	IN
en-wikipedia-org-3679	156	3	the	the	DT
en-wikipedia-org-3679	156	4	original	original	NN
en-wikipedia-org-3679	156	5	on	on	IN
en-wikipedia-org-3679	156	6	2015	2015	CD
en-wikipedia-org-3679	156	7	-	-	SYM
en-wikipedia-org-3679	156	8	08	08	CD
en-wikipedia-org-3679	156	9	-	-	SYM
en-wikipedia-org-3679	156	10	21	21	CD
en-wikipedia-org-3679	156	11	.	.	.
en-wikipedia-org-3679	157	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	157	2	August	August	NNP
en-wikipedia-org-3679	157	3	12	12	CD
en-wikipedia-org-3679	157	4	,	,	,
en-wikipedia-org-3679	157	5	2015	2015	CD
en-wikipedia-org-3679	157	6	.	.	.
en-wikipedia-org-3679	158	1	^	^	FW
en-wikipedia-org-3679	158	2	"	"	``
en-wikipedia-org-3679	158	3	Guide	Guide	NNP
en-wikipedia-org-3679	158	4	to	to	IN
en-wikipedia-org-3679	158	5	General	General	NNP
en-wikipedia-org-3679	158	6	Server	Server	NNP
en-wikipedia-org-3679	158	7	Security	Security	NNP
en-wikipedia-org-3679	158	8	"	"	''
en-wikipedia-org-3679	158	9	(	(	-LRB-
en-wikipedia-org-3679	158	10	PDF	PDF	NNP
en-wikipedia-org-3679	158	11	)	)	-RRB-
en-wikipedia-org-3679	158	12	.	.	.
en-wikipedia-org-3679	159	1	National	National	NNP
en-wikipedia-org-3679	159	2	Institute	Institute	NNP
en-wikipedia-org-3679	159	3	of	of	IN
en-wikipedia-org-3679	159	4	Standards	Standards	NNPS
en-wikipedia-org-3679	159	5	and	and	CC
en-wikipedia-org-3679	159	6	Technology	Technology	NNP
en-wikipedia-org-3679	159	7	.	.	.
en-wikipedia-org-3679	160	1	July	July	NNP
en-wikipedia-org-3679	160	2	2008	2008	CD
en-wikipedia-org-3679	160	3	.	.	.
en-wikipedia-org-3679	161	1	Archived	Archived	NNP
en-wikipedia-org-3679	161	2	(	(	-LRB-
en-wikipedia-org-3679	161	3	PDF	PDF	NNP
en-wikipedia-org-3679	161	4	)	)	-RRB-
en-wikipedia-org-3679	161	5	from	from	IN
en-wikipedia-org-3679	161	6	the	the	DT
en-wikipedia-org-3679	161	7	original	original	NN
en-wikipedia-org-3679	161	8	on	on	IN
en-wikipedia-org-3679	161	9	2011	2011	CD
en-wikipedia-org-3679	161	10	-	-	SYM
en-wikipedia-org-3679	161	11	10	10	CD
en-wikipedia-org-3679	161	12	-	-	HYPH
en-wikipedia-org-3679	161	13	08	08	CD
en-wikipedia-org-3679	161	14	.	.	.
en-wikipedia-org-3679	162	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	162	2	August	August	NNP
en-wikipedia-org-3679	162	3	12	12	CD
en-wikipedia-org-3679	162	4	,	,	,
en-wikipedia-org-3679	162	5	2015	2015	CD
en-wikipedia-org-3679	162	6	.	.	.
en-wikipedia-org-3679	163	1	^	^	NFP
en-wikipedia-org-3679	163	2	Sverre	Sverre	NNP
en-wikipedia-org-3679	163	3	H.	H.	NNP
en-wikipedia-org-3679	163	4	Huseby	Huseby	NNP
en-wikipedia-org-3679	163	5	(	(	-LRB-
en-wikipedia-org-3679	163	6	2004	2004	CD
en-wikipedia-org-3679	163	7	)	)	-RRB-
en-wikipedia-org-3679	163	8	.	.	.
en-wikipedia-org-3679	164	1	Innocent	innocent	JJ
en-wikipedia-org-3679	164	2	Code	Code	NNP
en-wikipedia-org-3679	164	3	:	:	:
en-wikipedia-org-3679	164	4	A	A	NNP
en-wikipedia-org-3679	164	5	Security	Security	NNP
en-wikipedia-org-3679	164	6	Wake	wake	NN
en-wikipedia-org-3679	164	7	-	-	HYPH
en-wikipedia-org-3679	164	8	Up	up	RP
en-wikipedia-org-3679	164	9	Call	call	NN
en-wikipedia-org-3679	164	10	for	for	IN
en-wikipedia-org-3679	164	11	Web	web	NN
en-wikipedia-org-3679	164	12	Programmers	Programmers	NNPS
en-wikipedia-org-3679	164	13	.	.	.
en-wikipedia-org-3679	165	1	John	John	NNP
en-wikipedia-org-3679	165	2	Wiley	Wiley	NNP
en-wikipedia-org-3679	165	3	&	&	CC
en-wikipedia-org-3679	165	4	Sons	Sons	NNP
en-wikipedia-org-3679	165	5	.	.	.
en-wikipedia-org-3679	166	1	pp	pp	NNP
en-wikipedia-org-3679	166	2	.	.	.
en-wikipedia-org-3679	167	1	91–92	91–92	CD
en-wikipedia-org-3679	167	2	.	.	.
en-wikipedia-org-3679	168	1	ISBN	ISBN	NNP
en-wikipedia-org-3679	168	2	 	  	_SP
en-wikipedia-org-3679	168	3	9780470857472	9780470857472	CD
en-wikipedia-org-3679	168	4	.	.	.
en-wikipedia-org-3679	169	1	Archived	archived	JJ
en-wikipedia-org-3679	169	2	from	from	IN
en-wikipedia-org-3679	169	3	the	the	DT
en-wikipedia-org-3679	169	4	original	original	NN
en-wikipedia-org-3679	169	5	on	on	IN
en-wikipedia-org-3679	169	6	2016	2016	CD
en-wikipedia-org-3679	169	7	-	-	HYPH
en-wikipedia-org-3679	169	8	04	04	CD
en-wikipedia-org-3679	169	9	-	-	HYPH
en-wikipedia-org-3679	169	10	01	01	CD
en-wikipedia-org-3679	169	11	.	.	.
en-wikipedia-org-3679	170	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	170	2	2015	2015	CD
en-wikipedia-org-3679	170	3	-	-	SYM
en-wikipedia-org-3679	170	4	08	08	CD
en-wikipedia-org-3679	170	5	-	-	HYPH
en-wikipedia-org-3679	170	6	12	12	CD
en-wikipedia-org-3679	170	7	.	.	.
en-wikipedia-org-3679	171	1	^	^	FW
en-wikipedia-org-3679	171	2	"	"	``
en-wikipedia-org-3679	171	3	List	List	NNP
en-wikipedia-org-3679	171	4	of	of	IN
en-wikipedia-org-3679	171	5	User	User	NNP
en-wikipedia-org-3679	171	6	-	-	HYPH
en-wikipedia-org-3679	171	7	Agents	Agents	NNP
en-wikipedia-org-3679	171	8	(	(	-LRB-
en-wikipedia-org-3679	171	9	Spiders	Spiders	NNPS
en-wikipedia-org-3679	171	10	,	,	,
en-wikipedia-org-3679	171	11	Robots	Robots	NNPS
en-wikipedia-org-3679	171	12	,	,	,
en-wikipedia-org-3679	171	13	Browser	Browser	NNP
en-wikipedia-org-3679	171	14	)	)	-RRB-
en-wikipedia-org-3679	171	15	"	"	''
en-wikipedia-org-3679	171	16	.	.	.
en-wikipedia-org-3679	172	1	User-agents.org	user-agents.org	UH
en-wikipedia-org-3679	172	2	.	.	.
en-wikipedia-org-3679	173	1	Archived	archived	JJ
en-wikipedia-org-3679	173	2	from	from	IN
en-wikipedia-org-3679	173	3	the	the	DT
en-wikipedia-org-3679	173	4	original	original	NN
en-wikipedia-org-3679	173	5	on	on	IN
en-wikipedia-org-3679	173	6	2014	2014	CD
en-wikipedia-org-3679	173	7	-	-	SYM
en-wikipedia-org-3679	173	8	01	01	CD
en-wikipedia-org-3679	173	9	-	-	HYPH
en-wikipedia-org-3679	173	10	07	07	CD
en-wikipedia-org-3679	173	11	.	.	.
en-wikipedia-org-3679	174	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	174	2	2013	2013	CD
en-wikipedia-org-3679	174	3	-	-	SYM
en-wikipedia-org-3679	174	4	12	12	CD
en-wikipedia-org-3679	174	5	-	-	SYM
en-wikipedia-org-3679	174	6	29	29	CD
en-wikipedia-org-3679	174	7	.	.	.
en-wikipedia-org-3679	175	1	^	^	FW
en-wikipedia-org-3679	175	2	"	"	``
en-wikipedia-org-3679	175	3	Access	Access	NNP
en-wikipedia-org-3679	175	4	Control	Control	NNP
en-wikipedia-org-3679	175	5	-	-	HYPH
en-wikipedia-org-3679	175	6	Apache	Apache	NNP
en-wikipedia-org-3679	175	7	HTTP	HTTP	NNP
en-wikipedia-org-3679	175	8	Server	Server	NNP
en-wikipedia-org-3679	175	9	"	"	''
en-wikipedia-org-3679	175	10	.	.	.
en-wikipedia-org-3679	176	1	Httpd.apache.org	Httpd.apache.org	NNP
en-wikipedia-org-3679	176	2	.	.	.
en-wikipedia-org-3679	177	1	Archived	archived	JJ
en-wikipedia-org-3679	177	2	from	from	IN
en-wikipedia-org-3679	177	3	the	the	DT
en-wikipedia-org-3679	177	4	original	original	NN
en-wikipedia-org-3679	177	5	on	on	IN
en-wikipedia-org-3679	177	6	2013	2013	CD
en-wikipedia-org-3679	177	7	-	-	SYM
en-wikipedia-org-3679	177	8	12	12	CD
en-wikipedia-org-3679	177	9	-	-	SYM
en-wikipedia-org-3679	177	10	29	29	CD
en-wikipedia-org-3679	177	11	.	.	.
en-wikipedia-org-3679	178	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	178	2	2013	2013	CD
en-wikipedia-org-3679	178	3	-	-	SYM
en-wikipedia-org-3679	178	4	12	12	CD
en-wikipedia-org-3679	178	5	-	-	SYM
en-wikipedia-org-3679	178	6	29	29	CD
en-wikipedia-org-3679	178	7	.	.	.
en-wikipedia-org-3679	179	1	^	^	FW
en-wikipedia-org-3679	179	2	"	"	``
en-wikipedia-org-3679	179	3	Deny	Deny	NNP
en-wikipedia-org-3679	179	4	Strings	Strings	NNP
en-wikipedia-org-3679	179	5	for	for	IN
en-wikipedia-org-3679	179	6	Filtering	filter	VBG
en-wikipedia-org-3679	179	7	Rules	Rules	NNPS
en-wikipedia-org-3679	179	8	 	  	_SP
en-wikipedia-org-3679	179	9	:	:	:
en-wikipedia-org-3679	179	10	The	the	DT
en-wikipedia-org-3679	179	11	Official	Official	NNP
en-wikipedia-org-3679	179	12	Microsoft	Microsoft	NNP
en-wikipedia-org-3679	179	13	IIS	iis	RB
en-wikipedia-org-3679	179	14	Site	Site	NNP
en-wikipedia-org-3679	179	15	"	"	''
en-wikipedia-org-3679	179	16	.	.	.
en-wikipedia-org-3679	180	1	Iis.net	Iis.net	NNP
en-wikipedia-org-3679	180	2	.	.	.
en-wikipedia-org-3679	181	1	2013	2013	CD
en-wikipedia-org-3679	181	2	-	-	SYM
en-wikipedia-org-3679	181	3	11	11	CD
en-wikipedia-org-3679	181	4	-	-	HYPH
en-wikipedia-org-3679	181	5	06	06	CD
en-wikipedia-org-3679	181	6	.	.	.
en-wikipedia-org-3679	182	1	Archived	archived	JJ
en-wikipedia-org-3679	182	2	from	from	IN
en-wikipedia-org-3679	182	3	the	the	DT
en-wikipedia-org-3679	182	4	original	original	NN
en-wikipedia-org-3679	182	5	on	on	IN
en-wikipedia-org-3679	182	6	2014	2014	CD
en-wikipedia-org-3679	182	7	-	-	SYM
en-wikipedia-org-3679	182	8	01	01	CD
en-wikipedia-org-3679	182	9	-	-	HYPH
en-wikipedia-org-3679	182	10	01	01	CD
en-wikipedia-org-3679	182	11	.	.	.
en-wikipedia-org-3679	183	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	183	2	2013	2013	CD
en-wikipedia-org-3679	183	3	-	-	SYM
en-wikipedia-org-3679	183	4	12	12	CD
en-wikipedia-org-3679	183	5	-	-	SYM
en-wikipedia-org-3679	183	6	29	29	CD
en-wikipedia-org-3679	183	7	.	.	.
en-wikipedia-org-3679	184	1	^	^	FW
en-wikipedia-org-3679	184	2	"	"	``
en-wikipedia-org-3679	184	3	Google	Google	NNP
en-wikipedia-org-3679	184	4	humans.txt	humans.txt	NNP
en-wikipedia-org-3679	184	5	"	"	''
en-wikipedia-org-3679	184	6	.	.	.
en-wikipedia-org-3679	185	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	185	2	October	October	NNP
en-wikipedia-org-3679	185	3	3	3	CD
en-wikipedia-org-3679	185	4	,	,	,
en-wikipedia-org-3679	185	5	2019	2019	CD
en-wikipedia-org-3679	185	6	.	.	.
en-wikipedia-org-3679	186	1	^	^	FW
en-wikipedia-org-3679	186	2	"	"	``
en-wikipedia-org-3679	186	3	Github	Github	NNP
en-wikipedia-org-3679	186	4	humans.txt	humans.txt	NNP
en-wikipedia-org-3679	186	5	"	"	''
en-wikipedia-org-3679	186	6	.	.	.
en-wikipedia-org-3679	187	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	187	2	October	October	NNP
en-wikipedia-org-3679	187	3	3	3	CD
en-wikipedia-org-3679	187	4	,	,	,
en-wikipedia-org-3679	187	5	2019	2019	CD
en-wikipedia-org-3679	187	6	.	.	.
en-wikipedia-org-3679	188	1	^	^	NNP
en-wikipedia-org-3679	188	2	Newman	Newman	NNP
en-wikipedia-org-3679	188	3	,	,	,
en-wikipedia-org-3679	188	4	Lily	Lily	NNP
en-wikipedia-org-3679	188	5	Hay	Hay	NNP
en-wikipedia-org-3679	188	6	(	(	-LRB-
en-wikipedia-org-3679	188	7	2014	2014	CD
en-wikipedia-org-3679	188	8	-	-	SYM
en-wikipedia-org-3679	188	9	07	07	CD
en-wikipedia-org-3679	188	10	-	-	HYPH
en-wikipedia-org-3679	188	11	03	03	CD
en-wikipedia-org-3679	188	12	)	)	-RRB-
en-wikipedia-org-3679	188	13	.	.	.
en-wikipedia-org-3679	189	1	"	"	``
en-wikipedia-org-3679	189	2	Is	be	VBZ
en-wikipedia-org-3679	189	3	This	this	DT
en-wikipedia-org-3679	189	4	a	a	DT
en-wikipedia-org-3679	189	5	Google	Google	NNP
en-wikipedia-org-3679	189	6	Easter	Easter	NNP
en-wikipedia-org-3679	189	7	Egg	Egg	NNP
en-wikipedia-org-3679	189	8	or	or	CC
en-wikipedia-org-3679	189	9	Proof	Proof	NNP
en-wikipedia-org-3679	189	10	That	that	IN
en-wikipedia-org-3679	189	11	Skynet	Skynet	NNP
en-wikipedia-org-3679	189	12	Is	be	VBZ
en-wikipedia-org-3679	189	13	Actually	actually	RB
en-wikipedia-org-3679	189	14	Plotting	plot	VBG
en-wikipedia-org-3679	189	15	World	world	NN
en-wikipedia-org-3679	189	16	Domination	Domination	NNP
en-wikipedia-org-3679	189	17	?	?	.
en-wikipedia-org-3679	189	18	"	"	''
en-wikipedia-org-3679	189	19	.	.	.
en-wikipedia-org-3679	190	1	Slate	Slate	NNP
en-wikipedia-org-3679	190	2	Magazine	Magazine	NNP
en-wikipedia-org-3679	190	3	.	.	.
en-wikipedia-org-3679	191	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	191	2	2019	2019	CD
en-wikipedia-org-3679	191	3	-	-	SYM
en-wikipedia-org-3679	191	4	10	10	CD
en-wikipedia-org-3679	191	5	-	-	HYPH
en-wikipedia-org-3679	191	6	03	03	CD
en-wikipedia-org-3679	191	7	.	.	.
en-wikipedia-org-3679	192	1	^	^	NNP
en-wikipedia-org-3679	192	2	"	"	``
en-wikipedia-org-3679	192	3	/killer	/killer	NN
en-wikipedia-org-3679	192	4	-	-	HYPH
en-wikipedia-org-3679	192	5	robots.txt	robots.txt	NNP
en-wikipedia-org-3679	192	6	"	"	''
en-wikipedia-org-3679	192	7	.	.	.
en-wikipedia-org-3679	193	1	2018	2018	CD
en-wikipedia-org-3679	193	2	-	-	SYM
en-wikipedia-org-3679	193	3	01	01	CD
en-wikipedia-org-3679	193	4	-	-	HYPH
en-wikipedia-org-3679	193	5	10	10	CD
en-wikipedia-org-3679	193	6	.	.	.
en-wikipedia-org-3679	194	1	Archived	archived	JJ
en-wikipedia-org-3679	194	2	from	from	IN
en-wikipedia-org-3679	194	3	the	the	DT
en-wikipedia-org-3679	194	4	original	original	NN
en-wikipedia-org-3679	194	5	on	on	IN
en-wikipedia-org-3679	194	6	2018	2018	CD
en-wikipedia-org-3679	194	7	-	-	SYM
en-wikipedia-org-3679	194	8	01	01	CD
en-wikipedia-org-3679	194	9	-	-	HYPH
en-wikipedia-org-3679	194	10	10	10	CD
en-wikipedia-org-3679	194	11	.	.	.
en-wikipedia-org-3679	195	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	195	2	2018	2018	CD
en-wikipedia-org-3679	195	3	-	-	SYM
en-wikipedia-org-3679	195	4	05	05	CD
en-wikipedia-org-3679	195	5	-	-	HYPH
en-wikipedia-org-3679	195	6	25	25	CD
en-wikipedia-org-3679	195	7	.	.	.
en-wikipedia-org-3679	196	1	^	^	NNP
en-wikipedia-org-3679	196	2	"	"	``
en-wikipedia-org-3679	196	3	To	to	TO
en-wikipedia-org-3679	196	4	crawl	crawl	NN
en-wikipedia-org-3679	196	5	or	or	CC
en-wikipedia-org-3679	196	6	not	not	RB
en-wikipedia-org-3679	196	7	to	to	TO
en-wikipedia-org-3679	196	8	crawl	crawl	VB
en-wikipedia-org-3679	196	9	,	,	,
en-wikipedia-org-3679	196	10	that	that	RB
en-wikipedia-org-3679	196	11	is	be	VBZ
en-wikipedia-org-3679	196	12	BingBot	BingBot	NNP
en-wikipedia-org-3679	196	13	's	's	POS
en-wikipedia-org-3679	196	14	question	question	NN
en-wikipedia-org-3679	196	15	"	"	''
en-wikipedia-org-3679	196	16	.	.	.
en-wikipedia-org-3679	197	1	3	3	CD
en-wikipedia-org-3679	197	2	May	May	NNP
en-wikipedia-org-3679	197	3	2012	2012	CD
en-wikipedia-org-3679	197	4	.	.	.
en-wikipedia-org-3679	198	1	Archived	archived	JJ
en-wikipedia-org-3679	198	2	from	from	IN
en-wikipedia-org-3679	198	3	the	the	DT
en-wikipedia-org-3679	198	4	original	original	NN
en-wikipedia-org-3679	198	5	on	on	IN
en-wikipedia-org-3679	198	6	2016	2016	CD
en-wikipedia-org-3679	198	7	-	-	SYM
en-wikipedia-org-3679	198	8	02	02	CD
en-wikipedia-org-3679	198	9	-	-	HYPH
en-wikipedia-org-3679	198	10	03	03	CD
en-wikipedia-org-3679	198	11	.	.	.
en-wikipedia-org-3679	199	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	199	2	9	9	CD
en-wikipedia-org-3679	199	3	February	February	NNP
en-wikipedia-org-3679	199	4	2016	2016	CD
en-wikipedia-org-3679	199	5	.	.	.
en-wikipedia-org-3679	200	1	^	^	FW
en-wikipedia-org-3679	200	2	"	"	``
en-wikipedia-org-3679	200	3	Change	Change	NNP
en-wikipedia-org-3679	200	4	Googlebot	Googlebot	NNP
en-wikipedia-org-3679	200	5	crawl	crawl	NN
en-wikipedia-org-3679	200	6	rate	rate	NN
en-wikipedia-org-3679	200	7	-	-	HYPH
en-wikipedia-org-3679	200	8	Search	Search	NNP
en-wikipedia-org-3679	200	9	Console	Console	NNP
en-wikipedia-org-3679	200	10	Help	Help	NNP
en-wikipedia-org-3679	200	11	"	"	''
en-wikipedia-org-3679	200	12	.	.	.
en-wikipedia-org-3679	201	1	support.google.com	support.google.com	NNP
en-wikipedia-org-3679	201	2	.	.	.
en-wikipedia-org-3679	202	1	Archived	archived	JJ
en-wikipedia-org-3679	202	2	from	from	IN
en-wikipedia-org-3679	202	3	the	the	DT
en-wikipedia-org-3679	202	4	original	original	NN
en-wikipedia-org-3679	202	5	on	on	IN
en-wikipedia-org-3679	202	6	2018	2018	CD
en-wikipedia-org-3679	202	7	-	-	SYM
en-wikipedia-org-3679	202	8	11	11	CD
en-wikipedia-org-3679	202	9	-	-	SYM
en-wikipedia-org-3679	202	10	18	18	CD
en-wikipedia-org-3679	202	11	.	.	.
en-wikipedia-org-3679	203	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	203	2	22	22	CD
en-wikipedia-org-3679	203	3	October	October	NNP
en-wikipedia-org-3679	203	4	2018	2018	CD
en-wikipedia-org-3679	203	5	.	.	.
en-wikipedia-org-3679	204	1	^	^	FW
en-wikipedia-org-3679	204	2	"	"	``
en-wikipedia-org-3679	204	3	Webmaster	Webmaster	NNP
en-wikipedia-org-3679	204	4	Help	Help	NNP
en-wikipedia-org-3679	204	5	Center	Center	NNP
en-wikipedia-org-3679	204	6	-	-	:
en-wikipedia-org-3679	204	7	How	how	WRB
en-wikipedia-org-3679	204	8	do	do	VBP
en-wikipedia-org-3679	204	9	I	-PRON-	PRP
en-wikipedia-org-3679	204	10	block	block	VB
en-wikipedia-org-3679	204	11	Googlebot	Googlebot	NNP
en-wikipedia-org-3679	204	12	?	?	.
en-wikipedia-org-3679	204	13	"	"	''
en-wikipedia-org-3679	204	14	.	.	.
en-wikipedia-org-3679	205	1	Archived	archived	JJ
en-wikipedia-org-3679	205	2	from	from	IN
en-wikipedia-org-3679	205	3	the	the	DT
en-wikipedia-org-3679	205	4	original	original	NN
en-wikipedia-org-3679	205	5	on	on	IN
en-wikipedia-org-3679	205	6	2010	2010	CD
en-wikipedia-org-3679	205	7	-	-	SYM
en-wikipedia-org-3679	205	8	08	08	CD
en-wikipedia-org-3679	205	9	-	-	HYPH
en-wikipedia-org-3679	205	10	01	01	CD
en-wikipedia-org-3679	205	11	.	.	.
en-wikipedia-org-3679	206	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	206	2	2007	2007	CD
en-wikipedia-org-3679	206	3	-	-	SYM
en-wikipedia-org-3679	206	4	11	11	CD
en-wikipedia-org-3679	206	5	-	-	SYM
en-wikipedia-org-3679	206	6	20	20	CD
en-wikipedia-org-3679	206	7	.	.	.
en-wikipedia-org-3679	207	1	^	^	FW
en-wikipedia-org-3679	207	2	"	"	``
en-wikipedia-org-3679	207	3	How	how	WRB
en-wikipedia-org-3679	207	4	do	do	VBP
en-wikipedia-org-3679	207	5	I	-PRON-	PRP
en-wikipedia-org-3679	207	6	prevent	prevent	VB
en-wikipedia-org-3679	207	7	my	-PRON-	PRP$
en-wikipedia-org-3679	207	8	site	site	NN
en-wikipedia-org-3679	207	9	or	or	CC
en-wikipedia-org-3679	207	10	certain	certain	JJ
en-wikipedia-org-3679	207	11	subdirectories	subdirectory	NNS
en-wikipedia-org-3679	207	12	from	from	IN
en-wikipedia-org-3679	207	13	being	be	VBG
en-wikipedia-org-3679	207	14	crawled	crawl	VBN
en-wikipedia-org-3679	207	15	?	?	.
en-wikipedia-org-3679	208	1	-	-	:
en-wikipedia-org-3679	208	2	Yahoo	Yahoo	NNP
en-wikipedia-org-3679	208	3	Search	Search	NNP
en-wikipedia-org-3679	208	4	Help	Help	NNP
en-wikipedia-org-3679	208	5	"	"	''
en-wikipedia-org-3679	208	6	.	.	.
en-wikipedia-org-3679	209	1	Archived	archived	JJ
en-wikipedia-org-3679	209	2	from	from	IN
en-wikipedia-org-3679	209	3	the	the	DT
en-wikipedia-org-3679	209	4	original	original	NN
en-wikipedia-org-3679	209	5	on	on	IN
en-wikipedia-org-3679	209	6	2007	2007	CD
en-wikipedia-org-3679	209	7	-	-	SYM
en-wikipedia-org-3679	209	8	10	10	CD
en-wikipedia-org-3679	209	9	-	-	SYM
en-wikipedia-org-3679	209	10	13	13	CD
en-wikipedia-org-3679	209	11	.	.	.
en-wikipedia-org-3679	210	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	210	2	2007	2007	CD
en-wikipedia-org-3679	210	3	-	-	SYM
en-wikipedia-org-3679	210	4	11	11	CD
en-wikipedia-org-3679	210	5	-	-	SYM
en-wikipedia-org-3679	210	6	20	20	CD
en-wikipedia-org-3679	210	7	.	.	.
en-wikipedia-org-3679	211	1	^	^	FW
en-wikipedia-org-3679	211	2	"	"	``
en-wikipedia-org-3679	211	3	Google	Google	NNP
en-wikipedia-org-3679	211	4	's	's	POS
en-wikipedia-org-3679	211	5	Hidden	Hidden	NNP
en-wikipedia-org-3679	211	6	Interpretation	Interpretation	NNP
en-wikipedia-org-3679	211	7	of	of	IN
en-wikipedia-org-3679	211	8	Robots.txt	Robots.txt	NNP
en-wikipedia-org-3679	211	9	"	"	''
en-wikipedia-org-3679	211	10	.	.	.
en-wikipedia-org-3679	212	1	Archived	archived	JJ
en-wikipedia-org-3679	212	2	from	from	IN
en-wikipedia-org-3679	212	3	the	the	DT
en-wikipedia-org-3679	212	4	original	original	NN
en-wikipedia-org-3679	212	5	on	on	IN
en-wikipedia-org-3679	212	6	2010	2010	CD
en-wikipedia-org-3679	212	7	-	-	SYM
en-wikipedia-org-3679	212	8	11	11	CD
en-wikipedia-org-3679	212	9	-	-	SYM
en-wikipedia-org-3679	212	10	20	20	CD
en-wikipedia-org-3679	212	11	.	.	.
en-wikipedia-org-3679	213	1	Retrieved	retrieve	VBN
en-wikipedia-org-3679	213	2	2010	2010	CD
en-wikipedia-org-3679	213	3	-	-	SYM
en-wikipedia-org-3679	213	4	11	11	CD
en-wikipedia-org-3679	213	5	-	-	SYM
en-wikipedia-org-3679	213	6	15	15	CD
en-wikipedia-org-3679	213	7	.	.	.
en-wikipedia-org-3679	214	1	^	^	FW
en-wikipedia-org-3679	214	2	"	"	``
en-wikipedia-org-3679	214	3	Yahoo	Yahoo	NNP
en-wikipedia-org-3679	214	4	!	!	.
en-wikipedia-org-3679	215	1	Search	Search	NNP
en-wikipedia-org-3679	215	2	Blog	Blog	NNP
en-wikipedia-org-3679	215	3	-	-	HYPH
en-wikipedia-org-3679	215	4	Webmasters	Webmasters	NNP
en-wikipedia-org-3679	215	5	can	can	MD
en-wikipedia-org-3679	215	6	now	now	RB
en-wikipedia-org-3679	215	7	auto	auto	NN
en-wikipedia-org-3679	215	8	-	-	HYPH
en-wikipedia-org-3679	215	9	discover	discover	NN
en-wikipedia-org-3679	215	10	with	with	IN
en-wikipedia-org-3679	215	11	Sitemaps	Sitemaps	NNP
en-wikipedia-org-3679	215	12	"	"	''
en-wikipedia-org-3679	215	13	.	.	.
en-wikipedia-org-3679	216	1	Archived	archived	JJ
en-wikipedia-org-3679	216	2	from	from	IN
en-wikipedia-org-3679	216	3	the	the	DT
en-wikipedia-org-3679	216	4	original	original	NN
en-wikipedia-org-3679	216	5	on	on	IN
en-wikipedia-org-3679	216	6	2009	2009	CD
en-wikipedia-org-3679	216	7	-	-	HYPH
en-wikipedia-org-3679	216	8	03	03	CD
en-wikipedia-org-3679	216	9	-	-	HYPH
en-wikipedia-org-3679	216	10	05	05	CD
en-wikipedia-org-3679	216	11	.	.	.
en-wikipedia-org-3679	217	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	217	2	2009	2009	CD
en-wikipedia-org-3679	217	3	-	-	HYPH
en-wikipedia-org-3679	217	4	03	03	CD
en-wikipedia-org-3679	217	5	-	-	HYPH
en-wikipedia-org-3679	217	6	23	23	CD
en-wikipedia-org-3679	217	7	.	.	.
en-wikipedia-org-3679	218	1	^	^	FW
en-wikipedia-org-3679	218	2	"	"	``
en-wikipedia-org-3679	218	3	Yandex	Yandex	NNP
en-wikipedia-org-3679	218	4	-	-	HYPH
en-wikipedia-org-3679	218	5	Using	use	VBG
en-wikipedia-org-3679	218	6	robots.txt	robots.txt	NN
en-wikipedia-org-3679	218	7	"	"	''
en-wikipedia-org-3679	218	8	.	.	.
en-wikipedia-org-3679	219	1	Archived	archived	JJ
en-wikipedia-org-3679	219	2	from	from	IN
en-wikipedia-org-3679	219	3	the	the	DT
en-wikipedia-org-3679	219	4	original	original	NN
en-wikipedia-org-3679	219	5	on	on	IN
en-wikipedia-org-3679	219	6	2013	2013	CD
en-wikipedia-org-3679	219	7	-	-	SYM
en-wikipedia-org-3679	219	8	05	05	CD
en-wikipedia-org-3679	219	9	-	-	HYPH
en-wikipedia-org-3679	219	10	09	09	CD
en-wikipedia-org-3679	219	11	.	.	.
en-wikipedia-org-3679	220	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	220	2	2013	2013	CD
en-wikipedia-org-3679	220	3	-	-	SYM
en-wikipedia-org-3679	220	4	05	05	CD
en-wikipedia-org-3679	220	5	-	-	SYM
en-wikipedia-org-3679	220	6	13	13	CD
en-wikipedia-org-3679	220	7	.	.	.
en-wikipedia-org-3679	221	1	^	^	FW
en-wikipedia-org-3679	221	2	"	"	``
en-wikipedia-org-3679	221	3	Robots.txt	Robots.txt	NNP
en-wikipedia-org-3679	221	4	Specifications	Specifications	NNPS
en-wikipedia-org-3679	221	5	"	"	''
en-wikipedia-org-3679	221	6	.	.	.
en-wikipedia-org-3679	222	1	Google	Google	NNP
en-wikipedia-org-3679	222	2	Developers	Developers	NNP
en-wikipedia-org-3679	222	3	.	.	.
en-wikipedia-org-3679	223	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	223	2	February	February	NNP
en-wikipedia-org-3679	223	3	15	15	CD
en-wikipedia-org-3679	223	4	,	,	,
en-wikipedia-org-3679	223	5	2020	2020	CD
en-wikipedia-org-3679	223	6	.	.	.
en-wikipedia-org-3679	224	1	^	^	NNP
en-wikipedia-org-3679	224	2	a	a	DT
en-wikipedia-org-3679	224	3	b	b	NNP
en-wikipedia-org-3679	224	4	"	"	``
en-wikipedia-org-3679	224	5	Robots	robot	VBZ
en-wikipedia-org-3679	224	6	meta	meta	JJ
en-wikipedia-org-3679	224	7	tag	tag	NN
en-wikipedia-org-3679	224	8	and	and	CC
en-wikipedia-org-3679	224	9	X	X	NNS
en-wikipedia-org-3679	224	10	-	-	HYPH
en-wikipedia-org-3679	224	11	Robots	robots	JJ
en-wikipedia-org-3679	224	12	-	-	HYPH
en-wikipedia-org-3679	224	13	Tag	tag	NN
en-wikipedia-org-3679	224	14	HTTP	http	NN
en-wikipedia-org-3679	224	15	header	header	NN
en-wikipedia-org-3679	224	16	specifications	specification	NNS
en-wikipedia-org-3679	224	17	-	-	:
en-wikipedia-org-3679	224	18	Webmasters	webmaster	NNS
en-wikipedia-org-3679	224	19	—	—	:
en-wikipedia-org-3679	224	20	Google	Google	NNP
en-wikipedia-org-3679	224	21	Developers	Developers	NNP
en-wikipedia-org-3679	224	22	"	"	''
en-wikipedia-org-3679	224	23	.	.	.
en-wikipedia-org-3679	225	1	Archived	archived	JJ
en-wikipedia-org-3679	225	2	from	from	IN
en-wikipedia-org-3679	225	3	the	the	DT
en-wikipedia-org-3679	225	4	original	original	NN
en-wikipedia-org-3679	225	5	on	on	IN
en-wikipedia-org-3679	225	6	2013	2013	CD
en-wikipedia-org-3679	225	7	-	-	SYM
en-wikipedia-org-3679	225	8	08	08	CD
en-wikipedia-org-3679	225	9	-	-	HYPH
en-wikipedia-org-3679	225	10	08	08	CD
en-wikipedia-org-3679	225	11	.	.	.
en-wikipedia-org-3679	226	1	Retrieved	Retrieved	NNP
en-wikipedia-org-3679	226	2	2013	2013	CD
en-wikipedia-org-3679	226	3	-	-	SYM
en-wikipedia-org-3679	226	4	08	08	CD
en-wikipedia-org-3679	226	5	-	-	SYM
en-wikipedia-org-3679	226	6	17	17	CD
en-wikipedia-org-3679	226	7	.	.	.
en-wikipedia-org-3679	227	1	External	External	NNP
en-wikipedia-org-3679	227	2	links[edit	links[edit	,
en-wikipedia-org-3679	227	3	]	]	-RRB-
en-wikipedia-org-3679	227	4	Official	official	JJ
en-wikipedia-org-3679	227	5	website	website	NN
en-wikipedia-org-3679	227	6	Robots	robot	NNS
en-wikipedia-org-3679	227	7	Database	Database	NNP
en-wikipedia-org-3679	227	8	(	(	-LRB-
en-wikipedia-org-3679	227	9	list	list	NN
en-wikipedia-org-3679	227	10	of	of	IN
en-wikipedia-org-3679	227	11	bot	bot	NN
en-wikipedia-org-3679	227	12	names	name	NNS
en-wikipedia-org-3679	227	13	)	)	-RRB-
en-wikipedia-org-3679	227	14	Robots.txt	Robots.txt	NNP
en-wikipedia-org-3679	227	15	checker	checker	NNP
en-wikipedia-org-3679	227	16	tool	tool	NN
en-wikipedia-org-3679	227	17	Microsoft	Microsoft	NNP
en-wikipedia-org-3679	227	18	bingbot	bingbot	NN
en-wikipedia-org-3679	227	19	official	official	JJ
en-wikipedia-org-3679	227	20	documentation	documentation	NN
en-wikipedia-org-3679	227	21	Internet	internet	NN
en-wikipedia-org-3679	227	22	portal	portal	NN
en-wikipedia-org-3679	227	23	Retrieved	retrieve	VBN
en-wikipedia-org-3679	227	24	from	from	IN
en-wikipedia-org-3679	227	25	"	"	``
en-wikipedia-org-3679	227	26	https://en.wikipedia.org/w/index.php?title=Robots_exclusion_standard&oldid=991385881	https://en.wikipedia.org/w/index.php?title=Robots_exclusion_standard&oldid=991385881	NNP
en-wikipedia-org-3679	227	27	"	"	''
en-wikipedia-org-3679	227	28	Categories	category	NNS
en-wikipedia-org-3679	227	29	:	:	:
en-wikipedia-org-3679	227	30	World	world	NN
en-wikipedia-org-3679	227	31	Wide	wide	JJ
en-wikipedia-org-3679	227	32	Web	web	NN
en-wikipedia-org-3679	227	33	Internet	internet	NN
en-wikipedia-org-3679	227	34	bots	bot	NNS
en-wikipedia-org-3679	227	35	Hidden	hide	VBN
en-wikipedia-org-3679	227	36	categories	category	NNS
en-wikipedia-org-3679	227	37	:	:	:
en-wikipedia-org-3679	227	38	CS1	cs1	NN
en-wikipedia-org-3679	227	39	maint	maint	NN
en-wikipedia-org-3679	227	40	:	:	:
en-wikipedia-org-3679	227	41	multiple	multiple	JJ
en-wikipedia-org-3679	227	42	names	name	NNS
en-wikipedia-org-3679	227	43	:	:	:
en-wikipedia-org-3679	227	44	authors	author	NNS
en-wikipedia-org-3679	227	45	list	list	NN
en-wikipedia-org-3679	227	46	Wikipedia	Wikipedia	NNP
en-wikipedia-org-3679	227	47	pending	pende	VBG
en-wikipedia-org-3679	227	48	changes	change	NNS
en-wikipedia-org-3679	227	49	protected	protect	VBD
en-wikipedia-org-3679	227	50	pages	page	NNS
en-wikipedia-org-3679	227	51	Articles	article	NNS
en-wikipedia-org-3679	227	52	with	with	IN
en-wikipedia-org-3679	227	53	short	short	JJ
en-wikipedia-org-3679	227	54	description	description	NN
en-wikipedia-org-3679	227	55	Short	short	JJ
en-wikipedia-org-3679	227	56	description	description	NN
en-wikipedia-org-3679	227	57	matches	match	VBZ
en-wikipedia-org-3679	227	58	Wikidata	Wikidata	NNP
en-wikipedia-org-3679	227	59	Official	official	JJ
en-wikipedia-org-3679	227	60	website	website	NN
en-wikipedia-org-3679	227	61	different	different	JJ
en-wikipedia-org-3679	227	62	in	in	IN
en-wikipedia-org-3679	227	63	Wikidata	Wikidata	NNP
en-wikipedia-org-3679	227	64	and	and	CC
en-wikipedia-org-3679	227	65	Wikipedia	Wikipedia	NNP
en-wikipedia-org-3679	227	66	AC	AC	NNP
en-wikipedia-org-3679	227	67	with	with	IN
en-wikipedia-org-3679	227	68	0	0	CD
en-wikipedia-org-3679	227	69	elements	element	NNS
en-wikipedia-org-3679	227	70	Navigation	Navigation	NNP
en-wikipedia-org-3679	227	71	menu	menu	NN
en-wikipedia-org-3679	227	72	Personal	personal	JJ
en-wikipedia-org-3679	227	73	tools	tool	NNS
en-wikipedia-org-3679	227	74	Not	not	RB
en-wikipedia-org-3679	227	75	logged	log	VBN
en-wikipedia-org-3679	227	76	in	in	IN
en-wikipedia-org-3679	227	77	Talk	Talk	NNP
en-wikipedia-org-3679	227	78	Contributions	Contributions	NNP
en-wikipedia-org-3679	227	79	Create	Create	NNP
en-wikipedia-org-3679	227	80	account	account	NN
en-wikipedia-org-3679	227	81	Log	Log	NNP
en-wikipedia-org-3679	227	82	in	in	IN
en-wikipedia-org-3679	227	83	Namespaces	Namespaces	NNP
en-wikipedia-org-3679	227	84	Article	Article	NNP
en-wikipedia-org-3679	227	85	Talk	talk	NN
en-wikipedia-org-3679	227	86	Variants	variant	VBZ
en-wikipedia-org-3679	227	87	Views	view	NNS
en-wikipedia-org-3679	227	88	Read	read	VBP
en-wikipedia-org-3679	227	89	Edit	edit	NN
en-wikipedia-org-3679	227	90	View	View	NNP
en-wikipedia-org-3679	227	91	history	history	NN
en-wikipedia-org-3679	227	92	More	More	JJR
en-wikipedia-org-3679	227	93	Search	Search	NNP
en-wikipedia-org-3679	227	94	Navigation	Navigation	NNP
en-wikipedia-org-3679	227	95	Main	Main	NNP
en-wikipedia-org-3679	227	96	page	page	NN
en-wikipedia-org-3679	227	97	Contents	content	VBZ
en-wikipedia-org-3679	227	98	Current	current	JJ
en-wikipedia-org-3679	227	99	events	event	NNS
en-wikipedia-org-3679	227	100	Random	random	JJ
en-wikipedia-org-3679	227	101	article	article	NN
en-wikipedia-org-3679	227	102	About	about	IN
en-wikipedia-org-3679	227	103	Wikipedia	Wikipedia	NNP
en-wikipedia-org-3679	227	104	Contact	Contact	NNP
en-wikipedia-org-3679	227	105	us	-PRON-	PRP
en-wikipedia-org-3679	227	106	Donate	Donate	NNP
en-wikipedia-org-3679	227	107	Contribute	contribute	NN
en-wikipedia-org-3679	227	108	Help	help	NN
en-wikipedia-org-3679	227	109	Learn	learn	VB
en-wikipedia-org-3679	227	110	to	to	TO
en-wikipedia-org-3679	227	111	edit	edit	VB
en-wikipedia-org-3679	227	112	Community	community	NN
en-wikipedia-org-3679	227	113	portal	portal	JJ
en-wikipedia-org-3679	227	114	Recent	recent	JJ
en-wikipedia-org-3679	227	115	changes	change	NNS
en-wikipedia-org-3679	227	116	Upload	upload	JJ
en-wikipedia-org-3679	227	117	file	file	NN
en-wikipedia-org-3679	227	118	Tools	tool	NNS
en-wikipedia-org-3679	227	119	What	what	WP
en-wikipedia-org-3679	227	120	links	link	NNS
en-wikipedia-org-3679	227	121	here	here	RB
en-wikipedia-org-3679	227	122	Related	relate	VBN
en-wikipedia-org-3679	227	123	changes	change	VBZ
en-wikipedia-org-3679	227	124	Upload	upload	JJ
en-wikipedia-org-3679	227	125	file	file	VBP
en-wikipedia-org-3679	227	126	Special	special	JJ
en-wikipedia-org-3679	227	127	pages	page	NNS
en-wikipedia-org-3679	227	128	Permanent	permanent	JJ
en-wikipedia-org-3679	227	129	link	link	NN
en-wikipedia-org-3679	227	130	Page	Page	NNP
en-wikipedia-org-3679	227	131	information	information	NN
en-wikipedia-org-3679	227	132	Cite	cite	VB
en-wikipedia-org-3679	227	133	this	this	DT
en-wikipedia-org-3679	227	134	page	page	NN
en-wikipedia-org-3679	227	135	Wikidata	wikidata	NN
en-wikipedia-org-3679	227	136	item	item	NN
en-wikipedia-org-3679	227	137	Print	print	NN
en-wikipedia-org-3679	227	138	/	/	SYM
en-wikipedia-org-3679	227	139	export	export	NN
en-wikipedia-org-3679	227	140	Download	Download	NNP
en-wikipedia-org-3679	227	141	as	as	IN
en-wikipedia-org-3679	227	142	PDF	PDF	NNP
en-wikipedia-org-3679	227	143	Printable	Printable	NNP
en-wikipedia-org-3679	227	144	version	version	NN
en-wikipedia-org-3679	227	145	Languages	language	VBZ
en-wikipedia-org-3679	227	146	العربية	العربية	NNP
en-wikipedia-org-3679	227	147	Boarisch	Boarisch	NNP
en-wikipedia-org-3679	227	148	Català	Català	NNP
en-wikipedia-org-3679	227	149	Čeština	Čeština	NNP
en-wikipedia-org-3679	227	150	Dansk	Dansk	NNP
en-wikipedia-org-3679	227	151	Deutsch	Deutsch	NNP
en-wikipedia-org-3679	227	152	Español	Español	NNP
en-wikipedia-org-3679	227	153	فارسی	فارسی	NNP
en-wikipedia-org-3679	227	154	Français	Français	NNP
en-wikipedia-org-3679	227	155	한국어	한국어	NN
en-wikipedia-org-3679	227	156	Bahasa	Bahasa	NNP
en-wikipedia-org-3679	227	157	Indonesia	Indonesia	NNP
en-wikipedia-org-3679	227	158	Italiano	Italiano	NNP
en-wikipedia-org-3679	227	159	עברית	עברית	NN
en-wikipedia-org-3679	227	160	Nederlands	Nederlands	NNP
en-wikipedia-org-3679	227	161	日本語	日本語	NNP
en-wikipedia-org-3679	227	162	Polski	Polski	NNP
en-wikipedia-org-3679	227	163	Português	Português	NNP
en-wikipedia-org-3679	227	164	Русский	Русский	NNP
en-wikipedia-org-3679	227	165	Simple	Simple	NNP
en-wikipedia-org-3679	227	166	English	English	NNP
en-wikipedia-org-3679	227	167	Српски	Српски	NNP
en-wikipedia-org-3679	227	168	/	/	SYM
en-wikipedia-org-3679	227	169	srpski	srpski	JJ
en-wikipedia-org-3679	227	170	Suomi	Suomi	NNP
en-wikipedia-org-3679	227	171	Svenska	Svenska	NNP
en-wikipedia-org-3679	227	172	Türkçe	Türkçe	NNP
en-wikipedia-org-3679	227	173	Українська	Українська	NNP
en-wikipedia-org-3679	227	174	中文	中文	NNP
en-wikipedia-org-3679	227	175	Edit	Edit	NNP
en-wikipedia-org-3679	227	176	links	link	NNS
en-wikipedia-org-3679	227	177	This	this	DT
en-wikipedia-org-3679	227	178	page	page	NN
en-wikipedia-org-3679	227	179	was	be	VBD
en-wikipedia-org-3679	227	180	last	last	RB
en-wikipedia-org-3679	227	181	edited	edit	VBN
en-wikipedia-org-3679	227	182	on	on	IN
en-wikipedia-org-3679	227	183	29	29	CD
en-wikipedia-org-3679	227	184	November	November	NNP
en-wikipedia-org-3679	227	185	2020	2020	CD
en-wikipedia-org-3679	227	186	,	,	,
en-wikipedia-org-3679	227	187	at	at	IN
en-wikipedia-org-3679	227	188	19:54	19:54	CD
en-wikipedia-org-3679	227	189	 	  	_SP
en-wikipedia-org-3679	227	190	(	(	-LRB-
en-wikipedia-org-3679	227	191	UTC	UTC	NNP
en-wikipedia-org-3679	227	192	)	)	-RRB-
en-wikipedia-org-3679	227	193	.	.	.
en-wikipedia-org-3679	228	1	Text	text	NN
en-wikipedia-org-3679	228	2	is	be	VBZ
en-wikipedia-org-3679	228	3	available	available	JJ
en-wikipedia-org-3679	228	4	under	under	IN
en-wikipedia-org-3679	228	5	the	the	DT
en-wikipedia-org-3679	228	6	Creative	Creative	NNP
en-wikipedia-org-3679	228	7	Commons	Commons	NNP
en-wikipedia-org-3679	228	8	Attribution	Attribution	NNP
en-wikipedia-org-3679	228	9	-	-	HYPH
en-wikipedia-org-3679	228	10	ShareAlike	ShareAlike	NNP
en-wikipedia-org-3679	228	11	License	License	NNP
en-wikipedia-org-3679	228	12	;	;	,
en-wikipedia-org-3679	228	13	additional	additional	JJ
en-wikipedia-org-3679	228	14	terms	term	NNS
en-wikipedia-org-3679	228	15	may	may	MD
en-wikipedia-org-3679	228	16	apply	apply	VB
en-wikipedia-org-3679	228	17	.	.	.
en-wikipedia-org-3679	229	1	By	by	IN
en-wikipedia-org-3679	229	2	using	use	VBG
en-wikipedia-org-3679	229	3	this	this	DT
en-wikipedia-org-3679	229	4	site	site	NN
en-wikipedia-org-3679	229	5	,	,	,
en-wikipedia-org-3679	229	6	you	-PRON-	PRP
en-wikipedia-org-3679	229	7	agree	agree	VBP
en-wikipedia-org-3679	229	8	to	to	IN
en-wikipedia-org-3679	229	9	the	the	DT
en-wikipedia-org-3679	229	10	Terms	term	NNS
en-wikipedia-org-3679	229	11	of	of	IN
en-wikipedia-org-3679	229	12	Use	Use	NNP
en-wikipedia-org-3679	229	13	and	and	CC
en-wikipedia-org-3679	229	14	Privacy	Privacy	NNP
en-wikipedia-org-3679	229	15	Policy	Policy	NNP
en-wikipedia-org-3679	229	16	.	.	.
en-wikipedia-org-3679	230	1	Wikipedia	Wikipedia	NNP
en-wikipedia-org-3679	230	2	®	®	NNPS
en-wikipedia-org-3679	230	3	is	be	VBZ
en-wikipedia-org-3679	230	4	a	a	DT
en-wikipedia-org-3679	230	5	registered	registered	JJ
en-wikipedia-org-3679	230	6	trademark	trademark	NN
en-wikipedia-org-3679	230	7	of	of	IN
en-wikipedia-org-3679	230	8	the	the	DT
en-wikipedia-org-3679	230	9	Wikimedia	Wikimedia	NNP
en-wikipedia-org-3679	230	10	Foundation	Foundation	NNP
en-wikipedia-org-3679	230	11	,	,	,
en-wikipedia-org-3679	230	12	Inc.	Inc.	NNP
en-wikipedia-org-3679	230	13	,	,	,
en-wikipedia-org-3679	230	14	a	a	DT
en-wikipedia-org-3679	230	15	non	non	JJ
en-wikipedia-org-3679	230	16	-	-	JJ
en-wikipedia-org-3679	230	17	profit	profit	JJ
en-wikipedia-org-3679	230	18	organization	organization	NN
en-wikipedia-org-3679	230	19	.	.	.
en-wikipedia-org-3679	231	1	Privacy	privacy	NN
en-wikipedia-org-3679	231	2	policy	policy	NN
en-wikipedia-org-3679	231	3	About	about	IN
en-wikipedia-org-3679	231	4	Wikipedia	Wikipedia	NNP
en-wikipedia-org-3679	231	5	Disclaimers	Disclaimers	NNPS
en-wikipedia-org-3679	231	6	Contact	Contact	NNP
en-wikipedia-org-3679	231	7	Wikipedia	Wikipedia	NNP
en-wikipedia-org-3679	231	8	Mobile	Mobile	NNP
en-wikipedia-org-3679	231	9	view	view	NN
en-wikipedia-org-3679	231	10	Developers	Developers	NNP
en-wikipedia-org-3679	231	11	Statistics	Statistics	NNP
en-wikipedia-org-3679	231	12	Cookie	Cookie	NNP
en-wikipedia-org-3679	231	13	statement	statement	NN