id	sid	tid	token	lemma	pos
www-robotstxt-org-3124	1	1	The	the	DT
www-robotstxt-org-3124	1	2	Web	web	NN
www-robotstxt-org-3124	1	3	Robots	Robots	NNPS
www-robotstxt-org-3124	1	4	Pages	Pages	NNP
www-robotstxt-org-3124	1	5	skip	skip	VBP
www-robotstxt-org-3124	1	6	to	to	TO
www-robotstxt-org-3124	1	7	content	content	VB
www-robotstxt-org-3124	1	8	Advertisement	Advertisement	NNP
www-robotstxt-org-3124	1	9	Navigation	Navigation	NNP
www-robotstxt-org-3124	1	10	The	the	DT
www-robotstxt-org-3124	1	11	/robots.txt	/robots.txt	.
www-robotstxt-org-3124	1	12	<	<	XX
www-robotstxt-org-3124	1	13	META	META	NNP
www-robotstxt-org-3124	1	14	>	>	NN
www-robotstxt-org-3124	1	15	tags	tag	NNS
www-robotstxt-org-3124	1	16	Frequently	frequently	RB
www-robotstxt-org-3124	1	17	Asked	ask	VBD
www-robotstxt-org-3124	1	18	Questions	Questions	NNPS
www-robotstxt-org-3124	1	19	Mailing	Mailing	NNP
www-robotstxt-org-3124	1	20	list	list	NN
www-robotstxt-org-3124	1	21	Other	other	JJ
www-robotstxt-org-3124	1	22	Sites	site	NNS
www-robotstxt-org-3124	1	23	About	about	IN
www-robotstxt-org-3124	1	24	robotstxt.org	robotstxt.org	NNP
www-robotstxt-org-3124	1	25	Tools	tool	NNS
www-robotstxt-org-3124	1	26	/robots.txt	/robots.txt	SYM
www-robotstxt-org-3124	1	27	checker	checker	NN
www-robotstxt-org-3124	1	28	Robots	Robots	NNPS
www-robotstxt-org-3124	1	29	Database	Database	NNP
www-robotstxt-org-3124	1	30	IP	IP	NNP
www-robotstxt-org-3124	1	31	lookup	lookup	NN
www-robotstxt-org-3124	1	32	Advertisement	advertisement	NN
www-robotstxt-org-3124	1	33	The	the	DT
www-robotstxt-org-3124	1	34	Web	web	NN
www-robotstxt-org-3124	1	35	Robots	Robots	NNPS
www-robotstxt-org-3124	1	36	Pages	Pages	NNPS
www-robotstxt-org-3124	1	37	Web	web	NN
www-robotstxt-org-3124	1	38	Robots	Robots	NNPS
www-robotstxt-org-3124	1	39	(	(	-LRB-
www-robotstxt-org-3124	1	40	also	also	RB
www-robotstxt-org-3124	1	41	known	know	VBN
www-robotstxt-org-3124	1	42	as	as	IN
www-robotstxt-org-3124	1	43	Web	Web	NNP
www-robotstxt-org-3124	1	44	Wanderers	Wanderers	NNPS
www-robotstxt-org-3124	1	45	,	,	,
www-robotstxt-org-3124	1	46	Crawlers	Crawlers	NNPS
www-robotstxt-org-3124	1	47	,	,	,
www-robotstxt-org-3124	1	48	or	or	CC
www-robotstxt-org-3124	1	49	Spiders	Spiders	NNPS
www-robotstxt-org-3124	1	50	)	)	-RRB-
www-robotstxt-org-3124	1	51	,	,	,
www-robotstxt-org-3124	1	52	are	be	VBP
www-robotstxt-org-3124	1	53	programs	program	NNS
www-robotstxt-org-3124	1	54	that	that	WDT
www-robotstxt-org-3124	1	55	traverse	traverse	VBP
www-robotstxt-org-3124	1	56	the	the	DT
www-robotstxt-org-3124	1	57	Web	web	NN
www-robotstxt-org-3124	1	58	automatically	automatically	RB
www-robotstxt-org-3124	1	59	.	.	.
www-robotstxt-org-3124	2	1	Search	search	NN
www-robotstxt-org-3124	2	2	engines	engine	NNS
www-robotstxt-org-3124	2	3	such	such	JJ
www-robotstxt-org-3124	2	4	as	as	IN
www-robotstxt-org-3124	2	5	Google	Google	NNP
www-robotstxt-org-3124	2	6	use	use	VBP
www-robotstxt-org-3124	2	7	them	-PRON-	PRP
www-robotstxt-org-3124	2	8	to	to	TO
www-robotstxt-org-3124	2	9	index	index	VB
www-robotstxt-org-3124	2	10	the	the	DT
www-robotstxt-org-3124	2	11	web	web	NN
www-robotstxt-org-3124	2	12	content	content	NN
www-robotstxt-org-3124	2	13	,	,	,
www-robotstxt-org-3124	2	14	spammers	spammer	NNS
www-robotstxt-org-3124	2	15	use	use	VBP
www-robotstxt-org-3124	2	16	them	-PRON-	PRP
www-robotstxt-org-3124	2	17	to	to	TO
www-robotstxt-org-3124	2	18	scan	scan	VB
www-robotstxt-org-3124	2	19	for	for	IN
www-robotstxt-org-3124	2	20	email	email	NN
www-robotstxt-org-3124	2	21	addresses	address	NNS
www-robotstxt-org-3124	2	22	,	,	,
www-robotstxt-org-3124	2	23	and	and	CC
www-robotstxt-org-3124	2	24	they	-PRON-	PRP
www-robotstxt-org-3124	2	25	have	have	VBP
www-robotstxt-org-3124	2	26	many	many	JJ
www-robotstxt-org-3124	2	27	other	other	JJ
www-robotstxt-org-3124	2	28	uses	use	NNS
www-robotstxt-org-3124	2	29	.	.	.
www-robotstxt-org-3124	3	1	On	on	IN
www-robotstxt-org-3124	3	2	this	this	DT
www-robotstxt-org-3124	3	3	site	site	NN
www-robotstxt-org-3124	3	4	you	-PRON-	PRP
www-robotstxt-org-3124	3	5	can	can	MD
www-robotstxt-org-3124	3	6	learn	learn	VB
www-robotstxt-org-3124	3	7	more	more	JJR
www-robotstxt-org-3124	3	8	about	about	IN
www-robotstxt-org-3124	3	9	web	web	NN
www-robotstxt-org-3124	3	10	robots	robot	NNS
www-robotstxt-org-3124	3	11	.	.	.
www-robotstxt-org-3124	4	1	About	about	RB
www-robotstxt-org-3124	4	2	/robots.txt	/robots.txt	NFP
www-robotstxt-org-3124	4	3	explains	explain	VBZ
www-robotstxt-org-3124	4	4	what	what	WP
www-robotstxt-org-3124	4	5	/robots.txt	/robots.txt	NFP
www-robotstxt-org-3124	4	6	is	be	VBZ
www-robotstxt-org-3124	4	7	,	,	,
www-robotstxt-org-3124	4	8	and	and	CC
www-robotstxt-org-3124	4	9	how	how	WRB
www-robotstxt-org-3124	4	10	to	to	TO
www-robotstxt-org-3124	4	11	use	use	VB
www-robotstxt-org-3124	4	12	it	-PRON-	PRP
www-robotstxt-org-3124	4	13	.	.	.
www-robotstxt-org-3124	5	1	The	the	DT
www-robotstxt-org-3124	5	2	FAQ	FAQ	NNP
www-robotstxt-org-3124	5	3	answers	answer	VBZ
www-robotstxt-org-3124	5	4	many	many	JJ
www-robotstxt-org-3124	5	5	frequently	frequently	RB
www-robotstxt-org-3124	5	6	asked	ask	VBN
www-robotstxt-org-3124	5	7	questions	question	NNS
www-robotstxt-org-3124	5	8	,	,	,
www-robotstxt-org-3124	5	9	such	such	JJ
www-robotstxt-org-3124	5	10	as	as	IN
www-robotstxt-org-3124	5	11	How	how	WRB
www-robotstxt-org-3124	5	12	do	do	VBP
www-robotstxt-org-3124	5	13	I	-PRON-	PRP
www-robotstxt-org-3124	5	14	stop	stop	VB
www-robotstxt-org-3124	5	15	robots	robot	NNS
www-robotstxt-org-3124	5	16	visiting	visit	VBG
www-robotstxt-org-3124	5	17	my	-PRON-	PRP$
www-robotstxt-org-3124	5	18	site	site	NN
www-robotstxt-org-3124	5	19	?	?	.
www-robotstxt-org-3124	6	1	and	and	CC
www-robotstxt-org-3124	6	2	How	how	WRB
www-robotstxt-org-3124	6	3	can	can	MD
www-robotstxt-org-3124	6	4	I	-PRON-	PRP
www-robotstxt-org-3124	6	5	get	get	VB
www-robotstxt-org-3124	6	6	the	the	DT
www-robotstxt-org-3124	6	7	best	good	JJS
www-robotstxt-org-3124	6	8	listing	listing	NN
www-robotstxt-org-3124	6	9	in	in	IN
www-robotstxt-org-3124	6	10	search	search	NN
www-robotstxt-org-3124	6	11	engines	engine	NNS
www-robotstxt-org-3124	6	12	?	?	.
www-robotstxt-org-3124	6	13	"	"	''
www-robotstxt-org-3124	7	1	The	the	DT
www-robotstxt-org-3124	7	2	Other	other	JJ
www-robotstxt-org-3124	7	3	Sites	Sites	NNPS
www-robotstxt-org-3124	7	4	page	page	NN
www-robotstxt-org-3124	7	5	links	link	NNS
www-robotstxt-org-3124	7	6	to	to	IN
www-robotstxt-org-3124	7	7	external	external	JJ
www-robotstxt-org-3124	7	8	resources	resource	NNS
www-robotstxt-org-3124	7	9	for	for	IN
www-robotstxt-org-3124	7	10	robot	robot	NN
www-robotstxt-org-3124	7	11	writers	writer	NNS
www-robotstxt-org-3124	7	12	and	and	CC
www-robotstxt-org-3124	7	13	webmasters	webmaster	NNS
www-robotstxt-org-3124	7	14	.	.	.
www-robotstxt-org-3124	8	1	The	the	DT
www-robotstxt-org-3124	8	2	Robots	Robots	NNP
www-robotstxt-org-3124	8	3	Database	Database	NNP
www-robotstxt-org-3124	8	4	has	have	VBZ
www-robotstxt-org-3124	8	5	a	a	DT
www-robotstxt-org-3124	8	6	list	list	NN
www-robotstxt-org-3124	8	7	of	of	IN
www-robotstxt-org-3124	8	8	robots	robot	NNS
www-robotstxt-org-3124	8	9	.	.	.
www-robotstxt-org-3124	9	1	The	the	DT
www-robotstxt-org-3124	9	2	/robots.txt	/robots.txt	NFP
www-robotstxt-org-3124	9	3	checker	checker	NN
www-robotstxt-org-3124	9	4	can	can	MD
www-robotstxt-org-3124	9	5	check	check	VB
www-robotstxt-org-3124	9	6	your	-PRON-	PRP$
www-robotstxt-org-3124	9	7	site	site	NN
www-robotstxt-org-3124	9	8	's	's	POS
www-robotstxt-org-3124	9	9	/robots.txt	/robots.txt	SYM
www-robotstxt-org-3124	9	10	file	file	NN
www-robotstxt-org-3124	9	11	and	and	CC
www-robotstxt-org-3124	9	12	meta	meta	JJ
www-robotstxt-org-3124	9	13	tags	tag	NNS
www-robotstxt-org-3124	9	14	.	.	.
www-robotstxt-org-3124	10	1	The	the	DT
www-robotstxt-org-3124	10	2	IP	IP	NNP
www-robotstxt-org-3124	10	3	Lookup	Lookup	NNP
www-robotstxt-org-3124	10	4	can	can	MD
www-robotstxt-org-3124	10	5	help	help	VB
www-robotstxt-org-3124	10	6	find	find	VB
www-robotstxt-org-3124	10	7	out	out	RP
www-robotstxt-org-3124	10	8	more	more	JJR
www-robotstxt-org-3124	10	9	about	about	IN
www-robotstxt-org-3124	10	10	what	what	WP
www-robotstxt-org-3124	10	11	robots	robot	NNS
www-robotstxt-org-3124	10	12	are	be	VBP
www-robotstxt-org-3124	10	13	visiting	visit	VBG
www-robotstxt-org-3124	10	14	you	-PRON-	PRP
www-robotstxt-org-3124	10	15	.	.	.
www-robotstxt-org-3124	11	1	Advertisement	advertisement	NN
www-robotstxt-org-3124	11	2	About	about	IN
www-robotstxt-org-3124	11	3	this	this	DT
www-robotstxt-org-3124	11	4	site	site	NN
www-robotstxt-org-3124	11	5	|	|	NNP
www-robotstxt-org-3124	11	6	Privacy	privacy	NN
www-robotstxt-org-3124	11	7	and	and	CC
www-robotstxt-org-3124	11	8	cookies	cookie	VBZ
www-robotstxt-org-3124	11	9	policy	policy	NN
www-robotstxt-org-3124	11	10	|	|	NNS
www-robotstxt-org-3124	11	11	Contact	Contact	NNP
www-robotstxt-org-3124	11	12	us	-PRON-	PRP
www-robotstxt-org-3124	11	13	|	|	CD
www-robotstxt-org-3124	11	14	©	©	NNP
www-robotstxt-org-3124	11	15	2007	2007	CD
www-robotstxt-org-3124	11	16	.	.	.
www-robotstxt-org-3124	12	1	All	all	DT
www-robotstxt-org-3124	12	2	rights	right	NNS
www-robotstxt-org-3124	12	3	reserved	reserve	VBD
www-robotstxt-org-3124	12	4	.	.	.
www-robotstxt-org-3124	13	1	|	|	NNS
www-robotstxt-org-3124	13	2	Hosted	host	VBN
www-robotstxt-org-3124	13	3	by	by	IN
www-robotstxt-org-3124	13	4	Mythic	Mythic	NNP
www-robotstxt-org-3124	13	5	Beasts	Beasts	NNPS