id	sid	tid	token	lemma	pos
clgiles-ist-psu-edu-8875	1	1	Measuring	measure	VBG
clgiles-ist-psu-edu-8875	1	2	the	the	DT
clgiles-ist-psu-edu-8875	1	3	web	web	NN
clgiles-ist-psu-edu-8875	1	4	crawler	crawler	NN
clgiles-ist-psu-edu-8875	1	5	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	1	6	Measuring	measure	VBG
clgiles-ist-psu-edu-8875	1	7	The	the	DT
clgiles-ist-psu-edu-8875	1	8	Web	web	NN
clgiles-ist-psu-edu-8875	1	9	Crawler	Crawler	NNP
clgiles-ist-psu-edu-8875	1	10	Ethics	Ethics	NNP
clgiles-ist-psu-edu-8875	1	11	C.	C.	NNP
clgiles-ist-psu-edu-8875	1	12	Lee	Lee	NNP
clgiles-ist-psu-edu-8875	1	13	Giles	Giles	NNP
clgiles-ist-psu-edu-8875	1	14	College	College	NNP
clgiles-ist-psu-edu-8875	1	15	of	of	IN
clgiles-ist-psu-edu-8875	1	16	Information	Information	NNP
clgiles-ist-psu-edu-8875	1	17	Sciences	Sciences	NNPS
clgiles-ist-psu-edu-8875	1	18	and	and	CC
clgiles-ist-psu-edu-8875	1	19	Technology	Technology	NNP
clgiles-ist-psu-edu-8875	1	20	Pennsylvania	Pennsylvania	NNP
clgiles-ist-psu-edu-8875	1	21	State	State	NNP
clgiles-ist-psu-edu-8875	1	22	University	University	NNP
clgiles-ist-psu-edu-8875	1	23	University	University	NNP
clgiles-ist-psu-edu-8875	1	24	Park	Park	NNP
clgiles-ist-psu-edu-8875	1	25	,	,	,
clgiles-ist-psu-edu-8875	1	26	PA	PA	NNP
clgiles-ist-psu-edu-8875	1	27	,	,	,
clgiles-ist-psu-edu-8875	1	28	USA	USA	NNP
clgiles-ist-psu-edu-8875	1	29	giles@ist.psu.edu	giles@ist.psu.edu	NNP
clgiles-ist-psu-edu-8875	1	30	Yang	Yang	NNP
clgiles-ist-psu-edu-8875	1	31	Sun	Sun	NNP
clgiles-ist-psu-edu-8875	1	32	AOL	AOL	NNP
clgiles-ist-psu-edu-8875	1	33	Research	Research	NNP
clgiles-ist-psu-edu-8875	1	34	888	888	CD
clgiles-ist-psu-edu-8875	1	35	Villa	Villa	NNP
clgiles-ist-psu-edu-8875	1	36	St	St	NNP
clgiles-ist-psu-edu-8875	1	37	Mountain	Mountain	NNP
clgiles-ist-psu-edu-8875	1	38	View	View	NNP
clgiles-ist-psu-edu-8875	1	39	,	,	,
clgiles-ist-psu-edu-8875	1	40	CA	CA	NNP
clgiles-ist-psu-edu-8875	1	41	,	,	,
clgiles-ist-psu-edu-8875	1	42	USA	USA	NNP
clgiles-ist-psu-edu-8875	1	43	yang.sun@corp.aol.com	yang.sun@corp.aol.com	NNP
clgiles-ist-psu-edu-8875	1	44	Isaac	Isaac	NNP
clgiles-ist-psu-edu-8875	1	45	G.	G.	NNP
clgiles-ist-psu-edu-8875	1	46	Councill	Councill	NNP
clgiles-ist-psu-edu-8875	1	47	Google	Google	NNP
clgiles-ist-psu-edu-8875	1	48	Inc.	Inc.	NNP
clgiles-ist-psu-edu-8875	1	49	76	76	CD
clgiles-ist-psu-edu-8875	1	50	Ninth	Ninth	NNP
clgiles-ist-psu-edu-8875	1	51	Avenue	Avenue	NNP
clgiles-ist-psu-edu-8875	1	52	4th	4th	NN
clgiles-ist-psu-edu-8875	1	53	Floor	Floor	NNP
clgiles-ist-psu-edu-8875	1	54	New	New	NNP
clgiles-ist-psu-edu-8875	1	55	York	York	NNP
clgiles-ist-psu-edu-8875	1	56	,	,	,
clgiles-ist-psu-edu-8875	1	57	NY	NY	NNP
clgiles-ist-psu-edu-8875	1	58	,	,	,
clgiles-ist-psu-edu-8875	1	59	USA	USA	NNP
clgiles-ist-psu-edu-8875	1	60	icouncill@gmail.com	icouncill@gmail.com	ADD
clgiles-ist-psu-edu-8875	1	61	ABSTRACT	ABSTRACT	NNP
clgiles-ist-psu-edu-8875	1	62	Web	web	NN
clgiles-ist-psu-edu-8875	1	63	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	1	64	are	be	VBP
clgiles-ist-psu-edu-8875	1	65	highly	highly	RB
clgiles-ist-psu-edu-8875	1	66	automated	automate	VBN
clgiles-ist-psu-edu-8875	1	67	and	and	CC
clgiles-ist-psu-edu-8875	1	68	seldom	seldom	RB
clgiles-ist-psu-edu-8875	1	69	regulated	regulate	VBD
clgiles-ist-psu-edu-8875	1	70	manually	manually	RB
clgiles-ist-psu-edu-8875	1	71	.	.	.
clgiles-ist-psu-edu-8875	2	1	The	the	DT
clgiles-ist-psu-edu-8875	2	2	diversity	diversity	NN
clgiles-ist-psu-edu-8875	2	3	of	of	IN
clgiles-ist-psu-edu-8875	2	4	crawler	crawler	NN
clgiles-ist-psu-edu-8875	2	5	activities	activity	NNS
clgiles-ist-psu-edu-8875	2	6	often	often	RB
clgiles-ist-psu-edu-8875	2	7	leads	lead	VBZ
clgiles-ist-psu-edu-8875	2	8	to	to	IN
clgiles-ist-psu-edu-8875	2	9	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	2	10	problems	problem	NNS
clgiles-ist-psu-edu-8875	2	11	such	such	JJ
clgiles-ist-psu-edu-8875	2	12	as	as	IN
clgiles-ist-psu-edu-8875	2	13	spam	spam	NN
clgiles-ist-psu-edu-8875	2	14	and	and	CC
clgiles-ist-psu-edu-8875	2	15	service	service	NN
clgiles-ist-psu-edu-8875	2	16	attacks	attack	NNS
clgiles-ist-psu-edu-8875	2	17	.	.	.
clgiles-ist-psu-edu-8875	3	1	In	in	IN
clgiles-ist-psu-edu-8875	3	2	this	this	DT
clgiles-ist-psu-edu-8875	3	3	research	research	NN
clgiles-ist-psu-edu-8875	3	4	,	,	,
clgiles-ist-psu-edu-8875	3	5	quantitative	quantitative	JJ
clgiles-ist-psu-edu-8875	3	6	models	model	NNS
clgiles-ist-psu-edu-8875	3	7	are	be	VBP
clgiles-ist-psu-edu-8875	3	8	proposed	propose	VBN
clgiles-ist-psu-edu-8875	3	9	to	to	TO
clgiles-ist-psu-edu-8875	3	10	measure	measure	VB
clgiles-ist-psu-edu-8875	3	11	the	the	DT
clgiles-ist-psu-edu-8875	3	12	web	web	NN
clgiles-ist-psu-edu-8875	3	13	crawler	crawler	NN
clgiles-ist-psu-edu-8875	3	14	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	3	15	based	base	VBN
clgiles-ist-psu-edu-8875	3	16	on	on	IN
clgiles-ist-psu-edu-8875	3	17	their	-PRON-	PRP$
clgiles-ist-psu-edu-8875	3	18	behaviors	behavior	NNS
clgiles-ist-psu-edu-8875	3	19	on	on	IN
clgiles-ist-psu-edu-8875	3	20	web	web	NN
clgiles-ist-psu-edu-8875	3	21	servers	server	NNS
clgiles-ist-psu-edu-8875	3	22	.	.	.
clgiles-ist-psu-edu-8875	4	1	We	-PRON-	PRP
clgiles-ist-psu-edu-8875	4	2	investigate	investigate	VBP
clgiles-ist-psu-edu-8875	4	3	and	and	CC
clgiles-ist-psu-edu-8875	4	4	define	define	VB
clgiles-ist-psu-edu-8875	4	5	rules	rule	NNS
clgiles-ist-psu-edu-8875	4	6	to	to	TO
clgiles-ist-psu-edu-8875	4	7	measure	measure	VB
clgiles-ist-psu-edu-8875	4	8	crawler	crawler	JJ
clgiles-ist-psu-edu-8875	4	9	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	4	10	,	,	,
clgiles-ist-psu-edu-8875	4	11	referring	refer	VBG
clgiles-ist-psu-edu-8875	4	12	to	to	IN
clgiles-ist-psu-edu-8875	4	13	the	the	DT
clgiles-ist-psu-edu-8875	4	14	extent	extent	NN
clgiles-ist-psu-edu-8875	4	15	to	to	TO
clgiles-ist-psu-edu-8875	4	16	which	which	WDT
clgiles-ist-psu-edu-8875	4	17	web	web	NN
clgiles-ist-psu-edu-8875	4	18	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	4	19	respect	respect	VBP
clgiles-ist-psu-edu-8875	4	20	the	the	DT
clgiles-ist-psu-edu-8875	4	21	regulations	regulation	NNS
clgiles-ist-psu-edu-8875	4	22	set	set	VBN
clgiles-ist-psu-edu-8875	4	23	forth	forth	RP
clgiles-ist-psu-edu-8875	4	24	in	in	IN
clgiles-ist-psu-edu-8875	4	25	robots.txt	robots.txt	CD
clgiles-ist-psu-edu-8875	4	26	configuration	configuration	NN
clgiles-ist-psu-edu-8875	4	27	files	file	NNS
clgiles-ist-psu-edu-8875	4	28	.	.	.
clgiles-ist-psu-edu-8875	5	1	We	-PRON-	PRP
clgiles-ist-psu-edu-8875	5	2	propose	propose	VBP
clgiles-ist-psu-edu-8875	5	3	a	a	DT
clgiles-ist-psu-edu-8875	5	4	vector	vector	NN
clgiles-ist-psu-edu-8875	5	5	space	space	NN
clgiles-ist-psu-edu-8875	5	6	model	model	NN
clgiles-ist-psu-edu-8875	5	7	to	to	TO
clgiles-ist-psu-edu-8875	5	8	represent	represent	VB
clgiles-ist-psu-edu-8875	5	9	crawler	crawler	NNP
clgiles-ist-psu-edu-8875	5	10	behav-	behav-	NNP
clgiles-ist-psu-edu-8875	5	11	ior	ior	NNP
clgiles-ist-psu-edu-8875	5	12	and	and	CC
clgiles-ist-psu-edu-8875	5	13	measure	measure	VB
clgiles-ist-psu-edu-8875	5	14	the	the	DT
clgiles-ist-psu-edu-8875	5	15	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	5	16	of	of	IN
clgiles-ist-psu-edu-8875	5	17	web	web	NN
clgiles-ist-psu-edu-8875	5	18	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	5	19	based	base	VBN
clgiles-ist-psu-edu-8875	5	20	on	on	IN
clgiles-ist-psu-edu-8875	5	21	the	the	DT
clgiles-ist-psu-edu-8875	5	22	behavior	behavior	NN
clgiles-ist-psu-edu-8875	5	23	vectors	vector	NNS
clgiles-ist-psu-edu-8875	5	24	.	.	.
clgiles-ist-psu-edu-8875	6	1	The	the	DT
clgiles-ist-psu-edu-8875	6	2	results	result	NNS
clgiles-ist-psu-edu-8875	6	3	show	show	VBP
clgiles-ist-psu-edu-8875	6	4	that	that	IN
clgiles-ist-psu-edu-8875	6	5	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	6	6	scores	score	NNS
clgiles-ist-psu-edu-8875	6	7	vary	vary	VBP
clgiles-ist-psu-edu-8875	6	8	significantly	significantly	RB
clgiles-ist-psu-edu-8875	6	9	among	among	IN
clgiles-ist-psu-edu-8875	6	10	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	6	11	.	.	.
clgiles-ist-psu-edu-8875	7	1	Most	Most	JJS
clgiles-ist-psu-edu-8875	7	2	commercial	commercial	JJ
clgiles-ist-psu-edu-8875	7	3	web	web	NN
clgiles-ist-psu-edu-8875	7	4	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	7	5	’	’	POS
clgiles-ist-psu-edu-8875	7	6	behaviors	behavior	NNS
clgiles-ist-psu-edu-8875	7	7	are	be	VBP
clgiles-ist-psu-edu-8875	7	8	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	7	9	.	.	.
clgiles-ist-psu-edu-8875	8	1	However	however	RB
clgiles-ist-psu-edu-8875	8	2	,	,	,
clgiles-ist-psu-edu-8875	8	3	many	many	JJ
clgiles-ist-psu-edu-8875	8	4	commer-	commer-	JJ
clgiles-ist-psu-edu-8875	8	5	cial	cial	JJ
clgiles-ist-psu-edu-8875	8	6	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	8	7	still	still	RB
clgiles-ist-psu-edu-8875	8	8	consistently	consistently	RB
clgiles-ist-psu-edu-8875	8	9	violate	violate	VBP
clgiles-ist-psu-edu-8875	8	10	or	or	CC
clgiles-ist-psu-edu-8875	8	11	misinterpret	misinterpret	VBP
clgiles-ist-psu-edu-8875	8	12	certain	certain	JJ
clgiles-ist-psu-edu-8875	8	13	robots.txt	robots.txt	NN
clgiles-ist-psu-edu-8875	8	14	rules	rule	NNS
clgiles-ist-psu-edu-8875	8	15	.	.	.
clgiles-ist-psu-edu-8875	9	1	We	-PRON-	PRP
clgiles-ist-psu-edu-8875	9	2	also	also	RB
clgiles-ist-psu-edu-8875	9	3	measure	measure	VBP
clgiles-ist-psu-edu-8875	9	4	the	the	DT
clgiles-ist-psu-edu-8875	9	5	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	9	6	of	of	IN
clgiles-ist-psu-edu-8875	9	7	big	big	JJ
clgiles-ist-psu-edu-8875	9	8	search	search	NN
clgiles-ist-psu-edu-8875	9	9	engine	engine	NN
clgiles-ist-psu-edu-8875	9	10	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	9	11	in	in	IN
clgiles-ist-psu-edu-8875	9	12	terms	term	NNS
clgiles-ist-psu-edu-8875	9	13	of	of	IN
clgiles-ist-psu-edu-8875	9	14	return	return	NN
clgiles-ist-psu-edu-8875	9	15	on	on	IN
clgiles-ist-psu-edu-8875	9	16	investment	investment	NN
clgiles-ist-psu-edu-8875	9	17	.	.	.
clgiles-ist-psu-edu-8875	10	1	The	the	DT
clgiles-ist-psu-edu-8875	10	2	re-	re-	JJ
clgiles-ist-psu-edu-8875	10	3	sults	sult	NNS
clgiles-ist-psu-edu-8875	10	4	show	show	VBP
clgiles-ist-psu-edu-8875	10	5	that	that	IN
clgiles-ist-psu-edu-8875	10	6	Google	Google	NNP
clgiles-ist-psu-edu-8875	10	7	has	have	VBZ
clgiles-ist-psu-edu-8875	10	8	a	a	DT
clgiles-ist-psu-edu-8875	10	9	higher	high	JJR
clgiles-ist-psu-edu-8875	10	10	score	score	NN
clgiles-ist-psu-edu-8875	10	11	than	than	IN
clgiles-ist-psu-edu-8875	10	12	other	other	JJ
clgiles-ist-psu-edu-8875	10	13	search	search	NN
clgiles-ist-psu-edu-8875	10	14	engines	engine	NNS
clgiles-ist-psu-edu-8875	10	15	for	for	IN
clgiles-ist-psu-edu-8875	10	16	a	a	DT
clgiles-ist-psu-edu-8875	10	17	US	US	NNP
clgiles-ist-psu-edu-8875	10	18	website	website	NN
clgiles-ist-psu-edu-8875	10	19	but	but	CC
clgiles-ist-psu-edu-8875	10	20	has	have	VBZ
clgiles-ist-psu-edu-8875	10	21	a	a	DT
clgiles-ist-psu-edu-8875	10	22	lower	low	JJR
clgiles-ist-psu-edu-8875	10	23	score	score	NN
clgiles-ist-psu-edu-8875	10	24	than	than	IN
clgiles-ist-psu-edu-8875	10	25	Baidu	Baidu	NNP
clgiles-ist-psu-edu-8875	10	26	for	for	IN
clgiles-ist-psu-edu-8875	10	27	Chinese	chinese	JJ
clgiles-ist-psu-edu-8875	10	28	websites	website	NNS
clgiles-ist-psu-edu-8875	10	29	.	.	.
clgiles-ist-psu-edu-8875	11	1	Categories	category	NNS
clgiles-ist-psu-edu-8875	11	2	and	and	CC
clgiles-ist-psu-edu-8875	11	3	Subject	Subject	NNP
clgiles-ist-psu-edu-8875	11	4	Descriptors	descriptor	NNS
clgiles-ist-psu-edu-8875	11	5	K.4.1	K.4.1	NNP
clgiles-ist-psu-edu-8875	11	6	[	[	-LRB-
clgiles-ist-psu-edu-8875	11	7	Public	Public	NNP
clgiles-ist-psu-edu-8875	11	8	Policy	Policy	NNP
clgiles-ist-psu-edu-8875	11	9	Issues	Issues	NNPS
clgiles-ist-psu-edu-8875	11	10	]	]	-RRB-
clgiles-ist-psu-edu-8875	11	11	:	:	:
clgiles-ist-psu-edu-8875	11	12	Ethics	ethic	NNS
clgiles-ist-psu-edu-8875	11	13	;	;	:
clgiles-ist-psu-edu-8875	11	14	K.4.1	K.4.1	NNP
clgiles-ist-psu-edu-8875	11	15	[	[	-LRB-
clgiles-ist-psu-edu-8875	11	16	Public	public	JJ
clgiles-ist-psu-edu-8875	11	17	Pol-	Pol-	NNP
clgiles-ist-psu-edu-8875	11	18	icy	icy	NN
clgiles-ist-psu-edu-8875	11	19	Issues	issue	NNS
clgiles-ist-psu-edu-8875	11	20	]	]	-RRB-
clgiles-ist-psu-edu-8875	11	21	:	:	:
clgiles-ist-psu-edu-8875	11	22	Privacy	Privacy	NNP
clgiles-ist-psu-edu-8875	11	23	General	General	NNP
clgiles-ist-psu-edu-8875	11	24	Terms	Terms	NNPS
clgiles-ist-psu-edu-8875	11	25	Measurement	Measurement	NNP
clgiles-ist-psu-edu-8875	11	26	,	,	,
clgiles-ist-psu-edu-8875	11	27	Design	Design	NNP
clgiles-ist-psu-edu-8875	11	28	,	,	,
clgiles-ist-psu-edu-8875	11	29	Experimentation	experimentation	NN
clgiles-ist-psu-edu-8875	11	30	,	,	,
clgiles-ist-psu-edu-8875	11	31	Algorithms	Algorithms	NNP
clgiles-ist-psu-edu-8875	11	32	Keywords	Keywords	NNPS
clgiles-ist-psu-edu-8875	11	33	robots.txt	robots.txt	NNP
clgiles-ist-psu-edu-8875	11	34	,	,	,
clgiles-ist-psu-edu-8875	11	35	web	web	NN
clgiles-ist-psu-edu-8875	11	36	crawler	crawler	NN
clgiles-ist-psu-edu-8875	11	37	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	11	38	,	,	,
clgiles-ist-psu-edu-8875	11	39	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	11	40	,	,	,
clgiles-ist-psu-edu-8875	11	41	privacy	privacy	NN
clgiles-ist-psu-edu-8875	11	42	1	1	CD
clgiles-ist-psu-edu-8875	11	43	.	.	.
clgiles-ist-psu-edu-8875	12	1	INTRODUCTION	introduction	NN
clgiles-ist-psu-edu-8875	12	2	Web	web	NN
clgiles-ist-psu-edu-8875	12	3	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	12	4	have	have	VBP
clgiles-ist-psu-edu-8875	12	5	been	be	VBN
clgiles-ist-psu-edu-8875	12	6	widely	widely	RB
clgiles-ist-psu-edu-8875	12	7	used	use	VBN
clgiles-ist-psu-edu-8875	12	8	for	for	IN
clgiles-ist-psu-edu-8875	12	9	search	search	NN
clgiles-ist-psu-edu-8875	12	10	engines	engine	NNS
clgiles-ist-psu-edu-8875	12	11	as	as	RB
clgiles-ist-psu-edu-8875	12	12	well	well	RB
clgiles-ist-psu-edu-8875	12	13	as	as	IN
clgiles-ist-psu-edu-8875	12	14	many	many	JJ
clgiles-ist-psu-edu-8875	12	15	other	other	JJ
clgiles-ist-psu-edu-8875	12	16	web	web	NN
clgiles-ist-psu-edu-8875	12	17	applications	application	NNS
clgiles-ist-psu-edu-8875	12	18	to	to	TO
clgiles-ist-psu-edu-8875	12	19	collect	collect	VB
clgiles-ist-psu-edu-8875	12	20	content	content	NN
clgiles-ist-psu-edu-8875	12	21	from	from	IN
clgiles-ist-psu-edu-8875	12	22	the	the	DT
clgiles-ist-psu-edu-8875	12	23	Web	web	NN
clgiles-ist-psu-edu-8875	12	24	.	.	.
clgiles-ist-psu-edu-8875	13	1	These	these	DT
clgiles-ist-psu-edu-8875	13	2	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	13	3	are	be	VBP
clgiles-ist-psu-edu-8875	13	4	highly	highly	RB
clgiles-ist-psu-edu-8875	13	5	automated	automate	VBN
clgiles-ist-psu-edu-8875	13	6	and	and	CC
clgiles-ist-psu-edu-8875	13	7	seldom	seldom	RB
clgiles-ist-psu-edu-8875	13	8	regulated	regulate	VBD
clgiles-ist-psu-edu-8875	13	9	manually	manually	RB
clgiles-ist-psu-edu-8875	13	10	.	.	.
clgiles-ist-psu-edu-8875	14	1	With	with	IN
clgiles-ist-psu-edu-8875	14	2	the	the	DT
clgiles-ist-psu-edu-8875	14	3	fast	fast	JJ
clgiles-ist-psu-edu-8875	14	4	growing	grow	VBG
clgiles-ist-psu-edu-8875	14	5	online	online	JJ
clgiles-ist-psu-edu-8875	14	6	services	service	NNS
clgiles-ist-psu-edu-8875	14	7	relying	rely	VBG
clgiles-ist-psu-edu-8875	14	8	on	on	IN
clgiles-ist-psu-edu-8875	14	9	Web	web	NN
clgiles-ist-psu-edu-8875	14	10	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	14	11	to	to	TO
clgiles-ist-psu-edu-8875	14	12	collect	collect	VB
clgiles-ist-psu-edu-8875	14	13	Web	web	NN
clgiles-ist-psu-edu-8875	14	14	pages	page	NNS
clgiles-ist-psu-edu-8875	14	15	,	,	,
clgiles-ist-psu-edu-8875	14	16	the	the	DT
clgiles-ist-psu-edu-8875	14	17	function-	function-	NN
clgiles-ist-psu-edu-8875	14	18	alities	alitie	NNS
clgiles-ist-psu-edu-8875	14	19	and	and	CC
clgiles-ist-psu-edu-8875	14	20	activities	activity	NNS
clgiles-ist-psu-edu-8875	14	21	of	of	IN
clgiles-ist-psu-edu-8875	14	22	web	web	NN
clgiles-ist-psu-edu-8875	14	23	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	14	24	have	have	VBP
clgiles-ist-psu-edu-8875	14	25	become	become	VBN
clgiles-ist-psu-edu-8875	14	26	extremely	extremely	RB
clgiles-ist-psu-edu-8875	14	27	diverse	diverse	JJ
clgiles-ist-psu-edu-8875	14	28	.	.	.
clgiles-ist-psu-edu-8875	15	1	Crawler	Crawler	NNP
clgiles-ist-psu-edu-8875	15	2	activities	activity	NNS
clgiles-ist-psu-edu-8875	15	3	typically	typically	RB
clgiles-ist-psu-edu-8875	15	4	include	include	VBP
clgiles-ist-psu-edu-8875	15	5	requests	request	NNS
clgiles-ist-psu-edu-8875	15	6	of	of	IN
clgiles-ist-psu-edu-8875	15	7	web	web	NN
clgiles-ist-psu-edu-8875	15	8	pages	page	NNS
clgiles-ist-psu-edu-8875	15	9	for	for	IN
clgiles-ist-psu-edu-8875	15	10	general	general	JJ
clgiles-ist-psu-edu-8875	15	11	-	-	HYPH
clgiles-ist-psu-edu-8875	15	12	purpose	purpose	NN
clgiles-ist-psu-edu-8875	15	13	text	text	NN
clgiles-ist-psu-edu-8875	15	14	indexing	indexing	NN
clgiles-ist-psu-edu-8875	15	15	and	and	CC
clgiles-ist-psu-edu-8875	15	16	searching	searching	NN
clgiles-ist-psu-edu-8875	15	17	,	,	,
clgiles-ist-psu-edu-8875	15	18	ex-	ex-	XX
clgiles-ist-psu-edu-8875	15	19	traction	traction	NN
clgiles-ist-psu-edu-8875	15	20	of	of	IN
clgiles-ist-psu-edu-8875	15	21	email	email	NN
clgiles-ist-psu-edu-8875	15	22	and	and	CC
clgiles-ist-psu-edu-8875	15	23	personal	personal	JJ
clgiles-ist-psu-edu-8875	15	24	identity	identity	NN
clgiles-ist-psu-edu-8875	15	25	information	information	NN
clgiles-ist-psu-edu-8875	15	26	for	for	IN
clgiles-ist-psu-edu-8875	15	27	busi-	busi-	JJ
clgiles-ist-psu-edu-8875	15	28	ness	ness	NN
clgiles-ist-psu-edu-8875	15	29	purposes	purpose	NNS
clgiles-ist-psu-edu-8875	15	30	as	as	RB
clgiles-ist-psu-edu-8875	15	31	well	well	RB
clgiles-ist-psu-edu-8875	15	32	as	as	IN
clgiles-ist-psu-edu-8875	15	33	for	for	IN
clgiles-ist-psu-edu-8875	15	34	malicious	malicious	JJ
clgiles-ist-psu-edu-8875	15	35	purposes	purpose	NNS
clgiles-ist-psu-edu-8875	15	36	.	.	.
clgiles-ist-psu-edu-8875	16	1	Accessing	access	VBG
clgiles-ist-psu-edu-8875	16	2	Copyright	Copyright	NNP
clgiles-ist-psu-edu-8875	16	3	is	be	VBZ
clgiles-ist-psu-edu-8875	16	4	held	hold	VBN
clgiles-ist-psu-edu-8875	16	5	by	by	IN
clgiles-ist-psu-edu-8875	16	6	the	the	DT
clgiles-ist-psu-edu-8875	16	7	author	author	NN
clgiles-ist-psu-edu-8875	16	8	/	/	,
clgiles-ist-psu-edu-8875	16	9	owner(s	owner(s	NN
clgiles-ist-psu-edu-8875	16	10	)	)	-RRB-
clgiles-ist-psu-edu-8875	16	11	.	.	.
clgiles-ist-psu-edu-8875	17	1	WWW	WWW	NNP
clgiles-ist-psu-edu-8875	17	2	2010	2010	CD
clgiles-ist-psu-edu-8875	17	3	,	,	,
clgiles-ist-psu-edu-8875	17	4	April	April	NNP
clgiles-ist-psu-edu-8875	17	5	26–30	26–30	CD
clgiles-ist-psu-edu-8875	17	6	,	,	,
clgiles-ist-psu-edu-8875	17	7	2010	2010	CD
clgiles-ist-psu-edu-8875	17	8	,	,	,
clgiles-ist-psu-edu-8875	17	9	Raleigh	Raleigh	NNP
clgiles-ist-psu-edu-8875	17	10	,	,	,
clgiles-ist-psu-edu-8875	17	11	North	North	NNP
clgiles-ist-psu-edu-8875	17	12	Carolina	Carolina	NNP
clgiles-ist-psu-edu-8875	17	13	,	,	,
clgiles-ist-psu-edu-8875	17	14	USA	USA	NNP
clgiles-ist-psu-edu-8875	17	15	.	.	.
clgiles-ist-psu-edu-8875	18	1	ACM	ACM	NNP
clgiles-ist-psu-edu-8875	18	2	978	978	CD
clgiles-ist-psu-edu-8875	18	3	-	-	HYPH
clgiles-ist-psu-edu-8875	18	4	1	1	CD
clgiles-ist-psu-edu-8875	18	5	-	-	HYPH
clgiles-ist-psu-edu-8875	18	6	60558	60558	CD
clgiles-ist-psu-edu-8875	18	7	-	-	HYPH
clgiles-ist-psu-edu-8875	18	8	799	799	CD
clgiles-ist-psu-edu-8875	18	9	-	-	HYPH
clgiles-ist-psu-edu-8875	18	10	8/10/04	8/10/04	CD
clgiles-ist-psu-edu-8875	18	11	.	.	.
clgiles-ist-psu-edu-8875	19	1	the	the	DT
clgiles-ist-psu-edu-8875	19	2	web	web	NN
clgiles-ist-psu-edu-8875	19	3	information	information	NN
clgiles-ist-psu-edu-8875	19	4	with	with	IN
clgiles-ist-psu-edu-8875	19	5	automated	automate	VBN
clgiles-ist-psu-edu-8875	19	6	Web	web	NN
clgiles-ist-psu-edu-8875	19	7	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	19	8	can	can	MD
clgiles-ist-psu-edu-8875	19	9	lead	lead	VB
clgiles-ist-psu-edu-8875	19	10	to	to	IN
clgiles-ist-psu-edu-8875	19	11	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	19	12	problems	problem	NNS
clgiles-ist-psu-edu-8875	19	13	of	of	IN
clgiles-ist-psu-edu-8875	19	14	privacy	privacy	NN
clgiles-ist-psu-edu-8875	19	15	and	and	CC
clgiles-ist-psu-edu-8875	19	16	security	security	NN
clgiles-ist-psu-edu-8875	19	17	.	.	.
clgiles-ist-psu-edu-8875	20	1	For	for	IN
clgiles-ist-psu-edu-8875	20	2	example	example	NN
clgiles-ist-psu-edu-8875	20	3	,	,	,
clgiles-ist-psu-edu-8875	20	4	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	20	5	can	can	MD
clgiles-ist-psu-edu-8875	20	6	extract	extract	VB
clgiles-ist-psu-edu-8875	20	7	personal	personal	JJ
clgiles-ist-psu-edu-8875	20	8	contact	contact	NN
clgiles-ist-psu-edu-8875	20	9	information	information	NN
clgiles-ist-psu-edu-8875	20	10	for	for	IN
clgiles-ist-psu-edu-8875	20	11	spam	spam	NN
clgiles-ist-psu-edu-8875	20	12	purposes	purpose	NNS
clgiles-ist-psu-edu-8875	20	13	and	and	CC
clgiles-ist-psu-edu-8875	20	14	identity	identity	NN
clgiles-ist-psu-edu-8875	20	15	theft	theft	NN
clgiles-ist-psu-edu-8875	20	16	.	.	.
clgiles-ist-psu-edu-8875	21	1	Crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	21	2	may	may	MD
clgiles-ist-psu-edu-8875	21	3	also	also	RB
clgiles-ist-psu-edu-8875	21	4	overload	overload	VB
clgiles-ist-psu-edu-8875	21	5	a	a	DT
clgiles-ist-psu-edu-8875	21	6	website	website	NN
clgiles-ist-psu-edu-8875	21	7	such	such	JJ
clgiles-ist-psu-edu-8875	21	8	that	that	IN
clgiles-ist-psu-edu-8875	21	9	normal	normal	JJ
clgiles-ist-psu-edu-8875	21	10	user	user	NN
clgiles-ist-psu-edu-8875	21	11	access	access	NN
clgiles-ist-psu-edu-8875	21	12	is	be	VBZ
clgiles-ist-psu-edu-8875	21	13	impeded	impede	VBN
clgiles-ist-psu-edu-8875	21	14	.	.	.
clgiles-ist-psu-edu-8875	22	1	Web	web	NN
clgiles-ist-psu-edu-8875	22	2	crawler	crawler	NN
clgiles-ist-psu-edu-8875	22	3	activities	activity	NNS
clgiles-ist-psu-edu-8875	22	4	can	can	MD
clgiles-ist-psu-edu-8875	22	5	be	be	VB
clgiles-ist-psu-edu-8875	22	6	regulated	regulate	VBN
clgiles-ist-psu-edu-8875	22	7	from	from	IN
clgiles-ist-psu-edu-8875	22	8	the	the	DT
clgiles-ist-psu-edu-8875	22	9	server	server	NN
clgiles-ist-psu-edu-8875	22	10	side	side	NN
clgiles-ist-psu-edu-8875	22	11	by	by	IN
clgiles-ist-psu-edu-8875	22	12	deploying	deploy	VBG
clgiles-ist-psu-edu-8875	22	13	Robots	Robots	NNP
clgiles-ist-psu-edu-8875	22	14	Exclusion	Exclusion	NNP
clgiles-ist-psu-edu-8875	22	15	Protocol	Protocol	NNP
clgiles-ist-psu-edu-8875	22	16	(	(	-LRB-
clgiles-ist-psu-edu-8875	22	17	a	a	DT
clgiles-ist-psu-edu-8875	22	18	set	set	NN
clgiles-ist-psu-edu-8875	22	19	of	of	IN
clgiles-ist-psu-edu-8875	22	20	rules	rule	NNS
clgiles-ist-psu-edu-8875	22	21	in	in	IN
clgiles-ist-psu-edu-8875	22	22	a	a	DT
clgiles-ist-psu-edu-8875	22	23	file	file	NN
clgiles-ist-psu-edu-8875	22	24	called	call	VBN
clgiles-ist-psu-edu-8875	22	25	robots.txt	robots.txt	NN
clgiles-ist-psu-edu-8875	22	26	)	)	-RRB-
clgiles-ist-psu-edu-8875	22	27	in	in	IN
clgiles-ist-psu-edu-8875	22	28	the	the	DT
clgiles-ist-psu-edu-8875	22	29	root	root	NN
clgiles-ist-psu-edu-8875	22	30	directory	directory	NN
clgiles-ist-psu-edu-8875	22	31	of	of	IN
clgiles-ist-psu-edu-8875	22	32	a	a	DT
clgiles-ist-psu-edu-8875	22	33	website	website	NN
clgiles-ist-psu-edu-8875	22	34	,	,	,
clgiles-ist-psu-edu-8875	22	35	allow-	allow-	CC
clgiles-ist-psu-edu-8875	22	36	ing	e	VBG
clgiles-ist-psu-edu-8875	22	37	webmasters	webmaster	NNS
clgiles-ist-psu-edu-8875	22	38	to	to	TO
clgiles-ist-psu-edu-8875	22	39	indicate	indicate	VB
clgiles-ist-psu-edu-8875	22	40	to	to	IN
clgiles-ist-psu-edu-8875	22	41	visiting	visit	VBG
clgiles-ist-psu-edu-8875	22	42	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	22	43	which	which	WDT
clgiles-ist-psu-edu-8875	22	44	parts	part	NNS
clgiles-ist-psu-edu-8875	22	45	of	of	IN
clgiles-ist-psu-edu-8875	22	46	their	-PRON-	PRP$
clgiles-ist-psu-edu-8875	22	47	sites	site	NNS
clgiles-ist-psu-edu-8875	22	48	should	should	MD
clgiles-ist-psu-edu-8875	22	49	not	not	RB
clgiles-ist-psu-edu-8875	22	50	be	be	VB
clgiles-ist-psu-edu-8875	22	51	visited	visit	VBN
clgiles-ist-psu-edu-8875	22	52	as	as	RB
clgiles-ist-psu-edu-8875	22	53	well	well	RB
clgiles-ist-psu-edu-8875	22	54	as	as	IN
clgiles-ist-psu-edu-8875	22	55	a	a	DT
clgiles-ist-psu-edu-8875	22	56	minimum	minimum	JJ
clgiles-ist-psu-edu-8875	22	57	time	time	NN
clgiles-ist-psu-edu-8875	22	58	interval	interval	NN
clgiles-ist-psu-edu-8875	22	59	between	between	IN
clgiles-ist-psu-edu-8875	22	60	visits	visit	NNS
clgiles-ist-psu-edu-8875	22	61	.	.	.
clgiles-ist-psu-edu-8875	23	1	A	a	DT
clgiles-ist-psu-edu-8875	23	2	recent	recent	JJ
clgiles-ist-psu-edu-8875	23	3	study	study	NN
clgiles-ist-psu-edu-8875	23	4	shows	show	VBZ
clgiles-ist-psu-edu-8875	23	5	more	more	JJR
clgiles-ist-psu-edu-8875	23	6	than	than	IN
clgiles-ist-psu-edu-8875	23	7	30	30	CD
clgiles-ist-psu-edu-8875	23	8	%	%	NN
clgiles-ist-psu-edu-8875	23	9	of	of	IN
clgiles-ist-psu-edu-8875	23	10	active	active	JJ
clgiles-ist-psu-edu-8875	23	11	websites	website	NNS
clgiles-ist-psu-edu-8875	23	12	employ	employ	VBP
clgiles-ist-psu-edu-8875	23	13	this	this	DT
clgiles-ist-psu-edu-8875	23	14	standard	standard	NN
clgiles-ist-psu-edu-8875	23	15	to	to	TO
clgiles-ist-psu-edu-8875	23	16	reg-	reg-	VB
clgiles-ist-psu-edu-8875	23	17	ulate	ulate	VB
clgiles-ist-psu-edu-8875	23	18	crawler	crawler	NN
clgiles-ist-psu-edu-8875	23	19	activities	activity	NNS
clgiles-ist-psu-edu-8875	23	20	[	[	-LRB-
clgiles-ist-psu-edu-8875	23	21	2	2	CD
clgiles-ist-psu-edu-8875	23	22	,	,	,
clgiles-ist-psu-edu-8875	23	23	3	3	CD
clgiles-ist-psu-edu-8875	23	24	]	]	-RRB-
clgiles-ist-psu-edu-8875	23	25	.	.	.
clgiles-ist-psu-edu-8875	24	1	However	however	RB
clgiles-ist-psu-edu-8875	24	2	,	,	,
clgiles-ist-psu-edu-8875	24	3	since	since	IN
clgiles-ist-psu-edu-8875	24	4	the	the	DT
clgiles-ist-psu-edu-8875	24	5	Robots	Robots	NNP
clgiles-ist-psu-edu-8875	24	6	Exclusion	Exclusion	NNP
clgiles-ist-psu-edu-8875	24	7	Protocol	Protocol	NNP
clgiles-ist-psu-edu-8875	24	8	(	(	-LRB-
clgiles-ist-psu-edu-8875	24	9	REP	REP	NNP
clgiles-ist-psu-edu-8875	24	10	)	)	-RRB-
clgiles-ist-psu-edu-8875	24	11	serves	serve	VBZ
clgiles-ist-psu-edu-8875	24	12	only	only	RB
clgiles-ist-psu-edu-8875	24	13	as	as	IN
clgiles-ist-psu-edu-8875	24	14	an	an	DT
clgiles-ist-psu-edu-8875	24	15	unenforced	unenforced	JJ
clgiles-ist-psu-edu-8875	24	16	ad-	ad-	CC
clgiles-ist-psu-edu-8875	24	17	visory	visory	JJ
clgiles-ist-psu-edu-8875	24	18	to	to	IN
clgiles-ist-psu-edu-8875	24	19	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	24	20	,	,	,
clgiles-ist-psu-edu-8875	24	21	web	web	NN
clgiles-ist-psu-edu-8875	24	22	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	24	23	may	may	MD
clgiles-ist-psu-edu-8875	24	24	ignore	ignore	VB
clgiles-ist-psu-edu-8875	24	25	the	the	DT
clgiles-ist-psu-edu-8875	24	26	rules	rule	NNS
clgiles-ist-psu-edu-8875	24	27	and	and	CC
clgiles-ist-psu-edu-8875	24	28	access	access	NN
clgiles-ist-psu-edu-8875	24	29	part	part	NN
clgiles-ist-psu-edu-8875	24	30	of	of	IN
clgiles-ist-psu-edu-8875	24	31	the	the	DT
clgiles-ist-psu-edu-8875	24	32	forbidden	forbidden	JJ
clgiles-ist-psu-edu-8875	24	33	information	information	NN
clgiles-ist-psu-edu-8875	24	34	on	on	IN
clgiles-ist-psu-edu-8875	24	35	a	a	DT
clgiles-ist-psu-edu-8875	24	36	website	website	NN
clgiles-ist-psu-edu-8875	24	37	.	.	.
clgiles-ist-psu-edu-8875	25	1	Vi-	vi-	VB
clgiles-ist-psu-edu-8875	25	2	olating	olate	VBG
clgiles-ist-psu-edu-8875	25	3	the	the	DT
clgiles-ist-psu-edu-8875	25	4	robots.txt	robots.txt	NN
clgiles-ist-psu-edu-8875	25	5	rules	rule	NNS
clgiles-ist-psu-edu-8875	25	6	can	can	MD
clgiles-ist-psu-edu-8875	25	7	lead	lead	VB
clgiles-ist-psu-edu-8875	25	8	to	to	IN
clgiles-ist-psu-edu-8875	25	9	serious	serious	JJ
clgiles-ist-psu-edu-8875	25	10	privacy	privacy	NN
clgiles-ist-psu-edu-8875	25	11	and	and	CC
clgiles-ist-psu-edu-8875	25	12	security	security	NN
clgiles-ist-psu-edu-8875	25	13	concerns	concern	NNS
clgiles-ist-psu-edu-8875	25	14	.	.	.
clgiles-ist-psu-edu-8875	26	1	Thus	thus	RB
clgiles-ist-psu-edu-8875	26	2	,	,	,
clgiles-ist-psu-edu-8875	26	3	measuring	measure	VBG
clgiles-ist-psu-edu-8875	26	4	crawler	crawler	NN
clgiles-ist-psu-edu-8875	26	5	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	26	6	becomes	become	VBZ
clgiles-ist-psu-edu-8875	26	7	an	an	DT
clgiles-ist-psu-edu-8875	26	8	important	important	JJ
clgiles-ist-psu-edu-8875	26	9	task	task	NN
clgiles-ist-psu-edu-8875	26	10	to	to	TO
clgiles-ist-psu-edu-8875	26	11	help	help	VB
clgiles-ist-psu-edu-8875	26	12	detecting	detect	VBG
clgiles-ist-psu-edu-8875	26	13	improper	improper	JJ
clgiles-ist-psu-edu-8875	26	14	crawler	crawler	NNP
clgiles-ist-psu-edu-8875	26	15	behav-	behav-	NN
clgiles-ist-psu-edu-8875	26	16	ior	ior	NNP
clgiles-ist-psu-edu-8875	26	17	in	in	IN
clgiles-ist-psu-edu-8875	26	18	early	early	JJ
clgiles-ist-psu-edu-8875	26	19	stages	stage	NNS
clgiles-ist-psu-edu-8875	26	20	as	as	RB
clgiles-ist-psu-edu-8875	26	21	well	well	RB
clgiles-ist-psu-edu-8875	26	22	as	as	IN
clgiles-ist-psu-edu-8875	26	23	identifying	identify	VBG
clgiles-ist-psu-edu-8875	26	24	unethical	unethical	JJ
clgiles-ist-psu-edu-8875	26	25	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	26	26	.	.	.
clgiles-ist-psu-edu-8875	27	1	The	the	DT
clgiles-ist-psu-edu-8875	27	2	issues	issue	NNS
clgiles-ist-psu-edu-8875	27	3	of	of	IN
clgiles-ist-psu-edu-8875	27	4	crawler	crawler	NNP
clgiles-ist-psu-edu-8875	27	5	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	27	6	,	,	,
clgiles-ist-psu-edu-8875	27	7	however	however	RB
clgiles-ist-psu-edu-8875	27	8	,	,	,
clgiles-ist-psu-edu-8875	27	9	did	do	VBD
clgiles-ist-psu-edu-8875	27	10	not	not	RB
clgiles-ist-psu-edu-8875	27	11	bring	bring	VB
clgiles-ist-psu-edu-8875	27	12	enough	enough	JJ
clgiles-ist-psu-edu-8875	27	13	attention	attention	NN
clgiles-ist-psu-edu-8875	27	14	to	to	IN
clgiles-ist-psu-edu-8875	27	15	the	the	DT
clgiles-ist-psu-edu-8875	27	16	research	research	NN
clgiles-ist-psu-edu-8875	27	17	community	community	NN
clgiles-ist-psu-edu-8875	27	18	and	and	CC
clgiles-ist-psu-edu-8875	27	19	are	be	VBP
clgiles-ist-psu-edu-8875	27	20	under	under	IN
clgiles-ist-psu-edu-8875	27	21	studied	study	VBN
clgiles-ist-psu-edu-8875	27	22	.	.	.
clgiles-ist-psu-edu-8875	28	1	Crawler	Crawler	NNP
clgiles-ist-psu-edu-8875	28	2	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	28	3	are	be	VBP
clgiles-ist-psu-edu-8875	28	4	not	not	RB
clgiles-ist-psu-edu-8875	28	5	limited	limited	JJ
clgiles-ist-psu-edu-8875	28	6	to	to	IN
clgiles-ist-psu-edu-8875	28	7	whether	whether	IN
clgiles-ist-psu-edu-8875	28	8	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	28	9	obeying	obey	VBG
clgiles-ist-psu-edu-8875	28	10	website	website	NN
clgiles-ist-psu-edu-8875	28	11	rules	rule	NNS
clgiles-ist-psu-edu-8875	28	12	,	,	,
clgiles-ist-psu-edu-8875	28	13	but	but	CC
clgiles-ist-psu-edu-8875	28	14	also	also	RB
clgiles-ist-psu-edu-8875	28	15	can	can	MD
clgiles-ist-psu-edu-8875	28	16	be	be	VB
clgiles-ist-psu-edu-8875	28	17	studied	study	VBN
clgiles-ist-psu-edu-8875	28	18	in	in	IN
clgiles-ist-psu-edu-8875	28	19	terms	term	NNS
clgiles-ist-psu-edu-8875	28	20	of	of	IN
clgiles-ist-psu-edu-8875	28	21	the	the	DT
clgiles-ist-psu-edu-8875	28	22	value	value	NN
clgiles-ist-psu-edu-8875	28	23	provide	provide	VBP
clgiles-ist-psu-edu-8875	28	24	to	to	IN
clgiles-ist-psu-edu-8875	28	25	websites	website	NNS
clgiles-ist-psu-edu-8875	28	26	.	.	.
clgiles-ist-psu-edu-8875	29	1	If	if	IN
clgiles-ist-psu-edu-8875	29	2	a	a	DT
clgiles-ist-psu-edu-8875	29	3	crawler	crawler	NN
clgiles-ist-psu-edu-8875	29	4	provides	provide	VBZ
clgiles-ist-psu-edu-8875	29	5	zero	zero	CD
clgiles-ist-psu-edu-8875	29	6	value	value	NN
clgiles-ist-psu-edu-8875	29	7	to	to	IN
clgiles-ist-psu-edu-8875	29	8	the	the	DT
clgiles-ist-psu-edu-8875	29	9	crawled	crawled	JJ
clgiles-ist-psu-edu-8875	29	10	website	website	NN
clgiles-ist-psu-edu-8875	29	11	,	,	,
clgiles-ist-psu-edu-8875	29	12	it	-PRON-	PRP
clgiles-ist-psu-edu-8875	29	13	should	should	MD
clgiles-ist-psu-edu-8875	29	14	also	also	RB
clgiles-ist-psu-edu-8875	29	15	be	be	VB
clgiles-ist-psu-edu-8875	29	16	considered	consider	VBN
clgiles-ist-psu-edu-8875	29	17	less	less	RBR
clgiles-ist-psu-edu-8875	29	18	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	29	19	than	than	IN
clgiles-ist-psu-edu-8875	29	20	those	those	DT
clgiles-ist-psu-edu-8875	29	21	who	who	WP
clgiles-ist-psu-edu-8875	29	22	provide	provide	VBP
clgiles-ist-psu-edu-8875	29	23	positive	positive	JJ
clgiles-ist-psu-edu-8875	29	24	values	value	NNS
clgiles-ist-psu-edu-8875	29	25	.	.	.
clgiles-ist-psu-edu-8875	30	1	In	in	IN
clgiles-ist-psu-edu-8875	30	2	this	this	DT
clgiles-ist-psu-edu-8875	30	3	research	research	NN
clgiles-ist-psu-edu-8875	30	4	,	,	,
clgiles-ist-psu-edu-8875	30	5	we	-PRON-	PRP
clgiles-ist-psu-edu-8875	30	6	propose	propose	VBP
clgiles-ist-psu-edu-8875	30	7	a	a	DT
clgiles-ist-psu-edu-8875	30	8	vector	vector	NN
clgiles-ist-psu-edu-8875	30	9	space	space	NN
clgiles-ist-psu-edu-8875	30	10	model	model	NN
clgiles-ist-psu-edu-8875	30	11	of	of	IN
clgiles-ist-psu-edu-8875	30	12	mea-	mea-	NNPS
clgiles-ist-psu-edu-8875	30	13	suring	suring	NN
clgiles-ist-psu-edu-8875	30	14	web	web	NN
clgiles-ist-psu-edu-8875	30	15	crawler	crawler	NN
clgiles-ist-psu-edu-8875	30	16	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	30	17	based	base	VBN
clgiles-ist-psu-edu-8875	30	18	on	on	IN
clgiles-ist-psu-edu-8875	30	19	the	the	DT
clgiles-ist-psu-edu-8875	30	20	Robots	Robots	NNP
clgiles-ist-psu-edu-8875	30	21	Exclusion	Exclusion	NNP
clgiles-ist-psu-edu-8875	30	22	Protocol	Protocol	NNP
clgiles-ist-psu-edu-8875	30	23	.	.	.
clgiles-ist-psu-edu-8875	31	1	We	-PRON-	PRP
clgiles-ist-psu-edu-8875	31	2	define	define	VBP
clgiles-ist-psu-edu-8875	31	3	the	the	DT
clgiles-ist-psu-edu-8875	31	4	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	31	5	metric	metric	JJ
clgiles-ist-psu-edu-8875	31	6	to	to	TO
clgiles-ist-psu-edu-8875	31	7	measure	measure	VB
clgiles-ist-psu-edu-8875	31	8	web	web	NN
clgiles-ist-psu-edu-8875	31	9	crawler	crawler	NN
clgiles-ist-psu-edu-8875	31	10	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	31	11	.	.	.
clgiles-ist-psu-edu-8875	32	1	We	-PRON-	PRP
clgiles-ist-psu-edu-8875	32	2	also	also	RB
clgiles-ist-psu-edu-8875	32	3	study	study	VBP
clgiles-ist-psu-edu-8875	32	4	the	the	DT
clgiles-ist-psu-edu-8875	32	5	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	32	6	of	of	IN
clgiles-ist-psu-edu-8875	32	7	big	big	JJ
clgiles-ist-psu-edu-8875	32	8	search	search	NN
clgiles-ist-psu-edu-8875	32	9	engine	engine	NN
clgiles-ist-psu-edu-8875	32	10	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	32	11	in	in	IN
clgiles-ist-psu-edu-8875	32	12	terms	term	NNS
clgiles-ist-psu-edu-8875	32	13	of	of	IN
clgiles-ist-psu-edu-8875	32	14	return	return	NN
clgiles-ist-psu-edu-8875	32	15	on	on	IN
clgiles-ist-psu-edu-8875	32	16	investment	investment	NN
clgiles-ist-psu-edu-8875	32	17	where	where	WRB
clgiles-ist-psu-edu-8875	32	18	crawler	crawler	NNP
clgiles-ist-psu-edu-8875	32	19	vis-	vis-	VBD
clgiles-ist-psu-edu-8875	32	20	its	-PRON-	PRP$
clgiles-ist-psu-edu-8875	32	21	are	be	VBP
clgiles-ist-psu-edu-8875	32	22	considered	consider	VBN
clgiles-ist-psu-edu-8875	32	23	investments	investment	NNS
clgiles-ist-psu-edu-8875	32	24	from	from	IN
clgiles-ist-psu-edu-8875	32	25	websites	website	NNS
clgiles-ist-psu-edu-8875	32	26	and	and	CC
clgiles-ist-psu-edu-8875	32	27	correspond-	correspond-	JJ
clgiles-ist-psu-edu-8875	32	28	ing	ing	NN
clgiles-ist-psu-edu-8875	32	29	search	search	NN
clgiles-ist-psu-edu-8875	32	30	engine	engine	NN
clgiles-ist-psu-edu-8875	32	31	traffic	traffic	NN
clgiles-ist-psu-edu-8875	32	32	is	be	VBZ
clgiles-ist-psu-edu-8875	32	33	considered	consider	VBN
clgiles-ist-psu-edu-8875	32	34	as	as	IN
clgiles-ist-psu-edu-8875	32	35	returns	return	NNS
clgiles-ist-psu-edu-8875	32	36	.	.	.
clgiles-ist-psu-edu-8875	33	1	The	the	DT
clgiles-ist-psu-edu-8875	33	2	results	result	NNS
clgiles-ist-psu-edu-8875	33	3	show	show	VBP
clgiles-ist-psu-edu-8875	33	4	that	that	IN
clgiles-ist-psu-edu-8875	33	5	Google	Google	NNP
clgiles-ist-psu-edu-8875	33	6	has	have	VBZ
clgiles-ist-psu-edu-8875	33	7	a	a	DT
clgiles-ist-psu-edu-8875	33	8	much	much	RB
clgiles-ist-psu-edu-8875	33	9	higher	high	JJR
clgiles-ist-psu-edu-8875	33	10	score	score	NN
clgiles-ist-psu-edu-8875	33	11	in	in	IN
clgiles-ist-psu-edu-8875	33	12	US	US	NNP
clgiles-ist-psu-edu-8875	33	13	websites	website	NNS
clgiles-ist-psu-edu-8875	33	14	but	but	CC
clgiles-ist-psu-edu-8875	33	15	has	have	VBZ
clgiles-ist-psu-edu-8875	33	16	a	a	DT
clgiles-ist-psu-edu-8875	33	17	lower	low	JJR
clgiles-ist-psu-edu-8875	33	18	score	score	NN
clgiles-ist-psu-edu-8875	33	19	than	than	IN
clgiles-ist-psu-edu-8875	33	20	Baidu	Baidu	NNP
clgiles-ist-psu-edu-8875	33	21	in	in	IN
clgiles-ist-psu-edu-8875	33	22	Chinese	chinese	JJ
clgiles-ist-psu-edu-8875	33	23	websites	website	NNS
clgiles-ist-psu-edu-8875	33	24	.	.	.
clgiles-ist-psu-edu-8875	34	1	2	2	LS
clgiles-ist-psu-edu-8875	34	2	.	.	.
clgiles-ist-psu-edu-8875	35	1	RELATED	related	JJ
clgiles-ist-psu-edu-8875	35	2	WORK	work	NN
clgiles-ist-psu-edu-8875	35	3	The	the	DT
clgiles-ist-psu-edu-8875	35	4	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	35	5	factors	factor	NNS
clgiles-ist-psu-edu-8875	35	6	are	be	VBP
clgiles-ist-psu-edu-8875	35	7	examined	examine	VBN
clgiles-ist-psu-edu-8875	35	8	from	from	IN
clgiles-ist-psu-edu-8875	35	9	three	three	CD
clgiles-ist-psu-edu-8875	35	10	perspectives	perspective	NNS
clgiles-ist-psu-edu-8875	35	11	[	[	-LRB-
clgiles-ist-psu-edu-8875	35	12	4	4	CD
clgiles-ist-psu-edu-8875	35	13	]	]	-RRB-
clgiles-ist-psu-edu-8875	35	14	:	:	:
clgiles-ist-psu-edu-8875	35	15	denial	denial	NN
clgiles-ist-psu-edu-8875	35	16	of	of	IN
clgiles-ist-psu-edu-8875	35	17	service	service	NN
clgiles-ist-psu-edu-8875	35	18	,	,	,
clgiles-ist-psu-edu-8875	35	19	cost	cost	NN
clgiles-ist-psu-edu-8875	35	20	,	,	,
clgiles-ist-psu-edu-8875	35	21	and	and	CC
clgiles-ist-psu-edu-8875	35	22	privacy	privacy	NN
clgiles-ist-psu-edu-8875	35	23	.	.	.
clgiles-ist-psu-edu-8875	36	1	An	an	DT
clgiles-ist-psu-edu-8875	36	2	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	36	3	crawl	crawl	NN
clgiles-ist-psu-edu-8875	36	4	guideline	guideline	NN
clgiles-ist-psu-edu-8875	36	5	is	be	VBZ
clgiles-ist-psu-edu-8875	36	6	described	describe	VBN
clgiles-ist-psu-edu-8875	36	7	for	for	IN
clgiles-ist-psu-edu-8875	36	8	crawler	crawler	NNP
clgiles-ist-psu-edu-8875	36	9	owners	owner	NNS
clgiles-ist-psu-edu-8875	36	10	to	to	TO
clgiles-ist-psu-edu-8875	36	11	follow	follow	VB
clgiles-ist-psu-edu-8875	36	12	.	.	.
clgiles-ist-psu-edu-8875	37	1	This	this	DT
clgiles-ist-psu-edu-8875	37	2	guideline	guideline	NN
clgiles-ist-psu-edu-8875	37	3	suggests	suggest	VBZ
clgiles-ist-psu-edu-8875	37	4	taking	take	VBG
clgiles-ist-psu-edu-8875	37	5	legal	legal	JJ
clgiles-ist-psu-edu-8875	37	6	action	action	NN
clgiles-ist-psu-edu-8875	37	7	or	or	CC
clgiles-ist-psu-edu-8875	37	8	initiating	initiate	VBG
clgiles-ist-psu-edu-8875	37	9	a	a	DT
clgiles-ist-psu-edu-8875	37	10	profes-	profes-	NN
clgiles-ist-psu-edu-8875	37	11	sional	sional	JJ
clgiles-ist-psu-edu-8875	37	12	organization	organization	NN
clgiles-ist-psu-edu-8875	37	13	to	to	TO
clgiles-ist-psu-edu-8875	37	14	regulate	regulate	VB
clgiles-ist-psu-edu-8875	37	15	web	web	NN
clgiles-ist-psu-edu-8875	37	16	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	37	17	.	.	.
clgiles-ist-psu-edu-8875	38	1	Our	-PRON-	PRP$
clgiles-ist-psu-edu-8875	38	2	research	research	NN
clgiles-ist-psu-edu-8875	38	3	adopts	adopt	VBZ
clgiles-ist-psu-edu-8875	38	4	these	these	DT
clgiles-ist-psu-edu-8875	38	5	perspectives	perspective	NNS
clgiles-ist-psu-edu-8875	38	6	of	of	IN
clgiles-ist-psu-edu-8875	38	7	crawler	crawler	JJ
clgiles-ist-psu-edu-8875	38	8	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	38	9	and	and	CC
clgiles-ist-psu-edu-8875	38	10	expands	expand	VBZ
clgiles-ist-psu-edu-8875	38	11	it	-PRON-	PRP
clgiles-ist-psu-edu-8875	38	12	to	to	IN
clgiles-ist-psu-edu-8875	38	13	a	a	DT
clgiles-ist-psu-edu-8875	38	14	computational	computational	JJ
clgiles-ist-psu-edu-8875	38	15	measure	measure	NN
clgiles-ist-psu-edu-8875	38	16	.	.	.
clgiles-ist-psu-edu-8875	39	1	The	the	DT
clgiles-ist-psu-edu-8875	39	2	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	39	3	issues	issue	NNS
clgiles-ist-psu-edu-8875	39	4	of	of	IN
clgiles-ist-psu-edu-8875	39	5	administrat-	administrat-	NNP
clgiles-ist-psu-edu-8875	39	6	ing	ing	NNP
clgiles-ist-psu-edu-8875	39	7	web	web	NN
clgiles-ist-psu-edu-8875	39	8	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	39	9	are	be	VBP
clgiles-ist-psu-edu-8875	39	10	discussed	discuss	VBN
clgiles-ist-psu-edu-8875	39	11	in	in	IN
clgiles-ist-psu-edu-8875	39	12	[	[	-LRB-
clgiles-ist-psu-edu-8875	39	13	1	1	CD
clgiles-ist-psu-edu-8875	39	14	]	]	-RRB-
clgiles-ist-psu-edu-8875	39	15	.	.	.
clgiles-ist-psu-edu-8875	40	1	It	-PRON-	PRP
clgiles-ist-psu-edu-8875	40	2	provides	provide	VBZ
clgiles-ist-psu-edu-8875	40	3	a	a	DT
clgiles-ist-psu-edu-8875	40	4	guideline	guideline	NN
clgiles-ist-psu-edu-8875	40	5	for	for	IN
clgiles-ist-psu-edu-8875	40	6	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	40	7	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	40	8	to	to	TO
clgiles-ist-psu-edu-8875	40	9	follow	follow	VB
clgiles-ist-psu-edu-8875	40	10	.	.	.
clgiles-ist-psu-edu-8875	41	1	The	the	DT
clgiles-ist-psu-edu-8875	41	2	guideline	guideline	NN
clgiles-ist-psu-edu-8875	41	3	also	also	RB
clgiles-ist-psu-edu-8875	41	4	gives	give	VBZ
clgiles-ist-psu-edu-8875	41	5	great	great	JJ
clgiles-ist-psu-edu-8875	41	6	insights	insight	NNS
clgiles-ist-psu-edu-8875	41	7	to	to	IN
clgiles-ist-psu-edu-8875	41	8	our	-PRON-	PRP$
clgiles-ist-psu-edu-8875	41	9	research	research	NN
clgiles-ist-psu-edu-8875	41	10	of	of	IN
clgiles-ist-psu-edu-8875	41	11	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	41	12	measurements	measurement	NNS
clgiles-ist-psu-edu-8875	41	13	.	.	.
clgiles-ist-psu-edu-8875	42	1	However	however	RB
clgiles-ist-psu-edu-8875	42	2	,	,	,
clgiles-ist-psu-edu-8875	42	3	WWW	WWW	NNP
clgiles-ist-psu-edu-8875	42	4	2010	2010	CD
clgiles-ist-psu-edu-8875	42	5	•	•	NNP
clgiles-ist-psu-edu-8875	42	6	Poster	poster	NN
clgiles-ist-psu-edu-8875	42	7	April	April	NNP
clgiles-ist-psu-edu-8875	42	8	26	26	CD
clgiles-ist-psu-edu-8875	42	9	-	-	SYM
clgiles-ist-psu-edu-8875	42	10	30	30	CD
clgiles-ist-psu-edu-8875	42	11	•	•	NNP
clgiles-ist-psu-edu-8875	42	12	Raleigh	Raleigh	NNP
clgiles-ist-psu-edu-8875	42	13	•	•	NNP
clgiles-ist-psu-edu-8875	42	14	NC	NC	NNP
clgiles-ist-psu-edu-8875	42	15	•	•	NN
clgiles-ist-psu-edu-8875	42	16	USA	USA	NNP
clgiles-ist-psu-edu-8875	42	17	1101	1101	CD
clgiles-ist-psu-edu-8875	42	18	none	none	NN
clgiles-ist-psu-edu-8875	42	19	of	of	IN
clgiles-ist-psu-edu-8875	42	20	the	the	DT
clgiles-ist-psu-edu-8875	42	21	above	above	RB
clgiles-ist-psu-edu-8875	42	22	mentioned	mention	VBN
clgiles-ist-psu-edu-8875	42	23	work	work	NN
clgiles-ist-psu-edu-8875	42	24	provides	provide	VBZ
clgiles-ist-psu-edu-8875	42	25	a	a	DT
clgiles-ist-psu-edu-8875	42	26	quantitative	quantitative	JJ
clgiles-ist-psu-edu-8875	42	27	measure	measure	NN
clgiles-ist-psu-edu-8875	42	28	of	of	IN
clgiles-ist-psu-edu-8875	42	29	web	web	NN
clgiles-ist-psu-edu-8875	42	30	crawler	crawler	NN
clgiles-ist-psu-edu-8875	42	31	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	42	32	.	.	.
clgiles-ist-psu-edu-8875	43	1	3	3	LS
clgiles-ist-psu-edu-8875	43	2	.	.	.
clgiles-ist-psu-edu-8875	44	1	CRAWLER	CRAWLER	NNP
clgiles-ist-psu-edu-8875	44	2	BEHAVIOR	BEHAVIOR	NNP
clgiles-ist-psu-edu-8875	44	3	MODEL	MODEL	NNP
clgiles-ist-psu-edu-8875	44	4	In	in	IN
clgiles-ist-psu-edu-8875	44	5	our	-PRON-	PRP$
clgiles-ist-psu-edu-8875	44	6	research	research	NN
clgiles-ist-psu-edu-8875	44	7	,	,	,
clgiles-ist-psu-edu-8875	44	8	each	each	DT
clgiles-ist-psu-edu-8875	44	9	web	web	NN
clgiles-ist-psu-edu-8875	44	10	crawler	crawler	NN
clgiles-ist-psu-edu-8875	44	11	’s	’s	POS
clgiles-ist-psu-edu-8875	44	12	behavior	behavior	NN
clgiles-ist-psu-edu-8875	44	13	is	be	VBZ
clgiles-ist-psu-edu-8875	44	14	modeled	model	VBN
clgiles-ist-psu-edu-8875	44	15	as	as	IN
clgiles-ist-psu-edu-8875	44	16	a	a	DT
clgiles-ist-psu-edu-8875	44	17	vector	vector	NN
clgiles-ist-psu-edu-8875	44	18	in	in	IN
clgiles-ist-psu-edu-8875	44	19	the	the	DT
clgiles-ist-psu-edu-8875	44	20	rule	rule	NN
clgiles-ist-psu-edu-8875	44	21	space	space	NN
clgiles-ist-psu-edu-8875	44	22	where	where	WRB
clgiles-ist-psu-edu-8875	44	23	rules	rule	NNS
clgiles-ist-psu-edu-8875	44	24	are	be	VBP
clgiles-ist-psu-edu-8875	44	25	specified	specify	VBN
clgiles-ist-psu-edu-8875	44	26	by	by	IN
clgiles-ist-psu-edu-8875	44	27	Robots	Robots	NNP
clgiles-ist-psu-edu-8875	44	28	Exclusion	Exclusion	NNP
clgiles-ist-psu-edu-8875	44	29	Protocol	Protocol	NNP
clgiles-ist-psu-edu-8875	44	30	to	to	TO
clgiles-ist-psu-edu-8875	44	31	regulate	regulate	VB
clgiles-ist-psu-edu-8875	44	32	the	the	DT
clgiles-ist-psu-edu-8875	44	33	crawler	crawler	NN
clgiles-ist-psu-edu-8875	44	34	behavior	behavior	NN
clgiles-ist-psu-edu-8875	44	35	.	.	.
clgiles-ist-psu-edu-8875	45	1	If	if	IN
clgiles-ist-psu-edu-8875	45	2	a	a	DT
clgiles-ist-psu-edu-8875	45	3	crawler	crawler	NN
clgiles-ist-psu-edu-8875	45	4	violates	violate	VBZ
clgiles-ist-psu-edu-8875	45	5	a	a	DT
clgiles-ist-psu-edu-8875	45	6	rule	rule	NN
clgiles-ist-psu-edu-8875	45	7	,	,	,
clgiles-ist-psu-edu-8875	45	8	the	the	DT
clgiles-ist-psu-edu-8875	45	9	corresponding	correspond	VBG
clgiles-ist-psu-edu-8875	45	10	vector	vector	NN
clgiles-ist-psu-edu-8875	45	11	element	element	NN
clgiles-ist-psu-edu-8875	45	12	is	be	VBZ
clgiles-ist-psu-edu-8875	45	13	larger	large	JJR
clgiles-ist-psu-edu-8875	45	14	than	than	IN
clgiles-ist-psu-edu-8875	45	15	0	0	CD
clgiles-ist-psu-edu-8875	45	16	.	.	.
clgiles-ist-psu-edu-8875	46	1	Websites	website	NNS
clgiles-ist-psu-edu-8875	46	2	can	can	MD
clgiles-ist-psu-edu-8875	46	3	also	also	RB
clgiles-ist-psu-edu-8875	46	4	be	be	VB
clgiles-ist-psu-edu-8875	46	5	modeled	model	VBN
clgiles-ist-psu-edu-8875	46	6	in	in	IN
clgiles-ist-psu-edu-8875	46	7	the	the	DT
clgiles-ist-psu-edu-8875	46	8	rules	rule	NNS
clgiles-ist-psu-edu-8875	46	9	space	space	NN
clgiles-ist-psu-edu-8875	46	10	that	that	IN
clgiles-ist-psu-edu-8875	46	11	if	if	IN
clgiles-ist-psu-edu-8875	46	12	a	a	DT
clgiles-ist-psu-edu-8875	46	13	website	website	NN
clgiles-ist-psu-edu-8875	46	14	includes	include	VBZ
clgiles-ist-psu-edu-8875	46	15	a	a	DT
clgiles-ist-psu-edu-8875	46	16	rule	rule	NN
clgiles-ist-psu-edu-8875	46	17	in	in	IN
clgiles-ist-psu-edu-8875	46	18	its	-PRON-	PRP$
clgiles-ist-psu-edu-8875	46	19	robots.txt	robots.txt	NN
clgiles-ist-psu-edu-8875	46	20	file	file	NN
clgiles-ist-psu-edu-8875	46	21	,	,	,
clgiles-ist-psu-edu-8875	46	22	the	the	DT
clgiles-ist-psu-edu-8875	46	23	corresponding	correspond	VBG
clgiles-ist-psu-edu-8875	46	24	vector	vector	NN
clgiles-ist-psu-edu-8875	46	25	element	element	NN
clgiles-ist-psu-edu-8875	46	26	is	be	VBZ
clgiles-ist-psu-edu-8875	46	27	larger	large	JJR
clgiles-ist-psu-edu-8875	46	28	than	than	IN
clgiles-ist-psu-edu-8875	46	29	0	0	CD
clgiles-ist-psu-edu-8875	46	30	.	.	.
clgiles-ist-psu-edu-8875	47	1	The	the	DT
clgiles-ist-psu-edu-8875	47	2	ac-	ac-	DT
clgiles-ist-psu-edu-8875	47	3	tual	tual	JJ
clgiles-ist-psu-edu-8875	47	4	value	value	NN
clgiles-ist-psu-edu-8875	47	5	for	for	IN
clgiles-ist-psu-edu-8875	47	6	a	a	DT
clgiles-ist-psu-edu-8875	47	7	rule	rule	NN
clgiles-ist-psu-edu-8875	47	8	element	element	NN
clgiles-ist-psu-edu-8875	47	9	can	can	MD
clgiles-ist-psu-edu-8875	47	10	be	be	VB
clgiles-ist-psu-edu-8875	47	11	defined	define	VBN
clgiles-ist-psu-edu-8875	47	12	based	base	VBN
clgiles-ist-psu-edu-8875	47	13	on	on	IN
clgiles-ist-psu-edu-8875	47	14	the	the	DT
clgiles-ist-psu-edu-8875	47	15	consequences	consequence	NNS
clgiles-ist-psu-edu-8875	47	16	or	or	CC
clgiles-ist-psu-edu-8875	47	17	cost	cost	NN
clgiles-ist-psu-edu-8875	47	18	of	of	IN
clgiles-ist-psu-edu-8875	47	19	violating	violate	VBG
clgiles-ist-psu-edu-8875	47	20	such	such	JJ
clgiles-ist-psu-edu-8875	47	21	rule	rule	NN
clgiles-ist-psu-edu-8875	47	22	.	.	.
clgiles-ist-psu-edu-8875	48	1	We	-PRON-	PRP
clgiles-ist-psu-edu-8875	48	2	define	define	VBP
clgiles-ist-psu-edu-8875	48	3	content	content	NN
clgiles-ist-psu-edu-8875	48	4	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	48	5	Ec	Ec	NNP
clgiles-ist-psu-edu-8875	48	6	and	and	CC
clgiles-ist-psu-edu-8875	48	7	access	access	NN
clgiles-ist-psu-edu-8875	48	8	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	48	9	Ea	Ea	NNP
clgiles-ist-psu-edu-8875	48	10	scores	score	NNS
clgiles-ist-psu-edu-8875	48	11	to	to	TO
clgiles-ist-psu-edu-8875	48	12	evaluate	evaluate	VB
clgiles-ist-psu-edu-8875	48	13	web	web	NN
clgiles-ist-psu-edu-8875	48	14	crawler	crawler	NN
clgiles-ist-psu-edu-8875	48	15	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	48	16	.	.	.
clgiles-ist-psu-edu-8875	49	1	In	in	IN
clgiles-ist-psu-edu-8875	49	2	content	content	NN
clgiles-ist-psu-edu-8875	49	3	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	49	4	,	,	,
clgiles-ist-psu-edu-8875	49	5	cost	cost	NN
clgiles-ist-psu-edu-8875	49	6	is	be	VBZ
clgiles-ist-psu-edu-8875	49	7	defined	define	VBN
clgiles-ist-psu-edu-8875	49	8	as	as	IN
clgiles-ist-psu-edu-8875	49	9	the	the	DT
clgiles-ist-psu-edu-8875	49	10	number	number	NN
clgiles-ist-psu-edu-8875	49	11	of	of	IN
clgiles-ist-psu-edu-8875	49	12	restricted	restrict	VBN
clgiles-ist-psu-edu-8875	49	13	web	web	NN
clgiles-ist-psu-edu-8875	49	14	pages	page	NNS
clgiles-ist-psu-edu-8875	49	15	or	or	CC
clgiles-ist-psu-edu-8875	49	16	web	web	NN
clgiles-ist-psu-edu-8875	49	17	directories	directory	NNS
clgiles-ist-psu-edu-8875	49	18	being	be	VBG
clgiles-ist-psu-edu-8875	49	19	unethically	unethically	RB
clgiles-ist-psu-edu-8875	49	20	accessed	access	VBN
clgiles-ist-psu-edu-8875	49	21	(	(	-LRB-
clgiles-ist-psu-edu-8875	49	22	see	see	VB
clgiles-ist-psu-edu-8875	49	23	Eq	Eq	NNP
clgiles-ist-psu-edu-8875	49	24	.	.	.
clgiles-ist-psu-edu-8875	50	1	1	1	LS
clgiles-ist-psu-edu-8875	50	2	)	)	-RRB-
clgiles-ist-psu-edu-8875	50	3	.	.	.
clgiles-ist-psu-edu-8875	51	1	Ec(C	Ec(C	NNP
clgiles-ist-psu-edu-8875	51	2	)	)	-RRB-
clgiles-ist-psu-edu-8875	51	3	=	=	NFP
clgiles-ist-psu-edu-8875	51	4	∑	∑	.
clgiles-ist-psu-edu-8875	51	5	wi∈W	wi∈w	XX
clgiles-ist-psu-edu-8875	51	6	||VC(wi)||	||VC(wi)||	NNP
clgiles-ist-psu-edu-8875	51	7	||D(wi)||	||d(wi)||	NN
clgiles-ist-psu-edu-8875	51	8	.	.	.
clgiles-ist-psu-edu-8875	52	1	(	(	-LRB-
clgiles-ist-psu-edu-8875	52	2	1	1	LS
clgiles-ist-psu-edu-8875	52	3	)	)	-RRB-
clgiles-ist-psu-edu-8875	52	4	Access	access	NN
clgiles-ist-psu-edu-8875	52	5	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	52	6	is	be	VBZ
clgiles-ist-psu-edu-8875	52	7	defined	define	VBN
clgiles-ist-psu-edu-8875	52	8	as	as	IN
clgiles-ist-psu-edu-8875	52	9	how	how	WRB
clgiles-ist-psu-edu-8875	52	10	a	a	DT
clgiles-ist-psu-edu-8875	52	11	crawler	crawler	NN
clgiles-ist-psu-edu-8875	52	12	respects	respect	VBZ
clgiles-ist-psu-edu-8875	52	13	the	the	DT
clgiles-ist-psu-edu-8875	52	14	desired	desire	VBN
clgiles-ist-psu-edu-8875	52	15	visit	visit	NN
clgiles-ist-psu-edu-8875	52	16	interval	interval	NN
clgiles-ist-psu-edu-8875	52	17	(	(	-LRB-
clgiles-ist-psu-edu-8875	52	18	crawl	crawl	NN
clgiles-ist-psu-edu-8875	52	19	-	-	HYPH
clgiles-ist-psu-edu-8875	52	20	delay	delay	NNP
clgiles-ist-psu-edu-8875	52	21	rule	rule	NN
clgiles-ist-psu-edu-8875	52	22	in	in	IN
clgiles-ist-psu-edu-8875	52	23	robots.txt	robots.txt	CD
clgiles-ist-psu-edu-8875	52	24	file	file	NN
clgiles-ist-psu-edu-8875	52	25	)	)	-RRB-
clgiles-ist-psu-edu-8875	52	26	of	of	IN
clgiles-ist-psu-edu-8875	52	27	the	the	DT
clgiles-ist-psu-edu-8875	52	28	website(see	website(see	.
clgiles-ist-psu-edu-8875	52	29	Eq	eq	RB
clgiles-ist-psu-edu-8875	52	30	.	.	.
clgiles-ist-psu-edu-8875	53	1	2	2	LS
clgiles-ist-psu-edu-8875	53	2	)	)	-RRB-
clgiles-ist-psu-edu-8875	53	3	.	.	.
clgiles-ist-psu-edu-8875	54	1	Ea(r	Ea(r	NNP
clgiles-ist-psu-edu-8875	54	2	)	)	-RRB-
clgiles-ist-psu-edu-8875	54	3	=	=	NFP
clgiles-ist-psu-edu-8875	54	4	∑	∑	NFP
clgiles-ist-psu-edu-8875	54	5	wi∈W	wi∈W	NNP
clgiles-ist-psu-edu-8875	54	6	e−(intervalC(wi)−delay(wi	e−(intervalC(wi)−delay(wi	NNP
clgiles-ist-psu-edu-8875	54	7	)	)	-RRB-
clgiles-ist-psu-edu-8875	54	8	)	)	-RRB-
clgiles-ist-psu-edu-8875	54	9	1	1	CD
clgiles-ist-psu-edu-8875	54	10	+	+	SYM
clgiles-ist-psu-edu-8875	54	11	e−(intervalC(wi)−delay(wi	e−(intervalc(wi)−delay(wi	ADD
clgiles-ist-psu-edu-8875	54	12	)	)	-RRB-
clgiles-ist-psu-edu-8875	54	13	)	)	-RRB-
clgiles-ist-psu-edu-8875	54	14	(	(	-LRB-
clgiles-ist-psu-edu-8875	54	15	2	2	LS
clgiles-ist-psu-edu-8875	54	16	)	)	-RRB-
clgiles-ist-psu-edu-8875	54	17	A	a	DT
clgiles-ist-psu-edu-8875	54	18	major	major	JJ
clgiles-ist-psu-edu-8875	54	19	advantage	advantage	NN
clgiles-ist-psu-edu-8875	54	20	for	for	IN
clgiles-ist-psu-edu-8875	54	21	websites	website	NNS
clgiles-ist-psu-edu-8875	54	22	allowing	allow	VBG
clgiles-ist-psu-edu-8875	54	23	search	search	NN
clgiles-ist-psu-edu-8875	54	24	engine	engine	NN
clgiles-ist-psu-edu-8875	54	25	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	54	26	to	to	TO
clgiles-ist-psu-edu-8875	54	27	crawl	crawl	VB
clgiles-ist-psu-edu-8875	54	28	their	-PRON-	PRP$
clgiles-ist-psu-edu-8875	54	29	web	web	NN
clgiles-ist-psu-edu-8875	54	30	pages	page	NNS
clgiles-ist-psu-edu-8875	54	31	is	be	VBZ
clgiles-ist-psu-edu-8875	54	32	that	that	IN
clgiles-ist-psu-edu-8875	54	33	the	the	DT
clgiles-ist-psu-edu-8875	54	34	search	search	NN
clgiles-ist-psu-edu-8875	54	35	engines	engine	NNS
clgiles-ist-psu-edu-8875	54	36	bring	bring	VBP
clgiles-ist-psu-edu-8875	54	37	traffic	traffic	NN
clgiles-ist-psu-edu-8875	54	38	back	back	RB
clgiles-ist-psu-edu-8875	54	39	to	to	IN
clgiles-ist-psu-edu-8875	54	40	them	-PRON-	PRP
clgiles-ist-psu-edu-8875	54	41	.	.	.
clgiles-ist-psu-edu-8875	55	1	From	from	IN
clgiles-ist-psu-edu-8875	55	2	this	this	DT
clgiles-ist-psu-edu-8875	55	3	perspective	perspective	NN
clgiles-ist-psu-edu-8875	55	4	,	,	,
clgiles-ist-psu-edu-8875	55	5	being	be	VBG
clgiles-ist-psu-edu-8875	55	6	ethical	ethical	JJ
clgiles-ist-psu-edu-8875	55	7	for	for	IN
clgiles-ist-psu-edu-8875	55	8	a	a	DT
clgiles-ist-psu-edu-8875	55	9	web	web	NN
clgiles-ist-psu-edu-8875	55	10	crawler	crawler	NN
clgiles-ist-psu-edu-8875	55	11	means	mean	VBZ
clgiles-ist-psu-edu-8875	55	12	bringing	bring	VBG
clgiles-ist-psu-edu-8875	55	13	more	more	JJR
clgiles-ist-psu-edu-8875	55	14	visits	visit	NNS
clgiles-ist-psu-edu-8875	55	15	back	back	RB
clgiles-ist-psu-edu-8875	55	16	to	to	IN
clgiles-ist-psu-edu-8875	55	17	the	the	DT
clgiles-ist-psu-edu-8875	55	18	crawled	crawled	JJ
clgiles-ist-psu-edu-8875	55	19	websites	website	NNS
clgiles-ist-psu-edu-8875	55	20	.	.	.
clgiles-ist-psu-edu-8875	56	1	The	the	DT
clgiles-ist-psu-edu-8875	56	2	effective	effective	JJ
clgiles-ist-psu-edu-8875	56	3	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	56	4	of	of	IN
clgiles-ist-psu-edu-8875	56	5	search	search	NN
clgiles-ist-psu-edu-8875	56	6	engine	engine	NN
clgiles-ist-psu-edu-8875	56	7	S	s	NN
clgiles-ist-psu-edu-8875	56	8	to	to	IN
clgiles-ist-psu-edu-8875	56	9	a	a	DT
clgiles-ist-psu-edu-8875	56	10	website	website	NN
clgiles-ist-psu-edu-8875	56	11	can	can	MD
clgiles-ist-psu-edu-8875	56	12	be	be	VB
clgiles-ist-psu-edu-8875	56	13	defined	define	VBN
clgiles-ist-psu-edu-8875	56	14	as	as	IN
clgiles-ist-psu-edu-8875	56	15	the	the	DT
clgiles-ist-psu-edu-8875	56	16	ratio	ratio	NN
clgiles-ist-psu-edu-8875	56	17	between	between	IN
clgiles-ist-psu-edu-8875	56	18	the	the	DT
clgiles-ist-psu-edu-8875	56	19	user	user	NN
clgiles-ist-psu-edu-8875	56	20	visits	visit	NNS
clgiles-ist-psu-edu-8875	56	21	referred	refer	VBN
clgiles-ist-psu-edu-8875	56	22	by	by	IN
clgiles-ist-psu-edu-8875	56	23	the	the	DT
clgiles-ist-psu-edu-8875	56	24	search	search	NN
clgiles-ist-psu-edu-8875	56	25	engine	engine	NN
clgiles-ist-psu-edu-8875	56	26	to	to	IN
clgiles-ist-psu-edu-8875	56	27	the	the	DT
clgiles-ist-psu-edu-8875	56	28	website	website	NN
clgiles-ist-psu-edu-8875	56	29	and	and	CC
clgiles-ist-psu-edu-8875	56	30	visits	visit	NNS
clgiles-ist-psu-edu-8875	56	31	generated	generate	VBN
clgiles-ist-psu-edu-8875	56	32	by	by	IN
clgiles-ist-psu-edu-8875	56	33	the	the	DT
clgiles-ist-psu-edu-8875	56	34	crawler	crawler	NNP
clgiles-ist-psu-edu-8875	56	35	r	r	NNP
clgiles-ist-psu-edu-8875	56	36	of	of	IN
clgiles-ist-psu-edu-8875	56	37	the	the	DT
clgiles-ist-psu-edu-8875	56	38	search	search	NN
clgiles-ist-psu-edu-8875	56	39	engine	engine	NN
clgiles-ist-psu-edu-8875	56	40	to	to	IN
clgiles-ist-psu-edu-8875	56	41	the	the	DT
clgiles-ist-psu-edu-8875	56	42	website	website	NN
clgiles-ist-psu-edu-8875	56	43	(	(	-LRB-
clgiles-ist-psu-edu-8875	56	44	see	see	VB
clgiles-ist-psu-edu-8875	56	45	Eq	Eq	NNP
clgiles-ist-psu-edu-8875	56	46	.	.	.
clgiles-ist-psu-edu-8875	57	1	3	3	LS
clgiles-ist-psu-edu-8875	57	2	)	)	-RRB-
clgiles-ist-psu-edu-8875	57	3	.	.	.
clgiles-ist-psu-edu-8875	58	1	Eeffective(r	eeffective(r	ADD
clgiles-ist-psu-edu-8875	58	2	)	)	-RRB-
clgiles-ist-psu-edu-8875	58	3	=	=	NFP
clgiles-ist-psu-edu-8875	58	4	Referenced(S	Referenced(S	NNP
clgiles-ist-psu-edu-8875	58	5	)	)	-RRB-
clgiles-ist-psu-edu-8875	58	6	Crawled(r	crawled(r	NN
clgiles-ist-psu-edu-8875	58	7	)	)	-RRB-
clgiles-ist-psu-edu-8875	58	8	(	(	-LRB-
clgiles-ist-psu-edu-8875	58	9	3	3	CD
clgiles-ist-psu-edu-8875	58	10	)	)	-RRB-
clgiles-ist-psu-edu-8875	58	11	4	4	CD
clgiles-ist-psu-edu-8875	58	12	.	.	.
clgiles-ist-psu-edu-8875	59	1	EXPERIMENTS	EXPERIMENTS	NNP
clgiles-ist-psu-edu-8875	59	2	Rank	Rank	NNP
clgiles-ist-psu-edu-8875	59	3	User	user	NN
clgiles-ist-psu-edu-8875	59	4	-	-	HYPH
clgiles-ist-psu-edu-8875	59	5	agent	agent	NN
clgiles-ist-psu-edu-8875	59	6	Content	Content	NNP
clgiles-ist-psu-edu-8875	59	7	Ethicality	Ethicality	NNP
clgiles-ist-psu-edu-8875	59	8	1	1	CD
clgiles-ist-psu-edu-8875	59	9	hyperestraier/1.4.9	hyperestraier/1.4.9	CD
clgiles-ist-psu-edu-8875	59	10	0.95621	0.95621	CD
clgiles-ist-psu-edu-8875	59	11	2	2	CD
clgiles-ist-psu-edu-8875	59	12	Teemer	teemer	NN
clgiles-ist-psu-edu-8875	59	13	0.01942	0.01942	CD
clgiles-ist-psu-edu-8875	59	14	3	3	CD
clgiles-ist-psu-edu-8875	59	15	msnbot	msnbot	NN
clgiles-ist-psu-edu-8875	59	16	-	-	HYPH
clgiles-ist-psu-edu-8875	59	17	media/1.0	media/1.0	NNP
clgiles-ist-psu-edu-8875	59	18	0.00632	0.00632	DT
clgiles-ist-psu-edu-8875	59	19	4	4	CD
clgiles-ist-psu-edu-8875	59	20	Yahoo	Yahoo	NNP
clgiles-ist-psu-edu-8875	59	21	!	!	.
clgiles-ist-psu-edu-8875	60	1	Slurp	Slurp	NNS
clgiles-ist-psu-edu-8875	60	2	0.00417	0.00417	CD
clgiles-ist-psu-edu-8875	60	3	5	5	CD
clgiles-ist-psu-edu-8875	60	4	charlotte/1.0b	charlotte/1.0b	CD
clgiles-ist-psu-edu-8875	60	5	0.00394	0.00394	CD
clgiles-ist-psu-edu-8875	60	6	6	6	CD
clgiles-ist-psu-edu-8875	60	7	gigabot/3.0	gigabot/3.0	NNP
clgiles-ist-psu-edu-8875	60	8	0.00370	0.00370	IN
clgiles-ist-psu-edu-8875	60	9	7	7	CD
clgiles-ist-psu-edu-8875	60	10	nutch	nutch	NN
clgiles-ist-psu-edu-8875	60	11	test	test	NN
clgiles-ist-psu-edu-8875	60	12	/	/	SYM
clgiles-ist-psu-edu-8875	60	13	nutch-0.9	nutch-0.9	NN
clgiles-ist-psu-edu-8875	60	14	0.00316	0.00316	CD
clgiles-ist-psu-edu-8875	60	15	8	8	CD
clgiles-ist-psu-edu-8875	60	16	googlebot	googlebot	NN
clgiles-ist-psu-edu-8875	60	17	-	-	HYPH
clgiles-ist-psu-edu-8875	60	18	image/1.0	image/1.0	NNP
clgiles-ist-psu-edu-8875	60	19	0.00315	0.00315	CD
clgiles-ist-psu-edu-8875	60	20	9	9	CD
clgiles-ist-psu-edu-8875	60	21	Ask	Ask	NNP
clgiles-ist-psu-edu-8875	60	22	Jeeves	Jeeves	NNPS
clgiles-ist-psu-edu-8875	60	23	/	/	SYM
clgiles-ist-psu-edu-8875	60	24	Teoma	Teoma	NNP
clgiles-ist-psu-edu-8875	60	25	0.00302	0.00302	CD
clgiles-ist-psu-edu-8875	60	26	10	10	CD
clgiles-ist-psu-edu-8875	60	27	googlebot/2.1	googlebot/2.1	NNP
clgiles-ist-psu-edu-8875	60	28	0.00282	0.00282	NNP
clgiles-ist-psu-edu-8875	60	29	Table	table	NN
clgiles-ist-psu-edu-8875	60	30	1	1	CD
clgiles-ist-psu-edu-8875	60	31	:	:	:
clgiles-ist-psu-edu-8875	60	32	Content	content	NN
clgiles-ist-psu-edu-8875	60	33	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	60	34	scores	score	NNS
clgiles-ist-psu-edu-8875	60	35	for	for	IN
clgiles-ist-psu-edu-8875	60	36	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	60	37	vis-	vis-	NN
clgiles-ist-psu-edu-8875	60	38	ited	ite	VBD
clgiles-ist-psu-edu-8875	60	39	our	-PRON-	PRP$
clgiles-ist-psu-edu-8875	60	40	test	test	NN
clgiles-ist-psu-edu-8875	60	41	site	site	NN
clgiles-ist-psu-edu-8875	60	42	.	.	.
clgiles-ist-psu-edu-8875	61	1	Table	table	NN
clgiles-ist-psu-edu-8875	61	2	1	1	CD
clgiles-ist-psu-edu-8875	61	3	and	and	CC
clgiles-ist-psu-edu-8875	61	4	2	2	CD
clgiles-ist-psu-edu-8875	61	5	list	list	NN
clgiles-ist-psu-edu-8875	61	6	the	the	DT
clgiles-ist-psu-edu-8875	61	7	content	content	NN
clgiles-ist-psu-edu-8875	61	8	and	and	CC
clgiles-ist-psu-edu-8875	61	9	access	access	NN
clgiles-ist-psu-edu-8875	61	10	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	61	11	results	result	NNS
clgiles-ist-psu-edu-8875	61	12	for	for	IN
clgiles-ist-psu-edu-8875	61	13	top	top	JJ
clgiles-ist-psu-edu-8875	61	14	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	61	15	that	that	WDT
clgiles-ist-psu-edu-8875	61	16	visited	visit	VBD
clgiles-ist-psu-edu-8875	61	17	our	-PRON-	PRP$
clgiles-ist-psu-edu-8875	61	18	test	test	NN
clgiles-ist-psu-edu-8875	61	19	website	website	NN
clgiles-ist-psu-edu-8875	61	20	during	during	IN
clgiles-ist-psu-edu-8875	61	21	the	the	DT
clgiles-ist-psu-edu-8875	61	22	time	time	NN
clgiles-ist-psu-edu-8875	61	23	of	of	IN
clgiles-ist-psu-edu-8875	61	24	the	the	DT
clgiles-ist-psu-edu-8875	61	25	study	study	NN
clgiles-ist-psu-edu-8875	61	26	.	.	.
clgiles-ist-psu-edu-8875	62	1	Higher	high	JJR
clgiles-ist-psu-edu-8875	62	2	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	62	3	scores	score	NNS
clgiles-ist-psu-edu-8875	62	4	represent	represent	VBP
clgiles-ist-psu-edu-8875	62	5	unethical	unethical	JJ
clgiles-ist-psu-edu-8875	62	6	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	62	7	.	.	.
clgiles-ist-psu-edu-8875	63	1	The	the	DT
clgiles-ist-psu-edu-8875	63	2	effective	effective	JJ
clgiles-ist-psu-edu-8875	63	3	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	63	4	of	of	IN
clgiles-ist-psu-edu-8875	63	5	Google	Google	NNP
clgiles-ist-psu-edu-8875	63	6	,	,	,
clgiles-ist-psu-edu-8875	63	7	Yahoo	Yahoo	NNP
clgiles-ist-psu-edu-8875	63	8	,	,	,
clgiles-ist-psu-edu-8875	63	9	MSN	MSN	NNP
clgiles-ist-psu-edu-8875	63	10	and	and	CC
clgiles-ist-psu-edu-8875	63	11	Baidu	Baidu	NNP
clgiles-ist-psu-edu-8875	63	12	are	be	VBP
clgiles-ist-psu-edu-8875	63	13	shown	show	VBN
clgiles-ist-psu-edu-8875	63	14	in	in	IN
clgiles-ist-psu-edu-8875	63	15	Table	table	NN
clgiles-ist-psu-edu-8875	63	16	3	3	CD
clgiles-ist-psu-edu-8875	63	17	.	.	.
clgiles-ist-psu-edu-8875	64	1	The	the	DT
clgiles-ist-psu-edu-8875	64	2	data	data	NN
clgiles-ist-psu-edu-8875	64	3	is	be	VBZ
clgiles-ist-psu-edu-8875	64	4	collected	collect	VBN
clgiles-ist-psu-edu-8875	64	5	between	between	IN
clgiles-ist-psu-edu-8875	64	6	2008/05/13	2008/05/13	CD
clgiles-ist-psu-edu-8875	64	7	Rank	Rank	NNP
clgiles-ist-psu-edu-8875	64	8	User	User	NNP
clgiles-ist-psu-edu-8875	64	9	-	-	HYPH
clgiles-ist-psu-edu-8875	64	10	agent	agent	NN
clgiles-ist-psu-edu-8875	64	11	Access	Access	NNP
clgiles-ist-psu-edu-8875	64	12	Ethicality	Ethicality	NNP
clgiles-ist-psu-edu-8875	64	13	1	1	CD
clgiles-ist-psu-edu-8875	64	14	msnbot	msnbot	NN
clgiles-ist-psu-edu-8875	64	15	-	-	HYPH
clgiles-ist-psu-edu-8875	64	16	media/1.0	media/1.0	NNP
clgiles-ist-psu-edu-8875	64	17	0.3317	0.3317	CD
clgiles-ist-psu-edu-8875	64	18	2	2	CD
clgiles-ist-psu-edu-8875	64	19	hyperestraier/1.4.9	hyperestraier/1.4.9	CD
clgiles-ist-psu-edu-8875	64	20	0.3278	0.3278	CD
clgiles-ist-psu-edu-8875	64	21	3	3	CD
clgiles-ist-psu-edu-8875	64	22	Yahoo	Yahoo	NNP
clgiles-ist-psu-edu-8875	64	23	!	!	.
clgiles-ist-psu-edu-8875	65	1	Slurp/3.0	Slurp/3.0	NNP
clgiles-ist-psu-edu-8875	65	2	0.2949	0.2949	CD
clgiles-ist-psu-edu-8875	65	3	4	4	CD
clgiles-ist-psu-edu-8875	65	4	Teemer	Teemer	NNP
clgiles-ist-psu-edu-8875	65	5	0.2744	0.2744	CD
clgiles-ist-psu-edu-8875	65	6	5	5	CD
clgiles-ist-psu-edu-8875	65	7	Arietis	Arietis	NNP
clgiles-ist-psu-edu-8875	65	8	/	/	SYM
clgiles-ist-psu-edu-8875	65	9	Nutch-0.9	Nutch-0.9	NNP
clgiles-ist-psu-edu-8875	65	10	0.0984	0.0984	CD
clgiles-ist-psu-edu-8875	65	11	6	6	CD
clgiles-ist-psu-edu-8875	65	12	msnbot/1.0	msnbot/1.0	NN
clgiles-ist-psu-edu-8875	65	13	0.098	0.098	CD
clgiles-ist-psu-edu-8875	65	14	7	7	CD
clgiles-ist-psu-edu-8875	65	15	disco	disco	NN
clgiles-ist-psu-edu-8875	65	16	/	/	SYM
clgiles-ist-psu-edu-8875	65	17	Nutch-1.0-dev	Nutch-1.0-dev	NNP
clgiles-ist-psu-edu-8875	65	18	0.0776	0.0776	CD
clgiles-ist-psu-edu-8875	65	19	8	8	CD
clgiles-ist-psu-edu-8875	65	20	ia	ia	NN
clgiles-ist-psu-edu-8875	65	21	archiver	archiver	NN
clgiles-ist-psu-edu-8875	65	22	0.077	0.077	CD
clgiles-ist-psu-edu-8875	65	23	9	9	CD
clgiles-ist-psu-edu-8875	65	24	gigabot/3.0	gigabot/3.0	$
clgiles-ist-psu-edu-8875	65	25	0.0079	0.0079	CD
clgiles-ist-psu-edu-8875	65	26	10	10	CD
clgiles-ist-psu-edu-8875	65	27	googlebot/2.1	googlebot/2.1	NNP
clgiles-ist-psu-edu-8875	65	28	0.0075	0.0075	CD
clgiles-ist-psu-edu-8875	65	29	Table	table	NN
clgiles-ist-psu-edu-8875	65	30	2	2	CD
clgiles-ist-psu-edu-8875	65	31	:	:	:
clgiles-ist-psu-edu-8875	65	32	Access	access	NN
clgiles-ist-psu-edu-8875	65	33	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	65	34	scores	score	NNS
clgiles-ist-psu-edu-8875	65	35	for	for	IN
clgiles-ist-psu-edu-8875	65	36	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	65	37	visited	visit	VBD
clgiles-ist-psu-edu-8875	65	38	our	-PRON-	PRP$
clgiles-ist-psu-edu-8875	65	39	test	test	NN
clgiles-ist-psu-edu-8875	65	40	site	site	NN
clgiles-ist-psu-edu-8875	65	41	.	.	.
clgiles-ist-psu-edu-8875	66	1	to	to	IN
clgiles-ist-psu-edu-8875	66	2	2008/06/21	2008/06/21	CD
clgiles-ist-psu-edu-8875	66	3	.	.	.
clgiles-ist-psu-edu-8875	67	1	Site	site	NN
clgiles-ist-psu-edu-8875	67	2	1	1	CD
clgiles-ist-psu-edu-8875	67	3	is	be	VBZ
clgiles-ist-psu-edu-8875	67	4	CiteSeerx	CiteSeerx	NNP
clgiles-ist-psu-edu-8875	67	5	,	,	,
clgiles-ist-psu-edu-8875	67	6	a	a	DT
clgiles-ist-psu-edu-8875	67	7	large	large	JJ
clgiles-ist-psu-edu-8875	67	8	scale	scale	NN
clgiles-ist-psu-edu-8875	67	9	academic	academic	JJ
clgiles-ist-psu-edu-8875	67	10	digital	digital	JJ
clgiles-ist-psu-edu-8875	67	11	library	library	NN
clgiles-ist-psu-edu-8875	67	12	for	for	IN
clgiles-ist-psu-edu-8875	67	13	computer	computer	NN
clgiles-ist-psu-edu-8875	67	14	science	science	NN
clgiles-ist-psu-edu-8875	67	15	.	.	.
clgiles-ist-psu-edu-8875	68	1	Site	site	NN
clgiles-ist-psu-edu-8875	68	2	2	2	CD
clgiles-ist-psu-edu-8875	68	3	is	be	VBZ
clgiles-ist-psu-edu-8875	68	4	a	a	DT
clgiles-ist-psu-edu-8875	68	5	Chinese	chinese	JJ
clgiles-ist-psu-edu-8875	68	6	movie	movie	NN
clgiles-ist-psu-edu-8875	68	7	information	information	NN
clgiles-ist-psu-edu-8875	68	8	website	website	NN
clgiles-ist-psu-edu-8875	68	9	.	.	.
clgiles-ist-psu-edu-8875	69	1	Site	site	NN
clgiles-ist-psu-edu-8875	69	2	3	3	CD
clgiles-ist-psu-edu-8875	69	3	is	be	VBZ
clgiles-ist-psu-edu-8875	69	4	guopi.com	guopi.com	ADD
clgiles-ist-psu-edu-8875	69	5	,	,	,
clgiles-ist-psu-edu-8875	69	6	an	an	DT
clgiles-ist-psu-edu-8875	69	7	online	online	JJ
clgiles-ist-psu-edu-8875	69	8	makeup	makeup	NN
clgiles-ist-psu-edu-8875	69	9	retail	retail	JJ
clgiles-ist-psu-edu-8875	69	10	store	store	NN
clgiles-ist-psu-edu-8875	69	11	.	.	.
clgiles-ist-psu-edu-8875	70	1	5	5	CD
clgiles-ist-psu-edu-8875	70	2	.	.	.
clgiles-ist-psu-edu-8875	71	1	CONCLUSIONS	conclusion	NNS
clgiles-ist-psu-edu-8875	71	2	We	-PRON-	PRP
clgiles-ist-psu-edu-8875	71	3	formally	formally	RB
clgiles-ist-psu-edu-8875	71	4	defined	define	VBD
clgiles-ist-psu-edu-8875	71	5	three	three	CD
clgiles-ist-psu-edu-8875	71	6	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	71	7	scores	score	NNS
clgiles-ist-psu-edu-8875	71	8	to	to	TO
clgiles-ist-psu-edu-8875	71	9	measure	measure	VB
clgiles-ist-psu-edu-8875	71	10	web	web	NN
clgiles-ist-psu-edu-8875	71	11	crawler	crawler	NN
clgiles-ist-psu-edu-8875	71	12	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	71	13	.	.	.
clgiles-ist-psu-edu-8875	72	1	Results	result	NNS
clgiles-ist-psu-edu-8875	72	2	show	show	VBP
clgiles-ist-psu-edu-8875	72	3	that	that	IN
clgiles-ist-psu-edu-8875	72	4	most	most	JJS
clgiles-ist-psu-edu-8875	72	5	commercial	commercial	JJ
clgiles-ist-psu-edu-8875	72	6	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	72	7	receive	receive	VBP
clgiles-ist-psu-edu-8875	72	8	a	a	DT
clgiles-ist-psu-edu-8875	72	9	good	good	JJ
clgiles-ist-psu-edu-8875	72	10	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	72	11	scores	score	NNS
clgiles-ist-psu-edu-8875	72	12	.	.	.
clgiles-ist-psu-edu-8875	73	1	However	however	RB
clgiles-ist-psu-edu-8875	73	2	,	,	,
clgiles-ist-psu-edu-8875	73	3	it	-PRON-	PRP
clgiles-ist-psu-edu-8875	73	4	is	be	VBZ
clgiles-ist-psu-edu-8875	73	5	surprising	surprising	JJ
clgiles-ist-psu-edu-8875	73	6	to	to	TO
clgiles-ist-psu-edu-8875	73	7	see	see	VB
clgiles-ist-psu-edu-8875	73	8	commercial	commercial	JJ
clgiles-ist-psu-edu-8875	73	9	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	73	10	constantly	constantly	RB
clgiles-ist-psu-edu-8875	73	11	disobeying	disobey	VBG
clgiles-ist-psu-edu-8875	73	12	or	or	CC
clgiles-ist-psu-edu-8875	73	13	misin-	misin-	JJ
clgiles-ist-psu-edu-8875	73	14	terpreting	terprete	VBG
clgiles-ist-psu-edu-8875	73	15	some	some	DT
clgiles-ist-psu-edu-8875	73	16	robots.txt	robots.txt	CD
clgiles-ist-psu-edu-8875	73	17	rules	rule	NNS
clgiles-ist-psu-edu-8875	73	18	.	.	.
clgiles-ist-psu-edu-8875	74	1	The	the	DT
clgiles-ist-psu-edu-8875	74	2	crawling	crawl	VBG
clgiles-ist-psu-edu-8875	74	3	algorithms	algorithm	NNS
clgiles-ist-psu-edu-8875	74	4	and	and	CC
clgiles-ist-psu-edu-8875	74	5	policies	policy	NNS
clgiles-ist-psu-edu-8875	74	6	that	that	WDT
clgiles-ist-psu-edu-8875	74	7	lead	lead	VBP
clgiles-ist-psu-edu-8875	74	8	to	to	IN
clgiles-ist-psu-edu-8875	74	9	such	such	JJ
clgiles-ist-psu-edu-8875	74	10	behaviors	behavior	NNS
clgiles-ist-psu-edu-8875	74	11	are	be	VBP
clgiles-ist-psu-edu-8875	74	12	unknown	unknown	JJ
clgiles-ist-psu-edu-8875	74	13	.	.	.
clgiles-ist-psu-edu-8875	75	1	How-	How-	NNP
clgiles-ist-psu-edu-8875	75	2	ever	ever	RB
clgiles-ist-psu-edu-8875	75	3	,	,	,
clgiles-ist-psu-edu-8875	75	4	obtaining	obtain	VBG
clgiles-ist-psu-edu-8875	75	5	more	more	JJR
clgiles-ist-psu-edu-8875	75	6	content	content	NN
clgiles-ist-psu-edu-8875	75	7	is	be	VBZ
clgiles-ist-psu-edu-8875	75	8	an	an	DT
clgiles-ist-psu-edu-8875	75	9	obvious	obvious	JJ
clgiles-ist-psu-edu-8875	75	10	reason	reason	NN
clgiles-ist-psu-edu-8875	75	11	for	for	IN
clgiles-ist-psu-edu-8875	75	12	most	most	JJS
clgiles-ist-psu-edu-8875	75	13	crawlers	crawler	NNS
clgiles-ist-psu-edu-8875	75	14	failing	fail	VBG
clgiles-ist-psu-edu-8875	75	15	to	to	TO
clgiles-ist-psu-edu-8875	75	16	obey	obey	VB
clgiles-ist-psu-edu-8875	75	17	certain	certain	JJ
clgiles-ist-psu-edu-8875	75	18	rules	rule	NNS
clgiles-ist-psu-edu-8875	75	19	.	.	.
clgiles-ist-psu-edu-8875	76	1	Website	website	NN
clgiles-ist-psu-edu-8875	76	2	Crawled	Crawled	NNP
clgiles-ist-psu-edu-8875	76	3	Referenced	Referenced	NNP
clgiles-ist-psu-edu-8875	76	4	Ereturn	Ereturn	NNP
clgiles-ist-psu-edu-8875	76	5	google	google	NNP
clgiles-ist-psu-edu-8875	76	6	Site	Site	NNP
clgiles-ist-psu-edu-8875	76	7	1	1	CD
clgiles-ist-psu-edu-8875	76	8	16799253	16799253	CD
clgiles-ist-psu-edu-8875	76	9	260898	260898	CD
clgiles-ist-psu-edu-8875	76	10	0.01553	0.01553	CD
clgiles-ist-psu-edu-8875	76	11	Site	site	NN
clgiles-ist-psu-edu-8875	76	12	2	2	CD
clgiles-ist-psu-edu-8875	76	13	872001	872001	CD
clgiles-ist-psu-edu-8875	76	14	46469	46469	CD
clgiles-ist-psu-edu-8875	76	15	0.05329	0.05329	CD
clgiles-ist-psu-edu-8875	76	16	Site	site	NN
clgiles-ist-psu-edu-8875	76	17	3	3	CD
clgiles-ist-psu-edu-8875	76	18	368417	368417	CD
clgiles-ist-psu-edu-8875	76	19	145115	145115	CD
clgiles-ist-psu-edu-8875	76	20	0.39389	0.39389	CD
clgiles-ist-psu-edu-8875	76	21	yahoo	yahoo	NNP
clgiles-ist-psu-edu-8875	76	22	Site	site	NN
clgiles-ist-psu-edu-8875	76	23	1	1	CD
clgiles-ist-psu-edu-8875	76	24	17375962	17375962	CD
clgiles-ist-psu-edu-8875	76	25	3919	3919	CD
clgiles-ist-psu-edu-8875	76	26	0.00023	0.00023	CD
clgiles-ist-psu-edu-8875	76	27	Site	site	NN
clgiles-ist-psu-edu-8875	76	28	2	2	CD
clgiles-ist-psu-edu-8875	76	29	502584	502584	CD
clgiles-ist-psu-edu-8875	76	30	1249	1249	CD
clgiles-ist-psu-edu-8875	76	31	0.00249	0.00249	CD
clgiles-ist-psu-edu-8875	76	32	Site	site	NN
clgiles-ist-psu-edu-8875	76	33	3	3	CD
clgiles-ist-psu-edu-8875	76	34	315119	315119	CD
clgiles-ist-psu-edu-8875	76	35	11819	11819	CD
clgiles-ist-psu-edu-8875	76	36	0.03751	0.03751	CD
clgiles-ist-psu-edu-8875	76	37	msn	msn	NNP
clgiles-ist-psu-edu-8875	76	38	Site	site	NN
clgiles-ist-psu-edu-8875	76	39	1	1	CD
clgiles-ist-psu-edu-8875	76	40	677181	677181	CD
clgiles-ist-psu-edu-8875	76	41	362	362	CD
clgiles-ist-psu-edu-8875	76	42	0.00054	0.00054	CD
clgiles-ist-psu-edu-8875	76	43	Site	site	NN
clgiles-ist-psu-edu-8875	76	44	2	2	CD
clgiles-ist-psu-edu-8875	76	45	16330	16330	CD
clgiles-ist-psu-edu-8875	76	46	5448	5448	CD
clgiles-ist-psu-edu-8875	76	47	0.33362	0.33362	CD
clgiles-ist-psu-edu-8875	76	48	Site	site	NN
clgiles-ist-psu-edu-8875	76	49	3	3	CD
clgiles-ist-psu-edu-8875	76	50	51128	51128	CD
clgiles-ist-psu-edu-8875	76	51	3801	3801	CD
clgiles-ist-psu-edu-8875	76	52	0.07434	0.07434	CD
clgiles-ist-psu-edu-8875	76	53	baidu	baidu	JJ
clgiles-ist-psu-edu-8875	76	54	Site	site	NN
clgiles-ist-psu-edu-8875	76	55	1	1	CD
clgiles-ist-psu-edu-8875	76	56	27	27	CD
clgiles-ist-psu-edu-8875	76	57	37	37	CD
clgiles-ist-psu-edu-8875	76	58	1.37037	1.37037	CD
clgiles-ist-psu-edu-8875	76	59	Site	site	NN
clgiles-ist-psu-edu-8875	76	60	2	2	CD
clgiles-ist-psu-edu-8875	76	61	622667	622667	CD
clgiles-ist-psu-edu-8875	76	62	61964	61964	CD
clgiles-ist-psu-edu-8875	76	63	0.09951	0.09951	CD
clgiles-ist-psu-edu-8875	76	64	Site	site	NN
clgiles-ist-psu-edu-8875	76	65	3	3	CD
clgiles-ist-psu-edu-8875	76	66	1830847	1830847	CD
clgiles-ist-psu-edu-8875	76	67	844786	844786	CD
clgiles-ist-psu-edu-8875	76	68	0.46142	0.46142	CD
clgiles-ist-psu-edu-8875	76	69	Table	table	NN
clgiles-ist-psu-edu-8875	76	70	3	3	CD
clgiles-ist-psu-edu-8875	76	71	:	:	:
clgiles-ist-psu-edu-8875	76	72	Comparison	comparison	NN
clgiles-ist-psu-edu-8875	76	73	of	of	IN
clgiles-ist-psu-edu-8875	76	74	the	the	DT
clgiles-ist-psu-edu-8875	76	75	effectiveness	effectiveness	NN
clgiles-ist-psu-edu-8875	76	76	of	of	IN
clgiles-ist-psu-edu-8875	76	77	Google	Google	NNP
clgiles-ist-psu-edu-8875	76	78	,	,	,
clgiles-ist-psu-edu-8875	76	79	Yahoo	Yahoo	NNP
clgiles-ist-psu-edu-8875	76	80	,	,	,
clgiles-ist-psu-edu-8875	76	81	MSN	MSN	NNP
clgiles-ist-psu-edu-8875	76	82	and	and	CC
clgiles-ist-psu-edu-8875	76	83	Baidu	Baidu	NNP
clgiles-ist-psu-edu-8875	76	84	.	.	.
clgiles-ist-psu-edu-8875	77	1	The	the	DT
clgiles-ist-psu-edu-8875	77	2	effective	effective	JJ
clgiles-ist-psu-edu-8875	77	3	ethicality	ethicality	NN
clgiles-ist-psu-edu-8875	77	4	scores	score	NNS
clgiles-ist-psu-edu-8875	77	5	of	of	IN
clgiles-ist-psu-edu-8875	77	6	search	search	NN
clgiles-ist-psu-edu-8875	77	7	engines	engine	NNS
clgiles-ist-psu-edu-8875	77	8	varies	vary	VBZ
clgiles-ist-psu-edu-8875	77	9	sig-	sig-	RB
clgiles-ist-psu-edu-8875	77	10	nificantly	nificantly	RB
clgiles-ist-psu-edu-8875	77	11	for	for	IN
clgiles-ist-psu-edu-8875	77	12	different	different	JJ
clgiles-ist-psu-edu-8875	77	13	websites	website	NNS
clgiles-ist-psu-edu-8875	77	14	.	.	.
clgiles-ist-psu-edu-8875	78	1	Ranking	rank	VBG
clgiles-ist-psu-edu-8875	78	2	by	by	IN
clgiles-ist-psu-edu-8875	78	3	the	the	DT
clgiles-ist-psu-edu-8875	78	4	referenced	referenced	JJ
clgiles-ist-psu-edu-8875	78	5	visits	visit	NNS
clgiles-ist-psu-edu-8875	78	6	,	,	,
clgiles-ist-psu-edu-8875	78	7	Google	Google	NNP
clgiles-ist-psu-edu-8875	78	8	plays	play	VBZ
clgiles-ist-psu-edu-8875	78	9	a	a	DT
clgiles-ist-psu-edu-8875	78	10	dominating	dominating	JJ
clgiles-ist-psu-edu-8875	78	11	role	role	NN
clgiles-ist-psu-edu-8875	78	12	in	in	IN
clgiles-ist-psu-edu-8875	78	13	the	the	DT
clgiles-ist-psu-edu-8875	78	14	US	US	NNP
clgiles-ist-psu-edu-8875	78	15	based	base	VBN
clgiles-ist-psu-edu-8875	78	16	site	site	NN
clgiles-ist-psu-edu-8875	78	17	1	1	CD
clgiles-ist-psu-edu-8875	78	18	and	and	CC
clgiles-ist-psu-edu-8875	78	19	ranks	rank	VBZ
clgiles-ist-psu-edu-8875	78	20	the	the	DT
clgiles-ist-psu-edu-8875	78	21	2nd	2nd	JJ
clgiles-ist-psu-edu-8875	78	22	and	and	CC
clgiles-ist-psu-edu-8875	78	23	3rd	3rd	NN
clgiles-ist-psu-edu-8875	78	24	in	in	IN
clgiles-ist-psu-edu-8875	78	25	the	the	DT
clgiles-ist-psu-edu-8875	78	26	two	two	CD
clgiles-ist-psu-edu-8875	78	27	China	China	NNP
clgiles-ist-psu-edu-8875	78	28	based	base	VBN
clgiles-ist-psu-edu-8875	78	29	websites	website	NNS
clgiles-ist-psu-edu-8875	78	30	.	.	.
clgiles-ist-psu-edu-8875	79	1	Baidu	Baidu	NNP
clgiles-ist-psu-edu-8875	79	2	leads	lead	VBZ
clgiles-ist-psu-edu-8875	79	3	in	in	IN
clgiles-ist-psu-edu-8875	79	4	the	the	DT
clgiles-ist-psu-edu-8875	79	5	search	search	NN
clgiles-ist-psu-edu-8875	79	6	market	market	NN
clgiles-ist-psu-edu-8875	79	7	in	in	IN
clgiles-ist-psu-edu-8875	79	8	China	China	NNP
clgiles-ist-psu-edu-8875	79	9	.	.	.
clgiles-ist-psu-edu-8875	80	1	6	6	CD
clgiles-ist-psu-edu-8875	80	2	.	.	.
clgiles-ist-psu-edu-8875	81	1	REFERENCES	reference	NNS
clgiles-ist-psu-edu-8875	81	2	[	[	-LRB-
clgiles-ist-psu-edu-8875	81	3	1	1	CD
clgiles-ist-psu-edu-8875	81	4	]	]	-RRB-
clgiles-ist-psu-edu-8875	81	5	D.	D.	NNP
clgiles-ist-psu-edu-8875	81	6	Eichmann	Eichmann	NNP
clgiles-ist-psu-edu-8875	81	7	.	.	.
clgiles-ist-psu-edu-8875	82	1	Ethical	ethical	JJ
clgiles-ist-psu-edu-8875	82	2	web	web	NN
clgiles-ist-psu-edu-8875	82	3	agents	agent	NNS
clgiles-ist-psu-edu-8875	82	4	.	.	.
clgiles-ist-psu-edu-8875	83	1	Computer	Computer	NNP
clgiles-ist-psu-edu-8875	83	2	Networks	Networks	NNPS
clgiles-ist-psu-edu-8875	83	3	and	and	CC
clgiles-ist-psu-edu-8875	83	4	ISDN	ISDN	NNP
clgiles-ist-psu-edu-8875	83	5	Systems	Systems	NNP
clgiles-ist-psu-edu-8875	83	6	,	,	,
clgiles-ist-psu-edu-8875	83	7	28(1	28(1	CD
clgiles-ist-psu-edu-8875	83	8	-	-	HYPH
clgiles-ist-psu-edu-8875	83	9	2):127–136	2):127–136	CD
clgiles-ist-psu-edu-8875	83	10	,	,	,
clgiles-ist-psu-edu-8875	83	11	1995	1995	CD
clgiles-ist-psu-edu-8875	83	12	.	.	.
clgiles-ist-psu-edu-8875	84	1	[	[	-LRB-
clgiles-ist-psu-edu-8875	84	2	2	2	LS
clgiles-ist-psu-edu-8875	84	3	]	]	-RRB-
clgiles-ist-psu-edu-8875	84	4	S.	S.	NNP
clgiles-ist-psu-edu-8875	84	5	Kolay	Kolay	NNP
clgiles-ist-psu-edu-8875	84	6	,	,	,
clgiles-ist-psu-edu-8875	84	7	P.	P.	NNP
clgiles-ist-psu-edu-8875	84	8	D’Alberto	D’Alberto	NNP
clgiles-ist-psu-edu-8875	84	9	,	,	,
clgiles-ist-psu-edu-8875	84	10	A.	A.	NNP
clgiles-ist-psu-edu-8875	84	11	Dasdan	Dasdan	NNP
clgiles-ist-psu-edu-8875	84	12	,	,	,
clgiles-ist-psu-edu-8875	84	13	and	and	CC
clgiles-ist-psu-edu-8875	84	14	A.	A.	NNP
clgiles-ist-psu-edu-8875	84	15	Bhattacharjee	Bhattacharjee	NNP
clgiles-ist-psu-edu-8875	84	16	.	.	.
clgiles-ist-psu-edu-8875	85	1	A	a	DT
clgiles-ist-psu-edu-8875	85	2	larger	large	JJR
clgiles-ist-psu-edu-8875	85	3	scale	scale	NN
clgiles-ist-psu-edu-8875	85	4	study	study	NN
clgiles-ist-psu-edu-8875	85	5	of	of	IN
clgiles-ist-psu-edu-8875	85	6	robots.txt	robots.txt	NNP
clgiles-ist-psu-edu-8875	85	7	.	.	.
clgiles-ist-psu-edu-8875	86	1	In	in	IN
clgiles-ist-psu-edu-8875	86	2	WWW	WWW	NNP
clgiles-ist-psu-edu-8875	86	3	’	'	''
clgiles-ist-psu-edu-8875	86	4	08	08	CD
clgiles-ist-psu-edu-8875	86	5	:	:	:
clgiles-ist-psu-edu-8875	86	6	Proceeding	proceed	VBG
clgiles-ist-psu-edu-8875	86	7	of	of	IN
clgiles-ist-psu-edu-8875	86	8	the	the	DT
clgiles-ist-psu-edu-8875	86	9	17th	17th	JJ
clgiles-ist-psu-edu-8875	86	10	international	international	JJ
clgiles-ist-psu-edu-8875	86	11	conference	conference	NN
clgiles-ist-psu-edu-8875	86	12	on	on	IN
clgiles-ist-psu-edu-8875	86	13	World	World	NNP
clgiles-ist-psu-edu-8875	86	14	Wide	Wide	NNP
clgiles-ist-psu-edu-8875	86	15	Web	Web	NNP
clgiles-ist-psu-edu-8875	86	16	,	,	,
clgiles-ist-psu-edu-8875	86	17	pages	page	NNS
clgiles-ist-psu-edu-8875	86	18	1171–1172	1171–1172	CD
clgiles-ist-psu-edu-8875	86	19	,	,	,
clgiles-ist-psu-edu-8875	86	20	New	New	NNP
clgiles-ist-psu-edu-8875	86	21	York	York	NNP
clgiles-ist-psu-edu-8875	86	22	,	,	,
clgiles-ist-psu-edu-8875	86	23	NY	NY	NNP
clgiles-ist-psu-edu-8875	86	24	,	,	,
clgiles-ist-psu-edu-8875	86	25	USA	USA	NNP
clgiles-ist-psu-edu-8875	86	26	,	,	,
clgiles-ist-psu-edu-8875	86	27	2008	2008	CD
clgiles-ist-psu-edu-8875	86	28	.	.	.
clgiles-ist-psu-edu-8875	87	1	ACM	ACM	NNP
clgiles-ist-psu-edu-8875	87	2	.	.	.
clgiles-ist-psu-edu-8875	88	1	[	[	-LRB-
clgiles-ist-psu-edu-8875	88	2	3	3	CD
clgiles-ist-psu-edu-8875	88	3	]	]	-RRB-
clgiles-ist-psu-edu-8875	88	4	Y.	y.	NN
clgiles-ist-psu-edu-8875	89	1	Sun	Sun	NNP
clgiles-ist-psu-edu-8875	89	2	,	,	,
clgiles-ist-psu-edu-8875	89	3	Z.	Z.	NNP
clgiles-ist-psu-edu-8875	89	4	Zhuang	Zhuang	NNP
clgiles-ist-psu-edu-8875	89	5	,	,	,
clgiles-ist-psu-edu-8875	89	6	and	and	CC
clgiles-ist-psu-edu-8875	89	7	C.	C.	NNP
clgiles-ist-psu-edu-8875	89	8	L.	L.	NNP
clgiles-ist-psu-edu-8875	89	9	Giles	Giles	NNP
clgiles-ist-psu-edu-8875	89	10	.	.	.
clgiles-ist-psu-edu-8875	90	1	A	a	DT
clgiles-ist-psu-edu-8875	90	2	large	large	JJ
clgiles-ist-psu-edu-8875	90	3	-	-	HYPH
clgiles-ist-psu-edu-8875	90	4	scale	scale	NN
clgiles-ist-psu-edu-8875	90	5	study	study	NN
clgiles-ist-psu-edu-8875	90	6	of	of	IN
clgiles-ist-psu-edu-8875	90	7	robots.txt	robots.txt	NNP
clgiles-ist-psu-edu-8875	90	8	.	.	.
clgiles-ist-psu-edu-8875	91	1	In	in	IN
clgiles-ist-psu-edu-8875	91	2	WWW	WWW	NNP
clgiles-ist-psu-edu-8875	91	3	’	'	''
clgiles-ist-psu-edu-8875	91	4	07	07	CD
clgiles-ist-psu-edu-8875	91	5	,	,	,
clgiles-ist-psu-edu-8875	91	6	2007	2007	CD
clgiles-ist-psu-edu-8875	91	7	.	.	.
clgiles-ist-psu-edu-8875	92	1	[	[	-LRB-
clgiles-ist-psu-edu-8875	92	2	4	4	CD
clgiles-ist-psu-edu-8875	92	3	]	]	-RRB-
clgiles-ist-psu-edu-8875	92	4	M.	M.	NNP
clgiles-ist-psu-edu-8875	92	5	Thelwall	Thelwall	NNP
clgiles-ist-psu-edu-8875	92	6	and	and	CC
clgiles-ist-psu-edu-8875	92	7	D.	D.	NNP
clgiles-ist-psu-edu-8875	92	8	Stuart	Stuart	NNP
clgiles-ist-psu-edu-8875	92	9	.	.	.
clgiles-ist-psu-edu-8875	93	1	Web	web	NN
clgiles-ist-psu-edu-8875	93	2	crawling	crawl	VBG
clgiles-ist-psu-edu-8875	93	3	ethics	ethic	NNS
clgiles-ist-psu-edu-8875	93	4	revisited	revisit	VBN
clgiles-ist-psu-edu-8875	93	5	:	:	:
clgiles-ist-psu-edu-8875	93	6	Cost	cost	NN
clgiles-ist-psu-edu-8875	93	7	,	,	,
clgiles-ist-psu-edu-8875	93	8	privacy	privacy	NN
clgiles-ist-psu-edu-8875	93	9	,	,	,
clgiles-ist-psu-edu-8875	93	10	and	and	CC
clgiles-ist-psu-edu-8875	93	11	denial	denial	NN
clgiles-ist-psu-edu-8875	93	12	of	of	IN
clgiles-ist-psu-edu-8875	93	13	service	service	NN
clgiles-ist-psu-edu-8875	93	14	.	.	.
clgiles-ist-psu-edu-8875	94	1	J.	J.	NNP
clgiles-ist-psu-edu-8875	95	1	Am	Am	NNP
clgiles-ist-psu-edu-8875	95	2	.	.	.
clgiles-ist-psu-edu-8875	96	1	Soc	Soc	NNP
clgiles-ist-psu-edu-8875	96	2	.	.	.
clgiles-ist-psu-edu-8875	97	1	Inf	inf	NN
clgiles-ist-psu-edu-8875	97	2	.	.	.
clgiles-ist-psu-edu-8875	98	1	Sci	Sci	NNP
clgiles-ist-psu-edu-8875	98	2	.	.	.
clgiles-ist-psu-edu-8875	99	1	Technol	Technol	NNP
clgiles-ist-psu-edu-8875	99	2	.	.	.
clgiles-ist-psu-edu-8875	99	3	,	,	,
clgiles-ist-psu-edu-8875	99	4	57(13):1771–1779	57(13):1771–1779	CD
clgiles-ist-psu-edu-8875	99	5	,	,	,
clgiles-ist-psu-edu-8875	99	6	November	November	NNP
clgiles-ist-psu-edu-8875	99	7	2006	2006	CD
clgiles-ist-psu-edu-8875	99	8	.	.	.
clgiles-ist-psu-edu-8875	100	1	WWW	WWW	NNP
clgiles-ist-psu-edu-8875	100	2	2010	2010	CD
clgiles-ist-psu-edu-8875	100	3	•	•	NNP
clgiles-ist-psu-edu-8875	100	4	Poster	poster	NN
clgiles-ist-psu-edu-8875	100	5	April	April	NNP
clgiles-ist-psu-edu-8875	100	6	26	26	CD
clgiles-ist-psu-edu-8875	100	7	-	-	SYM
clgiles-ist-psu-edu-8875	100	8	30	30	CD
clgiles-ist-psu-edu-8875	100	9	•	•	NNP
clgiles-ist-psu-edu-8875	100	10	Raleigh	Raleigh	NNP
clgiles-ist-psu-edu-8875	100	11	•	•	NNP
clgiles-ist-psu-edu-8875	100	12	NC	NC	NNP
clgiles-ist-psu-edu-8875	100	13	•	•	NN
clgiles-ist-psu-edu-8875	100	14	USA	USA	NNP
clgiles-ist-psu-edu-8875	100	15	1102	1102	CD