The Distant Reader, the Distant Reader Toolbox, and Study Carrels

The Distant Reader

The Distant Reader is a system designed to address two main problems: information overload and reading at scale. It takes collections of files — anything from a dozen journal articles to hundreds of books — as input, and transforms them into structured datasets called study carrels. These carrels are designed to be read by computers as well as people, meaning the same data set can be analyzed programmatically and browsed manually. The Distant Reader's home page is at https://distantreader.org.

Through the Distant Reader, students, researchers, and scholars can quickly address research questions ranging from the mundane ("How many items are in this corpus?") to the sublime ("What is St. Augustine's definition of love, and how does it compare to Rousseau's?").

The Distant Reader Toolbox

The Distant Reader Toolbox is a companion to the Distant Reader, implemented as a set of Python modules. It can be installed from the command line:

pip install reader-toolbox

Or installed from source via GitHub. Once installed, it exposes a command-line interface called rdr ("reader"), which provides a wide array of sub-commands for building, analyzing, and reporting on study carrels. Key commands include:

Command	Purpose
`build`	Create a study carrel from files in a directory
`info`	Output metadata describing a carrel
`bib`	Output bibliographic information
`concordance`	A "poor man's search engine" — locate words in context
`tm`	Apply topic modeling
`ent`	Filter named entities (people, places, organizations)
`pos`	Filter parts-of-speech (nouns, verbs, adjectives, etc.)
`wrd`	Filter statistically computed keywords
`search`	Full-text query against a carrel
`sql`	Use SQL queries against the carrel's database
`semantics`	Apply semantic indexing
`summarize`	Summarize a carrel
`read`	Open a carrel in a Web browser

The Toolbox can also be imported into Python scripts, allowing users to combine its functionality with other Python modules for custom analysis pipelines. Full documentation is available at https://reader-toolbox.readthedocs.io/.

Study Carrels

Study carrels are the output datasets produced by the Distant Reader and the Toolbox. They are organized as directories containing subdirectories and files, and they are designed to be platform-independent and network-independent — meaning no special software is required to read them, and once saved locally, no Internet connection is needed.

Directory Structure

Each carrel contains two primary content directories:

cache/ — the original documents used to create the carrel
txt/ — plain text versions of the cached content; nearly all analysis is performed against files here

Additional subdirectories contain tab-delimited files of extracted features:

Subdirectory	Contents
`adr/`	Email addresses
`bib/`	Bibliographics (authors, titles, dates, etc.)
`ent/`	Named entities (people, organizations, places)
`pos/`	Parts-of-speech (nouns, verbs, adjectives, etc.)
`urls/`	URLs and their domains
`wrd/`	Statistically significant keywords

Although none of these files carry .tsv or .tab extensions, they are all tab-delimited and can be imported into any spreadsheet, database, or programming language.

Two additional subdirectories exist:

figures/ — where images are saved
etc/ — "everything else," including the carrel's stop word list, bag-of-words representation, and a SQLite database file

Depending on how the carrel was modeled, several files may appear at the root:

File	Description
`index.csv`	Echoed metadata file (authors, titles, dates, etc.)
`index.gml`	Graph Modeling Language file for visualizing relationships
`index.htm`	HTML summary of extracted feature characteristics — "start here"
`index.json`	JSON version of the bibliographic report
`index.rdf`	RDF-encoded bibliographic data for the Semantic Web
`index.tsv`	Rudimentary list of characteristics
`index.txt`	Plain text bibliographic report
`index.xml`	Browsable interface to the carrel
`index.zip`	The entire carrel compressed for sharing/collaboration

Desktop Applications for Working with Carrels

Because carrels are platform-independent, a variety of desktop tools can be used to analyze them:

Text editors (BBEdit, NotePad++) — for opening and reading any file in a carrel; not word processors
Word cloud applications (Wordle) — for visualizing frequencies of specific features like keywords or entities
Concordances (AntConc) — for locating words in context and studying the company words keep
Spreadsheet-like tools (OpenRefine) — for importing delimited data and performing powerful faceting, sorting, and counting operations
Topic modeling tools (MALLET, Topic Modeling Tool) — for clustering a corpus into themes and examining how they ebb and flow
Network analysis tools (Gephi) — for visualizing relationships among authors, works, and keywords using graph theory

In Summary

The Distant Reader and its Toolbox take sets of narrative text as input and output study carrels — structured datasets readable by both machines and humans. Study carrels are designed to stand the test of time through their platform and network independence. Users can employ desktop software or the rdr command-line tool to build, search, browse, investigate, and report on carrel content. Extracted features serve as a back-of-the-book index, and they can feed into concordances for closer reading, topic models for theme discovery, and network graphs for relationship visualization. Ultimately, study carrels supplement traditional reading, making researchers more thorough and comprehensive when confronting large volumes of text.