Introducing Reader SRU

You've just done the perfect search. You used all the right controlled vocabulary terms, limited to the right fields, narrowed by date range. You got hundreds of results -- exactly what you needed. And now you face the soul-crushing part: click, save, authenticate, click, save, click, save...

This essay introduces the Distant Reader SRU server -- a tool used to search the Distant Reader catalog with natural language commands. But more importantly, the tool also evaluates search results, thus taking the search process to another level. In this way, Reader SRU functions much like a RAG (retrieval-augmented generation) system. I believe such functionality will eventually make its way into traditional library catalogs and discovery systems.

The balance of this essay describes the whys and wherefores of Reader SRU.

The Problems to Solve

Given dozens, hundreds, or even thousands of etexts and/or journal articles on a given topic, how are students, researchers, or scholars expected to read, use, and understand all of these things? An ecosystem I call the Distant Reader is designed to address (or rather, embrace) such a problem. ("Thank you Franco Moretti. I have taken the idea of distant reading to heart, and I have been practicing it for about a decade. Can we do dinner sometime?")

Given a set of etexts and/or journal articles, the Distant Reader transforms sets of files containing text into a collection of plain text files, performs various feature extractions against the files, and models them in a myriad of ways. Like a back-of-a-book index on steroids, these models enable people to quickly and easily address research questions that are traditionally too complicated or too time-consuming. Research questions range from the mundane to the sublime and might include: how much did Charles Dickens write and on what topics, to what degree are the writings of Henry David Thoreau chauvinistic, or what are some definitions of love appearing in Victorian literature and how did they change over time?

Ironically, the tools academics use to create collections for reading and analysis get in the way of actually creating collections. These tools (Google Scholar, Web of Science, Arxiv, JSTOR, library catalogs and discovery systems, etc.) might be great at search, but when it comes to getting the found items, the tools limit the academic to an all but endless process of click, save, click, save, click, save. Authenticate. Click, save, click, save, click, save. Authenticate again. Click, save, click, save, click, save,...

Suppose you did the perfect search against Web of Science. You used all the correct controlled vocabulary terms. You limited your query terms to specific fields. You employed Boolean logic to combine different ideas. You even limited the results to date ranges. After doing all this good work, how many items are in your search result? I'll bet you a coffee the number is more than a few. Instead, I'll bet the number of items is measured in at least the hundreds. You've done the perfect search, and now, how can you actually get the items you found? Click, save, click, save, click, save... Nobody is going to go through this process for very long, yet all of the found items, to some degree, can address your research question. Hmmm.

While we are really and truly suffering from information overload, it is actually very difficult to create collections of materials for the purposes of thoroughly and comprehensively reading and analyzing them. Moreover, I have a chicken-and-egg problem. How am I expected to demonstrate the utility of the Distant Reader -- a tool for analyzing large collections of text -- if the academic cannot create large collections in the first place? Such are the problems I am trying to address.

A Four-Fold Solution: Open Access Content, Cataloging, SRU, and MCP

My solution to the problems outlined above is four-fold. First, create a substantial collection of open access content, thus eliminating the need for authentication. Second, use traditional library practices to describe and catalog the content. Third, provide access to the catalog via a Web services protocol, making it easier to output search results in a myriad of formats and eliminating click, save, click, save... And lastly, drive the whole thing with natural language commands, thus reducing the need to understand controlled vocabularies, fielded searches, and Boolean logic.

The following sections describe each part of the solution in greater detail.

Open Access Content

I have been an advocate of open-source software and open access content/publishing since before the phrases were coined. I owe my advocacy to Richard Stallman, Michael Hart, and the principles of librarianship. ("Thank you Richard. Thank you Michael. If it were possible, I'd like to take both of you to dinner.") For example, starting with my Alex Catalogue of Electronic Texts, I have been creating and curating open access content for more than thirty years. Similarly, a long time ago, and in a galaxy far far away, even before HTTP (HyperText Transfer Protocol) was invented, I was systematically collecting, indexing, archiving, and providing access to scholarly open access journals with a thing I called the Mr. Serials Process.

Over the years and to date, I have amassed and curated about 1.7 million open access content items. These items come from various sources: my Alex Catalogue, Project Gutenberg, the HathiTrust, a project called EarlyPrint, a set of Catholic pamphlets from the University of Notre Dame, a set of articles on the topic of COVID-19, and bunches and bunches o' scholarly articles harvested from the Web.

Being a devout liberal artist, I have shaped the collection's strengths to surround "big" ideas such as truth, beauty, honor, justice, virtue, and love. Why? Because I am constantly asking myself questions such as: what is truth, how are truth and beauty related, and how have the answers to these and similar questions evolved over time?

Thus, the collection serves a dual purpose. First, it makes available a set of sample data to be used in the Distant Reader ecosystem. Second, it is my personal library, used to address sets of timeless questions.

The whole thing can be browsed in the "stacks" section of the Distant Reader website. [1] Take note. There is a very thin interface between you and the content, only pure HTTP. Thus, access to the content is not mediated by any middleware such as PHP, Python, or any other type of script that will eventually break and require maintenance. Additionally, access to the content requires zero authentication. No usernames. No passwords. Furthermore, a person can employ any number of Internet spider programs to crawl and download items from the collection. No click, save, click, save, click, save... Access to the content is designed to stand the test of time, or at least as long as there are computers connected to each other via the Transmission Control Protocol/Internet Protocol and those computers support HTTP. (I wonder how long that will be. Will it equal the length of time codices -- books -- have existed? Probably not. The Internet is very fragile.)

Cataloging

The second part of my solution is to use traditional library cataloging practices to provide enhanced access to the collection. In other words, I indexed the collection's content.

The collection, as described above, is not very useful. While access is easy, it is difficult to actually find anything of particular interest. With 1.7 million items in the collection, finding things is akin to finding the venerable needle in a haystack. Librarians have been addressing this problem for centuries, and the solution has been, and continues to be, to create a list of the items in the collection and supplement each item in the list with bibliographic characteristics -- authors, titles, dates, publishers, subjects/keywords, extents, notes, and locations. This process is called cataloging.

For better or for worse, the current best practice for library cataloging is to employ a data structure called MARC (Machine-Readable Cataloging). Articulated in 1965 or so, the MARC format produces records that mimic and echo traditional library catalog cards. (Remember them? Probably not.) When it comes to data structures, the current state of the art is certainly not MARC. To say the least, MARC is arcane, but it was most certainly innovative for its time. It really was. Thus, alas, if I am to apply traditional library cataloging practices to my collection, I need to use MARC. In other words, I need one MARC record for each and every item in my collection. That's 1.7 million MARC records, which is akin to a CSV (comma-separated value) file with 1.7 million rows. Open that in your favorite spreadsheet application.

A few of my collection's subsets (Project Gutenberg, the HathiTrust, and the pamphlets) came with MARC records, and I needed to create MARC records for the balance. Thoroughly describing this process is beyond the scope of this essay but worthy of an additional blog posting. That said, I wrote software that read the contents of databases, I wrote software that parsed XML (eXtensible Markup Language) files, and I applied natural language processing to my content and thus created rudimentary but well-formed and valid MARC records. Each MARC record includes various combinations of the bibliographic characteristics. Every MARC record includes a URL from which the item can be freely downloaded from the stacks, unfettered by authentication. That previous sentence is very important. Consider reading it again. For a good time, peruse the MARC records. They too are available in the stacks. [2]

To index the MARC records -- the descriptions of the items in my collection -- I used a thing called Koha. Here in Library Land, Koha is a type of software known as an open-source integrated library system (ILS). At the very least, an ILS takes MARC records as input, indexes them, and provides a search interface against the index. An ILS also keeps track of the people (patrons) who are authorized to use the library, and thus the ILS not only keeps track of what is in the library, but it also keeps track of who has borrowed what. Since there is only one patron in my library (me), and since all of the items in the library are freely available to anybody, I only needed to take advantage of Koha's indexing functions.

After using Koha to index the MARC records, I am able to use a Web browser to search the collection and get access to its items, the Distant Reader's catalog. [3] Be forewarned. The interface is bare bones, and unless somebody really knows their stuff (which I do), the interface requires click, save, click, save, click, save in order to download items of interest. Can I say, "Three steps forward and one step back"?

All that said, Koha is a model, jewel, and exemplar of library-related open-source software. It costs zero dollars to obtain, but like all open-source software, it is only as free as a free kitten, which is not free. Koha functions and functions well; Koha just goes. It is well supported by individuals, a community of individuals, and commercial communities. Installation is all but trivial for the person (librarian) with a modicum of systems administration skills. I installed it on a cloud-based Linux computer with 2 cores, 4 GB of RAM, and 80 GB of disk space. Since installation and indexing two years ago, I have not needed to touch or maintain anything. Again, Koha just goes. The whole thing costs me about $25/month to keep up and running. koha++

SRU (Search/Retrieve via URL)

The next part of the solution is to employ a machine-readable protocol against the index so search results can be divorced from business logic. A protocol called SRU (Search/Retrieve via URL) addresses this exact issue.

Koha is great, but its implementation is inseparable from traditional library practice. More specifically, the data returned from search results is: 1) formatted as HTML (HyperText Markup Language), which is designed to be read on a screen by people, and 2) very closely coupled with the processes of libraries. How could I expect anything else? After all, Koha is library software. But I'm trying to take libraries and librarianship to a different level, and I don't need or want HTML on a screen joined at the hip with books to be borrowed from a shelf. Instead, I want search results in a machine-readable form so I can repurpose them into something else, something I (or somebody else) might find useful for different purposes. In other words and as I've said elsewhere, I don't want the HTML interface. Instead, "Just give me the data."

SRU does this very well, and SRU is supported by Koha.

SRU is a standardized Web Services protocol. As the name implies, a standardized URL is sent to a Web server. The server interprets the URL, processes its inherent command, and returns the command's results in the form of a standardized data structure -- in this case, XML. In this way, SRU is a REST-ful (Representational State Transfer) protocol, but unlike most REST-ful implementations, the URL and the results are manifested in an industry-articulated fashion. The most difficult aspect of SRU is the query language, CQL (Common Query Language). While very flexible and expressive, its syntax can be daunting, and if the SRU query includes a syntax error, then the whole search will most likely fail. For more detail regarding SRU, see the pages at the Library of Congress. [4]

Under the hood, Koha supports SRU. To use SRU, it needs to be turned on and configured. The first part, turning it on, is relatively easy; to get SRU turned on, a Koha systems administrator "simply" needs to run a few commands. (See the documentation. [5]) The second part, configuration, is more difficult. More specifically, configuration requires a librarian to write what's called an SRU Explain response documenting what the SRU server supports, what it doesn't, and who you can call for technical support. For example, an Explain response addresses questions like: what the size and scope of the indexed collection are, whether the SRU implementation supports fielded queries and if so which ones, and what forms of XML are returned by the server. I did both of these things. I turned on my Koha's implementation of SRU, and I wrote an SRU Explain response. [6]

Like all standards, the SRU standard has gone through phases. Older versions of SRU merely returned search results, but newer versions return search results as well as faceted interpretations. What does this mean? It means categorical variables in the search results are: 1) counted and tabulated, 2) sorted by resulting frequencies, and 3) returned as a part of the search result. In the end, the facets enumerate what subjects are most frequently used to describe the found items, who the most frequently cited authors in the search results are, how many found items are etexts ("books"), how many of the found items are articles, and/or from what sub-collections the found items originated (Project Gutenberg, EarlyPrint, etc.). All of these things -- the facets -- kinda sorta summarize the results and make it easier to issue more nuanced, useful, and subsequent queries.

Out of the box, the Koha implementation of SRU does not support faceted search results. I lamented this fact on a mailing list, and a person named Andreas Roussos came to my rescue. He wrote a patch to the underlying SRU software. [7] After applying the patch, my SRU server flawlessly supported faceted search results. ("Andreas, let's make an appointment for dinner?")

"Okay," you say, "so what?" Well, I can now submit a query to the remote index, get back the research results (plus facets), and repurpose the data-only results in ways I desire. For example, I can format the results to include facets or not. I can format the results to include just titles coupled with drop-down lists to expose details, or not. I can return search results in the form of comma-separated value (CSV) files, JavaScript Object Notation (JSON) files, a network graph, etc.

What did I do? Initially, I: 1) got back the faceted results, 2) augmented the original search with a CQL-valid Boolean AND operator and a facet value, and 3) presented newly minted URLs making it easy for people to drill down and narrow their search results. I called this my Baroque implementation, and it worked quite well until Internet robots found my site, fell into endless loops of queries, and melted down my tiny Koha server.

Since then, I have greatly simplified my interface. It now includes only a single search box (no more pre-made hyperlinked example queries), lists of the resulting facets, and search results with the titles of found items hyperlinked back to the stacks. It does this work in the form of a "thin" SRU client where a query is sent to the SRU server via a Web browser, the server responds with a stream of XML, and the Web browser transforms the response into HTML (or some other format) through the use of a technology called XSL (eXtensible Stylesheet Language). See thin client, and search for the word "love". [8]

Like the collection and curation of the 1.7 million items and like the generation of the 1.7 million MARC records, the writing of the SRU Explain response and the writing of the XSL files -- which will break soon -- are the fodder for additional essays. Prod me, and I'll write them.

To the enterprising library student I submit the following challenge. Write a computer program that takes a simple SRU query as input (for example, "love"), sends the query to the SRU server, awaits the response, parses out each of the URLs from the Distant Reader stacks from the results, and downloads the full text of each item to a given directory. Such a program, especially if it were easy to use, would address all of the problems outlined at the beginning of this missive, namely, do the perfect search, and create a complete collection of the found items ripe for reading and analysis.

Finally, additional kudos go to the good folks at Index Data. Like me, they have been advocates of open-source software and open access content since before the phrases were coined, and Index Data was responsible for the underlying Koha indexer and SRU interface. If it weren't for their accessible, standards-compliant, and bullet-proof software, I would not have been able to get this far. ("Index Data, I'd love to take you to dinner too.")

MCP (Model Context Protocol)

Substantial content has been collected. It has been described and indexed. The interfaces used to query the index are expressive and functional, but the interfaces are geeky, reactive as opposed to proactive, and they do not take great advantage of the possibilities existing in the current computing environment. At the risk of having a solution looking for a problem, adding an MCP (Model Context Protocol) layer to the ecosystem addresses these issues.

MCP is a brand new computing protocol. Developed by the folks at Anthropic, the MCP protocol enables servers that insert LLMs (large-language models) between people and APIs (application programmer interfaces). Here is how the process works:

1. Agent software (like Claude, ChatGPT, ollmcp, etc.) is configured to use an MCP server which resides on a local or remote computer. Based on my experience, configuration is a bit of a pain.
2. The person (I hate the word "user") launches their agent, the agent connects to the configured MCP server(s), and internally enumerates the functions (API calls) the server supports. These API calls map to the features and functions of protocols like SRU.
3. The person issues one or more commands to the agent such as, "Find articles about love." The agent's LLM interprets the command(s) and maps them to one or more functions in the MCP server. The functions are executed, the results are returned, the LLM interprets the results, and the interpretations are returned to the person. In the case of SRU, there is no need to format a query into CQL, and the XML results are reformatted into person-friendly language.
4. Go to Step #3 until tired.

In short, MCP servers are middleware placed between people and APIs. Why is this important? It is important because APIs are cryptic, difficult for non-programmers to get their brains around, and often very complicated. Given a well-documented MCP server, an LLM will have little difficulty matching a person's command to an API call. More importantly, API calls almost always return well-structured data, and LLMs eat well-structured data for lunch; LLMs excel at identifying patterns and anomalies in the results of API calls. Thus, MCP servers make APIs (think "protocols") accessible to everyday people.

A few months ago, I found an MCP server designed to implement SRU. Sam Codefzer wrote the server to work against just about any SRU server, but his MCP server did not support facets, included functionality my SRU server did not support (like the "scan" operation), and imposed a number of default values that were invalid for my implementation. Consequently, I forked the source code and edited it to meet my specific needs; my version of his MCP SRU server is specific to the Distant Reader and can be found on GitHub. [9] ("Sam, you are invited to dinner too!")

[Please try to install my MCP server here, but your mileage may vary because I am not very good at the software distribution process.]

Once my MCP server is installed and configured, a person can submit commands using natural language and get back natural language responses. A demonstration is in order. Below are the commands I issued to my MCP server coupled with what happened behind the scenes and the results:

1. Thoroughly explain the Distant Reader Index, and use the word "love" as an example- The server issued an SRU Explain operation. It queried the index for the word love and output a narrative describing the scope and features of the SRU server as well as an abbreviated evaluation of search results. From the faceted results, I learned the word "fiction" would limit the results significantly.
2. Search for the word love and limit the results to items of fiction. Get a single record. Evaluate the facets- The words love and fiction were combined into an SRU-compatible query, the query was sent to the server, the metadata describing the first search result was retrieved, and the facets were characterized. I noticed an author's name which piqued my interest, Trollope.
3. Limit the results to items authored by Trollope.- This command augmented the previous query to limit by an author's name (Trollope); the query was sent, and a set of stories of fiction, about love, and authored by Trollope was returned. Based on the titles and keywords, the results were characterized and classified into a number of themes giving me an overview of the found documents.
4. Create a CSV file of the results- The server read the metadata of the results and locally saved a simple CSV file, a rudimentary bibliography.
5. Format the results as a bibliography in the Chicago style- The server read the search results' metadata again and then output a bibliography.
6. Save this entire session as an HTML file- The server read the whole of the session's context, reformatted it as an HTML file, and saved it locally.

The demonstration output two files. The first was a CSV file including author, title, date, and URL columns. [10] This file, especially if it is long, is intended to be curated (duplicates removed, names normalized, dates added, etc.). Once curated, the items could be downloaded for both close and distant reading. The software challenge, above, could be used for this purpose.

The second was an HTML file documenting what occurred during the MCP session. [11] Such is what a traditional reference librarian would have written for a student, researcher, or scholar after the reference librarian had performed a DIALOG (bibliographic database) search on the patron's behalf. Albeit a bit wordy, the HTML file is quite informative, and based on my reading, it contains very, very few inaccuracies. In this way, Reader SRU operates as if it were a RAG system. If you had done a search against your own library catalog, to what degree would you be able to summarize the session as the LLM did? Would you have been as thorough? To what degree would you have been able to identify patterns and anomalies, and summarize the session? How much time would it have taken you to do the work? For completeness's sake, take a gander at example SRU search-retrieve in the raw XML. [12]

To summarize this section, MCP servers act as intermediaries between people and APIs, and through the process, they reduce the need for a person to thoroughly understand the API's nitty-gritty details. Reader SRU addresses two large problems when it comes to searching bibliographic databases. First, it trivializes the process of articulating useful queries; Reader SRU makes it easy to submit queries complete with fielded searches, Boolean logic, etc. Second, and almost more importantly, the LLM embedded in Reader SRU interprets the search results faster and more thoroughly than a person can in the same amount of time. In this way, Reader SRU can be used to supplement the process of using and understanding search results. Ironically, this is exactly how SRU was designed to be used when it was articulated more than two decades ago.

Summary

This essay introduced an MCP server I call Reader SRU, a tool used to query a substantial collection of open access etexts and scholarly journal articles. The server's purpose is many-fold. First and foremost, Reader SRU is intended to make it easy to not only query the collection of etexts and articles but also evaluate search results and download (acquire) the items of interest. For the most part, I believe these goals have been accomplished, but "Software is never done."

Second, Reader SRU is intended to make it easy for people to create collections of sample data so they may more easily experiment with another tool of my own design, the Distant Reader. The Distant Reader is a tool used to address the problems of information overload. More specifically, the Distant Reader takes an arbitrary number of narrative files (like etexts or journal articles). The files are transformed into plain text files and then modeled in a myriad of ways. The models can then be read (analyzed) in a number of other ways for the ultimate purpose of using and understanding them. Ironically, the process of acquiring sets of etexts and/or articles is tedious and tiresome. Click, save, click, save... Reader SRU reduces the need for click, save, click, save and thus makes it easier for me to demonstrate the Distant Reader.

Third, as a liberal artist, I am interested in addressing questions surrounding "big" ideas: truth, justice, honor, beauty, trust, power, etc. Reader SRU makes it easier for me to create large, comprehensive collections of materials on just those topics so I might more thoroughly address my questions.

Reader SRU is built on top of three other things, and I described each. The first was a collection of 1.7 million open access etexts and scholarly journal articles. This collection is affectionately called the "stacks". The second was the description and indexing of those items, culminating in the creation of a Koha catalog. The third was the development and implementation of an SRU server designed specifically to work against the catalog.

Reader SRU itself is an MCP wrapper around the SRU server, above. As such, the MCP's LLM takes human language commands as input and maps them to SRU operations. In doing so, Reader SRU greatly reduces the need for a person to understand fielded searching, Boolean logic, let alone CQL. Furthermore, the MCP's underlying LLM interprets the SRU results in ways that are much more scalable when compared to the time and energy required for a person to do the same work. Reader SRU is a riff on the concept of RAG. See output documenting a search for the word love. [11]

Now, here's the leap. When doing whatever type of research, academics are usually expected to search bibliographic indexes (lists of books, articles, etc.). Current-day examples of such indexes include Google Scholar, Web of Science, library catalogs, etc. These tools are great at search, but unlike Reader SRU, they are weak when it comes to search results analysis. I ask you, "When you search a bibliographic index, how many items are typically in the search result, and to what degree are you presented with some sort of characterization or summary of those results?" I assert the degree is small, but with the advent of LLM technology, I predict automated characterization and summarization will happen to a greater degree; I assert the application of LLM technology to search results of academic bibliographic indexes will increase. In fact, you can already begin to see it happening when you search Internet indexes like Google or DuckDuckGo. My hope is: 1) the library community sees this trend too, 2) the library community will think about how functionality -- like Reader SRU -- can be applied to library resources more broadly, and 3) the library community will collaboratively explore ways to make it happen before they (we) feel forced to license such services instead of building them on our own. After all, I have been an advocate of open-source software since before the phrase was coined.

Software Is Never Done

I often say, "Software is never done, and if it were, then it would be called hardware." Such is true for everything described in this essay. The collection in the stacks can always be updated. The descriptions can be augmented, and the indexing can be improved. The developers of the most popular Web browsers have announced the end of built-in XSL support, and consequently, my thin SRU client is going to break. More importantly, I have only broached the problem of click, save, click, save,...

That said, I have addressed the enterprising library student challenge issued above, but I have yet to integrate it into Reader SRU. The linked Python script takes an arbitrary SRU query as input, performs the search, loops through the results, extracts the stacks' URLs and metadata, and caches the lot to a given directory. [13] What I have to do next is transform the script into an MCP function. When done, I ought to be able to issue a natural language command something like "Cache the items found from this search and save the items' metadata along the way." By doing so, no more click, save, click, save, click, save... Wish me luck.

Just for Fun

Just for fun I fed the blog posting to my desktop LLM and asked it to do two things. First, generate an imaginary dinner party with the invited guest. [14] Second, output a few bar jokes with featuring the same people. [15] I think the results are humorous and they help re-enforce some of the essay's salient points. Just for fun.

Links

[1] the "stacks" - https://distantreader.org/stacks/

[2] MARC records in the stacks - https://distantreader.org/stacks/marc/

[3] Distant Reader catalog - http://catalog.distantreader.org/

[4] SRU pages at the Library of Congress - http://www.loc.gov/standards/sru/

[5] Koha instructions for turning on SRU - https://wiki.koha-community.org/wiki/Setting_up_the_Z39.50_and_SRU_Server

[6] Sample SRU Explain response - ./sru-explain.xml

[7] SRU patch - https://gist.github.com/a-roussos/cdafddf9554b4b9f9a5a1d353e027be7

[8] SRU thin client - http://index.distantreader.org/

[9] Reader SRU - https://github.com/ericleasemorgan/reader-sru/

[10] CSV file of search results - ./sru-results.csv

[11] LLM-generated description and summary of search results - ./sru-results.xml

[12] Example SRU search-retrieve results in the raw XML - ./sru-results.xml

[13] Given an SRU query and an output directory, search the SRU server and cache the results - ./reader-sru-retrieve.py

[14] Imaginary dinner party - ./dinner-party.htm

[15] Bar jokes - ./bar-jokes.htm