Bibliographic Wilderness Bibliographic Wilderness Code that Lasts: Sustainable And Usable Open Source Code A presentation I gave at online conference Code4Lib 2021, on Monday March 21. I have realized that the open source projects I am most proud of are a few that have existed for years now, increasing in popularity, with very little maintenance required. Including traject and bento_search. While community aspects matter for open source sustainability, … Continue reading Code that Lasts: Sustainable And Usable Open Source Code → Product management In my career working in the academic sector, I have realized that one thing that is often missing from in-house software development is “product management.” But what does that mean exactly? You don’t know it’s missing if you don’t even realize it’s a thing and people can use different terms to mean different roles/responsibilities. Basically, … Continue reading Product management → Rails auto-scaling on Heroku We are investigating moving our medium-small-ish Rails app to heroku. We looked at both the Rails Autoscale add-on available on heroku marketplace, and the hirefire.io service which is not listed on heroku marketplace and I almost didn’t realize it existed. I guess hirefire.io doesn’t have any kind of a partnership with heroku, but still uses … Continue reading Rails auto-scaling on Heroku → Managed Solr SaaS Options I was recently looking for managed Solr “software-as-a-service” (SaaS) options, and had trouble figuring out what was out there. So I figured I’d share what I learned. Even though my knowledge here is far from exhaustive, and I have only looked seriously at one of the ones I found. The only managed Solr options I … Continue reading Managed Solr SaaS Options → Gem authors, check your release sizes Most gems should probably be a couple hundred kb at most. I’m talking about the package actually stored in and downloaded from rubygems by an app using the gem. After all, source code is just text, and it doesn’t take up much space. OK, maybe some gems have a couple images in there. But if … Continue reading Gem authors, check your release sizes → Every time you decide to solve a problem with code… Every time you decide to solve a problem with code, you are committing part of your future capacity to maintaining and operating that code. Software is never done. Software is drowning the world by James Abley Updating SolrCloud configuration in ruby We have an app that uses Solr. We currently run a Solr in legacy “not cloud” mode. Our solr configuration directory is on disk on the Solr server, and it’s up to our processes to get our desired solr configuration there, and to update it when it changes. We are in the process of moving … Continue reading Updating SolrCloud configuration in ruby → Are you talking to Heroku redis in cleartext or SSL? In “typical” Redis installation, you might be talking to redis on localhost or on a private network, and clients typically talk to redis in cleartext. Redis doesn’t even natively support communications over SSL. (Or maybe it does now with redis6?) However, the Heroku redis add-on (the one from Heroku itself) supports SSL connections via “Stunnel”, … Continue reading Are you talking to Heroku redis in cleartext or SSL? → Comparing performance of a Rails app on different Heroku formations I develop a “digital collections” or “asset management” app, which manages and makes digitized historical objects and their descriptions available to the public, from the collections here at the Science History Institute. The app receives relatively low level of traffic (according to Google Analytics, around 25K pageviews a month), although we want it to be … Continue reading Comparing performance of a Rails app on different Heroku formations → Deep Dive: Moving ruby projects from Travis to Github Actions for CI So this is one of my super wordy posts, if that’s not your thing abort now, but some people like them. We’ll start with a bit of context, then get to some detailed looks at Github Actions features I used to replace my travis builds, with example config files and examination of options available. For … Continue reading Deep Dive: Moving ruby projects from Travis to Github Actions for CI → Unexpected performance characteristics when exploring migrating a Rails app to Heroku I work at a small non-profit research institute. I work on a Rails app that is a “digital collections” or “digital asset management” app. Basically it manages and provides access (public as well as internal) to lots of files and description about those files, mostly images. It’s currently deployed on some self-managed Amazon EC2 instances … Continue reading Unexpected performance characteristics when exploring migrating a Rails app to Heroku → faster_s3_url: Optimized S3 url generation in ruby Subsequent to my previous investigation about S3 URL generation performance, I ended up writing a gem with optimized implementations of S3 URL generation. github: faster_s3_url It has no dependencies (not even aws-sdk). It can speed up both public and presigned URL generation by around an order of magnitude. In benchmarks on my 2015 MacBook compared … Continue reading faster_s3_url: Optimized S3 url generation in ruby → Delete all S3 key versions with ruby AWS SDK v3 If your S3 bucket is versioned, then deleting an object from s3 will leave a previous version there, as a sort of undo history. You may have a “noncurrent expiration lifecycle policy” set which will delete the old versions after so many days, but within that window, they are there. What if you were deleting … Continue reading Delete all S3 key versions with ruby AWS SDK v3 → Github Actions tutorial for ruby CI on Drifting Ruby I’ve been using travis for free automated testing (“continuous integration”, CI) on my open source projects for a long time. It works pretty well. But it’s got some little annoyances here and there, including with github integration, that I don’t really expect to get fixed after its acquisition by private equity. They also seem to … Continue reading Github Actions tutorial for ruby CI on Drifting Ruby → More benchmarking optimized S3 presigned_url generation In a recent post, I explored profiling and optimizing S3 presigned_url generation in ruby to be much faster. In that post, I got down to using a Aws::Sigv4::Signer instance from the AWS SDK, but wondered if there was a bunch more optimization to be done within that black box. Julik posted a comment on that … Continue reading More benchmarking optimized S3 presigned_url generation → Delivery patterns for non-public resources hosted on S3 I work at the Science History Institute on our Digital Collections app (written in Rails), which is kind of a “digital asset management” app combined with a public catalog of our collection. We store many high-resolution TIFF images that can be 100MB+ each, as well as, currently, a handful of PDFs and audio files. We … Continue reading Delivery patterns for non-public resources hosted on S3 → Speeding up S3 URL generation in ruby It looks like the AWS SDK is very slow at generating S3 URLs, both public and presigned, and that you can generate around an order of magnitude faster in both cases. This can matter if you are generating hundreds of S3 URLs at once. My app The app I work is a “digital collections” or … Continue reading Speeding up S3 URL generation in ruby → A custom local OHMS front-end Here at the Science History Institute, we’ve written a custom OHMS viewer front-end, to integrate seamlessly with our local custom “content management system” (a Rails-based digital repository app with source available), and provide some local functionality like the ability to download certain artifacts related to the oral history. We spent quite a bit of energy … Continue reading A custom local OHMS front-end → Encrypting patron data (in Rails): why and how Special guest post by Eddie Rubeiz I’m Eddie Rubeiz. Along with the owner of this blog, Jonathan Rochkind, and our system administrator, Dan, I work on the Science History Institute’s digital collections website, where you will find, among other marvels, this picture of the inventor of Styrofoam posing with a Santa “sculpture”, which predates the … Continue reading Encrypting patron data (in Rails): why and how → Intentionally considering fixity checking In our digital collections app rewrite at Science History Institute, we took a moment to step back and  be intentional about how we approach “fixity checking” features and UI, to make sure it’s well-supporting the needs it’s meant to.  I think we do a good job of providing UI to let repository managers and technical … Continue reading Intentionally considering fixity checking →