Bibliographic Wilderness Bibliographic Wilderness logging URI query params with lograge The lograge gem for taming Rails logs by default will lot the path component of the URI, but leave out the query string/query params. For instance, perhaps you have a URL to your app /search?q=libraries. lograge will log something like: method=GET path=/search format=html… The q=libraries part is completely left out of the log. I kinda … Continue reading logging URI query params with lograge → Notes on Cloudfront in front of Rails Assets on Heroku, with CORS Heroku really recommends using a CDN in front of your Rails app static assets — which, unlike in non-heroku circumstances where a web server like nginx might be taking care of it, otherwise on heroku static assets will be served directly by your Rails app, consuming limited/expensive dyno resources. After evaluating a variety of options … Continue reading Notes on Cloudfront in front of Rails Assets on Heroku, with CORS → ActiveSupport::Cache via ActiveRecord (note to self) There are a variety of things written to use flexible back-end key/value datastores via the ActiveSupport::Cache API. For instance, say, activejob-status. I have sometimes in the past wanted to be able to use such things storing the data in an rdbms, say vai ActiveRecord. Make a table for it. Sure, this won’t be nearly as … Continue reading ActiveSupport::Cache via ActiveRecord (note to self) → Heroku release phase, rails db:migrate, and command failure If you use capistrano to deploy a Rails app, it will typically run a rails db:migrate with every deploy, to apply any database schema changes. If you are deploying to heroku you might want to do the same thing. The heroku “release phase” feature makes this possible. (Introduced in 2017, the release phase feature is … Continue reading Heroku release phase, rails db:migrate, and command failure → Code that Lasts: Sustainable And Usable Open Source Code A presentation I gave at online conference Code4Lib 2021, on Monday March 21. I have realized that the open source projects I am most proud of are a few that have existed for years now, increasing in popularity, with very little maintenance required. Including traject and bento_search. While community aspects matter for open source sustainability, … Continue reading Code that Lasts: Sustainable And Usable Open Source Code → Product management In my career working in the academic sector, I have realized that one thing that is often missing from in-house software development is “product management.” But what does that mean exactly? You don’t know it’s missing if you don’t even realize it’s a thing and people can use different terms to mean different roles/responsibilities. Basically, … Continue reading Product management → Rails auto-scaling on Heroku We are investigating moving our medium-small-ish Rails app to heroku. We looked at both the Rails Autoscale add-on available on heroku marketplace, and the hirefire.io service which is not listed on heroku marketplace and I almost didn’t realize it existed. I guess hirefire.io doesn’t have any kind of a partnership with heroku, but still uses … Continue reading Rails auto-scaling on Heroku → Managed Solr SaaS Options I was recently looking for managed Solr “software-as-a-service” (SaaS) options, and had trouble figuring out what was out there. So I figured I’d share what I learned. Even though my knowledge here is far from exhaustive, and I have only looked seriously at one of the ones I found. The only managed Solr options I … Continue reading Managed Solr SaaS Options → Gem authors, check your release sizes Most gems should probably be a couple hundred kb at most. I’m talking about the package actually stored in and downloaded from rubygems by an app using the gem. After all, source code is just text, and it doesn’t take up much space. OK, maybe some gems have a couple images in there. But if … Continue reading Gem authors, check your release sizes → Every time you decide to solve a problem with code… Every time you decide to solve a problem with code, you are committing part of your future capacity to maintaining and operating that code. Software is never done. Software is drowning the world by James Abley Updating SolrCloud configuration in ruby We have an app that uses Solr. We currently run a Solr in legacy “not cloud” mode. Our solr configuration directory is on disk on the Solr server, and it’s up to our processes to get our desired solr configuration there, and to update it when it changes. We are in the process of moving … Continue reading Updating SolrCloud configuration in ruby → Are you talking to Heroku redis in cleartext or SSL? In “typical” Redis installation, you might be talking to redis on localhost or on a private network, and clients typically talk to redis in cleartext. Redis doesn’t even natively support communications over SSL. (Or maybe it does now with redis6?) However, the Heroku redis add-on (the one from Heroku itself) supports SSL connections via “Stunnel”, … Continue reading Are you talking to Heroku redis in cleartext or SSL? → Comparing performance of a Rails app on different Heroku formations I develop a “digital collections” or “asset management” app, which manages and makes digitized historical objects and their descriptions available to the public, from the collections here at the Science History Institute. The app receives relatively low level of traffic (according to Google Analytics, around 25K pageviews a month), although we want it to be … Continue reading Comparing performance of a Rails app on different Heroku formations → Deep Dive: Moving ruby projects from Travis to Github Actions for CI So this is one of my super wordy posts, if that’s not your thing abort now, but some people like them. We’ll start with a bit of context, then get to some detailed looks at Github Actions features I used to replace my travis builds, with example config files and examination of options available. For … Continue reading Deep Dive: Moving ruby projects from Travis to Github Actions for CI → Unexpected performance characteristics when exploring migrating a Rails app to Heroku I work at a small non-profit research institute. I work on a Rails app that is a “digital collections” or “digital asset management” app. Basically it manages and provides access (public as well as internal) to lots of files and description about those files, mostly images. It’s currently deployed on some self-managed Amazon EC2 instances … Continue reading Unexpected performance characteristics when exploring migrating a Rails app to Heroku → faster_s3_url: Optimized S3 url generation in ruby Subsequent to my previous investigation about S3 URL generation performance, I ended up writing a gem with optimized implementations of S3 URL generation. github: faster_s3_url It has no dependencies (not even aws-sdk). It can speed up both public and presigned URL generation by around an order of magnitude. In benchmarks on my 2015 MacBook compared … Continue reading faster_s3_url: Optimized S3 url generation in ruby → Delete all S3 key versions with ruby AWS SDK v3 If your S3 bucket is versioned, then deleting an object from s3 will leave a previous version there, as a sort of undo history. You may have a “noncurrent expiration lifecycle policy” set which will delete the old versions after so many days, but within that window, they are there. What if you were deleting … Continue reading Delete all S3 key versions with ruby AWS SDK v3 → Github Actions tutorial for ruby CI on Drifting Ruby I’ve been using travis for free automated testing (“continuous integration”, CI) on my open source projects for a long time. It works pretty well. But it’s got some little annoyances here and there, including with github integration, that I don’t really expect to get fixed after its acquisition by private equity. They also seem to … Continue reading Github Actions tutorial for ruby CI on Drifting Ruby → More benchmarking optimized S3 presigned_url generation In a recent post, I explored profiling and optimizing S3 presigned_url generation in ruby to be much faster. In that post, I got down to using a Aws::Sigv4::Signer instance from the AWS SDK, but wondered if there was a bunch more optimization to be done within that black box. Julik posted a comment on that … Continue reading More benchmarking optimized S3 presigned_url generation → Delivery patterns for non-public resources hosted on S3 I work at the Science History Institute on our Digital Collections app (written in Rails), which is kind of a “digital asset management” app combined with a public catalog of our collection. We store many high-resolution TIFF images that can be 100MB+ each, as well as, currently, a handful of PDFs and audio files. We … Continue reading Delivery patterns for non-public resources hosted on S3 →