Lessons Learned from Upgrading Rails from Version 7.1 to 7.2

Our Ruby on Rails web application was running on version 7.1, and we decided it was time to upgrade to version 7.2. This release included several improvements that would benefit our application, particularly two features we were excited about:

Query performance analysis: The new .explain method on ActiveRecord::Relation makes it much easier to analyze and optimize database queries directly from our models (https://blog.saeloun.com/2024/11/21/rails-7-2-adds-support-for-explain-method-to-activerecord-relation)
Safer background job execution: A critical fix that prevents jobs enqueued within transactions from executing immediately—they now properly wait until the transaction commits, avoiding potential race conditions and data inconsistencies (https://guides.rubyonrails.org/7_2_release_notes.html#prevent-jobs-from-being-scheduled-within-transactions)

While Rails 8.0 was already available, we chose an incremental upgrade path to minimize risk and isolate any compatibility issues. We expected the move to 7.2 to be relatively straightforward. It wasn't. The upgrade broke 101 out of 5,333 tests, with some failures proving surprisingly difficult to diagnose and fix.

Context

Each test runs inside a transaction that is rolled back at the end to ensure isolation. We use the following setting:

RSpec.configure do |config|
  config.use_transactional_fixtures = true
end

Our web application connects to multiple databases, and we frequently perform join operations between them to fetch or update data.

class RealEstate < ApplicationRecord
  connects_to database: { writing: :real_estates, reading: :real_estates_replica }

  has_many :projects, dependent: :destroy
end

class Project < ApplicationRecord
  connects_to database: { writing: :default, reading: :default_replica }

  belongs_to :real_estate
end

① `ActiveRecord::LockWaitTimeout` Exceptions

For some reason, the following error occurred in some of our tests:

ActiveRecord::LockWaitTimeout:
  Mysql2::Error::TimeoutError: Lock wait timeout exceeded; try restarting transaction

The ActiveRecord::LockWaitTimeout error occurs when a transaction waits to acquire a lock on a table or row but exceeds the timeout period.

After investigating the tests and code where this error occurred, we found some common patterns:

Cross-database JOINs were present
The error happened during bulk updates (using update_all or import)

# spec/services/update_projects_service_spec.rb
require "rails_helper"

RSpec.describe UpdateProjectsService, type: :service do
  describe "#perform" do
    let!(:project) { create(:project) }
    let!(:real_estate) { create(:real_estate, available: 1) }

    before do
      project.real_estate = real_estate
      project.save!
    end

    it "runs successfully" do
      projects = Project.all.joins(:real_estate).where(real_estate: { available: 1 })
      projects.update_all(title: "test") # ActiveRecord::LockWaitTimeout
    end
  end
end

Our Hypothesis 🕵

The probable reason is that our tests are already running inside a transaction. update_all and import need to lock rows for writing. The exact cause remained unclear.

As a quick test, we disabled use_transactional_fixtures:

config.use_transactional_fixtures = false

The tests passed! The transaction was the culprit. However, we didn't want to keep it this way because it would mean our tests wouldn't be isolated anymore—changes would persist in the database between tests.

We tried conditionally disabling use_transactional_fixtures with the following configuration:

config.before do |example|
  if example.metadata[:use_truncation]
    config.use_transactional_fixtures = false
  end
end

config.after do |example|
  if example.metadata[:use_truncation]
    # logic for truncating the database
  end
end

But it didn't work. It seems that once you set use_transactional_fixtures, it cannot be changed dynamically during test execution.

# spec/services/update_projects_service_spec.rb
require "rails_helper"

RSpec.describe UpdateProjectsService, type: :service do
  describe "#perform" do
    ...
    it "runs successfully", :use_truncation do
      ...
    end
  end
end

Solution ✅

We got our hands dirty and decided to manually close open transactions by committing them before executing the test.

config.before do |example|
  if example.metadata[:use_truncation]
    # Forcefully exit ALL transactions using raw SQL
    ActiveRecord::Base.connection_handler.connection_pool_list.each do |pool|
      connection = pool.lease_connection
      transaction_count = connection.open_transactions

      # Force close all transactions using raw SQL COMMIT
      # This bypasses ActiveRecord's transaction tracking
      transaction_count.times do
        connection.execute("COMMIT")
      rescue StandardError
        # If COMMIT fails, try ROLLBACK
        begin
          connection.execute("ROLLBACK")
        rescue StandardError
          nil
        end
      end

      # Reset ActiveRecord's internal transaction counter
      # This is necessary because we bypassed its tracking with raw SQL
      connection.instance_variable_set(:@transaction_manager, ActiveRecord::ConnectionAdapters::TransactionManager.new(connection))
    end
  end
end

config.after do |example|
  if example.metadata[:use_truncation]
    # Clean up using TRUNCATE across all database connections
    ActiveRecord::Base.connection_handler.connection_pool_list.each do |pool|
      connection = pool.lease_connection

      # Make sure we're not in a transaction
      begin
        connection.execute("COMMIT") if connection.open_transactions.positive?
      rescue StandardError
        nil
      end

      tables_to_truncate = connection.tables - ["schema_migrations", "ar_internal_metadata"]
      next if tables_to_truncate.empty?

      connection.execute("SET FOREIGN_KEY_CHECKS = 0")
      tables_to_truncate.each do |table|
        connection.execute("TRUNCATE TABLE #{table}")
      end
      connection.execute("SET FOREIGN_KEY_CHECKS = 1")
    end
  end
end

Not pretty, but it worked! Now we could make the tests pass by using the use_truncation metadata for tests that needed it.

# spec/services/update_projects_service_spec.rb
require "rails_helper"

RSpec.describe UpdateProjectsService, type: :service do
  describe "#perform" do
    ...
    it "runs successfully", :use_truncation do
      ...
    end
  end
end

② Cross-Database JOINs Not Working Properly

In some tests, records were created but were not retrievable when fetching them with SQL queries that joined multiple databases.

RealEstate.joins(:company).count
=> 0

RealEstate.first.company
=> #<Company:0x00007fffd1ac3410
  id: 9912
  ...

Because of this, many tests failed.

Our Hypothesis 🕵

Each database starts its own transaction, and no changes are committed to the database. Remember, each test is encapsulated inside a transaction (actually multiple transactions in our case, since we have multiple databases).

Even if real estates are created in the real_estates database, when trying to join the company table (from the default database) with the real_estates table to get companies with real estates, the query might not see anything due to transaction isolation.

Solution ✅

After extensive research, I discovered it was possible to adjust the transaction isolation level to allow reading uncommitted changes within a transaction.

There are four possible isolation levels: READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, and SERIALIZABLE.

Reference: https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html

The default isolation level for our database was READ COMMITTED, where a transaction only sees data that has been committed by other transactions.

We changed the isolation level to READ UNCOMMITTED. Rails provides a way to set this in the config/database.yml file:

test:
  default:
    <<: *default
    database: default_test
    variables:
      sql_mode: TRADITIONAL
      foreign_key_checks: 0
      transaction_isolation: 'READ-UNCOMMITTED' # Here!
  real_estates:
    <<: *real_estates
    database: real_estates_test
    variables:
      sql_mode: TRADITIONAL
      foreign_key_checks: 0
      transaction_isolation: 'READ-UNCOMMITTED' # Here!

After running the problematic tests again, they passed just like in version 7.1! 🎉

This feels like a workaround rather than a proper fix. We didn't find any other solution, but since the problem only occurs in the test environment, we decided to proceed with it for now.

What Caused These Problems?

We didn't identify the exact piece of code responsible for these issues.

Rails 7.2 introduced several changes to transaction handling:

These changes likely introduced breaking behavior in our multi-database setup. Further investigation is needed to understand the root cause.

Conclusion

Upgrading major library versions can be challenging, especially for large frameworks like Rails. When things break, it forces you to dive deeper into the problem and the codebase, which helps you grow as an engineer.

We finally succeeded in upgrading our Rails application (what a relief!), and no issues have surfaced yet. But we'll keep our eyes open!

Lessons Learned from Upgrading Rails from Version 7.1 to 7.2

Lessons Learned from Upgrading Rails from Version 7.1 to 7.2

Context

① `ActiveRecord::LockWaitTimeout` Exceptions

Our Hypothesis 🕵

Solution ✅

② Cross-Database JOINs Not Working Properly

Our Hypothesis 🕵

Solution ✅

What Caused These Problems?

Conclusion

Tony Duong

Related Articles

An interface is not an interface - Recent thoughts about clean coding