Category Archives: Programming

Understanding Rails secrets/credentials

What this feature is for

The credentials feature is a way of storing secrets that you don’t want to keep in plaintext, like AWS credentials for example. (In fact, the one and only thing I keep in my main Rails project’s credentials are my Active Storage AWS credentials.)

Why the credentials feature is difficult to learn about

I personally have found Rails credentials really hard to understand. I think there are three reasons why this is.

  1. The official Rails docs about the credentials feature are a little terse and not easily discoverable.
  2. The feature changed somewhat drastically from Rails 4 to Rails 5, even changing names from “secrets” to “credentials”.
  3. I only need to use the feature once in a while, when I’m deploying an application for the first time, meaning there’s lots of time in between to forget everything I knew.

How to work with credentials without frustration

There are perhaps five important things to understand about credentials:

  • Where they’re stored
  • How the master key works
  • How editing credentials work
  • What the deal is with secrets.yml
  • The steps you need to take to set up credentials on a fresh production machine
  • Credentials are stored in config/credentials.yml.enc

    At the risk of stating the obvious, your secrets are stored in config/credentials.yml.enc. That file is encrypted.

    There of course needs to be a way to securely decrypt this encrypted file. That’s done using something called a master key. The master key is just a hash string that gets stored in one of two places: a file called config/master.key (which should NOT be committed to version control) or a RAILS_MASTER_KEY environment variable.

    The development master key and production master key are the same by default

    A key thing to understand, which I found counterintuitive, is that unless you go out of your way to configure it differently, the master key you use in production should be the same as the master key you use in development. If your development master key is stored in config/master.key, create an identical config/master.key on production, containing the same exact key. If your development master key is stored in the RAILS_MASTER_KEY environment variable, set the production RAILS_MASTER_KEY to the exact same value.

    I found this counterintuitive because usually I try to make all my passwords, etc. different for each environment I have. I thought I would need to create a different master key for my production environment. No, I need to not create a different master key.

    (After I originally posted this article, it was pointed out to me that it is possible to configure different keys per environment. According to this article, keys can be configured specific to e.g. production by running rails credentials:edit --environment production.)

    The credentials.yml.enc file is edited in a special way

    Since it’s encrypted, the config/credentials.yml.enc file can’t be edited directly. It can only be edited using the rails credentials:edit command.

    What often throws me for a loop is that a prerequisite to using rails credentials:edit is having the EDITOR environment variable set, which on a fresh production machine I usually don’t. I’m a Vim guy, so I run export EDITOR=vim and then I’m good to go. Then I can run rails credentials:edit and the command will open the credential file, decrypted, in Vim.

    secrets.yml is obsolete

    If you find something online that refers to secrets.yml, you’re looking at an old post. Before Rails 5.2, there was a secrets.yml and secrets.yml.enc instead of the new credentials-related files. Don’t make the mistake of conflating Rails secrets with Rails credentials (like I did several times before learning better!).

    The steps for setting up credentials in production

    1. Take the same master key you’re using in development and put it either in config/master.key or the RAILS_MASTER_KEY environment variable.
    2. Set the EDITOR environment variable to your favorite terminal-based editor.
    3. Run rails credentials:edit to verify that your master key is working properly.

    Helpful links

    I hope my credentials guide is the new best guide on the internet but I’ll link to the sources that helped me put this together.

    Best of luck with your credential management endeavors.

What kinds of Rails tests I write and what kinds I don’t

The challenge of deciding what kinds of tests to write

There are a lot of different kinds of tests a developer could possibly write. In RSpec there are model specs, feature specs, view specs, helper specs, routing specs, controller specs, and request specs. Do you need to write all these different types of tests? If not, which ones should you skip?

The two types of tests I write

In my Rails apps I usually only write two kinds of tests. To use framework-agnostic language, I write model tests and acceptance tests. To use RSpec-specific language, I write model specs and feature specs. Let’s talk about each of these two in a little more detail.

Model tests are relatively isolated. A model test will test the behavior of a particular Ruby class in my Rails application, e.g. the Customer model. Model tests are what give me confidence that my features are working at a fine grain. (I often use the terms “class” and “model” somewhat interchangeably since a Rails model takes the form of a Ruby class.) For example, if I have a method that returns the grand total of the payments a certain customer has made, I’ll write a model test for that method.

What about acceptance tests? Whereas model tests are relatively isolated, acceptance tests are the opposite; they’re relatively integrated. An acceptance test exercises the whole stack, browser and all. Acceptance tests give me confidence that everything works together. For example, if I have CRUD functionality for a Customer resource, I’ll write acceptance tests for creating a customer, updating a customer, and deleting a customer.

A note on terminology

As of the time of this writing there not complete consensus in the testing world on testing terminology (and I don’t expect this to change anytime soon). For example, some developers might consider the terms end-to-end-test, integration test, and acceptance test to refer to basically the same thing, whereas other developers might consider these three terms to refer to three completely distinct types of tests.

What does this mean for you in terms of your testing journey? It means that if you, for example, see a reference to end-to-end tests that doesn’t square with your understanding of the term end-to-end tests, that doesn’t necessarily mean that your understanding is mistaken, it just means that not everyone in the testing world talks about the same concepts using the same exact terms. I think understanding this communication challenge is helpful since otherwise one might get distracted by all the apparent discrepancies.

Testing terminology + RSpec

How do model tests and acceptance tests map to RSpec? Model tests—or model “specs” in RSpec terminology—are the tests that live in the spec/models directory. Nothing but bare RSpec is needed in order to write model specs. What the outside world calls acceptance tests (or again, integration tests or end-to-end tests) are known in RSpec as feature specs. Since feature specs exercise features via the browser, the Capybara library (a library that allows us to manipulate the browser using Ruby) is necessary.

Why, by the way, is an RSpec test called a “spec” instead of a “test”? The idea is that each RSpec file is an “executable specification”. Each test case is actually called an example in RSpec terminology, an example of how that feature is supposed to behave. A set of examples makes up a spec. Personally, I’m willing to play along with this terminology to some extent, but I frankly find it kind of annoying. You’ll find that most of the time I say “test” instead of “spec” or “example”.

The types of tests I don’t write

I typically don’t write view specs, routing specs, or request specs/controller specs. Let’s discuss each of these.

Request specs/controller specs

First, a bit of terminology: request specs and controller specs are two very slightly different things, although for our purposes we can treat the two terms as meaning the same thing. A close enough approximation to the truth is that “controller spec” is the old name and “request spec” is the new name.

I usually don’t write controller/request specs (although I do sometimes). The reason is that I try to have as little code in my controllers as possible. For the most part, my controllers don’t do anything, they just call model objects that do things. Any code that might have been covered by a controller/request spec, I prefer to cover via a feature spec.

View specs, routing specs

I don’t write view specs and I don’t believe I’ve ever encountered anyone who does. The reason is that I’ve never been able to conceive of how a view spec could have value that a feature spec doesn’t already provide.

I also don’t write routing specs, and for the same reason. I find them to be redundant to feature specs.

Conclusion

If you’re just getting started with Rails/RSpec testing and you’re wondering which types of tests to focus on, I would recommend focusing on model specs and feature specs.

How to deploy a Ruby on Rails application to AWS Elastic Beanstalk

Overview

This tutorial will show you how to deploy a Rails application to AWS Elastic Beanstalk.

Using Elastic Beanstalk is just one of many (perhaps an infinite number of!) AWS deployment options. Each approach has different pros and cons. I’ll briefly go over some of them because it’s good to understand the pros and cons of various approaches (at least to an extent) before choosing one.

Manual EC2 deployment

One option is to do things “the old fashioned way” and manually set up a Rails application on a single EC2 instance. This is the approach I go over in this AWS/Rails deployment post and it’s perfectly fine for hobby projects where the stakes are low.

The downside to manual EC2 deployment is you end up with a snowflake server, a server with a one-of-a-kind configuration that’s hard to understand, modify, or replicate.

Elastic Beanstalk

Elastic Beanstalk is kind of analogous to Heroku. The basic idea is the same in that both Elastic Beanstalk and Heroku are abstraction layers on top of AWS services. The big difference is that Heroku is generally really easy and Elastic Beanstalk is a giant pain in the ass.

But the upside is that EB provides more easily replicable and understandable server instances than a manual EC2 instance. The server configuration is expressed in files, and then that configuration can be applied to an indefinite number of servers. This makes scaling easier. It’s also nice to know that if I somehow accidentally blow away one of my EC2 instances, EB will just automatically spin up a new identical one for me.

Another drawback to EB is that I understand EB can be kind of overly rigid. I ran into this trouble myself on a project where I needed to set up Sidekiq. I discovered that EB boxed me in in a way that made Sidekiq setup very difficult. So for a production project that grows over time, EB is perhaps a good place to start, but it should be expected that you might want to migrate to something more flexible sometime in the future.

An infrastructure-as-code approach

An infrastructure-as-code approach is probably the best long-term solution, although it currently also seems to be the most difficult and time-consuming to set up initially.

Options in this area include Ansible, Chef, Puppet, ECS, and probably a lot more. I’ve personally only used Ansible. I found Ansible to be great. This post will of course only cover Elastic Beanstalk though.

Configuration steps

Here are the steps we’ll be carrying out in this tutorial.

  1. Install the Elastic Beanstalk CLI
  2. Create the Elastic Beanstalk application
  3. Create the Elastic Beanstalk environment
  4. Create the RDS instance
  5. Deploy

Let’s get started.

Install the Elastic Beanstalk CLI

Much of what we’ll be doing involves the Elastic Beanstalk CLI (command-line interface). It can be installed with this command:

$ brew update && brew install awsebcli

Create the Elastic Beanstalk application

Now cd into the directory that contains your Rails project and run eb init.

$ eb init

When prompted, Select Create new Application. Accept the defaults for all other options.

When this command finishes running you’ll end up with a file called .elasticbeanstalk/config.yml that looks something like this:

branch-defaults:
  master:
    environment: null
    group_suffix: null
global:
  application_name: discuss_with
  branch: null
  default_ec2_keyname: aws-eb-cwj-post
  default_platform: Ruby 2.6 (Passenger Standalone)
  default_region: us-east-1
  include_git_submodules: true
  instance_profile: null
  platform_name: null
  platform_version: null
  profile: personal
  repository: null
  sc: git
  workspace_type: Application

Now that we’ve created our Elastic Beanstalk application, we’ll need to create an Elastic Beanstalk environment inside of that application. I typically set up one production environment and one staging environment inside a single application.

Create the Elastic Beanstalk environment

The command to create an environment is eb create.

$ eb create

You’ll be prompted for what to call your environment. I called mine discuss-with-production.

For load balancer type, choose application.

This step will take a long time. When it finishes, health will probably be “Severe”. Ignore this.

Set up SECRET_KEY_BASE

We’ll need to set a value for the SECRET_KEY_BASE environment variable. This can be done using the following eb setenv command which just sets the variable to a random string.

$ eb setenv SECRET_KEY_BASE=$(ruby -e "require 'securerandom';puts SecureRandom.hex(64)")

Set up ebextensions

With Elastic Beanstalk, you can add files in an .ebextensions directory at your project root to control how your server is configured. We need to add three ebextensions files.

The first, .ebextensions/01_ruby.config, looks like this:

packages:
  yum:
    git: []

container_commands:
  01_assets:
    command: RAILS_ENV=production bundle exec rake assets:precompile
    leader_only: true

The second, .ebextensions/02_yarn.config, looks like this:

commands:

  01_node_get:
    cwd: /tmp
    command: 'sudo curl --silent --location https://rpm.nodesource.com/setup_13.x | sudo bash -'

  02_node_install:
    cwd: /tmp
    command: 'sudo yum -y install nodejs'

  03_yarn_get:
    cwd: /tmp
    # don't run the command if yarn is already installed (file /usr/bin/yarn exists)
    test: '[ ! -f /usr/bin/yarn ] && echo "yarn not installed"'
    command: 'sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo'

  04_yarn_install:
    cwd: /tmp
    test: '[ ! -f /usr/bin/yarn ] && echo "yarn not installed"'
    command: 'sudo yum -y install yarn'

The last, .ebextensions/gem_install_bundler.config, looks like this:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/09_gem_install_bundler.sh" :
    mode: "000775"
    owner: root
    group: users
    content: |
      #!/usr/bin/env bash

      EB_APP_STAGING_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_staging_dir)
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      # Source the application's Ruby
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      cd $EB_APP_STAGING_DIR
      echo "Installing compatible bundler"
      gem install bundler -v 2.0.2

Deploy the application

Now we can make our first deployment attempt.

$ eb deploy

Unfortunately, it doesn’t work. We get an error that says: /opt/elasticbeanstalk/hooks/appdeploy/pre/12_db_migration.sh failed.

Why does this happen? Because he haven’t set up a database yet. Let’s do that now.

Create the RDS instance

In the AWS console, go to RDS, go to Databases, and click Create database.

Choose Postgresql and Free tier.

Choose whatever name you like for your database. I’m calling mine discuss-with-production

For size, choose t2.micro.

Make sure to set public accessibility to Yes so you can remotely connect to your database from your development machine.

Click Create database.

On the next screen, click View credential details.

Copy what’s there to a separate place for later use. You’ll also need the RDS instance’s endpoint URL which is found in a different place, under Connectivity & security and then Endpoint & port.

Make sure your database’s security group has port 5432 open.

Set the database credentials and create the database instance

Our production server will need to know the database endpoint URL and database credentials. Run the eb setenv command, with your own values of course replaced for mine, to set these values.

$ eb setenv RDS_DATABASE=discuss-with-production RDS_USERNAME=postgres RDS_PASSWORD=your-password RDS_HOST=your-endpoint-url

Even though the RDS instance exists, our actual PostgreSQL instance doesn’t exist yet. The RDS instance itself is more like just a container. We can run the rails db:create command remotely on the RDS instance by supplying the RDS endpoint URL when we run rails db:create.

Before running this command, make sure the production section of config/database.yml matches up with these environment variable names as follows:

production:
  <<: *default
  database: <%= ENV['RDS_DATABASE'] %>
  username: <%= ENV['RDS_USERNAME'] %>
  password: <%= ENV['RDS_PASSWORD'] %>
  host: <%= ENV['RDS_HOST'] %>
  port: 5432

Now create the database.

$ RDS_DATABASE=discuss-with-production RDS_USERNAME=postgres RDS_PASSWORD=your-password RDS_HOST=your-endpoint-url rails db:create

Deploy the application

Now the application can finally be deployed for real using eb deploy.

$ eb deploy

Once this finishes you can use the eb open command to visit your environment’s URL in the browser.

$ eb open

Where to start with introducing TDD to a new Rails app

A Code With Jason reader recently wrote me with the following question:

How do I introduce TDD into a new Rails app? Where do I start? I am deeply knowledgable on RSpec and use it a lot. I am not sure how to get started with testing in Rails. When should testing be introduced? (hopefully as soon as possible) What should be testing vs. what should I assume works because it was tested in some other way? For example, why test generated code?

There are a lot of good questions here. I’ll pick a couple and address them individually.

How do I introduce TDD into a new Rails app?

This depends on your experience level with Rails and with testing.

I personally happen to be very experienced with both Rails and testing. When I build a new Rails application, I always start writing tests from the very beginning of the project.

If you’re new to Rails AND new to testing, I wouldn’t recommend trying to learn both at the same time. That’s too much to learn at once. Instead, get comfortable with Rails first and then start learning testing.

What if you’re comfortable with Rails but not testing yet? How do you introduce testing to a new Rails app then?

I would suggest starting with testing from the very beginning. Adding tests to your application is never going to be easier than it will be at the very beginning. In fact, retroactively adding tests to an existing application is notoriously super hard.

The kinds of tests to start with

What kinds of tests should you add at the beginning? I would recommend starting with acceptance tests, also known to some as integration tests, end-to-end tests or, in RSpec terminology, feature specs. The reason I recommend starting with feature specs is that they’re conceptually easy to grasp. They’re a simulation of a human opening a browser and clicking on things and typing stuff.

You can start with a feature spec “hello world” where you just visit a static page and verify that it says “hello world”. Once you’re comfortable with that, you can add feature specs to all your CRUD operations. I have a step-by-step formula for writing feature specs that you can use too.

Once you’re somewhat comfortable with feature specs, you can move on to model specs.

Testing vs. TDD

I’ve been sloppy with my language so far. I’ve been talking about testing but the question was specifically about TDD. How do you introduce TDD into a new Rails app?

First let me say that I personally don’t do TDD 100% of the time. I maybe do it 60% of the time.

I mainly write two types of tests in Rails: feature specs and model specs. When I’m building a new feature, I often don’t know exactly what form the UI is going to take or what labels or IDs all the page elements are going to have. This makes it really hard for me to try to write a feature spec before I write any code. So, usually, I don’t bother.

Another reason I don’t always do TDD is that I often use scaffolds, and scaffolds and TDD are incompatible. With TDD you write the tests first and the code after. Since scaffolds give you all the code up front, it’s of course impossible to write the tests before the code because the code already exists.

So again, the only case when I do TDD is when I’m writing model code, and I would guess this is maybe 60% of the time.

How, then, should you start introducing TDD to a new Rails app? I personally start TDD’ing when I start writing my first model code. The model code is often pretty trivial for the first portion of a Rails app’s lifecycle, so it’s usually a while before I get into “serious TDD”.

What should I be testing vs. what should I assume works because it was tested in some other way? For example, why test generated code?

I’ll answer the second part of this question first.

Why test generated code?

I assume when we say “generated code” we’re mainly talking about scaffolding.

If you’re never ever going to modify the generated code there’s little value in writing tests for it. Scaffold-generated CRUD features pretty much always just work.

However, it’s virtually never the case that a scaffolded feature comes out exactly the way we want with no modification required. For anything but the most trivial models, a little tweaking of the scaffold-generated code is necessary in order to get what we need, and each tweak carries a risk of introducing a bug. So it’s not really generated code. Once a human starts messing with it, all bets are off.

That’s not even the main value of writing tests for generated code though. The main value of the tests covering generated code (which, again, is rarely 100% actual generated code) is that the tests help protect against regressions that may be introduced later. If you have tests, you can refactor freely and be reasonably confident that you’re not breaking anything.

Having said that, I don’t try to test all parts of the code generated by a scaffold. There are some things that are pointless to test.

What should I test and what should I assume works?

When I generate a scaffold, there are certain types of tests I put on the scaffold-generated code. I have a step-by-step process I use to write specific test scenarios.

I typically write three feature specs: a feature spec for creating a record using valid inputs, a feature spec for trying to create a record using invalid inputs, and a feature spec for updating a record. That’s enough to give me confidence that things aren’t horribly broken.

At the model level, I’ll typically add tests for validations (I am TDD’ing in this case) and then add the validations. Usually I only need a few presence validations and maybe a uniqueness validation. It’s not common that I’ll need anything fancier than that at that time.

There are certain other types of tests I’ve seen that I think are pointless. These include testing the presence of associations, testing that a model responds to certain methods, testing for the presence of callbacks, and testing for database columns and indexes. There’s no value in testing these things directly. These tests are tautological. What’s better is to write tests for the behaviors that these things (i.e. these associations, methods, etc.) enable.

Suggestions for further reading

Extracting a tidy PORO from a messy Active Record model

Skinny controller, fat model – the old advice

In 2006 Jamis Buck wrote a famous post called Skinny Controller, Fat Model. In it Jamis observed that Rails developers often put too much logic in controllers, making the code harder to understand and to test than it needs to be.

“Skinny controllers, fat models” became a piece of accepted wisdom in the Rails community. It was a step forward in the overall state of affairs, but a problem remained: moving code out of controllers and into models often resulted in fat models.

The new advice: skinny everything

Over time developers apparently came to a realization that while it’s better in certain ways to put bulky code in models than in controllers, the fatness, wherever you put it, is still kind of a problem.

Other ways of combating digital obesity were adopted or devised, including concerns, service objects, domain objects, Interactors, factories, something called Trailblazer, and probably others. These solutions vary in how good they are and where they’re appropriate to use.

Not all weight loss methods are equally good

When we talk about making e.g. a controller less fat, we of course aren’t talking about obliterating the bulk from existence but rather just moving it to a more appropriate place. The world is complex and we can’t make the complexity go away. The art is in distributing the complexity throughout the code such that a human can understand what the application does, in spite of the complexity.

For whatever reason, it seems lately that “service objects” (a term which means different things to different people, hence the quotes) and Interactors are widely esteemed as great ways to reorganize complexity in a Rails application. I personally find Interactors and service objects unappealing. In fact, I think they’re terrible. Avdi Grimm seems to have a similar opinion.

I think service objects and Interactors add unnecessary complexity to an application without adding any meaning to justify their added complexity. To take every action and make an object named after it—e.g. AuthenticateUser, PlaceOrder, SendThankYou—seems like a superfluous, tautological extra step. I think we can do better.

The case for POROs and domain objects

In Domain Driven Design, Eric Evans describes the concept of domain objects, which I take to mean: objects that represent concepts from the domain.

To me this makes a lot of sense. Most Rails applications of course already make use of this concept. If there’s something called an appointment in the domain, there will be an Appointment class in the application. If there’s something called a Patient in the domain, there will be a Patient class in the application.

It’s also possible to invent objects that don’t exist in the domain, that only exist to make the code easier to understand. It took me personally a long time to realize this. Examples of object-oriented programming often include things like creating a Dog and a Cat class, each of which inherit from Animal. These examples seem sensible and straightforward, but think of how many objects in Ruby and Rails have nothing to do with any corresponding real-world idea, e.g. Request, MimeNegotiation and SingularAssociation.

These types of objects take a little extra imagination to conceive of and name, but I think once a person opens up their mind to this way of coding, it gets easier and easier.

The form that I like to give these sorts of objects is POROs (Plain Old Ruby Objects).

My example code: a messy Active Record class

For the remainder of this post I’m going to take a certain big, messy Active Record class and refactor it into some number of POROs.

The code below is taken from a real production application. It was written by a terrible programmer: me in 2012. The class represents a stylist at a hair salon.

I invite you to have a nice scroll through the code and give yourself a light familiarity with it.

class Stylist < ActiveRecord::Base
  include ActionView::Helpers::NumberHelper
  scope :active, -> { where(active: true) }
  scope :inactive, -> { where(active: false) }
  scope :with_active_salon, -> { joins(:salon).merge(Salon.active) }

  scope :non_receptionist, -> {
    joins('LEFT JOIN user_role ON user_role.user_id = stylist.user_id')
    .joins('LEFT JOIN role ON user_role.role_id = role.id')
    .where('role.code != ? OR role.code IS NULL', 'RECEPTIONIST')
  }

  belongs_to :salon
  belongs_to :user
  has_many :roles, through: :user
  belongs_to :stylist_employment_type
  has_many :stylist_services
  has_many :services, through: :stylist_services
  has_many :rent_payments
  has_many :appointments
  has_many :clients, through: :appointments
  attr_accessor :skip_saving_other_stylists

  accepts_nested_attributes_for :user, :allow_destroy => true, :reject_if => :all_blank

  after_initialize -> { self.skip_saving_other_stylists = false }
  before_save -> { make_sure_self_gets_correct_order_index }
  after_save -> { salon.reload.resave_stylists unless skip_saving_other_stylists }

  validates_presence_of :name
  validates_presence_of :salon

  validates_each :name do |model, attr, value|
    stylist = Stylist.find_by_name_and_salon_id(value, model.salon_id)
    if stylist != nil and stylist.id != model.id
      model.errors.add(attr, "A stylist with name \"#{value}\" already exists.")
    end
  end

  def service_length(service)
    ss = stylist_services.find_by_service_id(service.id)
    ss ? ss.length_in_minutes : ""
  end

  def service_price(service)
    ss = stylist_services.find_by_service_id(service.id)
    ss ? ss.price : ""
  end

  def unique_clients_ordered_by_name
    Client.select("client.*, TRIM(UPPER(client.name)) upper_name")
          .joins(:appointments)
          .where("appointment.stylist_id = ?", id)
          .uniq
          .order("upper_name")
  end

  def save_services(lengths, prices)
    stylist_services.destroy_all
    salon.services.active.each_with_index do |service, i|
      if lengths[i].to_i > 0 || prices[i].to_i > 0
        StylistService.create!(
          stylist_id: self.id,
          service_id: service.id,
          length_in_minutes: lengths[i],
          price: prices[i]
        )
      end
    end
  end

  def report(date)
    sql = "
      SELECT a.start_time,
             c.name client_name,
             ti.label item_label,
             CASE WHEN ti.is_taxed = true THEN ti.price * #{salon.tax_rate + 1}
                  ELSE ti.price
             END
        FROM appointment a
        JOIN transaction_item ti ON ti.appointment_id = a.id
        JOIN client c ON a.client_id = c.id
        JOIN stylist s ON a.stylist_id = s.id
       WHERE s.id = #{self.id}
         AND a.start_time >= '#{Date.parse(date).strftime}'
         AND a.start_time <= '#{(Date.parse(date) + 7).strftime}'
    ORDER BY a.start_time,
             c.name,
             ti.label
    "
    result = self.connection.select_all(sql)
    result
  end

  def save_services_for_demo
    [{:name => "Men's Haircut",   :price => 20, :length => 30},
     {:name => "Women's Haircut", :price => 20, :length => 40}].each do |service|
      s = Service.create(
        :name => service[:name],
        :price => service[:price],
        :salon_id => self.salon_id
      )
      StylistService.create(
        :service_id => s.id,
        :stylist_id => self.id,
        :length_in_minutes => service[:length]
      )
    end
  end

  def formatted_rent
    if self.rent > 0
      number_with_precision(self.rent, :precision => 2)
    else
      ""
    end
  end

  def formatted_service_commission_rate
    if self.service_commission_rate > 0
      number_with_precision(self.service_commission_rate, :precision => 2)
    else
      ""
    end
  end

  def formatted_retail_commission_rate
    if self.retail_commission_rate > 0
      number_with_precision(self.retail_commission_rate, :precision => 2)
    else
      ""
    end
  end

  def clean_values
    if self.rent == nil
      self.rent = 0
    end

    if self.service_commission_rate = nil
      self.service_commission_rate = 0
    end

    if self.retail_commission_rate = nil
      self.retail_commission_rate = 0
    end
  end

  def pay_rent(date = Time.zone.now)
    payment = RentPayment.new
    payment.date = date
    payment.stylist_id = self.id
    payment.amount = self.rent
    payment.save
  end

  def rent_payments
    RentPayment.find_all_by_stylist_id(self.id)
  end

  def new_rent_payment
    rp = RentPayment.new
    rp.date = Time.zone.today
    rp.stylist_id = self.id
    rp.amount = self.rent
    rp
  end

  def gross_service_sales(start_date, end_date)
    self.salon.earnings(start_date, end_date, self.id)['services'].to_f
  end

  def net_service_sales(start_date, end_date)
    self.gross_service_sales(start_date, end_date) * self.service_commission_rate
  end

  def gross_product_sales(start_date, end_date)
    self.salon.earnings(start_date, end_date, self.id)['products'].to_f
  end

  def net_product_sales(start_date, end_date)
    self.gross_product_sales(start_date, end_date) * self.retail_commission_rate
  end

  def self.everyone_stylist
    self.new(id: 0, name: 'All Stylists')
  end

  def make_sure_self_gets_correct_order_index
    self.cached_order_index = order_index
    self.manual_order_index = manual_position if salon.has_manual_order?
  end

  def manual_position
    new_record? ? salon.highest_manual_order_index + 1 : manual_order_index
  end

  def alphabetical_position
    stylist_names = salon.ordered_stylist_names
    stylist_names << name unless stylist_names.member? name
    stylist_names.map(&:downcase).sort!.index(name.downcase)
  end

  def order_index
    return -1 unless active
    salon.has_manual_order? ? manual_position : alphabetical_position
  end

  def has_appointment_at?(start_time)
    appointments.where("start_time = ? and is_cancelled = false", start_time).any?
  end

  def stylist_employment_type_code
    stylist_employment_type ? stylist_employment_type.code : ""
  end
end

The PORO to extract: calendar-related code

The class is so big and the code is so bad that it’s hard to decide where to start. I could spend quite a lot of time refactoring this class, but to make it all fit into a reasonable blog post, I’ll extract just one POROs out of this class.

There are a few methods that I happen to know are related to displaying the stylist’s name on the calendar:

def manual_position
  new_record? ? salon.highest_manual_order_index + 1 : manual_order_index
end

def alphabetical_position
  stylist_names = salon.ordered_stylist_names
  stylist_names << name unless stylist_names.member? name
  stylist_names.map(&:downcase).sort!.index(name.downcase)
end

def order_index
  return -1 unless active
  salon.has_manual_order? ? manual_position : alphabetical_position
end

This is relatively easy low-hanging fruit. I can just conceive of a Calendar object and move these methods into it.

class Calendar
  def manual_position
    new_record? ? salon.highest_manual_order_index + 1 : manual_order_index
  end

  def alphabetical_position
    stylist_names = salon.ordered_stylist_names
    stylist_names << name unless stylist_names.member? name
    stylist_names.map(&:downcase).sort!.index(name.downcase)
  end

  def order_index
    return -1 unless active
    salon.has_manual_order? ? manual_position : alphabetical_position
  end
end

This won’t exactly work, though. Many of the values referred to inside this new class only exist in the context of a Stylist. We’ll need to pass in a Stylist instance.

class Calendar
  def initialize(stylist)
    @stylist = stylist
  end

  def manual_position
    @stylist.new_record? ? salon.highest_manual_order_index + 1 : @stylist.manual_order_index
  end

  def alphabetical_position
    stylist_names = salon.ordered_stylist_names
    stylist_names << @stylist.name unless stylist_names.member? @stylist.name
    stylist_names.map(&:downcase).sort!.index(@stylist.name.downcase)
  end

  def order_index
    return -1 unless @stylist.active
    salon.has_manual_order? ? @stylist.manual_position : @stylist.alphabetical_position
  end

  private

  def salon
    @stylist.salon
  end
end

Now that I see this much, I realize I’ve chosen an unfitting name for this class. All this code deals with figuring out where to show the current stylist in a list of all the stylists. There’s no way it makes sense to have a Calendar instance with just one single Stylist inside it. This class name needs to change.

Perhaps CalendarStylistListItem is a better name.

class CalendarStylistListItem
  def initialize(stylist)
    @stylist = stylist
  end

  def manual_position
    @stylist.new_record? ? salon.highest_manual_order_index + 1 : @stylist.manual_order_index
  end

  def alphabetical_position
    stylist_names = salon.ordered_stylist_names
    stylist_names << @stylist.name unless stylist_names.member? @stylist.name
    stylist_names.map(&:downcase).sort!.index(@stylist.name.downcase)
  end

  def order_index
    return -1 unless @stylist.active
    salon.has_manual_order? ? @stylist.manual_position : @stylist.alphabetical_position
  end

  private

  def salon
    @stylist.salon
  end
end

Now I can take a closer look at the contents of this class itself. I’m going to focus on this method first:

def manual_position
  @stylist.new_record? ? salon.highest_manual_order_index + 1 : @stylist.manual_order_index
end

Why is salon concerned with highest_manual_order_index? That seems like too fine-grained a detail for it to care about. That also seems like perhaps a responsibility of the CalendarStylistListItem class. Let’s bring it in.

class CalendarStylistListItem
  def initialize(stylist)
    @stylist = stylist
  end

  def manual_position
    @stylist.new_record? ? highest_manual_order_index + 1 : @stylist.manual_order_index
  end

  def alphabetical_position
    stylist_names = salon.ordered_stylist_names
    stylist_names << @stylist.name unless stylist_names.member? @stylist.name
    stylist_names.map(&:downcase).sort!.index(@stylist.name.downcase)
  end

  def order_index
    return -1 unless @stylist.active
    salon.has_manual_order? ? @stylist.manual_position : @stylist.alphabetical_position
  end

  private

  def highest_manual_order_index
    salon.stylists.map(&:manual_order_index).max
  end

  def salon
    @stylist.salon
  end
end

Now I’ll focus on the next method down, alphabetical_position.

def alphabetical_position
  stylist_names = salon.ordered_stylist_names
  stylist_names << @stylist.name unless stylist_names.member? @stylist.name
  stylist_names.map(&:downcase).sort!.index(@stylist.name.downcase)
end

Some of this code could certainly be refactored to be tidier, but I don’t see a lot with respect to object composition that needs to change. One thing that does stick out to me is salon.ordered_stylist_names. This seems like a concern of the CalendarStylistListItem, not of the salon. Let’s bring this method in.

class CalendarStylistListItem
  def initialize(stylist)
    @stylist = stylist
  end

  def manual_position
    @stylist.new_record? ? highest_manual_order_index + 1 : @stylist.manual_order_index
  end

  def alphabetical_position
    stylist_names = ordered_stylist_names
    stylist_names << @stylist.name unless stylist_names.member? @stylist.name
    stylist_names.map(&:downcase).sort!.index(@stylist.name.downcase)
  end

  def order_index
    return -1 unless @stylist.active
    salon.has_manual_order? ? @stylist.manual_position : @stylist.alphabetical_position
  end

  private

  def highest_manual_order_index
    salon.stylists.map(&:manual_order_index).max
  end

  def ordered_stylist_names
    salon.stylists.active.non_receptionist
      .order(salon.attribute_by_which_to_order_stylists)
      .map(&:name)
  end

  def attribute_by_which_to_order_stylists
    salon.has_manual_order? ? "stylist.manual_order_index" : "LOWER(stylist.name)"
  end

  def salon
    @stylist.salon
  end
end

In the process of doing this I see one more method that’s currently in Salon that would be more appropriate in CalendarStylistListItem: the has_manual_order? method. Let’s bring that in and rename it to salon_has_manual_order?.

class CalendarStylistListItem
  def initialize(stylist)
    @stylist = stylist
  end

  def manual_position
    @stylist.new_record? ? highest_manual_order_index + 1 : @stylist.manual_order_index
  end

  def alphabetical_position
    stylist_names = ordered_stylist_names
    stylist_names << @stylist.name unless stylist_names.member? @stylist.name
    stylist_names.map(&:downcase).sort!.index(@stylist.name.downcase)
  end

  def order_index
    return -1 unless @stylist.active
    salon_has_manual_order? ? @stylist.manual_position : @stylist.alphabetical_position
  end

  private

  def highest_manual_order_index
    salon.stylists.map(&:manual_order_index).max
  end

  def ordered_stylist_names
    salon.stylists.active.non_receptionist
      .order(salon.attribute_by_which_to_order_stylists)
      .map(&:name)
  end

  def attribute_by_which_to_order_stylists
    salon_has_manual_order? ? "stylist.manual_order_index" : "LOWER(stylist.name)"
  end

  def salon_has_manual_order?
    salon.stylists.sum(:manual_order_index) > 0
  end

  def salon
    @stylist.salon
  end
end

Lastly, I’m pretty sure the only method needed from the outside is order_index. Let’s make manual_position and alphabetical_position private.

class CalendarStylistListItem
  def initialize(stylist)
    @stylist = stylist
  end

  def order_index
    return -1 unless @stylist.active
    salon_has_manual_order? ? @stylist.manual_position : @stylist.alphabetical_position
  end

  private

  def manual_position
    @stylist.new_record? ? highest_manual_order_index + 1 : @stylist.manual_order_index
  end

  def alphabetical_position
    stylist_names = ordered_stylist_names
    stylist_names << @stylist.name unless stylist_names.member? @stylist.name
    stylist_names.map(&:downcase).sort!.index(@stylist.name.downcase)
  end

  def highest_manual_order_index
    salon.stylists.map(&:manual_order_index).max
  end

  def ordered_stylist_names
    salon.stylists.active.non_receptionist
      .order(salon.attribute_by_which_to_order_stylists)
      .map(&:name)
  end

  def attribute_by_which_to_order_stylists
    salon_has_manual_order? ? "stylist.manual_order_index" : "LOWER(stylist.name)"
  end

  def salon_has_manual_order?
    salon.stylists.sum(:manual_order_index) > 0
  end

  def salon
    @stylist.salon
  end
end

Making use of the new CalendarStylistListItem class

This tidy new PORO is of course only valuable if we can actually use it. How do we do so?

All the logic inside of CalendarStylistListItem seems to exist for the purpose of figuring out the order_index. Externally, it seems that order_index is the only method on the class that gets called.

And it looks like the one place that order_index gets called is in Stylist#make_sure_self_gets_correct_order_index. Currently, that method looks like this:

class Stylist
  def make_sure_self_gets_correct_order_index
    self.cached_order_index = order_index
    self.manual_order_index = manual_position if salon.has_manual_order?
  end
end

On the line self.cached_order_index = order_index, it’s of course the Stylist class’s own implementation of order_index that’s getting called. It was of course our whole aim in this refactoring exercise to be able to remove methods like order_index from the Stylist class (because, again, the stylist’s order index on the calendar is a detail that’s peripheral to the essence of the Stylist class). So, let’s stop calling Stylist#order_index and start calling CalendarStylistListItem#order_index.

This can be achieved in the following way:

class Stylist
  def make_sure_self_gets_correct_order_index
    self.cached_order_index = CalendarStylistListItem.new(self).order_index
    self.manual_order_index = manual_position if salon.has_manual_order?
  end
end

With that change, the tie is severed, and we can now safely delete the order_index from the Stylist class as well as the following methods that now live inside CalendarStylistListItem:

  • Stylist#manual_position
  • Stylist#alphabetical_position
  • Salon#highest_manual_order_index
  • Salon#has_manual_order?
  • Salon#attribute_by_which_to_order_stylists
  • Salon#ordered_stylist_names

That’s a big improvement! Previously, we had a lot of logic scattered across two classes, adding distractions and possible confusions to both classes. Now all that logic is kept in one single cohesive class.

Conclusion

The process of creating my PORO, CalendarStylistListItem, didn’t require importing any libraries or learning any radically new techniques. All I had to do was identify a “missing abstraction” in my Stylist class and then conceive of a new object to bundle it up in.

The overall impact to the Stylist class was not all that profound, but cleaning up a big messy class is of course going to take quite a lot of work. I feel like this was a meaningful step in a positive direction.

Next time you encounter a bloated Active Record model, I hope that instead of reaching for a service object or Interfactor, you might consider refactoring to a PORO.

My top five Rails performance tips for performance noobs

Performance is a topic that’s widely discussed and also, sadly, widely misunderstood.

I’m going to share five performance tips that have worked well for me in my 9+ years of using Rails and even before that, since much of what’s here is technology-agnostic.

I want to add a caveat: I’ve never worked for a big B2C startup that has needed to scale to millions of users. I’ve worked mostly applications with modest user bases and modest performance requirements. Luckily, I imagine my career experiences are probably pretty similar to that of most Rails developers, so it’s highly likely that what has worked for me performance-wise will also worked for you.

If you already know the basics of performance optimization and want to hear from someone who does Rails performance work all day every day, I recommend checking out the work of my friend Nate Berkopec, who I know as “The Rails Performance Guy”. (I’ve also had Nate on my podcast.)

Now onto my tips.

1. Don’t optimize prematurely

Before I came to Rails I worked with PHP. At one of my PHP jobs, I worked at a place where our official policy was to use single-quoted strings instead of double-quoted strings. The reason for this policy was that, supposedly, single-quoted strings were parsed ever so slightly faster than double-quoted strings.

In other words, someone in leadership took great care to implement and enforce a policy that was probably responsible for like 0.0001% of the total performance picture for any one of our apps. Penny wise, pound foolish.

Don’t do this. When you’re first building an application, the right approach to performance 99% of the time is to not think about it at all. Unlike bugs, which are best never deployed to production, performance problems are usually best dealt with after they’re discovered in production, after they prove themselves to be legitimate problems.

2. Profile and benchmark before optimizing

When I talk about performance profiling, I mean taking stock of performance across an application and judging where optimization effort is best spent. Some judgment and common sense come into the picture here. For example, if two pages are equally slow, and one of them is visited 100 times a day and the other one 1 time a month, obviously fixing the busier page will give a higher ROI.

In practice I rarely bother to do any actual profiling though. This is partly due to the nature of the work I personally have done historically. Most of my projects have been back-office internal-facing apps where I have a direct line of communication with the users or someone representative of the users. If there’s an important page that’s unacceptably slow, I’ll usually hear complaints about it or I’ll observe the sluggishness firsthand.

The next step I take before setting to work on optimization is to get a benchmark. This is just a fancy way of saying that I measure how fast the page currently loads. I of course can’t know how much of an improvement I made unless I know the difference between the “before” speed and the “after” speed.

The way I typically perform my benchmarking is to simply load a page in my browser while watching the output of the logs in a different window. At the bottom of the log it will say, e.g. “Completed 200 OK in 760ms”. I also use the sql_queries_count gem because sometimes it’s helpful to be aware of how many queries are being executed on the page.

3. The key to performance is usually not to add clever things but to remove stupid things

Performance optimization is often more akin to digging a ditch than to solving a Rubik’s cube.

Typically, the thing responsible for most of the performance cost of a page is waste. There’s a thing that could easily be done in an efficient way, but for whatever reason it was done in an inefficient way. So the task when optimizing a page is to find the waste and remove it. More on this in the next tip.

4. It’s almost always the database

The cause of most performance problems is waste. Most waste takes the form of making too many trips to the database.

This fact is so true and important that, in my 15+ years of programming experience, almost every performance optimization I’ve made has been some variation of “hit the database less”.

In Rails, the most common form of hitting the database too much is the famous “N+1 query problem”. Let’s say you have a page that shows a list of 10 things—let’s say clients—where each list item shows not only the client’s name but the client’s address. This means that a database query occurs for each one of those 10 clients. That’s the N in N+1. The +1 part of N+1 is the original query behind Client.all that grabs all the clients in the first place.

A common solution to the N+1 query problem, one I use a lot is eager loading. Instead of doing Client.limit(10), you can do Client.includes(:address).limit(10), which will grab all associated addresses ahead of time in just one query instead of in 10 separate queries. So instead of a total of 11 queries (1 for all clients, 1 for each of 10 addresses) you’ll have a total of just 2 queries (1 for all clients, 1 for all addresses).

Another way to make fewer trips to the database is to cache the results of expensive queries. I often use Rails’ low-level caching for this. However, this adds a certain amount of complexity that I’m not always eager to add. So I prefer to try to make fewer trips to the databases and/or optimize my queries themselves, resorting to caching only if I have to.

5. If it seems faster, it is faster

The only thing that ultimately matters is how fast users perceive the application to be. If a certain change makes an application feel faster, then for all intents and purposes, it is faster.

I once heard a story about a condo where the residents complained about the elevators being slow. Management’s reaction was not to immediately try to speed up the elevators somehow, which would presumably have been difficult and expensive. Instead, management installed mirrors on the walls next to the elevators. The complaints went away.

So, sometimes the right initial response to a performance problem is not necessarily to embark upon a big, expensive optimization project. Sometimes a change can be introduced that’s less invasive but still effective. For example, let’s say I have a calendar page with back/forward buttons to go from month to month, and let’s say each month takes 800ms to load. If I click the button and nothing at all happens for 800ms, it might feel a little slow. But if I change it so that immediately upon clicking the button I see an animated loading indicator, the page will feel faster. It sounds dumb but it’s absolutely true. Sometimes those little changes that improve the responsiveness of a page will increase its perceived performance, without needing to touch the actual performance at all.

6. Bonus tip: it’s almost never Ruby

Despite what you might read online about Ruby being “slow”, I’ve never ever encountered a performance problem where the culprit was the Ruby language itself. Whether an application is written in Ruby, Go, Python or whatever else, the performance bottleneck is almost always the database.

If all you know about performance is the five tips above, you’ll probably be able to tackle 80% of the performance challenges you encounter.

How to perform hotfixes without side effects or extra stress

A reader of mine recently wrote to me with a question which I’ll paraphrase.

The “tagalong” problem

Let’s say I have a production server. Let’s also say that in addition to that, I have a staging server where I push most changes before production so the changes can be reviewed. This works out fine most of the time.

But let’s say occasionally there’s a critical bug in production that needs to be fixed ASAP. I need to make my fix and push straight to production without the delay of going to staging first. Unfortunately when I do this I also push the last however many commits to production as well.

In other words, as a side effect of my bugfix, I end up pushing work to production that potentially wasn’t ready for production yet.

The fix

The solution is to always keep the master branch in a deployable state. I’ll mention a few tactics that can make this easier.

The first tactic is frequent deployments. I try to do all my work in the form of atomic commits which could each conceivably be deployed to production. And in fact, I tend to perform a deployment almost every time I commit to master or merge something into master.

The second tactic is feature branches. Feature branches help keep partway-done work out of master, helping ensure that a deployment of what’s on master never means deploying incomplete work. In my experience it’s good for a feature branch to have a lifespan of a few hours or a few days. Anything much longer than that gets dangerous because it leaves too much room for the feature branch to diverge from master.

But what if you need to work on a feature that spans, say, 4 weeks, and can’t be released until it’s fully complete? This is where my third tactic comes in, dark launching. Dark launching is where you separate the concept of deployment from the concept of release. Just because the code for a feature gets deployed to production doesn’t mean that that feature necessarily needs to be surfaced to users. The UI of that feature can be hidden using a feature flag, which can be as simple as something that says if false until you’re ready to unveil that feature and remove the code that hides it.

Recommended reading

The things I’ve shared above are basically DevOps principles. To learn more you can check out the books The DevOps Handbook and Continuous Delivery.

How to set up Rails secrets

This is part 4 of my series on how to deploy a Ruby on Rails application to AWS. If you found this page via search, I recommend starting from the beginning.

Overview of this step

We need to set up the Rails secrets security feature. It’s a relatively simple step although it does require us to jump through a few hoops.

1. Run rails credentials:edit

First we need to give permission to the current user, ubuntu, so we can make changes.

cd /var/www/hello_world
sudo chown -R ubuntu:ubuntu .

When we run the rails credentials:edit command, it will have us edit a credential file. We need to specify the editor that should be used for this action. In this case I’ll specify Vim.

export EDITOR=vim

Now we need to delete the existing config/credentials.yml.enc or else there will be a conflict.

rm config/credentials.yml.enc

With all these things out of the way, we can finally edit our credential file. No changes to the file are necessary. Just save and exit.

rails credentials:edit

Lastly, we need to give permissions back to the nginx user, www-data. Restart nginx afterward.

sudo chown -R www-data:www-data .
sudo service nginx restart

2. Verify success

Now, if you visit your EC2 instance’s URL in the browser, you should get this error:

The significant thing about this error is that it’s coming from Rails, not nginx. So we’ve made it all the way “to Rails”.

If you run tail -f log/production.log before refreshing the page, you should be able to see the exact error that’s occurring. It should be something like this:

This is telling us there’s no PostgreSQL server running, which is true. We can fix this problem in the next step: setting up our RDS database.

How to deploy a Ruby on Rails application to AWS

Overview

This tutorial will show you how to deploy a Rails application to AWS.

There are a number of ways this task could be tackled. It can be done manually or it can be done using an infrastructure-as-code approach, with a tool like Ansible.

This tutorial shows how to deploy Rails to AWS manually.

I wouldn’t recommend using a manual setup like this for a production Rails project, although I do recommend the experience of going through this manual process for the sake of learning what’s involved. It’s also find to start out hosting an application this way because it’s easy enough to migrate later to a more sophisticated hosting setup.

Before you dive in, be forewarned: it’s kind of a monster of a task. There are a large number of steps involved, many of them tricky and error-prone. Be prepared for the full process to involve hours or even days of potentially frustrating work.

Contents

The size of the setup process makes it impractical to put everything into one post, so each step is its own post.

  1. Launch EC2 instance
  2. Install nginx and Passenger
  3. Add the Rails application to the nginx server
  4. Set up secrets
  5. Create RDS database

Don’t be discouraged if not everything works on the first try. It most likely won’t. My advice if something goes wrong is to just blow everything away and start again from the beginning. I find that that approach is, paradoxically, often the fastest.

Good luck!

How I approach test coverage metrics

Different developers have different opinions about test coverage. Some engineering organizations not only measure test coverage but have rules around it. Other developers think test coverage is basically BS and don’t measure it at all.

I’m somewhere in between. I think test coverage is a useful metric but only in a very approximate and limited sort of way.

If I encounter two codebases, one with 10% coverage and another with 90% coverage, I can of course probably safely conclude that the latter codebase has a healthier test suite. But if there’s a difference of 90% and 100% I’m not convinced that that means much.

I personally measure test coverage on my projects, but I don’t try to optimize for it. Instead, I make testing a habit and let my habitual coding style be my guiding force instead of the test coverage metrics.

If you’re curious what type of test coverage my normal workflow naturally results in, I just checked the main project I’ve been working on for the last year or so and the coverage level is 96.62%, according to simplecov. I feel good about that number, although more important to me than the test coverage percentage is what it feels like to work with the codebase on a day-to-day basis. Are annoying regressions popping up all the time? Is new code hard to write tests for due to the surrounding code not having been written in an easily testable way? Then the codebase could probably benefit from more tests.