API Sessions With Redis in Rails

Updated: 10/20/2013 Updated with better working code samples

If you’re creating an API that you yourself are consuming through a pure front-end client (along the lines of how Twitter does it) you’ll probably end up needing to track user state – sessions, basically. RESTful APIs should not be keeping track of state but unfortunately this is often unavoidable. Here I’ll go over one way to manage user state for users using your front-end client and authenticating users accessing the API programmatically (using HTTP Basic, Digest, Token authentication or something similar). My experience with this is Rails-centric as I’m rebuilding my favorite project as a pure (mostly) RESTful back-end API with a front-end API consuming client. For this particular use case you probably want to avoid reading and writing from the database as much as possible. I’ve been wanting to try out Redis for a while now and this use case has given me a perfect opportunity for that. Here’s one way to manage the state and authentication of users in a Rails app using Redis instead of a traditional session or database solution.

A hypothetical situation

If you haven’t actually built any of this, just imagine you have for the purpose of this post…

So at this point you’ve got a RESTful API on your server. It’s a “pure” API in the sense that all it does is takes HTTP verbs along with some data as input and outputs data in a serialized format (JSON, XML, etc.). The API does not respond to requests with a view object ever.

At the same time you’ve also built an API consumer client. This can be a server-side application or a client-side app that uses a JavaScript framework or something. It doesn’t matter really. The point is that your API client is a totally separate application from the API itself even if they’re hosted on the same server. In addition to your own “official” API client you have opened up your API to third party developers to build and expand upon. Your own application consumes your API just like other clients would and you require authentication through HTTP token headers.

Your app is given permissions to access all user accounts and third parties can only access public data or their own account data.

So that’s the situation we’re working with here.

Defining the problem

Our problem now is knowing if a user is a) authorized to access certain controller actions in your API and b) keeping track of authorized users.

As it relates to what’s really “RESTful”, there aren’t many good solutions here. You also want a fast API so lots of reading and writing from disk are out of the question. The three possible solutions are to use sessions, the database, or keep everything in memory (that’s where Redis comes in). Sessions aren’t really RESTful and since you’re dealing with an API you’d be forced to send a cookie to the client which only works if the client is a browser. If you use any kind of sessions you have now just barred any non-browser client from consuming your API. You could store all active tokens in your database with an expiration time and a cron job to kill sessions after that time but keeping track of sessions like that is not RESTful and it puts a lot of strain on your database with tons of reads and writes that have to take place upon every request. My solution is Redis.

Solutions

So we’ve decided to use Redis now to handle active sessions. Before we go on let’s talk about some things to consider. Redis is an in-memory key-value store. It’s like a super fast NoSQL database except it has all sorts of cool data types and functions which are perfect for temporary and/or frequently changing data. A database is best for data that needs to be persisted for long periods. Redis is great for when the data can be destroyed with no consequences. Redis can only hold as much data as you have free memory – that’s the catch. But in this case you’re only storing short strings for active users for a limited period of time. If you’ve got a server with 1G of RAM then you should be able to handle at least a few thousands active users at the same time. In my case I have Redis backed up by my database so if the server or the app restarts and Redis loses all its data the database is used as a fallback. During normal operation, however, you’d never hit the database at all.

Our tokens in this application are temporary. Each user is given an access token by first submitting their API key. From there the app verifies the identity of the key and returns an access token along with an expiration time. When the user nears the expiration time they need to resubmit their key for a new token in order to continue making requests to the app. We don’t allow the expiration to be extended just in case an attacker gets a hold of a user’s access token. They’d need their API key in order to get a new valid session. We do have facilities for applications to get special “non-expiring” tokens but we’re not going into that here.

So, in summary, we’re using Redis to track active API users via their tokens because it’s fast and cheap.

Simple token authentication

My first stab at the problem brought me to a simple solution. Just store access tokens in a database table with a belongs_to relationship to the users table. The idea is that we’d use Rails’ authenticate_or_request_with_http_token to capture and validate the token before any request went through as well as grab any relevant user details to pass along to the next controller which would finish processing the request. To implement this our ApplicationController would look something like this:

def authenticate_with_token
  if authenticate_with_http_token { |token, options| Token.exists?(token) && @token = token }
    @current_user = Token.find_by token: @token
  else
    render json: "Invalid token", status: :unauthorized
  end
end

That would work but then on top of having to do some crazy regexing to get the actual token value from the header (the token variable outside of the authenticate_with_http_token is out of scope on the third line so it needs regexing which I haven’t included here) you’d also have to hit the database with at least once upon every single request to the server. This works for authenticating a valid token but you’re still hitting the database. If you’ve got to authenticate a token on each request you really don’t want to be hitting the database every time. I’ve got thousands of users to support. Thousands of users can bring millions of requests during high load periods and there’s no way I’m going to deal with so many database queries.

Redis to the rescue

So we don’t really need to change our Application Controller much to support Redis instead of the database. We just need to install Redis and create a Module to help us interact with it.

Installing and Configuring Redis

The first step to getting Redis running alongside your Rails app is to install it on your server and dev machines. That’s beyond the scope of this post but you can find great instructions on the Redis website. If you’re on a Mac and have Homebrew installed, just run brew install redis and you’re good to go. If you’re on a Linux distro, follow the official Redis website steps, don’t use your package manager as it has outdated versions of Redis. If you’re on Windows… well I wonder how you get any web development work done at all outside of using IIS and .NET.

Once Redis is installed on your machine you need a way for Rails to interact with it. Open up your Gemfile and add these lines to it:

gem 'redis'
gem 'redis-namespace'

These give you access to all the Redis commands you can find in the Redis docs and let you namespace different data you’d store in Redis.

Now let’s make sure Redis is initialized when Rails starts up. Create a file in your initializers config/initializers/redis.rb and put the following in it:

$redis = Redis::Namespace.new("your_app_name", :redis => Redis.new)

And now you can use $redis in your Rails controllers and models. But you’ll soon find that you need to access Redis from many different controllers. I first had a number of methods for interacting with Redis within my Token controller alone but then I found I needed those same methods elsewhere and didn’t want to be including random calls to Token.redis_method() from unrelated controllers so I created a new Redis module in ~~lib/redis.rb~~ lib/redis.rb and put the following code in it:

Module RedisStore # Calling it Redis will interfere with the Gem
  # Gets value of a single Redis hash key
  def hget(hash, field)
    redis.hget(hash, field)
  end

  # Get value of multiple keys in a Redis hash
  def hmget(hash, *fields)
    redis.hmget(hash, *fields)
  end

  # Get all fields and values from Redis at once by passing in the hash name
  def hgetall(hash)
    redis.hgetall(hash)
  end

  # Saves the current token as a hash with user info
  def hmset(hash, *field_value)
    redis.hmset(hash, *field_value)
  end

  # Expire an item
  def expire(hash, time)
    redis.expire(hash, time)
  end

  private
    # Create our own instance of the $redis global so as not to interfere with its use elsewhere
    def redis
      $redis
    end
end

Using this we can now just include RedisStore in any controller (Application Controller if you want it available everywhere) then in the controllers you need it you just do this:

RedisStore::hget(hash, field)

As a way to access data stored in your Redis datastore. So with this new module included, we can rewrite our authenticate_with_token method like this:

def authenticate_with_token
  if authenticate_with_http_token { |token, options| RedisStore::hexists(token, 'user') && @token = token }
    @current_user = RedisStore::hgetall(@token).with_indifferent_access
  else
    render json: "Invalid token", status: :unauthorized
  end
end

At this point you can now use ~~redis.hget(token, 'user_id')~~ RedisStore::hgetall(@token).with_indifferent_access in the authenticate_with_token method of ApplicationController instead of calling from the database. Using with_indifferent_access will allow you to get the token values from anywhere in your application using symbols. So rather than calling @current_user_var[:username] rather than @current_user_var['username'] – a small difference but I like it for readability. Using this refactored method you can now use authenticate_with_token as a before_action filter in front of every request and the result is just about instant because you get the result from memory, never touching the database. For tokens that do need to be stored and read from the database you could just add an elsif clause to the method which checks for the existence of the token in the database and then stores it in Redis temporarily anyway. The point is that even if the token doesn’t expire, you only need to keep it in memory for quick access for as long as the user will need an active session.

But how do you get the token into Redis to begin with?

In our case our users will exchange their API keys for a one-time-use access token which expires after a given amount of time. When the token expires the user must reauthenticate by exchanging their key for another token.

# **Corrected Code Sample (10/20/13)**
# controllers/tokens_controller.rb
skip_before_action :authenticate_with_token, only: [:exchange_key_for_token]

def exchange_key_for_token
  api_key = Key.find_by secret_key: params[:secret_key]

  if api_key
    @token = Token.new(user_id => api_key.user_id, api_key => api_key.secret_key ...) # Create a new Token object (Token is a model here)

    # Store the Token object's attributes (@token) in Redis THEN set an expiration on it
    RedisStore::hmset(@token.token, 'user_id', @token.user_id, ...)
    RedisStore::expire(@token.token, '300') # 5 minute expiration (but you can set this to whatever - shorter is more secure)

    render json: { token: @token.token, expiration: @token.expiration }, status: :ok

  else
    render json: { error: "Invalid key" }, status: unauthorized
  end
end

Now you can extend this idea by adding more authorization code and permissions as you like. But at this point you should have a way to store and retrieve data from Redis. You’ll have to define all of the methods you need in the Redis module first though. My example only gives a few of the hash and other common commands. If you name the methods in the Redis module the same as the commands Redis uses to work on data then all you need to do is learn the Redis commands and they translate directly to your Rails app. Want to use the Redis hmget command? Just call RedisStore::hmget(). Get it? No? Read it through again and check out the Redis docs. Use their interactive examples too, they’re super awesome for learning.

In the future you can start tracking failed login attempts, spam, and create leader boards and such. The possibilities are endless. Tracking API client sessions is just the start for me.