1 Introduction
There comes a time in the development lifecycle of most web applications when a third-party integration becomes necessary. One of the simplest ways to do so is to expose a REST API for consumption. This article will walk you through a possible approach to designing and implementing a REST API in an intentionally simplistic task management web application, and will cover some best practices to ensure maintainability of the code.
2 Requirements, Assumptions
This article is going to assume some familiarity with the Rails framework, the Ruby programming language in general, and at least some familiarity with the Rails ecosystem by way of Devise, RSpec, and Capybara. When this does not significantly impact DRYness (DRY stands for “Don`t Repeat Yourself,” a mantra of software development), the more verbose syntax was intentionally chosen for readability.
2.1 The theoretical application
We will build out an API which corresponds to a task management application. It will contain a User model to represent users with access to the system, a Project model to represent projects, and a Todo model representing specific tasks to be done within a project. As such, a User will have many Projects, and a Project will have many Todos.
3 Theory. What makes an API RESTful?
-
Statelessness: Client state should not be stored on the server between requests. Said another way, each individual request should have no context of the requests that came before it.
-
Resource identification per request: Each request should uniquely identify a single resource. Said differently, each request that modifies the database should act on one and only one row of one and only one table.Requests that only fetch information should get zero or more rows from one table.
-
Representational state transfer: The resource endpoints should return representations of the resource as data, usually XML or JSON. The returned information should be sufficient for the client to uniquely identify and manipulate the database row(s) in question.
While there are other criteria which need to be fulfilled for a theoretically pure REST API, these suffice in practice.
4. A basic REST API in Rails
4.1 Routes
Rails provides a fantastic tool for defining endpoints in the form of routes:
rubyApiDemoApp::Application.routes.draw do scope `/api` do scope `/v1` do scope `/projects` do get `/` => `api_projects#index` post `/` => `api_projects#create` scope `/:name` do get `/` => `api_projects#show` put `/` => `api_projects#update` scope `/todos` do get `/` => `api_todos#index` post `/` => `api_todos#create` scope `/:todo_name` do get `/` => `api_todos#show` put `/` => `api_todos#update` end end end end end end end
4.1.1 Naming
While Rails will let you use PUT and POST more or less interchangeably, your API will be consumed by other developers. As such, it is best practice to go for the principle of least surprise – POST for create, PUT for update, PATCH for upsert (update and insert). Another reason to use the correct methods is that your application will be maintained by developers who are not you, possibly long after you are gone. Using methods which conform to standard practice (index, show, create, and update) ensures that this is a simpler task.
4.1.3 Versioning
Once your API is exposed, you need to assume that somebody is consuming it. Therefore, an existing API should never be modified, except for critical bugfixes. Rather than changing existing endpoints, expose a new version. One way to do this might be to create versioned controllers and routes (as above). With comprehensive test coverage, backwards compatibility can be ensured. A corollary of the above is that one should only expose an API that has gone through rigorous internal testing.
4.1.4 Route Parameters
While it is possible to identify routes based on the id column of the targeted resource, this does presume a higher degree of knowledge by the consuming application. A deliberate tradeoff — that needs to be made on a case by case basis — using unique database ids in the route chain allows users to access short routes, and simplifies resource lookup, while exposing internal database ids to the consumer and requiring the consumer to maintain a reference to ids on their end. Using public, but possibly not unique, identifiers like name reduces the amount of system internals that needs to be exposed, while allowing the client to easily lookup any data needed to make a request. The downfall is longer nested routes.
4.2 Controllers
4.2.1 BaseController and Authentication
A base API controller is useful to handle authentication and extract common API functionality. There are many possible schemes, but a common approach is to require reauthentication on a per-request level. This is probably the simplest way to ensure statelessness.
javascriptclass BaseApiController < ApplicationController before_filter :parse_request, :authenticate_user_from_token! private def authenticate_user_from_token! if !@json[`api_token`] render nothing: true, status: :unauthorized else @user = nil User.find_each do |u| if Devise.secure_compare(u.api_token, @json[`api_token`]) @user = u end end end end def parse_request @json = JSON.parse(request.body.read) end end
4.2.1.1 Security
Devise.secure_compare helps avoid timing attacks. While the comparison algorithm used by Devise is not strictly speaking constant time as it uses newly allocated memory and is capable of invoking the Garbage Collector as a result, it is much nearer constant time then custom comparison routines. Similarly, the Users loop does not break, thus preventing an attacker from establishing api token validity based on response time.
4.2.2 ProjectsController
javascriptclass ApiProjectsController < BaseApiController before_filter :find_project, only: [:show, :update] before_filter only: :create do unless @json.has_key?(`project`) && @json[`project`].responds_to?(:[]) && @json[`project`][`name`] render nothing: true, status: :bad_request end end before_filter only: :update do unless @json.has_key?(`project`) render nothing: true, status: :bad_request end end before_filter only: :create do @project = Project.find_by_name(@json[`project`][`name`]) end def index render json: Project.where(`owner_id = ?`, @user.id) end def show render json: @project end def create if @project.present? render nothing: true, status: :conflict else @project = Project.new @project.assign_attributes(@json[`project`] if @project.save render json: @project else render nothing: true, status: :bad_request end end end def update @project.assign_attributes(@json[`project`]) if @project.save render json: @project else render nothing: true, status: :bad_request end end private def find_project @project = Project.find_by_name(params[:name]) render nothing: true, status: :not_found unless @project.present? && @project.user == @user end end
The todos controller will look similar, even having to find a project in order to fulfill the RESTfulness requirement (one uniquely identifiable resource per endpoint).
4.2.2.1 Defensive Programming
Defensive programming is a software design principle that dictates that a piece of software should be designed to continue functioning in unforeseen circumstances. Because your API will be exposed to third-party developers, allowing them to submit arbitrary inputs, it is important to apply this practice in API design.
This is why has_key? is used above rather than simple existence checking. If @json[`project`] came in as false, or null, a simple existence check would return false, even though we can not say with any degree of accuracy that no JSON request body will ever have a false node at a depth of one.
The more significant defensive choice was the inclusion of responds_to?(:[]). This is due to the fact that, if the request body were say {“project”: “noteIAmNotAnObject”}, @json[`project`][`name`] would result in a server error. In the spirit of defensive programming, when building APIs, ask yourself, “what is the least expected, most random, or most malicious input a user can submit?” Then write code to handle it.
4.2.2.2 HTTP Status Codes
While it is tempting to simply return 200 (OK), 404 (Not Found), and 500 (Internal Server Error), HTTP status codes — very much like HTTP verbs — have well-defined meanings. For the benefit of other developers, use status codes that make sense. For example, if we have a uniqueness constraint on Project name, we should return a conflict status on creation attempt for name clashes to let the developers understand where they went wrong.
The exception to the rule is when an item exists but the API user does not have the right access privileges to act on it. While it might be tempting to return a 403 (Forbidden), this in and of itself provides an attacker with information about the existence of the item. Instead, return a 404 if the user cannot access the record, no matter what the reason.
4.2.2.3 Code DRY?
While the above does achieve the desired result, it is far from DRY. There are repeated JSON validations and a duplicate database read, once for read only on update, and once for update rejection on a non existing item for create. Additionally, any nested route, such as /api/v1/projects/:name/todos will require the same find methods so as to uniquely identify the correct resource.
5 A better way
First, we`ll extract functionality common to all API endpoints into the BaseApiController.
javascriptdef validate_json(condition) unless condition render nothing: true, status: :bad_request end end def update_values(ivar, attributes) instance_variable_get(ivar).assign_attributes(attributes) if instance_variable_get(ivar).save render nothing: true, status: :ok else render nothing: true, status: :bad_request end end def check_existence(ivar, object, finder) instance_variable_set(ivar, instance_eval(object+"."+finder)) end
This turns the create and update methods into calls to update_values, while the Project JSON validations call validate_json:
javascriptbefore_filter only: :create do |c| meth = c.method(:validate_json) meth.call (@json.has_key?(`project`) && @json[`project`].responds_to?(:[]) && @json[`project`][`name`]) end before_filter only: :update do |c| meth = c.method(:validate_json) meth.call (@json.has_key?(`project`)) end before_filter only: :create do |c| meth = c.method(:check_existence) meth.call(@project, "Project", "find_by_name(@json[`project`][`name`])" end def create if @project.present? render nothing: true, status: :conflict else @project = Project.new update_values :@project, @json[`project`] end end
The other, possibly more significant improvement is introducing a multi-tiered inheritance hierarchy into your controllers. A good way to break this out is based on route nesting. Todos are nested within Projects; therefore, any Todo will need to find a Project.
javascriptclass ApiProjectRouteController < BaseApiController private def find_project @project = Project.find_by_name(params[:name]) render nothing: true, status: :not_found unless @project.present? && @project.user == @user end def find_todo @todo = #... end #other finders also go here. end end
ApiProjectsController now needs to inherit from ApiProjectRouteController rather then BaseApiController.
javascriptclass ApiProjectsController < ApiProjectRouteController #... end class ApiTodosController < ApiProjectRouteController #... end
6 Custom behavior
Unfortunately, APIs do not exist in a vacuum. The system will have its own set of predefined behaviors, and on occasion, despite the best of intentions it will be necessary to expose behavior through the API that is not available otherwise, or may even clash with the application`s existing behavior. A specific, fairly common example would be the relaxing of validations.
For the sake of clarity, let us walk through an example. Let`s assume that our system`s Project model has a priority column that is required.
javascriptclass Project < ActiveRecord::Base has_many :todos belongs_to :user validates :priority, presence: :true validates :name, presence: :true end
Now let us assume that a major customer would like to integrate an in-house solution that allows Project Managers to bulk create Project entries without priority, as priorities have not yet been decided. However, the regular UI users should not gain these increased privileges. How could this be achieved? In short, we need the model to be aware of the request source, and apply one set of validations in case the request came from the API, and another otherwise.
The first thing to do, is to add this concept at the controller level. Adding a method to the BaseApiController:
javascriptbefore_filter :indicate_source def indicate_source @api = true end
The second thing to do is to allow the model to validate conditionally.
javascriptclass Project < ActiveRecord::Base after_initialize :set_ivars has_many :todos belongs_to :user validates :priority, presence: :true, if: lambda { |model| model.instance_variable_get(:@strict_priority_validation)} validates :name, presence: :true private def set_ivars @strict_priority_validation = true end end
Now every time a model is instantiated it will set its validations to be strict, and apply the priority validation if strictness is true.
The final piece of the puzzle is sharing state between the controller and the model to override model level instance variables. To do this, we can define a PORO (Plain Old Ruby Object):
javascriptclass ProjectValidationPicker def pick_a_validation(project, api=false) if api project.instance_variable_set(:@strict_priority_validation, false) else project.instance_variable_set(:@strict_priority_validation, true) end end end
And for the final piece of the puzzle, this object needs to be used on update/create. Handily, we already have a function on BaseApiController to do just that. Lets change this method to pull in a validation picker if it exists and to set the api appropriately.
javascript
def update_values(ivar, attributes) instance_variable_get(ivar).assign_attributes(attributes) validation_picker = check_validation_picker_existence(ivar) validation_picker.send(:new).pick_a_validation(instance_variable_get(ivar), @api) if validation_picker if instance_variable_get(ivar).save render nothing: true, status: :ok else render nothing: true, status: :bad_request end end def check_validation_picker_existence(ivar) Module.const_get (ivar.to_s[1..-1] + "_validation_picker").camelize rescue false end end
7 Debugging
Unfortunately, no code is ever perfect the first time it is written. While a detailed analysis of either of these tools is outside the scope of this article, it is worthy of note that both the Unix curl command, and the Google Apps Postman REST client allow crafting of custom requests to arbitrary endpoints.
8 Testing
With the code above, we have created a flexible, DRY, RESTful API. It allows us to have behavior that is unique to the API, while maintaining the same Database Models. But how can this be tested? Since the bulk of the lifting is done at the controller level, and since every request is a stateless transaction, request level specs are the most obvious fit. The simplest way to achieve this is controller tests, which do not differ significantly from the usual model for controller endpoint tests. Fundamentally, we craft JSON requests, and send them to the endpoints. A simple example might look something like this:
javascriptdescribe ApiProjectsController do before do @base_json = { api_token: @user.api_token } @project_json = {project: {priority: "4", name: "foo"}} @new_project_json = @base_json.merge(@project_json) end describe "actions" do describe "#create" do before do @request.env[`RAW_POST_DATA`] = @new_project_json.to_json lambda do post :create end.should change(Project, :count).by(1) end it "returns ok" do expect(response.status).to eq(200) expect(Project.all.last.priority).to eq(@new_project_json[:priority]) expect(Project.all.last.name).to eq(@new_project_json[:name]) end end end end
9 Other approaches and considerations
It is worth mentioning that the code outlined above represents only one possible approach to building a RESTful API in a Rails app. In fact it has made two, relatively significant tradeoffs.
The first, is that every request requires its own authentication. It is possible to avoid making this sacrifice using session cookies which Devise can also generate. I would argue that this complicates third party integration as it requires the developer to keep track of the session cookie, passing it in with every request instead of the api_token, as well as handling session expiration. In addition, it leaves the system exposed to attack, if the cookie origin is not scrutinized sufficiently, or even if the API consumer stands up to go to the restroom.
The second is that the in-house UI does not by default hit the same API endpoints. One solution would be to have Devise generate and expire a secondary, UI-only auth token, which does not trigger the API validation (maintains @api as false in BaseApiController), this may be stored in another column, or a different datastore entirely. Non-persistent in-memory datastores like Redis are particularly well suited to this application.
Finally it is worth noting that while this demo simply returns the entire resource, it is best practice to use a serializer to limit the subset of keys returned to what it is safe and/or necessary to expose.
10 Summary
We have covered how to build a REST api in an existing Rails application from the ground up, how to expose the endpoints, how to route to them, and how to allow custom behavior. We have touched on testing and debugging. While the example was fairly trivial, it is my hope that you can use it as a template for building scalable, reusable APIs.
11 Acknowledgements
A large thank you to my friend Joshua Yanovski who proofread this article and noticed the problem with Devise.secure_compare. He can be found at https://github.com/pythonesque