Friday, 25 June 2010

Rails migrations for multi-column uniqueness checking in the database

I was adding some checks to my database in response to some app race conditions I discovered, and ran into some issues that were slightly puzzling.

The initial problem was 'How do I check multiple columns for uniqueness?'. That is, how do I ensure that for every row in my table, the n-tuple [col1, col2, ..., colN ] is unique (elipsis included to shorten the example), and have this enforced by the database? It turn out the answer is easy; just add an index like so (as part of a Rails DB migration), replacing the elipsis '...' as appropriate:

add_index :my_table, [ :col1, :col2, :col3, ..., :colN ], :unique => true

Easy! However, I found that my constraint didn't seem to be working:

describe MyTable do
  it "should police uniqueness in DB" do
    mt1 = nil
    lambda {
      mt1 = MyTable.create(:col1 => 1, :col2 => 2, ..., :colN => N)
      mt2 = MyTable.create(:col1 => 1, :col2 => 2, ..., :colN => N)
    }.should raise_error(ActiveRecord::StatementInvalid)

The Rspec case didn't pass; there was no exception raised, and the duplicate entry was being created. Looking in the logs, I found something suspicious:

Index name on table is too long; the limit is 64 characters. Skipping.

It turns out that the autogenerated DB index name (something like "by_#{columns.join('_and_')}") breached this 64-character limit, and was ignored, without the database migration failing!

Turn out the solution is to override the name explicitly to fit the 64-character limit:

add_index :my_table,
          [ :col1, :col2, :col3, ..., :colN ],
          :name => 'by_my_index_columns',
          :unique => true

With this the index is added properly, and the uniqueness of the column-tuples worked as intended.

As an aside, I also had some fun adding a uniqueness index for a text column; it wouldn't let me! I had to add a migration first to convert the column type from :text to ':string, :limit => 64':

class AddListLabelUniqueness < ActiveRecord::Migration
  def self.up
    remove_index :list_labels, :label
    change_column :list_labels, :label, :string, :limit => 64
    add_index :list_labels, :label, :unique => true
    add_index :list_label_instances,
              [ :user_id, :list_id, :list_label_id ],
              :name => 'by_list_label_indices',
              :unique => true

  def self.down
    remove_index :list_label_instances, [ :user_id, :list_id, :list_label_id ]
    remove_index :list_labels, :label
    change_column :list_labels, :label, :text
    #add_index :list_labels, :label

With that, I had the database ensuring that app races would not allow duplicate database rows to be created (with subsequent fun and bugs). Rails Model validations are good for a first line of defense, but if multiple requests are serviced in parallel, you need DB rules to protect data integrity.

Saturday, 19 June 2010

SEO-friendly URLs and RESTful routes in Rails

Following the advice of PluginSEO's Paint-by-Numbers Guides to SEO, I wanted the URLs for my pages to include the most important keywords. For recipes this would include the author and recipe title. What I wanted was something along the lines of this:

However, RESTful resources tend by default in Rails to have rather shorter names:

The above example is the List object (which happens in this instance to be a recipe) with database id 27. Not very descriptive!

I had a look at what obvious options I could see, and the :path_prefix option for resource mapping looked possible; a :path_prefix of 'recipes/:author/:title' would result in the following URLs:

There are a couple of things I didn't like about this. One is stylistic; I didn't like the extra 'lists' component. The other is more practical; I didn't like that the code would need to explicitly provide the author and title whenever building URLs. I want the path helpers to automatically build the URL given just the List object.

What I ended up doing was leaving the RESTful routes in place, but add separate routes for the 'pretty' URLs, and I monkey patched the path helpers using alias_method_chain to rewrite the RESTful URL as a pretty one. The code is as follows:

Starting with the routes, we map the RESTful routes normally, then provide 'pretty' routes at a lower priority. This ensures they are recognized, but the path helpers do not try to create them. This is imprtant because the default path helpers have no idea where to get the extra path components specified as :url_author and :url_title.

  # Normal RESTful routes
  map.resources :lists

  # Handle SEO-friendly URLs produced by ListsHelper list_url and list_path overrides.
  map.connect 'recipes/:url_author/:url_title/:id', :controller => :lists, :action => :show, :conditions => { :method => :get }
  map.connect 'recipes/:url_author/:url_title/:id.:format', :controller => :lists, :action => :show, :conditions => { :method => :get }
  map.connect 'recipes/:url_author/:url_title/:id/edit', :controller => :lists, :action => :edit, :conditions => { :method => :get }
  map.connect 'recipes/:url_author/:url_title/:id', :controller => :lists, :action => :update, :conditions => { :method => :put }
  map.connect 'recipes/:url_author/:url_title/:id.:format', :controller => :lists, :action => :update, :conditions => { :method => :put }
  map.connect 'recipes/:url_author/:url_title/:id', :controller => :lists, :action => :destroy, :conditions => { :method => :delete }
  map.connect 'recipes/:url_author/:url_title/:id.:format', :controller => :lists, :action => :destroy, :conditions => { :method => :delete }

Now in the module ListsHelper, we monkey patch the normal helpers, using alias method chain. We patch the versions generated by route_set.rb for both ActionView and ActionController. This new wrapper for the path helpers calls list.url_author and list.url_title to get the author and title in a form suitable for embedding in a URL.

require 'list'

module ListsHelper
  def self.list_helper_with_seo(mod, helper)
    mod.module_eval do
      helper_with = :"#{helper.to_s}_with_seo"
      helper_without = :"#{helper.to_s}_without_seo"
      unless method_defined?(helper_with)"Redefine #{helper} for #{}") if logger
        define_method(helper_with) do |*args|
"Create URL for #{args[1..-1].inspect}") if logger
          if List === args[0]
            list = args[0]
            base_url = self.send(helper_without, *args)
            md = base_url.match(/^(.*)\/lists\/(.*)/)
            if md
            self.send(helper_without, *args)
        alias_method_chain helper, :seo

  [ActionController::Base, ActionView::Base].each do |mod|
    [ :list_path,
    ].each do |helper|
      # Test so that specs without routing work.
      if mod.method_defined?(helper)
        self.list_helper_with_seo(mod, helper)

We now get URL in the form.

The ListsController can lookup the List from the id at the end, and SEO has its relevant path content.

Finally, we implement the List routines url_author and url_title:

class List < ActiveRecord::Base
  def url_author
    if author and (author.size != 0)
      url_author = author
    elsif user and ( != 0)
      url_author =
      url_author = 'anonymous'
    URI.escape(CGI.escape(url_author.gsub(/ /, '-').downcase),'.')

  def url_title
    url_title = 'default'
    if title and (title.size != 0)
      url_title = title
    URI.escape(CGI.escape(url_title.gsub(/ /, '-').downcase),'.')

Now we get pretty URLs from the normal helpers "list_path(list)", "edit_list_path(list)", etc.

I'm sure there is probably a better way to do this, but it worked for me...

Problems with class_inheritable_accessor and class reloading in Rails

I was having some very odd issues with WEBrick having odd ActiveRecord errors for the second and subsequent page views; the first page I loaded always worked, then other requests fail with errors like the following:

TypeError in ListsController#show

can't dup NilClass

RAILS_ROOT: /home/russ/git_repos/myrecipecart
Application Trace | Framework Trace | Full Trace

/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/base.rb:2220:in `dup'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/base.rb:2220:in `scoped_methods'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/base.rb:2177:in `with_scope'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/base.rb:2187:in `with_exclusive_scope'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:368:in `find_associated_records'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:255:in `preload_has_many_association'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:120:in `block in preload_one_association'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:114:in `each'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:114:in `preload_one_association'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:91:in `preload_associations'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:90:in `block in preload_associations'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:90:in `each'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/association_preload.rb:90:in `preload_associations'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/base.rb:1581:in `find_every'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/base.rb:1614:in `find_one'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/base.rb:1600:in `find_from_ids'
/home/russ/git_repos/myrecipecart/vendor/rails/activerecord/lib/active_record/base.rb:619:in `find'
/home/russ/git_repos/myrecipecart/app/models/list.rb:79:in `block in find_for_show'
/home/russ/git_repos/myrecipecart/app/models/list.rb:92:in `block in with_scope_for_show'
/home/russ/git_repos/myrecipecart/app/models/list.rb:91:in `with_scope_for_show'
/home/russ/git_repos/myrecipecart/app/models/list.rb:78:in `find_for_show'
/home/russ/git_repos/myrecipecart/app/controllers/lists_controller.rb:45:in `show'

After seeing a few different callstacks, it seemed to be that the problem was that class instance variables were suddenly becoming nil in some derived classes. For the callstack above, it was this one, from active_record/base.rb:

    # Stores the default scope for the class
    class_inheritable_accessor :default_scoping, :instance_writer => false
    self.default_scoping = []

I still haven't tracked down what is causing the problem, but I have found a workaround that helps me get on with my work; re-enable class caching in WEBrick development mode:

config.cache_classes = true

The result of this is that application classes are not reloaded with every request, and as a result I don't see the wierd crashes. It is annoying that I then have to restart WEBrick by hand whenever I need it to update the running code, but at least I can test behaviour that requires more than one HTTP request to complete!

If anyone can tell me why I'm seeing these problems with class_inheritable_accessor, I'd appreciate it! I have some supicions (my class definition for List includes a module that dynamically defines some other classes derived from ActiveRecord::Base), but nothing concrete just yet.

Tuesday, 15 June 2010

Ruby 1.9 dynamic creation of local variables

In order to DRY up my code, I wanted to refactor a couple of dozen lines of similar code initializing some local variables in a block iterating over an array of symbols to act on (details changed to get to the kernel of the issue).

lcl1 = options[:lcl1]
lcl2 = options[:lcl2]
lcl3 = options[:lcl3]

I initially tried something like

[ :lcl1, :lcl2, :lcl3 ].each do |sym|
  eval "#{sym.to_s} = options[:#{sym.to_s}]"
puts lcl1

This didn't seem to work (undefined local variable or method `lcl1'), so I thought maybe the variables were being created as local to the block (so out of scope by the time we reached the puts). I then tried something else, passing the bindings from the outer scope to the eval:

outer_binding = binding
[ :lcl1, :lcl2, :lcl3 ].each do |sym|
  eval "#{sym.to_s} = options[:#{sym.to_s}]", outer_binding
puts lcl1

This still gave pretty much the same error. I tried something else:

lcl1, lcl2, lcl3 = nil
[ :lcl1, :lcl2, :lcl3 ].each do |sym|
  eval "#{sym.to_s} = options[:#{sym.to_s}]"
puts lcl1

This worked pretty much as desired, but doesn't extend well to dynamically defined variables. I found another blog post describing very well exactly what I was seeing (problems creating local variables dynamically using eval), but it didn't present a solution.

I eventually found this forum post from Matz himself indicating that local variables that are defined in eval() cannot be accessed outside of eval.

It therefore seems what I want is impossible. It is possible to proceed with the ugly approach above of pre-declaring variables, but that only works if you know the variable names in advance.

One other thing that works is defining accessors dynamically for each dynamic 'variable', as follows:

eigenclass = class << self; self; end
[ :lcl1, :lcl2, :lcl3 ].each do |sym|
  eigenclass.class_eval { attr_accessor sym }
  eval "self.#{sym.to_s} = options[:#{sym.to_s}]"
puts lcl1

This is not creating local variables (it is a proliferation of class attributes), but that might be acceptable, depending on your requirements. Note here that the eval uses "self.sym =" rather than just "sym =", as it is now assigning to a class instance variable, not a local variable. The "puts lcl1" statement can still unambiguously refer to the attribute reader.

If anyone knows how to programatically create dynamic local variables in Ruby (presumably not using eval), I'd be very interested.

Saturday, 5 June 2010

Bookshelf of an Entrepreneur

So I've finally left my old job after 12+ years and am the CEO of my new company Limited (more on that in a later post). It's not yet got very far, but I thought it would be worth a quick look back at some of the resources that helped me get to this point.
  • The Beermat Entrepreneur - This was the first business book I read, shortly after ideas started sparking. It was an easy read, but got me thinking along the right lines at a very high level (people that would need to be involved, and how to test the business model for obvious flaws).
  • Agile Web Development with Rails (Pragmatic Programmers) - This (and the rest of the Pragmatic Programmers books) are the standard reference for the Ruby on Rails web development framework. I devoured about a dozen of the books in this series and used the skills to try mocking up some of my ideas with quick and dirty prototypes. This really helped to refine the ideas and see what was important to focus on first.
  • The Definitive Business Plan (Financial Times Series) - This one I read much later (over a year). I was still working for my old company, but the ideas had kept on ticking over, and I was thinking more seriously about whether I could make a business out of them. This book helped me do a much more critical analysis of the strengths, weaknesses and threats I would face, and how to actually write a plan for success.
  • The Non-Designer's Design Book - As mentioned in a previous post, this helped my wife and I do the initial design for the website. It covers basic principles to help you achieve a strong, effective look with good usability.
  • School for Startups - I attended several of these excellent events, and like the business books it helped me ask questions to establish the weaknesses and strengths of my business proposition.
  • Smarta's company registration tool helped me easily create and register my business.
There are a lot of other resources I've used to get to this point, but those ones above were the key ones for me.

Friday, 4 June 2010

AWS S3 and attachment_fu Cache-Control

For both performance and cost reasons, it is desirable to have sensible Cache-Control and Expires headers on all static objects served up from S3 in our web application.

To achieve this for new objects uploaded to S3, I followed the instructions in this excellent blog post.

However, that left me with a one-off problem of how to deal with the 6000+ objects I'd already uploaded. For this, I tried Bucket Explorer (with a free trial licence) and it worked a treat (using the Batch processing to add metadata for { key => "Cache-Control", value => "max-age=315360000" }, { key => "Expires", value => "315360000" }. I'd definitely recommend it! If I ever have an ongoing use case (rather than this one-off event), I'll definitely buy the full version rather than the time-limited trial version.

Thursday, 3 June 2010

Invalid User IQ

I was trying to set up "Amazon Associates" links for my last post, and had no end of trouble; it just didn't want to display. The Blogger editor interface didn't even show the "Amazon Associates" Panel properly; it just showed a stuck/frozen "Initializing" message that never got anywhere.

I eventually tried it in IE (I normally use Firefox, currently 3.6.3) and it all worked fine, and then it dawned on me -- My AdBlocker Pro Firefox plugin was preventing me from seeing my own Amazon associate links. D'oh!

A few mouse clicks later and I disabled the rules that were blocking the Amazon product link for the design book I was referencing, and all was fine and dandy! I also directly copied the link URL from the served ad and embedded it in the original post so that even people with the same AdBlock settings as I originally had would have a product link that worked as intended.

Details on ABP settings:
  • CTRL-SHIFT-V to open the blockable items pane
  • You'll see some items blocked by the Filters "^$third-party" and "*/widgets/q?$third-party". Click the icon at the far right to disable the rule.
  • Reload the page; the Amazon Associates panel/ads should work now.