Question

I’m building a rack server app that has to filter domains that can access the data.
This is currently how i do that and it works fine:

   authorized_domains = /domain1.com|domain2.com|domain3.com/
   return [417, {"Content-Type" => "text/html"}, ["Expectation Failed"]] unless env['HTTP_REFERER'].match(authorized_domains)

The problem is that i would get probably half a million request a day
or even more, and on top of that i would have about 1000-3000 domains on the list.

Is this the most efficient filtering there is? doing a regex on the list of domains?

Answer 1

Store the domains as a Set in a class constant so that it doesn’t get instantiated on each request. Then get the domain.tld from the HTTP_REFERER and do a look-up in the set.

Something like

require 'rubygems'
require 'rack'
require 'set'
require 'URI'

class FontServer
  DOMAINS = %w[domain1.com domain2.com domain3.com].to_set
  def call(env)
    uri = URI.parse(env['HTTP_REFERER'])
    domain_tld = uri.host.split('.')[-2..-1].join('.')
    return [417, {"Content-Type" => "text/html"}, ["Expectation Failed"]] unless DOMAINS.include? domain
  end
end

@pguardiario argued that the above is slower than his regex solution. To benchmark is to know.

require 'rubygems'
require 'rack'
require 'set'
require 'URI'
require 'benchmark'

domains=[]
3000.times {|i| domains<<"domain#{i}.com"}
DOMAINS = domains.to_set
re = Regexp.new('^https?:\/\/(' + domains.join('|') + ')')
env = {'HTTP_REFERER' => 'http://domain4711.com/'}

Benchmark.bm do |benchmark|
puts "pguardiario regex"
benchmark.report do
    100000.times do
      env['HTTP_REFERER'] =~ re
    end
  end
  # Parsing the URL with the URI gem and getting domain.tld
  # About three times faster than the regex solution
  puts "jonelf original"
  benchmark.report do
    100000.times do
      uri = URI.parse(env['HTTP_REFERER'])
      domain_tld = uri.host.split('.')[-2..-1].join('.')
      DOMAINS.include? domain_tld
    end
  end
  # Set::include? is really fast. If we only got a
  # fast way to get the domain it would rock.
  puts "Only the set look-up"
  benchmark.report do
    domain_tld="domain4711.com"
    100000.times do
      DOMAINS.include? domain_tld
    end
  end
  # Gets the full host instead of just domain.tld
  # so that it does the same as pguardiario did
  # About 3.8 times faster than the regex on my machine.
  puts "A tweaked solution"
  benchmark.report do
    100000.times do
      DOMAINS.include? URI.parse(env['HTTP_REFERER']).host
    end
  end
end

      user     system      total        real
pguardiario regex
  8.437000   0.000000   8.437000 (  8.538086)
jonelf original
  2.719000   0.000000   2.719000 (  2.734375)
Only the set look-up
  0.047000   0.000000   0.047000 (  0.040039)
A tweaked solution
  2.203000   0.000000   2.203000 (  2.222656)