Adam Gotterer

rake spec is slow

I spent some time debugging rake spec today to try and figure out why it seemed really slow compared to running other rake tasks (and its not because of my tests, those are actually fast). My observation was that the application took a long time to load. So I set out to do some debugging. Durring my benchmarking I realized that the application was getting loaded twice. The first time to load rake and the second time when an external call is made to ruby to run spec. The time through rake is twice as slow.

rake spec: ~27 seconds
ruby spec: ~12 seconds

I posted this as a bug on the rspec-rails github. For now the work around is to call rspec directly and skip rake.

/usr/local/rvm/rubies/ruby-1.9.3-p194/bin/ruby -S rspec spec/

Update: This is not an rspec issue. It’s a problem with rails and the way environments work. This problem also affects unit/test and minitest. The default rails environment is dev and since tests need to run in a test environment it has no choice but to run as a new process, thus loading it a second time. Thanks @dchelimsky.

Rails postgres db encoding issues

This error has given me quite the headache today and I could barely find any worthwhile information when searching around. Hopefully this post saves someone the same frustration.

PG::Error: ERROR:  encoding "UTF8" does not match locale "en_US"
DETAIL:  The chosen LC_CTYPE setting requires encoding "LATIN1".

I was running into this when running rake db:create or rake db:setup. I’m not positive, but I believe the issue is that the template and init settings I used to create postgres are different then the rails default. I started messing around with my database.yml file and came up with these additional options:

encoding: utf8
template: template0
collate: en_US.UTF8 
ctype: en_US.UTF8   

The rails create_database method took the encoding and template settings but was ignoring the lc_ctype and lc_collate setting which was needed to get past that error. After looking at the rails create_database code I realized that as of rails 3.2.8 there is no support for those options. Fortunately someone has added a pull request which adds support and will hopefully be part of the next rails release. As a temporary work-around I’ve updated the create_database method with the new code.

Twitter bootstrap: Lock the nav bar to top of screen when the page scrolls (CoffeeScript)

Apparently the Twitter bootstap navbar doesn’t natively support locking to the top of the screen when the page scrolls. A little bit of googling and I found a lot of solution. For some reason everything I found was dirty and long. Most of them didn’t work at all. Then I realized I was spending more time searching then it would take to just write it myself. This was the shortest solution I could come up with. It’s in CoffeeScript because thats what I’m learning this week :)

$(document).scroll ->
    nav = $('.navbar')
    return if nav.hasClass('navbar-fixed-top')

    if (nav.offset().top - nav.outerHeight() <= $(this).scrollTop())
        nav.addClass('navbar-fixed-top')
    else
        nav.removeClass('navbar-fixed-top')

Hacking the .SVN directory (Archive)

This comes from a blog post I wrote on 01/26/2009. I see people are still searching for it and landing at my old site. Unfortunately I can’t redirect the URL. But here’s the original content in all its glory. Kind of funny to look back at this almost 4 years later. Especially since I haven’t used svn in 3 :)

The other day on College Humor and Bustedtees we discovered a fairly serious security vulnerability. Fortunately because of the layout of our code nothing malicious could be exploited (more in another post). We thought our “push” script was skipping .svn folders, it turned out to not be operating correctly.

The hack is simple, documented and easily overlooked. Once the vulnerability was found, I did my best to exploit the shit out of it. I did so very successfully. I even tried it on some other popular websites and was able to access files I should have never been able to. In one instance I gained limited access to a sites admin. I emailed all of these sites to notify them of the security vulnerability. They were most gracious, once company even sent me a gift card!

The hack obviously starts in .svn directory, specifically at the entries file. You can access this file by browsing to:

somedomain.com/.svn/entires?

This document contains all of the files and folders svn manages in that directory. In some instances you can locate admin directories and the same thing applies…

somedomain.com/admin/.svn/entries?

So at this point all you have are a bunch of file names. Sometimes you can get some fun information and access to files that were meant to be hidden. Security by obscurity is not a solution, protect files you don’t want the public to access!

Now this is where things get interesting… Any file that has been checked in I can now execute. Either directly or through an svn folder that holds file revisions. Pick any file in the list and browse to:

somedomain.com/.svn/text-base/filename.php.svn-base

In this example the PHP file will be put through the PHP parser and executed. The results really depend on the layout of the code. Depending on the way the coder uses includes/requires decides on how much access and what kind of output you get. If a file is included using a relative path, the includes won’t be included since your working directory is the text-base dir. If they are using absolute paths, includes will continue through the execution. In one of the sites I poked around in, I found their admin wrapped through some kind of lite template/framework. I was able to bypass the system and go directly to the file without a using password. From there I had limited additional actions, but I still gained access to where I wasn’t welcome.

To do some additional testing I setup a test site to play with other file types. I found that files without a PHP extension, for example .inc files were NOT parsed and instead the contents were spit out to the page. In this test case the .inc file contained passwords and locations to databases. The possible additional damage I could cause from here is endless…

I’m not the first one to discover this hack, although a quick search only revealed obvious prevention methods. Protecting your site is really simple. Add this to your htacces file:

RewriteRule (\.svn)/(.*?) - [F,L]

Another option is blocking .svn folders through your web server config file for all sites.

Update
A number of people have mentioned a better prevention technique… They recommend doing an SVN export instead of a checkout or rsync. This was something I thought about after discovering the exploit. But I am by no means a system admin or the person who deals with that stuff at work. I’m glad these people were able to confirm that idea. Thanks!

Mountain Lion python command line issues

Looks like Mountain Lion wipes out your site-packages folder. You’ll need to sudo easy_install pip and then use pip to reinstall any packages that were wiped out.

How to be a successful tech recruiter: read the resume and proofread

In a given day I receive at least one recruitment email. Sometimes more, lots more. I’m generally very cordial to recruiters. I know they are just doing there job. Usually I’ll be pleasant and write back a short note telling them that I’m not considering new opportunities. I honestly believe that you never know when your paths might cross. Also this sometimes limits the number of follow up messages.

But every once in a while I’ll receive a message that does’t respect me or my time. If your job as a recruiter is to find talent and get them excited about jumping ship to a “better” opportunity, you can’t be sending out messages like this:

Adam, 

Your background looks great and I Have an Awesome Linux Infrastructure Engineer role Mid level 90-140k plus great bonus up to 60% wanted to see if you or any of your friends would be intrested, This is a Great laid back cutting edge environment. Hours 8:30-5:30pm. Great Benefits and perks.. Top Buy & Sell side Financial Services Firm on Park Ave, Your search is confidential. If interested or you know a friend you can send updated word resume and exact comp asap to: redacted@redacted.com Interviews are immediate, 

I have quite a number of problems with this message. Let’s review:

• “I Have an Awesome Linux Infrastructure Engineer role Mid level” - To start, besides a footnote or two mentioning Linux usage. I’ve never worked as a systems engineer. Second, i’d say my most recent experience has been at a fairly senior level.  Not entirely sure why you’d think I would want a mid level position after last being a manager? These things would be obvious if you read my resume. I’m just going to assume that a few keywords matched and he sent me this canned message. I guess spray-and-pray is a legit business model.
• I don’t claim to be an English major or have the greatest grammar and spelling. But my god the whole message has horrible grammar, spelling errors, weird capitalization and comma usage throughout. This is a horrible first impression. A horrible first impression for you personally, for the company you’re representing as well as the recruiting firm you work for. This is also a sign of what it would be like to have to interact with you.
• “Top Buy & Sell side Financial Services Firm” - Not a single thing on my resume says I would be a good fit at a Financial services firm. I’ve never worked in finance. Actually most of my resume is consumer web, media and commerce.
• “90-140k plus great bonus up to 60%” - Seems like a nice salary for mid level. But that range is all over the place. $140K seems pretty senior to me. But it’s nice that you show a range at all.
• Last but not least. And this is an offense for almost all recruiting attempts. There is no mention of the company name! No good software engineer (or systems engineer) is going to be so excited by a generic / poorly written / butchered description that they MUST email you back to find out more. No one cares that its a top mobile social geo enabled consumer facing app that just closed another round. We want to know what company it’s for. Plain and simple.

What’s the takeaway here? If you want to be a successful tech recruiter, start with reading the resume and proofreading your email.

Exposing Grape entities follow up

After spending some more time thinking about exposing Grape entities. I decided to write a mixin to make it easier. Below is the class and under that the usage. The mixing adds #expose_all and #hide methods. #expose_all can take an optional arguments option like #expose takes. #hide can be used to explicitly prevent an entity from being shown. Order matter when using these methods. The usage section should make it clear.

module EntityMixin
  def self.included(base)
    base.extend ClassMethods
  end

  module ClassMethods
    def expose_all?
      @expose_all ||= false
    end

    def expose_all(*args)
      @expose_all = true
      @expose_all_options = args.last.is_a?(Hash) ? args.pop : {}
    end

    def expose_all_options
      @expose_all_options ||= {}
    end

    def hide(*args)
      @hide ||= []
      args.each { |attr| @hide.push(attr.to_sym) }
      @hide
    end
  end

  def initialize(object, options={})
    # If expose_all? is set to true
    # run through all the objects attributes and
    # add send them to #expose. Options will be passed, blocks are not allowed.
    # If an attribute has already been exposed directly, expose_all
    # will ignore the attribute and not overwrite it.
    if self.class.expose_all?
      args = object.attributes.keys.map { |attr| attr.to_sym }
      args.each do |attr|
        if !self.class.exposures.include?(attr) 
          self.class.expose(*[attr].push(self.class.expose_all_options))
        end
      end
    end

    super(object, options)

    # Hide elements by removing them from the exposures list
    self.class.hide.each { |attr| self.class.exposures.delete(attr) }
  end
end

Usage:

class User < Grape::Entity
  include EntityMixin

  expose_all # Must go first, calls to expose overwrite default behavior
  expose :email, :if { :type => 'admin' }
  hide :password # hides must go last to remove exposures
end

Exposing entities in Grape by default

Entities in Grape are “exposed” instead of hidden by default (white labeling instead of black labeling). For the particular model I’m working with I has dynamic field names in a MongoDB that I don’t reliably know the names of. But I do know which fields I want to hide or have additional conditions for. My goal is to white label all fields by default and hide/condition the ones I don’t want exposed. Pretty much the opposite of how entities currently work. This is a working example:

module API
  module Entities
    class User < Grape::Entity
      expose :password, :if => {:type => :admin}

      def initialize(object, options = {})

        exposures = self.class.exposures
        object.attributes.keys.each do |key| 
          key = key.to_sym
          if !exposures.include?(key)
            self.class.expose(key) 
          end
        end

        super(object, options)
      end
    end
  end
end

I’ll probably write an entity extension to expose some additional functionality to make it easier to work with these types of scenarios. But in the mean time, this is what I cam up with.

A rake task to create Mongoid indexes

If you’re not using Mongoid within Rails you don’t get access to a rake task that you can run to create indexes. I whipped up this little rake task that automatically runs through a model directory and checks for Mongoid documents and then adds indexes.

# Based on: http://vikinghammer.com/2012/05/16/rake-task-mongoid-index/
# Usage: rake mongoid:create_indexes[ENV_TYPE_HERE]

require 'mongoid'
require 'logger'
logger = Logger.new($stdout)

ROOT = File.expand_path('.', File.dirname(__FILE__))

namespace :mongoid do
  task :create_indexes, :environment do |t, args|
    def determine_model(path)
      path =~ /(.*)\/(.*).rb/
      $2.camelize.constantize
    end

    unless args[:environment]
        logger.fatal "Mongoid: Must provide an environment"
        exit
    end

    yaml = YAML.load_file("config/mongoid.yml")
    env_info = yaml[args[:environment]]
    unless env_info
        logger.fatal "Mongoid: Unknown environment"
        exit
    end

    Mongoid.configure do |config|
        config.from_hash(env_info)
    end

    Dir.glob(File.join(ROOT, 'models', '*.rb')).each do |model|
      require model
      klass = determine_model(model)

      if klass.ancestors.include?(Mongoid::Document)
        klass.create_indexes
        logger.info "Mongoid: Indexes processed for \"#{klass.class.name}\""
      end
    end
  end
end 

Testing Grape without Rails

I was having some trouble getting rspec to work with Grape outside of Rails. While the solution was simple and was mainly about getting the requires correct, I wanted to share an example.

require 'rspec'
require 'rack/test'
require 'myapi'

describe MyAPI do
  include Rack::Test::Methods

  def app
    MyAPI
  end

  describe MyAPI do
    describe 'GET /somepath' do
      it 'returns a test string' do
        get '/test', {}, { 'SOME_HEADER' => 'VALUE' }
        last_response.status.should == 200
        JSON.parse(last_response.body).should == []
      end
    end
  end
end