Scaling, Resque, and cross-site authentication www.heaps.co.nz 

Heaps www.heaps.co.nz • Personal Finance tool • Developed by Social Capital + Kiwibank • Public launch on March 4th 2010 • 14,500 Users • currently over $1.2b spending tracked 

Scaling • Importer - Validate permissions for every user in the system • Writing to large database tables • default_scope is usually a bad idea • Need to offload long running blocking operations to run in the background 

Scaling • Code smart, but don’t pre-optimise • New Relic RPM • Log number of calls to the db • Index database 

Resque • Used for running background tasks • redis-server daemon, queue for pending jobs • Resque worker(s) backed by redis client pull jobs off the queue and perform them 

Resque Installation Compile redis-server and redis-client from source install gems config.gem "redis", :lib=> "redis" config.gem "redis-namespace", :lib=>"redis/namespace" Start the server: redis-server Start worker: QUEUE=* rake environment resque:work 

Resque class VerifyViewPermissions @queue = :high def self.perform(view_id) v = AccountView.find(view_id) v.verify_view_permissions! if v end end Resque.enqueue(VerifyViewPermissions, view_id) 

Resque • Server (queue) can only hold string parameters for jobs, so pass id’s, arrays, hashes etc not models/ custom objects through for each job • Define class method perform • need to define which queue to place it on • Can have workers only responding to high priority queues to prevent getting slowed 

Resque • Run multiple workers to create high priority queues. • Can dynamically set queue (e.g. to run on a particular server) • Useful for fileupload processing where the file is only on one server 

Resque class UploadProcessor def self.enqueue(upload_id, job_queue) Resque::Job.create(job_queue, self, upload_id) end def self.perform(upload_id) @upload = Upload.find_by_id(upload_id) if @upload begin @upload.process! rescue => e @upload.register_error!(e) end end end end UploadProcesser.enqueue(@upload.id, `hostname`) 

Cross Site Authentication • Needed to pass through user identification (access number) • Link needs to time out • Need to encrypt parameters 

Cross Site Authentication • Encryption/Decryption at either end is done via shared secret/key • payload (users’s access number, time of link generation - both encrypted) • initialisation vector - random salt • checksum - signature of the first two 

Cross Site Authentication Site authentication coming from: 1. Take payload and initialisation vector, encrypt both using shared key. 2. Create checksum of both using shared secret. 3. Url encode all 3, and generate link. Site receiving: 1. un-encode parameters. 2. Checksum payload and initialisation vector with secret, compare to sent checksum. 3. Decrypt payload with shared key. 4. Profit. 

heaps! www.heaps.co.nz