Services, Scale, Backgrounding and WTF is going on here?!??!

@davetron5000

http://www.naildrivin5.com/blog

A story

about reasonable developers and reasonable decisions

creating a service-oriented architecture

Tech Lead, Payments

(i.e. the money)

Our story begins…

class PeopleController << ApplicationController
  def create
    @person = Person.create(params[:person])
    if @person.valid?
      UserMailer.send_welcome_email(person)
      redirect_to people_path
    else
      flash[:error] = 'Invalid Data'
      redirect_to new_person_path
    end
  end
end

Problem

Background Processing!

def create
  @person = Person.create(params[:person])
  if @person.valid?
Resque.enqueue(NewPersonEvent,@person.id)
redirect_to people_path else # ... end end
class NewPersonEvent
  def self.perform(id)
    person = Person.find(id)
    UserMailer.send_welcome_email(person)
  end
end

A few months later…

def create
  @person = Person.create(params[:person])
  if @person.valid?
Resque.enqueue(NewPersonEvent,@person.id) #! ETIMEDOUT
else # ... end end

Timeout to Redis

>  bundle exec rails console production
irb> has_a_sad

Solution

def create
Person.transaction do
@person = Person.create(params[:person]) if @person.valid? Resque.enqueue(NewPersonEvent,@person.id) redirect_to people_path else # ...
end
end end

A few weeks later…

class NewPersonEvent
  def self.perform(id)
person = Person.find(id) #! ActiveRecord::NotFoundError
UserMailer.send_welcome_email(person) end end

WTF?

class NewPersonEvent
  def self.perform(id)
    person = Person.find(id)
    UserMailer.send_welcome_email(person)
  end
end
class NewPersonEvent
include ResqueRetryOTron7000
def self.perform(id)
person = Person.find_by_id(id) retry_on(:times => 5, :when => lambda { person.nil? }) do
UserMailer.send_welcome_email(person)
end
end end

Better(ish)

Acquisition

Bulk Upload some Users

After we create a Person…

…run our business logic

class Person < ActiveRecord::Base
  # ...

after_create :send_new_person_event
# ... private def send_new_person_event
Resque.enqueue(NewPersonEvent,@person.id)
end end
def create
Person.transaction do
@person = Person.create(params[:person]) if @person.valid?
Resque.enqueue(NewPersonEvent,@person.id)
redirect_to people_path else # ... end
end
end
def create
  @person = Person.create(params[:person])
  if @person.valid?
    redirect_to people_path
  else
     # ...
  end
end

Reasonable

A few months later…

…we need to log stats

def send_new_person_event
  Resque.enqueue(NewPersonEvent,@person.id)
Stats.ping(:new_person)
end

Weeks go by…

…and we now have a cache to warm

def send_new_person_event
  Resque.enqueue(NewPersonEvent,@person.id)
  Stats.ping(:new_person)
PersonCache.put(:name,@person.id,@person.name)
end

Skunkworks project!

def send_new_person_event
  Resque.enqueue(NewPersonEvent,@person.id)
  Stats.ping(:new_person)
  PersonCache.put(:name,@person.id,@person.name)
PetProject.frobnosticate( HexDigest.md5_sha1_hash_rot13(@person.inspect))
end

Reasonable?

Mailers

class NewPersonEvent
  include ResqueRetryOTron

  def self.perform(id)
    person = Person.find_by_id(id)
    retry_on(:times => 5,
             :when  => lambda { person.nil? }) do
MailerService.mail(:send_welcome_email,person)
end end end

Meanwhile…

a Refactoring's afoot!

def send_new_person_event
  Resque.enqueue(NewPersonEvent,@person.id)
Stats.ping(:new_person) PersonCache.put(:name,@person.id,@person.name) PetProject.frobnosticate( HexDigest.md5_sha1_hash_rot13(@person.inspect))
end
def send_new_person_event
  Resque.enqueue(NewPersonEvent,@person.id)
Stats.ping(:new_person) PersonCache.put(:name,@person.id,@person.name) PetProject.frobnosticate( HexDigest.md5_sha1_hash_rot13(@person.inspect))
end
def send_new_person_event
  Resque.enqueue(NewPersonEvent,@person.id)
end
def self.perform(id)
  person = Person.find_by_id(id)
  retry_on(:times => 5,
           :when  => lambda { person.nil? }) do
    MailerService.mail(:send_welcome_email,person)
Stats.ping(:new_person) PersonCache.put(:name,@person.id,@person.name) PetProject.frobnosticate( HexDigest.md5_sha1_hash_rot13(@person.inspect))
end end

An event fails!

Just replay!

MailService's Design

is too "dumb"

What "smarts" would help?

Make Idempotent

Call mail a zillion times

Additional calls

What if…

No system is perfect

Where do we begin?

Historical Record

Prevent Bad Data

Fix Errors

Extract Services

In Review

Does this sound fun?

Thank You!
@davetron5000



Slides (bit.ly/services-wtf)
Make your own with Trickster (search github)