Services, Scale, Backgrounding and WTF is going on here?!??!
@davetron5000
http://www.naildrivin5.com/blog
A story
about reasonable developers and reasonable decisions
creating a service-oriented architecture
Tech Lead, Payments
(i.e. the money)
class PeopleController << ApplicationController
def create
@person = Person.create(params[:person])
if @person.valid?
UserMailer.send_welcome_email(person)
redirect_to people_path
else
flash[:error] = 'Invalid Data'
redirect_to new_person_path
end
end
end
Problem
- Submit user info
- Save it to DB
- Wait for Mailer?
- Bad resource allocation
def create
@person = Person.create(params[:person])
if @person.valid?
Resque.enqueue(NewPersonEvent,@person.id)
redirect_to people_path
else
# ...
end
end
class NewPersonEvent
def self.perform(id)
person = Person.find(id)
UserMailer.send_welcome_email(person)
end
end
def create
@person = Person.create(params[:person])
if @person.valid?
Resque.enqueue(NewPersonEvent,@person.id) #! ETIMEDOUT
else
# ...
end
end
Timeout to Redis
- Person record created…
- …but email wasn't sent
- Person can't be validated
- Unique email constraints anyone?
> bundle exec rails console production
irb> has_a_sad
Solution
- Person created and email sent
- or
- No email sent and no person created
- Transactions
def create
Person.transaction do
@person = Person.create(params[:person])
if @person.valid?
Resque.enqueue(NewPersonEvent,@person.id)
redirect_to people_path
else
# ...
end
end
end
class NewPersonEvent
def self.perform(id)
person = Person.find(id) #! ActiveRecord::NotFoundError
UserMailer.send_welcome_email(person)
end
end
WTF?
- Got the id from the database
- Event created
- Event processed
- Transaction not yet committed
class NewPersonEvent
def self.perform(id)
person = Person.find(id)
UserMailer.send_welcome_email(person)
end
end
class NewPersonEvent
include ResqueRetryOTron7000
def self.perform(id)
person = Person.find_by_id(id)
retry_on(:times => 5,
:when => lambda { person.nil? }) do
UserMailer.send_welcome_email(person)
end
end
end
Better(ish)
- Likely to complete in 5 retries
- Can replay any that bubble up
- No grand re-architecture required
Bulk Upload some Users
- Send welcome emails
- Same business logic
After we create a Person…
…run our business logic
class Person < ActiveRecord::Base
# ...
after_create :send_new_person_event
# ...
private
def send_new_person_event
Resque.enqueue(NewPersonEvent,@person.id)
end
end
def create
Person.transaction do
@person = Person.create(params[:person])
if @person.valid?
Resque.enqueue(NewPersonEvent,@person.id)
redirect_to people_path
else
# ...
end
end
end
def create
@person = Person.create(params[:person])
if @person.valid?
redirect_to people_path
else
# ...
end
end
Reasonable
- Other approaches exist
- Some might be better
- Nothing inherently wrong here
A few months later…
…we need to log stats
def send_new_person_event
Resque.enqueue(NewPersonEvent,@person.id)
Stats.ping(:new_person)
end
Weeks go by…
…and we now have a cache to warm
def send_new_person_event
Resque.enqueue(NewPersonEvent,@person.id)
Stats.ping(:new_person)
PersonCache.put(:name,@person.id,@person.name)
end
def send_new_person_event
Resque.enqueue(NewPersonEvent,@person.id)
Stats.ping(:new_person)
PersonCache.put(:name,@person.id,@person.name)
PetProject.frobnosticate(
HexDigest.md5_sha1_hash_rot13(@person.inspect))
end
Reasonable?
- It looks a mess now
- But each line was reasonable
- Technical Debt
Mailers
- Deprecated
- New Awesome Mail Service
- Just a Simple REST Call
class NewPersonEvent
include ResqueRetryOTron
def self.perform(id)
person = Person.find_by_id(id)
retry_on(:times => 5,
:when => lambda { person.nil? }) do
MailerService.mail(:send_welcome_email,person)
end
end
end
Meanwhile…
a Refactoring's afoot!
def send_new_person_event
Resque.enqueue(NewPersonEvent,@person.id)
Stats.ping(:new_person)
PersonCache.put(:name,@person.id,@person.name)
PetProject.frobnosticate(
HexDigest.md5_sha1_hash_rot13(@person.inspect))
end
def send_new_person_event
Resque.enqueue(NewPersonEvent,@person.id)
Stats.ping(:new_person)
PersonCache.put(:name,@person.id,@person.name)
PetProject.frobnosticate(
HexDigest.md5_sha1_hash_rot13(@person.inspect))
end
def send_new_person_event
Resque.enqueue(NewPersonEvent,@person.id)
end
def self.perform(id)
person = Person.find_by_id(id)
retry_on(:times => 5,
:when => lambda { person.nil? }) do
MailerService.mail(:send_welcome_email,person)
Stats.ping(:new_person)
PersonCache.put(:name,@person.id,@person.name)
PetProject.frobnosticate(
HexDigest.md5_sha1_hash_rot13(@person.inspect))
end
end
An event fails!
MailerService.mail(:send_welcome_email,person)
Stats.ping(:new_person)
PersonCache.put(:name,@person.id,@person.name)
- BOOM!
MailService's Design
is too "dumb"
What "smarts" would help?
- Idempotent calls
- Additional calls to expose state
Make Idempotent
- Require a client-generated request id
- Perform operation once per request id
Call mail a zillion times
- Only one mail sent…ever
- Same result goes back
Additional calls
- What emails have been sent?
- Client checks first before sending
What if…
- …the mail fails at the gateway?
- …I forget to check first?
- …two requests come in at the same time?
- …my request ids clash?
Historical Record
- Log, log, log
- Audit Activity
Prevent Bad Data
- AR Validations cannot be trusted
- Database constraints
- Sanity Check the rest
- Fix bad data - don't code around it
Fix Errors
- Email
redalert@yourcompany.com
- Fix the problem or prevent the error
- Downgrade to warnings as needed
Extract Services
- Don't just "be dumb"
- Design for Idempotence
- Provide help for the client
In Review
- Record history
- Prevent bad data
- Fix errors
- Make Extracted Services "smart"
