Poor man's PagerDuty

Amazon is a really cool place with very interesting challenges to tackle. The group I was in when I interned there was the Item Authority team. They are the ones who determine if two items being sold on the Amazon platform are actually the same thing. Anyway, Amazon engineers are all on pager rotation for fixing stuff when things go down. How frequently one needs to carry pager varies depending on the team size. For my startup AvidTap, I also needed something to monitor the health of our services and be alerted when things go bad. We used to have some issues with the VM on our host, but thankfully it hasn’t happened lately.

I’m aware of the start-up PagerDuty that probably makes this very easy, but throwing together something simple is also very easy with the technology now. Below I walk through how you can get your own poor man’s PagerDuty.

Accounts You Need

1. Sign up a free account with AppFog
2. Sign up for a trial account with Twilio
3. AWS account

Personally I was already using all of the above services for my personal projects, so putting this together was quick.

Grab The Source Code

My monitoring repo on github contains the code needed for this.

1. Start by git clone the repo: git clone https://github.com/dannysu/monitoring.git
2. cd monitoring
3. Grab the node.js modules needed: npm install
4. Modify the test function to access your services and check for status

Setting Things Up For AppFog

1. Install AppFog CLI
2. af login and provide your credentials
3. af push monitoring
4. Answer Y to “Would you like to deploy from the current directory? [Yn]:”
5. Answer n to “Detected a Node.js Application, is this correct? [Yn]:”
6. Type the number associated with Standalone. At this time it’s 6: “Select Application Type: 6”
7. Select node08 as runtime
8. Use node app.js as the Start Command
9. Choose wherever you want the monitoring service to be running from
10. Then just choose default options rest of the way

The app should now be deployed.

Filling in AWS Details

1. Login to AWS
2. Grab your security credential (Available by hovering over your name and click on it)
3. Login to AppFog and select the monitoring app
4. Click on the “Env Variables” tab on the side
5. Provide values for both AWSAccessKeyID and AWSSecretKey
6. Modify app.js to fill in the email addresses you want to send alerts to

Filling in Twilio Details

1. Login to Twilio
2. Your account SID and auth token are right at the top of the dashboard
3. Login to AppFog and select the monitoring app
4. Click on the “Env Variables” tab on the side
5. Provide values for both TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN
6. Modify app.js to fill in the phone numbers you want to send alerts to

You also need to have a TwiML file accessible somewhere. You can see Twilio documentation for more info on what can be in the TwiML file. I included an example in my code here. Just host it somewhere and modify the code to point the url variable to it.

Finally, after all these changes, run af update monitoring to upload changes to AppFog. Enjoy! Let’s hope you don’t ever get alerts from this. However, should your service go down, this will alert you via email/sms/phone.