Easy Retries with RabbitMQ

Overview

Here’s the problem…you’re using RabbitMQ to solve some business problem. Basically, a message is put on an exchange, then routed to one or more queues and some other application picks the message off the queue and does something with it. But what happens if the application consuming the messages from the queue fails? Sometimes you’ll want to just move onto the next thing (maybe even log an error if you’re nice). Other times you’re going to want to retry. However, this is where the problem is…

Some background on Rabbit MQ.  Since version 2.7.0 rabbit mq tries to guarantee message ordering.  This means that RMQ does it’s best to maintain the order of messages in the queue.  This even applies to messages which are requeued after reading it for the first time.  This can lead to consumers bogged down or stuck on the failed message which can also lead to heavy load on the consumers processing the message or any of the downstream systems these consumers rely on.

First Approach

The first time I tried to do retries I did the following:

  1. Categorize the types of errors that were retryable and which were permanent
  2. Implement a way in my application to determine if an error I encountered is retryable or not.
  3. I implement a function which I can call when I encounter a retryable error event.  This is typically done by calling the reject function on a queue and passing it the the requeue flag set to true.  See https://www.rabbitmq.com/nack.html for more details.

This approach is flawed because of a couple reasons:

  1. When a message is requeued, the message is put on the head of the queue.  Which means it will be immediately picked up by a worker process.  This is undesirable because you could get stuck in a tight loop of retrying the same set of messages and you’ll never move on to newer messages which may be successful.
  2. If I mis-categorized a error as retryable when it was not, then I will retry indefinitely.  If this happens to enough message, then the messages stuck in the retry loop will block any new message from getting though.

A Better Way

I considered a few options when trying to find solutions to the previously mentioned flaws with my original approach for retries.  I looked into acking the message and republishing to the same queue/exchange.  I also added a retry counter to my message which I increment, giving me the ability to give up after n number of retries.  This is better, because I’m putting the message I’m retrying on the back of the queue and I now give up after a certain number of retries.  But there is still a possibility of retries happening in a tight loop, which is undesirable.

I want to be able to say “retry this message in 30 seconds”.  Or even have a expodential backoff so that the time between retries grows as I get closer to my max retry count.

One option for implementing this is in code.  But that will need to be done in each of my consumers and is prone to errors.  Rabbit MQ actually has a nice feature which can do this for you.  It’s called Dead Letter Exchanges (https://www.rabbitmq.com/dlx.html).  Here is an overview of how this new approach works:

  1. A message is published to an exchange is routed to a queue
  2. Consumer reads message off the queue
  3. Retryable error is caught
  4. Ack original message
  5. Update original message with a retry count of +1
  6. Publish message to Dead Letter Exchange (instead of your normal exchange) with the same routing key that was received originally.  This will put the message in a retry_message queue
  7. Message will be republished based on how you setup the Dead Letter Exchange

Setting Up the Dead Letter Exchange

So, here is what I do to get things working:

  1. Create the dead letter exchange, which is just a normal exchange with a special name
  2. Create a retry_message queue and have all messages published to the dead letter exchange route here
  3. When you setup the retry_message queue, be sure to default the following parameter values of the queue
    1. x-message-ttl: 30000 – This will set a ttl on any message published to the queue.  When the ttl expires, the message will be republished to the exchange specified in the `x-dead-letter-exchange` parameter.
    2. x-dead-letter-exchange: original_exchange_name – This is where the message will get republished to once the message ttl expires.  We normally want this be the name of the exchange where the message was originally published.

The result of the above configuration means that any message sent to the dead letter exchange will wait in the retry_message queue for 30 seconds before being put back on the tail of the original queue for reprocessing.

Using OAuth2 on Spray-can

Overview

Over the past few days I’ve been working on a application which will make use of my data from sites like Github, Twitter and LinkedIn. Each of these sites use OAuth2 for their authentication mechanism. When I worked on a similar project in Python this was easy, I simply used a library someone already created and all the hard work was done for me. However, I did not find that to be the case for Scala and Spray. Now, there are couple OAuth libraries out there for Scala and I could have also used one of the Java options. But I wanted to keep my application a pure Scala and Spray application…not bringing in extra dependencies these libraries would require. When completed and looking back at the implementation, this wasn’t all that difficult…the apis I’m interfacing with are RESTful after all. So the next few sections will walk you though how to incorporate apis like Github, Twitter and LinkedIn into your application.

Resources and Gathering Required Info

First, lets review important resources and discuss what you will need before you begin. As previously mentioned, we’ll focus on Github, Twitter and LinkedIn’s apis. Each implement OAuth 2 similarly, but there are some subtle differences between the three. Also, each provide there own documentation, which is a great resource if you get stuck.

After reviewing each of the links above, you will discover that you will need some basic credential information. To get this information follow the steps below for each:

Github

  1. Go to your Github profile settings via https://github.com/settings/profile
  2. Go to the `Applications` section
  3. From the `Developer applications` section select `Register new application`
  4. Once you fill out the details for your new application you will be provide the credentials you need to continue: Client ID, Client Secret

Twitter

  1. Go to your Twitter profile application settings page via https://dev.twitter.com/apps
  2. From this page select `Create a new application`
  3. Once you fill out the details for your new application you will be provided the credentials you need to continue: Consumer Key, Consumer Secret

LinkedIn

  1. Go to your LinkedIn profile application settings page via https://www.linkedin.com/secure/developer
  2. From this page select `Add New Application`
  3. Once you fill out the details for your new application you will be provided the credentials you need to continue: Api Key, Secret Key

Workflows

Now that we have our application registered with Twitter, LinkedIn and Github let’s review the general authentication workflow these sites have implemented:

Github

  1. Make a Temporary Redirect request to github’s authorize end point (https://github.com/login/oauth/authorize?client_id=$clientId&state=$apiAuthState&redirect_uri=$apiRedirectUri).
    Note: Replace $clientId with the Client ID github provided you during the registration process.
    Replace $apiAuthState with a unique string of our choice. This will be used later when validating the response.
    And lastly, replace $apiRedirectUrl with a url back to your site. This will be where Github will send the authentication token you will be using for those subsequent API calls.
  2. Github will direct you to a page which requests your approval for allowing your application access to information in your account.
  3. Upon accepting the approval in step 2, Github will send a redirect request to your redirect url (which you provided in step 1). This request will contain two important pieces of data as parameters, code and state. The code is the authentication token you will need for subsequent API requests, but first you want to make sure the value of the state parameter matches the value you provided in the $apiAuthState field during step 1.
  4. At this point you should have the token you can use to make subsequent API requests

Twitter

  1. Make a Post request to Twitter’s authorize endpont (https://api.twitter.com/oauth2/token). You will need to provide your client credentials in the request header.
  2. Upon successful validation of the request, Twitter will return a token you will use (also called a Bearer Token).
  3. At this point you should have the token you can use to make subsequent API requests

LinkedIn

LinkedIn’s workflow is very similar to Github’s. The only different is the redirect url you will be using. Here it is: https://www.linkedin.com/uas/oauth2/authorization?response_type=code&client_id=$apiKey&state=$apiAuthState&redirect_uri=$apiRedirectUri

Authentication Clients

Now, let’s review some of the code we can write to make the workflows described in the previous section a reality. This next section will cover creating a spray-can client for each of the three APIs we want to communicate with. As you may have noticed, Github and LinkedIn have very similar workflows, so we will discuss them at the same time. LinkedIn has a simpler workflow which we will discuss afterwards. So the end goal for each will be to have a method we can call which will return our authentication token that we will be able to use with subsequent calls to the various APIs.

Let’s first describe what our authentication clients will need to do. First, we need to be able to get an existing token, let’s call it getAccessToken. If we don’t have one yet we can simply return nothing which can indicate to the caller that we need to kick off the process to generate one. Which leads to the next call we need to support, generateAccessToken.

Github and LinkedIn

Let’s first describe what our authentication clients will need to do. First, we need to be able to get an existing token, let’s call it getAccessToken. If we don’t have one yet we can simply return nothing which can indicate to the caller that we need to kick off the process to generate one. Which leads to the next call we need to support, generateAccessToken. Here is what this may look like:

case class GithubAccessToken() {
  import ExecutionContext.Implicits.global
  implicit val system = ActorSystem()
  import system.dispatcher // execution context for futures

  import com.gagnechris.backend.BackendConfig.GithubConfig

  val clientId = GithubConfig.clientId
  val clientSecret = GithubConfig.clientSecret
  val apiRedirectUri = GithubConfig.apiRedirectUri
  var apiAccessToken: Option[GithubToken] = None
  val apiAuthState = UUID.randomUUID().toString()
  val authRequestUri = s"https://github.com/login/oauth/authorize?client_id=$clientId&state=$apiAuthState&redirect_uri=$apiRedirectUri"

  def generateAccessToken(authCode: String, state: String): Future[Option[GithubToken]] = {
    import com.gagnechris.backend.model.GithubJsonProtocol._
    import SprayJsonSupport._

    if (state == apiAuthState) {
      val pipeline: HttpRequest => Future[Option[GithubToken]] = (
        encode(Gzip)
        ~> addHeader(Accept(`application/json`))
        ~> sendReceive
        ~> decode(Deflate)
        ~> unmarshal[Option[GithubToken]]
      )
      pipeline {
        Post(s"https://github.com/login/oauth/access_token?code=$authCode&redirect_uri=$apiRedirectUri&client_id=$clientId&client_secret=$clientSecret")
      }
    }
    else Future.failed(new Exception("Auth States did not match.  Possibly due to CSRF."))
  }

  def getAccessToken: Future[Option[GithubToken]] = {
    apiAccessToken match {
      case Some(token) => Future.successful(apiAccessToken)
      case None        => Future.successful(None)
    }
  }
}

One thing you may see above is that we are unmarshalling our request to a Option[GithubToken] object, which basically means that we have a GithubToken case class defined somewhere in our application and we want to deserialize the response to an instance of it. I found that Spray-json is a very nice library for serializing and deserializing json, so we will use it.

import spray.json.{JsonFormat, DefaultJsonProtocol}

case class GithubToken(access_token: String, scope: String, token_type: String)

object GithubJsonProtocol extends DefaultJsonProtocol {
  implicit def GithubTokenFormat = jsonFormat3(GithubToken)
}

You may have also observed that we do not have any redirect calls yet. That is because the redirect should be handled by the calling application. Here is an example how how this may be used:

  def githubRoute: Route =
    getPath("GithubRepos") {
      import com.gagnechris.backend.model.GithubJsonProtocol._

      onSuccess(githubAccessToken.getAccessToken) { token =>
        token match {
          case Some(tokenValue) => {
            cache(cache30min) {
              complete(GithubRepos.getUserAndRepos(tokenValue.access_token))
            }
          }
          case None => redirect(githubAccessToken.authRequestUri, TemporaryRedirect)
        }
      }
    } ~
    getPath("GithubAuthCode") {
      import com.gagnechris.backend.model.GithubJsonProtocol._

      parameters('code, 'state) { (code, state) =>
        onComplete(githubAccessToken.generateAccessToken(code, state)) {
          case Success(token) => {
            githubAccessToken.apiAccessToken = token
            redirect("/api/GithubRepos", TemporaryRedirect)
          }
          case Failure(ex) => {
            val errorMsg = ex.getMessage
            complete(InternalServerError, s"githubRoute Error: $errorMsg")
          }
        }
      }
    }

Here, we provide a GithubRepos endpoint to our users. This will call the getAccessToken method. If we have one, we will continue and call another Github API using that token. Otherwise, we’ll issue the redirect. If the user accepts, a call is made to the GithubAuthCode endpoint where we extract the code and state parameters and pass them to the generateAccessToken method. If this is successful we can save the token for next time.

Twitter

As I indicated earlier, Twitter’s workflow is much simpler and therefore requires less explanation.

object TwitterBearerToken extends TwitterBearerToken {
  import com.gagnechris.backend.BackendConfig.TwitterConfig

  val twitterBaseUrl = TwitterConfig.url
  val consumerKey = TwitterConfig.consumerKey
  val consumerSecret = TwitterConfig.consumerSecret
  val credentials = Base64.encodeBase64String(s"$consumerKey:$consumerSecret".getBytes())

  def getBearerToken: String = {
    generateBearerToken(credentials, twitterBaseUrl)
  }
}

trait TwitterBearerToken {
  import ExecutionContext.Implicits.global
  implicit val system = ActorSystem()
  import system.dispatcher // execution context for futures

  def generateBearerToken(credentials: String, baseUrl: String): String = {
    import com.gagnechris.backend.model.TwitterJsonProtocol.TwitterTokenFormat
    import SprayJsonSupport._

    val pipeline: HttpRequest => Future[TwitterToken] = (
      addHeader("Authorization", s"Basic $credentials")
      ~> encode(Gzip)
      ~> sendReceive
      ~> decode(Deflate)
      ~> unmarshal[TwitterToken]
    )
    val response = pipeline {
      Post(s"$baseUrl/oauth2/token", FormData(Map("grant_type" -> "client_credentials")))
    }
    Await.result(response, 5 seconds).access_token
  }
}

Example of how this may be used:

  def twitterRoute: Route = getPath("TwitterTweets") {
    import com.gagnechris.backend.model.TwitterJsonProtocol.TweetFormat

    cache(cache30min) {
      onComplete(TwitterTweets.tweets(TwitterBearerToken.getBearerToken)) {
        case Success(tweets) => {
          respondWithStatus(OK) {
            complete(tweets.toJson.toString)
          }
        }
        case Failure(ex) => {
          val errorMsg = ex.getMessage
          complete(InternalServerError, s"twitterRoute Error: $errorMsg")
        }
      }
    }
  }

Conclusion

The above is a basic implementation for authenticating to Github, Twitter and LinkedIn using basic OAuth2 workflows. The complete project which I’m continuing to work on can be found here. One of the issues I’ve identified with the implementation above is how error conditions are handled. What I’d like to have is a way to deserialize the error response, so we can better handle the error. But for now we are simply failing the request during the deserialization process and returning the error to the user. There is an open spray-can issue that aims to resolve our deserialization issue here.

Learning Scala : A Reference Guide

Overview

During the past 8 months I have been learning Scala alongside the Dev team I lead. I have a Java background, but I also have some experience with C# and a couple other languages. Our Dev team here at Getty Images, with the backing of senior management, has decided to move away from Microsoft/.NET/C# technology stack and embrace open-source technologies. Our preference being Scala and the JVM for moving forward projects.

Over the past few months we’ve learned an incredible amount about Scala, Functional Programming and the various tools, frameworks and libraries that came along with this move. So, as we wrap up the first major phase of our first Scala project, I thought I’d share the various resource the team and myself have used to learn the language.

Scala References:

  • Scala for Java Devs – This is a quick, high level, overview of the language targeted towards Java developers. I’ve also found that it’s helpful to C# developers as well.
  • Scala Quick Guide – A good overview of things you must know about syntax and other fundamental aspects of the language.
  • Scala in Action – This is a great book, which goes over the things you must know to build a Scala application.
  • @typesafe – Follow typesafe on Twitter, they will keep you up to date on the latest webinars going on.
  • Coursera Course – Take the Functional Programming course offered by coursera. It’s free, teaches Functional Programming using Scala and taught by Martin Odersky…enough said

Scala Libraries/Frameworks You Must Learn

There are many great frameworks and libraries available for Scala, but here is a list of resources for the frameworks and libraries we used for our project.

  • Spray – Also know as Spray on Akka and soon to be known as Akka Http. If you are building a Web API, then this is the way to go.
  • Spray Learning – Take a look at my co-worker’s tutorials for Spray.
  • Akka – In my opinion, Akka is the only way to build concurrent / distributed applications with Scala. Also, if you’re using Spray, then you must learn Akka.
  • SBT – Scala Build Tool
  • Slick – Using a database? I know, dumb question…who isn’t. So check out slick…a great way to talk to a database using Scala.
  • Swagger – Building a API? Do you have users of the API? Be kind to them and use Swagger to Document your API for them.

Conclusion

The above is a quick list of resources I have been using when developing in Scala. I plan to update this as I come across more useful resources.

Application Logging with Aspect-Oriented Programming: Part 1

Overview

Application logging is a common problem every software developer needs to deal with. Logs are valuable, providing various types of information about your application while it is running. This may include how many times a particular section of code is execute or how long is takes certain code to execute. However, sometimes it is difficult to know what logging you need before you build and deploy your application. This article is the first in a series about building a logging system using Aspect-Oriented Programming (AOP) allowing you to control what you log outside of your application. This article focuses on finding a solution for Scala and Java applications.

AOP Terminology:

AOP is a large subject, so let’s first review the terminology which applies to what we are trying to accomplish here.

pointcut – A pointcut is responsible for locating join points (points in code…methods in our case…that matches a given pattern). Pointcuts are also able to expose data from these join points (i.e. param values).

advice – Advice is where behavior is defined. In our case, this is where we will implement our logging logic. This behavior is executed for each join point (i.e. method) identified by the related Pointcut.

around – Around is a type of Advice (other Advice include before and after..which are less expensive than Around). This advice allows you to do pre-processing and post-processing. Where the other two types of Advice allow you to do one or the other.

Goal

Let’s first define what our first iteration of our Application Logger will look like. As mentioned earlier, what we want is a way to add/remove/change out logging without having to rebuild and deploy our application. What we will do to accomplish this is build a Scala java agent that we can include in any Java or Scala application. All we need will need to do is define what we want to log in a separate xml file, include our java agent in our application’s startup parameters and add our xml file to our application’s WEB-INF directory.

Building a Simple Logger Application

We will build our logging application with Scala. This application is actually pretty simple since we will use aspectj to do most of the real work. So let’s first define the core of our application, the monitor abstract class.

@Aspect
abstract class ScalaMonitor extends Logging {
    @Around(value = "methodPerformancePointcut()")
    def logMethodPerformance(jp: ProceedingJoinPoint) = {
        val startTime = currentTime
        jp.proceed()
        val endTime = currentTime
        // JMXClient.recordExecution(jp.getSignature().toString(), startTime, endTime)
        logger.info(s"type=perf elapsedTime: ${endTime-startTime}, ${jp.getSignature()}")
    }
}

The code snippet above adds logging logic to our application which measures how long it takes the targeted methods to execute. We’re using the @Around annotation which allows us to put our timing code `before` and `after` the calling method (aka JoinPoint). If we wanted to instead just add logging before or after the JoinPoint, then we’d use with the @Before or @After annotation.

The next step is to define which methods you want to add this logging to. This is done in the the aop.xml file which will be different for each application you are measuring. The xml below says we want to monitor two methods, ElevationService.process and TimezoneService.process. Here we are defining the concrete monitoring class and also defining the Pointcut. The Pointcut syntax can also include wildcards.

<aspects>
    <concrete-aspect name="com.christophergagne.scalaperformance.ScalaMonitorImpl" 
        extends="com.christophergagne.scalaperformance.ScalaMonitor">

        <pointcut name="methodPerformancePointcut" expression="execution(void com.christophergagne.sampleapp.ElevationService.process(..)) || 
            execution(void com.christophergagne.sampleapp.TimezoneService.process(..))"/>
    </concrete-aspect>
  </aspects>

Conclusion

This is all you need to start building your own logging application that you can use on any of your projects. However, I did intentionally leave out some of the details, so you can review a fully working sample application here: https://github.com/gagnechris/ScalaPerformanceMonitorDemo. Our next articles will go into more detail on the following:

  • How to define your Pointcuts with aspectj
  • How to extract method parameter values from your Joinpoint
  • Building a JMX Client which we can log to allowing us to review our data in jconsole
  • Other more advanced aspectj topics

Building RESTful APIs with Scala using Spray

Overview

There are many ways to build RESTful APIs with Scala and Spray (spray.io). Here we will review how to create end points which expose functionality and also how to call other RESTful web services taking advantages of APIs you or someone else may have written. We will use Google’s Elevations and Timezone API to demonstrate the later functionality.

Libraries Used

Setting Up the Project

The first thing we need to do is create the directory structure and add some of the other plumbing so that we can get to the real work. First you will want to create the project directory. I created a directory named SprayApiDemo in the place where I keep all of my other projects. This is the directory where we will want to create the following directory structure:

├── SprayApiDemo
│   ├── project
│   │   build.properties
│   │   plugins.sbt
│   └── src
│       ├── main
│       │   ├── resources
│       │   └── scala
│       └── test
│           └── scala
└── build.sbt

Here is the code required to create a compiling scala projects that brings in the proper dependencies for the libraries we will be using:

build.properties

sbt.version=0.13.0

plugins.sbt

addSbtPlugin("io.spray" % "sbt-revolver" % "0.7.1")
addSbtPlugin("org.ensime" % "ensime-sbt-cmd" % "0.1.2")

build.sbt

name := "SprayApiDemo"

version := "0.1"

scalaVersion := "2.10.2"

scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")

resolvers ++= Seq(
  "spray repo" at "http://repo.spray.io/"
)

libraryDependencies ++= {
  val sprayVersion = "1.2-M8"
  val akkaVersion = "2.2.0-RC1"
  Seq(
  "io.spray" % "spray-can" % sprayVersion,
  "io.spray" % "spray-routing" % sprayVersion,
  "io.spray" % "spray-testkit" % sprayVersion,
  "io.spray" % "spray-client" % sprayVersion,
  "io.spray" %%  "spray-json" % "1.2.5",
  "com.typesafe.akka" %% "akka-actor" % akkaVersion,
  "com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
  "com.typesafe.akka" %% "akka-testkit" % akkaVersion % "test",
  "ch.qos.logback" % "logback-classic" % "1.0.12",
  "org.scalatest" %% "scalatest" % "2.0.M7" % "test"
  )
}

seq(Revolver.settings: _*)

You should now be able to compile the project which won’t do much since we haven’t written anything yet. Compiling what we have so far will help confirm our build files were written properly and sbt is working as expected. You may do this by executing `sbt` while in the root directory of your project. When this is done you should see the specified dependencies being installed if they have not been already.

Creating Our Tests

Now that we have created a skeleton project, we can start adding functionality. But before we get too ahead of ourselves, let’s write a few tests. This will help us think about what we want our application to do. We will use the FreeSpec trait provided to us by the ScalaTest library. Let’s begin by adding two files to the `scala` directory in the `src\test` folder; ElevationServiceSpec.scala and TimezoneServiceSpec.scala. Each test will confirm that when the our ElevationService or TimezoneService API is called we receive the appropriate response. Obviously these tests will fail until we write the services.

ElevationServiceSpec.scala

package com.christophergagne.sprayapidemo

import scala.concurrent.future
import org.scalatest.BeforeAndAfterAll
import org.scalatest.FreeSpec
import org.scalatest.Matchers
import spray.http.StatusCodes._
import spray.testkit.ScalatestRouteTest

class ElevationServiceSpec extends FreeSpec with SprayApiDemoService with ScalatestRouteTest with Matchers {
  def actorRefFactory = system

  "The Elevation Service" - {
    "when calling GET api/ElevationService/39/80" - {
      "should return '1159.288940429688'" in {
        Get("/api/ElevationService/39/80") ~> sprayApiDemoRoute ~> check {
          status should equal(OK)
          entity.toString should include("1159.288940429688")
        }
      }
    }
  }
}

TimezoneServiceSpec.scala

package com.christophergagne.sprayapidemo

import scala.concurrent.future
import org.scalatest.BeforeAndAfterAll
import org.scalatest.FreeSpec
import org.scalatest.Matchers
import spray.http.StatusCodes._
import spray.testkit.ScalatestRouteTest

class TimezoneServiceSpec extends FreeSpec with SprayApiDemoService with ScalatestRouteTest with Matchers {
  def actorRefFactory = system

  "The Timezone Service" - {
    "when calling GET /api/TimezoneService/39/-119/1331161200" - {
      "should return 'Pacific Standard Time'" in {
        Get("/api/TimezoneService/39/-119/1331161200") ~> sprayApiDemoRoute ~> check {
          status should equal(OK)
          entity.toString should include("Pacific Standard Time")
        }
      }
    }
  }
}

Defining the Services and Routes

The next step is to create our service class and define our routes. We will create our service in a way where we can easily test each end point we want to expose. In the `scala` directory inside the `src/main` folder we will create the SprayApiDemoService.scala file. This will be where we define our parent akka actor and the service trait. Creating the service as a trait gives us the option to mix-in our service into other our parent akka actor or a test. This is a small change, with a lot of benefits.

Please also observe our routings. You can see that each service endpoint has a path prefix of `api`. This means just that, both endpoints will be found under `api`, as in http://xyz.com/api/&#8230; We are defining two endpoint which take varying parameters; ElevationService and TimezoneService. The first requires two double values to be passed, as in http://xyz.com/api/ElevationService/39/80. The seconds requires two double values and an additional segment parameter to be passed, as in http://xyz.com/api/TimezoneService/39/-119/1331161200.

You may notice that our application doesn’t compile yet. This is because we are missing the pieces that will do the real work (ElevationService and TimezoneService). Don’t worry, we’ll get to building those pieces later, which is also when you will see how we can consume someone else’s API.

SprayApiDemoService.scala

package com.christophergagne.sprayapidemo

import akka.actor.{Actor, Props}
import akka.event.Logging
import spray.routing._
import spray.http._
import MediaTypes._

class SprayApiDemoServiceActor extends Actor with SprayApiDemoService {
  
  def actorRefFactory = context

  def receive = runRoute(sprayApiDemoRoute)
}

trait SprayApiDemoService extends HttpService {
  val sprayApiDemoRoute =
    pathPrefix("api") {
      path("ElevationService" / DoubleNumber / DoubleNumber) { (long, lat) =>
        requestContext =>
          val elevationService = actorRefFactory.actorOf(Props(new ElevationService(requestContext)))
          elevationService ! ElevationService.Process(long, lat)
      } ~
      path("TimezoneService" / DoubleNumber / DoubleNumber / Segment) { (long, lat, timestamp) =>
        requestContext =>  
          val timezoneService = actorRefFactory.actorOf(Props(new TimezoneService(requestContext)))
          timezoneService ! TimezoneService.Process(long, lat, timestamp)
      }
    }
}

Bootstrapping the Service

Let’s bootstrap our application by creating the class which has the main method. We will call this `Boot.scala` and we will put it in the same directory as our service class. This class will also initialize our Akka actor system and create, initialize logging, create an instance of our service and start our HTTP server.

Boot.scala

package com.christophergagne.sprayapidemo

import akka.actor.{ActorSystem, Props}
import akka.event.Logging
import akka.io.IO
import spray.can.Http

object Boot extends App {

  // we need an ActorSystem to host our application in
  implicit val system = ActorSystem("spray-api-service")
  val log = Logging(system, getClass)

  // create and start our service actor
  val service = system.actorOf(Props[SprayApiDemoServiceActor], "spray-service")

  // start a new HTTP server on port 8080 with our service actor as the handler
  IO(Http) ! Http.Bind(service, interface = "localhost", port = 8080)
}

Creating the ElevationService

Let’s now build something that will do something. The elevation service will actually take advantage of Google’s Elevation service, which you may want to take a quick look at. Focus on the JSON response that will look something like this:

{
  results: [{
    elevation: 4838.74072265625,
    location: {
      lat: 30,
      lng: 100
    },
    resolution: 152.7032318115234
  }],
  status: "OK"
}

With this knowledge, we can now create the class which will help us deserialize the results from Google. For this you will want to create a the Elevation.scala file. Here we will use spray-json to serialize the JSON response. The classes and objects in this source file will be used when we create the Elevation Service next.

Elevation.scala

package com.christophergagne.sprayapidemo

import spray.json.{ JsonFormat, DefaultJsonProtocol }

case class Elevation(location: Location, elevation: Double)
case class Location(lat: Double, lng: Double)
case class GoogleElevationApiResult[T](status: String, results: List[T])

object ElevationJsonProtocol extends DefaultJsonProtocol {
  implicit val locationFormat = jsonFormat2(Location)
  implicit val elevationFormat = jsonFormat2(Elevation)
  implicit def googleElevationApiResultFormat[T :JsonFormat] = jsonFormat2(GoogleElevationApiResult.apply[T])
}

Now create the ElevationService.scala source file. This file will define our ElevationService class, which will be a Akka Actor and it’s companion object. When the ElevationService receives a message, it will then take the two Double parameters passed to it and use Spray’s client API to send a message to Google’s API, receive a response, serialize the response to our Elevation object and return the elevation value. If for some reason the request to Google fails, we will send back a failure as well.

ElevationService.scala

package com.christophergagne.sprayapidemo

import akka.actor.{Actor, ActorRef}
import akka.event.Logging
import akka.io.IO

import spray.routing.RequestContext
import spray.httpx.SprayJsonSupport
import spray.client.pipelining._

import scala.util.{ Success, Failure }

object ElevationService {
  case class Process(long: Double, lat: Double)
}

class ElevationService(requestContext: RequestContext) extends Actor {

  import ElevationService._

  implicit val system = context.system
  import system.dispatcher
  val log = Logging(system, getClass)

  def receive = {
    case Process(long,lat) =>
      process(long,lat)
      context.stop(self)
  }

  def process(long: Double, lat: Double) = { 

    log.info("Requesting elevation long: {}, lat: {}", long, lat)

    import ElevationJsonProtocol._
    import SprayJsonSupport._
    val pipeline = sendReceive ~> unmarshal[GoogleElevationApiResult[Elevation]]

    val responseFuture = pipeline{
      Get(s"http://maps.googleapis.com/maps/api/elevation/json?locations=$long,$lat&sensor=false")
    }
    responseFuture onComplete {
      case Success(GoogleElevationApiResult(_, Elevation(_, elevation) :: _)) =>
        log.info("The elevation is: {} m", elevation)
        requestContext.complete(elevation.toString)

      case Failure(error) =>
        requestContext.complete(error)
    }
  }
}

Creating the TimezoneService

The final step to creating our service is to write the TimezoneService. This process is very similar to creating the ElevationService, however this service will demonstrate that we’re able to send requests to APIs using https (a secure connection).

Timezone.scala

package com.christophergagne.sprayapidemo

import spray.json.{ JsonFormat, DefaultJsonProtocol }

case class Timezone(status: String, timeZoneId: String, timeZoneName: String)
case class GoogleTimezoneApiResult[T](status: String, timeZoneId: String, timeZoneName: String)

object TimezoneJsonProtocol extends DefaultJsonProtocol {
  implicit val timezoneFormat = jsonFormat3(Timezone) 
  implicit def googleTimezoneApiResultFormat[T :JsonFormat] = jsonFormat3(GoogleTimezoneApiResult.apply[T])
}

TimezoneService.scala

package com.christophergagne.sprayapidemo

import akka.actor.{Actor, ActorRef}
import akka.event.Logging
import akka.io.IO

import spray.routing.RequestContext
import spray.httpx.SprayJsonSupport
import spray.client.pipelining._

import scala.util.{ Success, Failure }

object TimezoneService {
  case class Process(long: Double, lat: Double, timestamp: String)
}

class TimezoneService(requestContext: RequestContext) extends Actor {

  import TimezoneService._

  implicit val system = context.system
  import system.dispatcher
  val log = Logging(system, getClass)

  def receive = {
    case Process(long,lat,timestamp) =>
      process(long,lat,timestamp)
      context.stop(self)
  }

  def process(long: Double, lat: Double, timestamp: String) = { 

    log.info("Requesting timezone long: {}, lat: {}, timestamp: {}", long, lat, timestamp)

    import TimezoneJsonProtocol._
    import SprayJsonSupport._
    val pipeline = sendReceive ~> unmarshal[GoogleTimezoneApiResult[Timezone]]

    val responseFuture = pipeline {
      Get(s"https://maps.googleapis.com/maps/api/timezone/json?location=$long,$lat&timestamp=$timestamp&sensor=false")
    }
    responseFuture onComplete {
      case Success(GoogleTimezoneApiResult(_, _, timeZoneName)) =>
        log.info("The timezone is: {} m", timeZoneName)
        requestContext.complete(timeZoneName)

      case Failure(error) =>
        requestContext.complete(error)
    }
  }
}

Conclusion

Above we created a RESTful service with Scala and Spray. You should now be able to compile, run and call the endpoints. To compile execute `sbt compile`. To run execute `sbt run`. And to call the endpoints try the following in your favorite browser (we’ve only implemented GET requests).

You should also verify our tests work by executing `sbt test`. Now, take what you’ve learned here and make something great.

If you want to get your hands on a complete implementation checkout the project at github.com/gagnechris/SprayApiDemo.

Git Workflow

Overview

About a year ago my team of developers migrated our application’s source code to git.  During this transition we did a lot of research to help prepare for the move.  Our main concerns were around workflow.  The team was ready to become freed from a lot of the pain they were used to while working with TFS (e.g. how difficult it is to create a feature branch).  Recently we revisited our workflow and made some improvements with how we use git.  Additionally, we moved over to gitlab (we were previously using gitblit) and started taking advantages of the features gitlab has to offer.  This article attempts to describe our workflow in it’s current state.  Maybe a similar workflow will work for your team.

Branching Strategy

The project will use two main branches; master and develop.  The master branch reflects a ‘production-ready’ state.  The develop branch reflects the latest delivered development changes for the next release.  Feature branches are also created when development on a new feature begins.  These branches are created off of develop.  While the new feature is being developed, it may get rebased with changes made to develop on a regular basis.  Lastly, when the new feature is complete, it is merged back into develop.

Two important rules to follow

  1. There will only be one level of feature branches (never create a feature branch off of a feature branch)
  2. Communicate frequently with other devs who share the same feature branch as you

Branching Details

  • origin/master
    • to be the main branch where the source code reflects production-ready state
    • tags would be used as identifiers for releases
  • origin/develop
    • to be the branch where source code reflects the latest delivered changes for the next release.  This branch is where our continuous integration environment builds, runs tests, checks code tests, deploys, etc.
      Image
  • origin/{feature branch}
    • Branch off of origin/develop which represents a new feature being developed
    • Branch names should include the story name and number
    • Rebase your feature branch with changes in origin/develop regularly
    • Create a merge request and discuss your changes with the team before merging the feature branch to origin/develop
    • Feature branch will be deleted once it’s merged back to origin/develop.  This is the responsibility of the developer who created the feature branch.
      Image
  • origin/{hotfix branch}
    • A hotfix branch may be created off of master for special circumstances.
    • Hotfix branches are deleted after the changes are deployed to production and merged to master.
    • A hotfix branch should be short lived (one day or less)

`New Feature` Lifecycle

  1. Team is assigned a new feature to work on.  A new feature branch is created for this story.  In the event that we are working on a large feature, multiple feature branches may be created.
  2. Changes related to the new feature are worked on in the feature branch.  If there is an operations support issue (bug) that needs to be worked on while this feature work is being done, then that work is done directly in the develop branch.
  3. During the life of the feature branch we may rebase it with changes happening in develop.  We typically rebase no less than once a week.
  4. Once the development of a feature is complete a merge request is created and other devs have an opportunity to review, comment and request changes to the feature branch.
  5. Once the merge request is approved by the team the feature branch is merged into develop.  The scope of the features branch changes are hopefully small enough so that we can complete them within a week or two.
  6. We have a Jenkins server which polls our develop branch for changes.  When changes are merged into develop our unit tests are ran.  Hopefully there are no test failures, but if there are they are fixed immediately.
  7. Once all tests are passing in develop we are then able to deploy the changes to other lower environments and eventually production.  This is done via Jenkins as well.
  8. After the changes are deployed to production we merge out changes to master.
  9. (Optional) In the case where a bug is found and needs to be prioritized higher than other changes already in develop, then a hotfix branch is created.  The fix for this bug is then checked into the hotfix branch deployed to production.  The hotfix branch will get merged to master and develop may be rebased with that fix.

Rebase vs. Merge

There has been numerous discussions concerning when to rebase and when to merge.  There are many other blogs that review the differences between these two commands so I will not go into much detail there.  What we basically decided is similar to what other teams are doing.  First, everyone should be setup to do a rebase when they `pull` changes from origin.  This can be done in one of two ways; `git pull –rebase` or add `autosetuprebase = true` in your .gitconfig.

Next, when creating a feature branch, it is good practice to make sure you keep your feature branch up to date with changes going on below you (develop). You can do this by doing frequent rebases.  This is preferred over merging your changes into the feature branch from develop.  Once the merge is complete you may need to do a `git push –force`, this is the ONLY time it is acceptable to force a push to origin.

Finally, when a feature branch is ready and you confirmed that ALL test cases pass on your feature branch, you will want to `merge` your changes into develop.

Deploying Django to AWS

I recently updated my personal website (this site), which is written in Python using the Django framework. I previously used Heroku to host this site because it was free assuming your only using one Web Dyno. Unfortunately if you are setup this way, the Web Dyno goes to sleep if there has been no recent activity. I found this particularly annoying because it seemed to take 10 – 20 seconds for the Web Dyno to wakeup. I could have gotten around this by adding another Web Dyno, but that would cost $34.50 per month ($414 per year).

After deploying a new technology stack to AWS recently at work, I got to thinking that AWS may be a better alternative. I figure I only need micro instances. I decided to use Elastic Beanstalk to deploy my personal website because it looked like the most straight forward way to deploy a python application in EC2 and it supports a simple deployment process using git. Additionally, I took a look at the cost of using AWS and Elastic Beanstalk and using micro instances it comes to about 14 cents per month or a little less than 2 bucks per year. This is assuming very light usage of my site, but this sure beats the $414 per year when using Heroku.

What follows is a step-by-step walkthrough of what I had to do to get my existing application up and running in AWS.
  1. If you don’t already, get an AWS account here: http://aws.amazon.com. It’s free…well, kind of. Signing up is free, but you need to pay as you go.
  2. Make sure you have a AWS Access Key and Secret Key you can use. This will be needed in future steps so you may as well get them now. If you are not sure, you can generate a new set of keys here: https://console.aws.amazon.com/iam/home?#security_credential.
  3. Install the Elastic Beanstalk (eb) command-line tools and add it to your path. You may download this from here: http://aws.amazon.com/code/6752709412171743. This will allow us to control AWS Elastic Beanstalk from the command-line.
  4. In the root directory of your repo execute `eb init`. This will help get our project ready to be deployed into AWS. Please note: during this step you will be asked to provide security credentials. Use the credentials you obtained during step 2.
  5. Execute the `eb start` command. Because we haven’t pushed our repo to AWS yet, this will simply deploy a sample application to AWS. Once this command completes you may execute `eb status –verbose` to check the provisioning status. When successful a URL will be provided. You can use this URL to confirm that the sample application is up and running.
  6. Now is the time to look at the changes `eb init` and `eb start` made to our project. You will notice the following
    a) .gitignore was modified. We are now ignoring the `.elasticbeanstalk` directory.
    b) There are now two new directories; `.ebextensions` and `.elasticbeanstalk`. These files contain additional files which hold the configuration options you selected when executing the `eb` command in steps 4 and 5.
    c) Updates were made to files in the .git directory. These updates make it possible for you to deploy your application to AWS using git. We will show you how to do this later on.
  7. Let’s now take a look at the configuration files. First you will want to create a config file in `.ebextensions` for your application. This file may look like this (or something similar, based on the specifics of your application)…https://github.com/gagnechris/syte/blob/master/.ebextensions/syte.config.If your applications require any specific configuration or packages, then you can make the necessary updates there. Here I can installing the `python-devel` and `postgresql-devel` packages which are required by my application. But please note: if your application includes a requirements.txt file, then those dependencies will automatically be installed. So those dependencies do not need to be repeated here.You will also want to review the files created for you in `.elasticbeanstalk` directory. In particular, take a look at opensettings.XXX-env (where XXX-env is the name of your environment which you specified in step 4). Make any changes necessary here.
  8. Execute `eb update` to ensure the above changes are not reverted on you (I struggled with this at first, so don’t make the same mistakes as me).
  9. You are now ready to deploy your application. This can be done executing `git aws.push`. That’s right, we’re deploying our application to AWS using git!
  10. The above command may take a while to complete, but when it does you can check the status by executing `eb status –verbose`. If there are any issues deploying your application, then I would recommend loggining into AWS, go to Elastic Beanstalk console, select Logs and select `snapshot Logs`. This will retrieve the logs from the EC2 instance and display them to you. You can then review the logs for errors specific to your issue.

Congrats! You now have your django application up and running in AWS!