The Fun of Experimenting with a more Advanced Microservice Application - Building a Slack “Done This” Tracker

On By Ad van der Veer in tech

Guest Post: Ad Van der Veer is a student at TU Delft and the developer of dockpit.io. He also was an intern Giant Swarm for several months in 2014/2015, where he tested Dockpit on the swarm and worked on some more advanced microservice examples as well as his master thesis on microservices.

We cannot escape complexity. As a workaround we often delegate complexity by reusing libraries. However, that doesn’t really solve the problem or at least it creates new problems.

Microservice systems suffer similar problems. Although individual services are simple and loosely coupled, complexity now ends up in the architectural layer. Managing architecture can be hard, luckily the microservice concept is really catching up and one of the companies that allows you to stop worrying about this layer, but provides enough freedom to build the architecture you have in mind, is Giant Swarm.

Of course, the value of having an expert manage the complexity of your microservice architecture only becomes apparent when you move beyond the basic hello world examples and start creating and deploying more serious setups. In this post I’ll walk through the creation of such a more complex app called “Slack Topscore” by using the swarm.

Slack Topscore

Slack is a team communication platform that has the concept of channels that allow team members to chat about certain topics. At Giant Swarm (and some other companies) one of those channels (#donethis) is specifically used to allow every team member to report on the tasks they completed that day. (You can read more about the tools used at Giant Swarm in a dedicated blog post.)

I wanted to create a microservice app that analyzes the content of above-mentioned channel and keeps a scoreboard with the total number of todos each team member ever completed. Users should be able to query the current score board by issuing a ‘top’ command to the company’s chat room bot “swarmbot”.

Schematic components overview

In spirit of the microservices philosophy three services (listed below) where planned that each had their single purpose. By splitting up the application in separate services we don’t have to stick to a single language but can choose the best tool for the job for each service. To reduce coupling between the services we also introduce a message queue. This has the additional benefit of making the setup extendable (more on this later).

  • A “harvester” service written in Go that polls the Slack API once every second and checks if there are any new messages in the #donethis channel. This service publishes slack messages on an NSQ message queue.
  • A Go service called “topscore” that listens to the message queue and analyzes messages for todos and stores them in a MongoDB database. It also exposes an HTTP API for retrieving a JSON encoded scoreboard.
  • A NodeJS Hubot robot that allows users to query for their current topscore by using the API of the ‘topscore’ service.

This microservices setup already requires us to manage 5 different Docker containers that communicate both internally (on the same host) and over the network (in a cluster of hosts). Normally, we would have to worry not only about how they should be linked or how to discover them in a cluster setup, but also make sure that they remain stable (running) and manage virtual machines using Amazon or Google compute for example.

As my application was still evolving I didn’t want to spend many hours of frustrating scripting to quickly re-deploy my application to try out configurations.

Using the Swarm

Giant Swarm allowed me to quickly sketch out a way, in which the app is composed by filling a single JSON file. Quick and representative feedback is important and the swarm makes it possible that a live version is always only a single command away.

For the impatient, the final results can be found here:

github.com/giantswarm/giantswarm-advancedexample

The Harvester

The responsibility of the harvester is to connect to the Slack API and fetch messages for broadcasting on the message queue – its configuration looks like this (note: this and the following json snippets are only parts of the whole swarm.json):

    "harvester/nsqd": {
      "image": "nsqio/nsqd",
      "ports": [
        "4150/tcp"
      ]
    },
    "harvester/process": {
      "image": "registry.giantswarm.io/$username/ext-harvester",
      "args": [
        "app",
        "--slack-channel=$channel",
        "--slack-token=$slack_token",
        "--nsqd-addr=nsqd:4150"
      ],
      "links": [
        {
          "component": "harvester/nsqd",
          "alias": "nsqd",
          "target_port": "4150/tcp"
        }
      ]
    },

A lot is going on here, so lets break it down: The configuration allows me to easily couple the Golang container with a NSQ message queue and only link its TCP port 4150 to the harvester process. A private registry is provided just for my application images that we can then use in the configuration file.

We want to keep the access token of Slack out of our repository. For this Giant Swarm provides a handy way of moving specific variables outside the main config and into a dedicated file, which we could keep out of public repositories for example.

The Analyzer

The analyzer listens to incoming messages and as such should also be linked to the NSQ process:

    "topscore/mongo": {
      "image": "mongo",
      "ports": [
        "27017/tcp"
      ],
      "volumes": [
        {
          "path": "/data/db",
          "size": "10 GB"
        }
      ]
    },
    "topscore/process": {
      "image": "registry.giantswarm.io/$username/ext-topscore",
      "ports": [
        "9000/tcp"
      ],
      "args": [
        "app",
        "--nsqd-addr=nsqd:4150",
        "--mongo-addr=mongodb:27017/topscore"
      ],
      "links": [
        {
          "component": "harvester/nsqd",
          "alias": "nsqd",
          "target_port": "4150/tcp"
        },
        {
          "component": "topscore/mongo",
          "alias": "mongodb",
          "target_port": "27017/tcp"
        }
      ]
    }

The configuration file makes this easy by being able to specify dependencies to components in other services, namely the NSQ process, and making sure that components are started in the correct order. We also specify a MongoDB instance for persisting the scoreboards for each member. In this case we also specify a volume for MongoDB to make sure the data remains intact after the container is restarted.

The Bot

Linking our hubot to the topscore process is again trivial and done in the same fashion as the services above. The bot does require some specific environment variables, these are also easily configurable through the swarm.json:

    "bot/hubot": {
      "image": "registry.giantswarm.io/$username/ext-bot",
      "env": [
        "HUBOT_IRC_ROOMS=#$channel",
        "HUBOT_IRC_SERVER=$irc_server",
        "HUBOT_IRC_NICK=$irc_nickname",
        "HUBOT_IRC_PASSWORD=$irc_password",
        "HUBOT_IRC_UNFLOOD=true",
        "HUBOT_IRC_USESSL=true",
        "HIGHSCORE_ENDPOINT=http://topscore:9000"
      ],
      "links": [
        {
          "component": "topscore/process",
          "alias": "topscore",
          "target_port": "9000/tcp"
        }
      ]
    },

Conclusion

Experimenting with new application architectures is great fun, and not having to struggle with fragile bash scripts makes it even better. Giant Swarm indeed makes it easier to play around with different tools and architecture configurations.

As for the application itself, it is a fun way to get content from the Slack API in general. More specifically it is a simple way of tracking tasks and maybe even have some fun motivating yourself to accomplish more (for this also see Trade Your To-Do List for a Done List). I already got some fun ideas to play around with what I built above. One could think of applying some NLP algorithms for example to see what people are working on or analyze the co-working network that emerges from people mentioning others in tasks (e.g. “finalized new docs with @colleague”). All of these could be new separate components that plug into the microservice system easily through the message queue.

If this little application gave you some other awesome ideas for playing around with Slack messages, feel free to fork the repo and send me some pull requests and if you’re not on Giant Swarm, yet, you should request an invite.

Picture of Ad van der Veer
Ad van der Veer
Ad is a student at TU Delft. He was an intern at Giant Swarm in 2014/2015, where he tested his tool dockpit.io on the swarm and worked on his master thesis on microservices.

Let’s start your journey towards microservices

Contact us Now