The Sweet Spot

Andrew Hao's thoughts about software engineering, design, and anything shiny.

Toolbox: learning Swift and VIPER

| Comments

The following are some notes I’m compiling as I’m beginning a journey down the rabbit hole, writing an app in Swift utilizing the VIPER app development methodology

  • I had trouble importing nested source code into XCode before realizing that I needed to import the folder with corresponding Groups. This is done by clicking the checkbox “Create Groups for any Added Folders”

    Reference: https://developer.apple.com/library/ios/technotes/iOSStaticLibraries/Articles/configuration.html

    Without doing this, the compiler was not able to build the project.

    • Since there is no way to do method swizzling in Swift, there are no real easy ways to do mocking/stubbing the way we used to do so in Ruby. Instead, this is forcing me to rely on plain old Swift structs. There are some simple ways to stub, but it ends up looking kind of awkward and very wiring-intensive like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class NewRidePresenterSpec: QuickSpec {
  override func spec() {
    describe("#startRecordingGpsTrack") {
      class MockInteractor: NewRideInteractor {
        var wasCalled: Bool = false

        @objc private override func startRecordingGpsTrack() {
          wasCalled = true
        }
      }

      var subject = NewRidePresenter()

      it("tells the interactor to start recording") {
        let mockInteractor = MockInteractor()
        subject.interactor = mockInteractor
        subject.startRecordingGpsTrack()

        expect(mockInteractor.wasCalled).to(beTrue())
      }
    }
  }
}
  • Using the vipergen and boa scaffolding generators helped me understand the concepts behind the view.
  • Tip: Build a VIPER module, but don’t build it all at once. Just focus on the Presenter-Interactor-Wireframe component, or the DataStore-Entity-Interactor component. This will keep your head from exploding.
  • Dude. I miss vim. Alcatraz + xvim helped a little…
  • xcodebuild + xcpretty + Guard-shell == some sort of CI feedback loop.
  • Manually creating mocks in Swift = kind of painful. If you override (subclass) a NSObject in Swift, you must provide it with the @objc pragma, otherwise it throws a segfault error
  • You must contact CircleCI manually if you want to activate an iOS build (it’s still in beta). What are some other good CI tools to use with iOS?

Building GPX stats through FRP principles with Bacon.js

| Comments

With my current fascination with tracking workouts and location-based-activities, I have been interested in how I might be able to rewrite some of my stats logic with FRP principles.

What is FRP?

FRP, or Functional Reactive Programming, is often defined as “functional programming over values that change over time”. It uses functional composition for streams of data that may appear in an infinite stream of data for some far indeterminate future – these types of use cases are served well by FRP which “(simplifies) these problems by explicitly modeling time”.

GPS – your location, varied over time.

A great application of this would be a workout. Let’s say I wanted to build an app that received realtime updates on a person’s position. Say the app was a Node server that received this JSON blob from a web API as a location update:

1
2
3
4
{ 'lat': 29.192414,
  'lon': 148.113241,
  'ele': 122.1,
  'time': '2015-04-18T13:54:56Z' }

Say that some time later, the API receives this JSON blob:

1
2
3
4
{ 'lat': 29.192424,
  'lon': 148.113251,
  'ele': 123.1,
  'time': '2015-04-18T13:55:26Z' }

So we have these data points, that the user has moved +0.00001 latitude points and +0.00001 longitude points, climbing a total of +1.0 meters, over a period of 30 seconds.

Exercise: Get my instantaneous velocity

If we performed this imperatively, we would write it something like this:

1
2
3
4
5
6
7
var locations = [{ /*json*/ }, { /*json*/ } /*, ...*/];
var last = locations[locations.length-1];
var secondToLast = locations[locations.length-2];
var timeDelta = last.time - secondToLast.time;
var distanceDelta = getDistance(last.lon, last.lat, secondToLast.lon, secondToLast.lat);
var velocity = distanceDelta / timeDelta;
console.log(velocity);

With FRP, it might look more like this:

1
2
3
4
5
6
7
8
9
10
var locationStream = [{ /*json*/ }, { /*json*/ } /*, ...some JSON objects that might appear in the future */];
locationStream.slidingWindow(2)
              .map(function(pairs) {
                var timeDelta = pairs[1].time - pairs[0].time;
                var distanceDelta = getDistance(pairs[1].lon, pairs[1].lat, pairs[0].lon, pairs[0].lat);
                return distanceDelta / timeDelta
              })
              .onValue(function(velocity) {
                console.log(velocity);
              });

There is a key difference that is not easily demonstrated here – that the former imperative example requires that all JSON arrays be materialized at once – via db query, in-memory store, etc. It doesn’t account for change in time.

However, the latter functional example accounts for changing values of time as they appear over the stream – as soon as a new value shows up in the stream, the velocity is changed instantly.

Some more location-based experiments: rxlocation

I wrote up a library to parse various facts from a changing stream of GPS events, from instantaneous velocity, average velocity, moving/stopped status, etc.

I investigated different reactive frameworks, mainly RxJS and Bacon.js. My takeaways were that RxJS does everything and the kitchen sink, but I got lost trying to reconcile Node streams with RxJS cold streams. Bacon.js just seemed to work for me, out of the box. I’m still learning, so I hope to have a better understanding of the core issues here.

You can check it out here: rxlocation.

Docker, Rails, and Docker Compose in your development workflow

| Comments

(This post originally appeared on the Carbon Five blog.)

We’ve been trialing the usage of Docker and Docker Compose (previously known as fig) on a Rails project here at Carbon Five. In the past, my personal experience with Docker had been that the promise of portable containerized apps was within reach, but the tooling and development workflow were still awkward – commands were complex, configuration and linking steps were complicated, and the overall learning curve was high.

My team decided to take a peek at the current landscape of Docker tools (primarily boot2docker and Docker Compose) and see how easily we could spin up a new app and integrate it into our development workflow on Mac OS X.

In the end, I’ve found my experience with Docker tools to be surprisingly pleasant; the tooling easily integrates with existing Rails development workflows with only a minor amount of performance overhead. Docker Compose offers a seamless way to build containers and orchestrate their dependencies, and helps lower the learning curve to build Dockerized applications. Read on to find out how we built ours.

Introduction to docker-compose (née Fig).

Docker Compose acts as a wrapper around Docker – it links your containers together and provides syntactic sugar around some complex container linking commands.

We liked Docker Compose for its ability to coordinate and spin up your entire application and dependencies with one command. In the past, frameworks like Vagrant were easy ways to generate a standard image for your development team to use and get started on. Docker Compose offers similar benefits of decoupling the app from the host environment, but also provides the container vehicle for the app to run in all environments – that is, the container you develop in will often be the same container that you deploy to production with.

Docker (with the orchestration tooling provided by Compose) provides us the ability to:

  • Upgrade versions of Ruby or Node (or whatever runtime your app requires) in production with far less infrastructure coordination than normally required.
  • Reduce the number of moving parts in the deployment process. Instead of writing complex Puppet and Capistrano deployment scripts, our deployments will now center around moving images around and starting containers.
  • Simplify developer onboarding by standardizing your team on the same machine images.

In this example, we will run two Docker containers – a Rails container and a MySQL container – and rely on Compose to build, link, and run them.

Installing boot2docker, Docker, and Docker Compose.

Docker runs in a VirtualBox VM through an image called boot2docker. The reason we have to use boot2docker and VirtualBox is because the Mac OSX filesystem is not compatible with the type of filesystem required to support Docker. Hence, we must run our Docker containers within yet another virtual machine.

  1. Download and install VirtualBox.
  2. Now install boot2docker and Docker Compose.
1
$ brew install boot2docker docker-compose
  1. Initialize and start up boot2docker
1
2
$ boot2docker init
$ boot2docker start
  1. Configure your Docker host to point to your boot2docker image.
1
$ $(boot2docker shellinit)

You’ll need to run this for every terminal session that invokes the docker or docker-compose command – better export this line into your .zshrc or .bashrc.

Creating a Dockerfile

Let’s start by creating a Dockerfile for this app. This specifies the base dependencies for our Rails application. We will need:

  • Ruby 2.2 – for our Rails instance
  • NodeJS and NPM – for installation of Karma, jshint, and other JS dependencies.
  • MySQL client – for ActiveRecord tasks
  • PhantomJS – for executing JS-based tests
  • vim – for inspecting and editing files within our container

Create a Dockerfile from within your Rails app directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
FROM ruby:2.2.0
RUN apt-get update -qq && apt-get install -y build-essential nodejs npm nodejs-legacy mysql-client vim
RUN npm install -g phantomjs

RUN mkdir /myapp

WORKDIR /tmp
COPY Gemfile Gemfile
COPY Gemfile.lock Gemfile.lock
RUN bundle install

ADD . /myapp
WORKDIR /myapp
RUN RAILS_ENV=production bundle exec rake assets:precompile --trace
CMD ["rails","server","-b","0.0.0.0"]

Let’s start by breaking this up line-by-line:

1
FROM ruby:2.2.0

The FROM directive specifies the library/ruby base image from Docker Hub, and uses the 2.2.0 tag, which corresponds to the Ruby 2.2.0 runtime.

From here on, we are going to be executing commands that will build on this reference image.

1
2
RUN apt-get update -qq && apt-get install -y build-essential nodejs npm nodejs-legacy mysql-client vim
RUN npm install -g phantomjs

Each RUN command builds up the image, installing specific application dependencies and setting up the environment. Here we install our app dependencies both from apt and npm.

An aside on how a Docker image is built

One of the core concepts in Docker is the concept of “layers”. Docker runs on operating systems that support layering filesystems such as aufs or btrfs. Changes to the filesystem can be thought of as atomic operations that can be rolled forward or backwards.

This means that Docker can effectively store its images as snapshots of each other, much like Git commits. This also has implications as to how we can build up and cache copies of the container as we go along.

The Dockerfile can be thought of as a series of rolling incremental changes to a base image – each command builds on top of the line before. This allows Docker to quickly rebuild changes to the reference image by understanding which lines have changed – and not rebuild the image from scratch each time.

Keep these concepts in mind as we talk about speeding up your Docker build in the following section.

Fast Docker builds by caching your Gemfiles

The following steps install the required Ruby gems for Bundler, within your app container:

1
2
3
4
WORKDIR /tmp
COPY Gemfile Gemfile
COPY Gemfile.lock Gemfile.lock
RUN bundle install

Note how we sneak the gems into /tmp, then run the bundle install which downloads and installs gems into Bundler’s vendor/bundle directory. This is a cache hack – whereas in the past we would have kept the Gemfiles in with the rest of the application directory in /myapp.

Keeping Gemfiles inline with the app would have meant that the entire bundle install command would have been re-run on each docker-compose build — without any caching — due to the constant change in the code in the /myapp directory.

By separating out the Gemfiles into their own directory, we logically separate the Gemfiles, which are far less likely to change, from the app code, which are far more likely to change. This reduces the number of times we have to wait for a clean bundle install to complete.

HT: Brian Morearty: “How to skip bundle install when deploying a Rails app to Docker”

Adding the app

Finally, we finish our Dockerfile by adding our current app code to the working directory.

1
2
3
4
ADD . /myapp
WORKDIR /myapp
RUN RAILS_ENV=production bundle exec rake assets:precompile --trace
CMD ["rails","server","-b","0.0.0.0"]

This links the contents of the app directory on the host to the /myapp directory within the container.

Note that we precompile all our assets before the container boots up – this ensures that the container is preloaded and ready to run and jives with Docker tenets that a container should be the same container that runs in development, test, and production environments.

Setting up Docker Compose

Now that we’ve defined a Dockerfile for booting our Rails app, we turn to the Compose piece that orchestrates the linking phase between the Rails app and its dependencies – in this case, the DB.

A docker-compose.yml file automatically configures our application ecosystem. Here, it defines our Rails container and its db container:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
web:
  build: .
  volumes:
    - .:/myapp
  ports:
    - "3000:3000"
  links:
    - db
  env_file:
    - '.env.web'
db:
  image: library/mysql:5.6.22
  ports:
    - "13306:3306"
  env_file:
    - '.env.db'

A simple:

$ docker-compose up

will spin up both the web and db instances.

One of the most powerful tools of using Docker Compose is the ability to abstract away the configuration of your server, no matter whether it is running as a development container on your computer, a test container on CI, or on your production Docker host.

The directive:

1
2
links:
  - db

will add an entry for db into the Rails’ container’s /etc/hosts, linking the hostname to the correct container. This allows us to write our database.yml like so:

1
2
3
# config/database.yml
development: &default
  host: db

Another important thing to note is the volumes configuration:

1
2
3
# docker-compose.yml
volumes:
  - .:/myapp

This mounts the current directory . on the host Mac to the /myapp directory in the container. This allows us to make live code changes on the host filesystem and see code changes reflected in the container.

Also note that we make use of Compose’s env_file directive, which allows us to specify environment variables to inject into the container at runtime:

1
2
env_file:
  - '.env.web'

A peek into .env.web shows:

1
2
3
4
5
6
7
PORT=3000
PUMA_WORKERS=1
MIN_THREADS=4
MAX_THREADS=16
SECRET_KEY_BASE=<Rails secret key>
AWS_REGION=us-west-2
# ...

Note that the env_file is powerful in that it allows us to swap out environment configurations when you deploy and run your containers. Perhaps your container needs separate configurations on dev than when on CI, or when deployed to staging or on production.

Creating containers and booting them up.

Now it’s time to assemble the container. From within the Rails app, run:

$ docker-compose build

This downloads and builds the containers that your web app and your db will live in, linking them up. You will need to re-run the docker-compose build command every time you change the Dockerfile or Gemfile.

Running your app in containers

You can bring up your Rails server and associated containers by running:

$ docker-compose up

This is a combination of build, link, and start-services command for each container. You should see output that indicates that both our web and db containers, as configured in the docker-compose.yml file, are booting up.

Development workflow

I was pleasantly surprised to discover that developing with Docker added very little overhead to the development process. In fact, most commands that you would run for Rails simply needed to be prepended with a docker-compose run web.

When you want to run: With Docker Compose, you would run:
bundle install docker-compose run web bundle install
rails s docker-compose run web rails s
rspec spec/path/to/spec.rb docker-compose run web rspec spec/path/to/spec.rb
RAILS_ENV=test rake db:create docker-compose run -e RAILS_ENV=test web rake db:create
tail -f log/development.log docker-compose run web tail -f log/development.log

Protips

Here are some nice development tricks I found useful when working with Docker:

  • Add a dockerhost entry to your /etc/hosts file so you can visit dockerhost from your browser.
1
2
$ boot2docker ip
192.168.59.104

Then add the IP to your /etc/hosts

1
192.168.59.104  dockerhost

Now you can pull up your app from dockerhost:3000:

Screenshot of your URL bar

  • Debugging containers with docker exec

    Sometimes you need to get inside a container to see what’s really happening. Perhaps you need to test whether a port is truly open, or verify that a process is truly running. This can be accomplished by grabbing the container ID with a docker ps, then passing that ID into the docker exec command:

1
2
3
4
5
$ docker ps
CONTAINER ID        IMAGE
301fa6331388        myrailsapp_web:latest
$ docker exec -it 301fa6331388 /bin/bash
root@301fa6331388:/myapp#
  • Showing environment variables in a container with docker-compose run web env
1
2
3
4
5
6
7
8
$ docker-compose run web env
AWS_SECRET_KEY=
MAX_THREADS=16
MIN_THREADS=4
AWS_REGION=us-west-2
BUNDLE_APP_CONFIG=/usr/local/bundle
HOME=/root
#...
  • Running an interactive debugger (like pry) in your Docker container

    It takes a little extra work to get Docker to allow interactive terminal debugging with tools like byebug or pry. Should you desire to start your web server with debugging capabilities, you will need to use the --service-ports flag with the run command.

1
$ docker-compose run --service-ports web

This works due to two internal implementations of docker-compose run:

  • docker-compose run creates a TTY session for your app to connect to, allowing interactive debugging. The default docker-compose up command does not create a TTY session.
  • The run command does not map ports to the Docker host by default. The --service-ports directive maps the container’s ports to the host’s ports, allowing you to visit the container from your web browser.

  • Use slim images when possible on production

Oftentimes, your base image will come supplied with a -slim variant on Docker Hub. This usually means that the image maintainer has supplied a trimmed-down version of the container for you to use with source code and build-time files stripped and removed. You can oftentimes shave a couple hundred megabytes off your resulting image — we did when we switched our ruby image from 2.2.1 to 2.2.1-slim. This results in faster deployment times due to less network I/O from the registry to the deployment target.

Gotchas

  • Remember that your app runs in containers – so every time you do a docker-compose run, remember that Compose is spinning up entirely new containers for your code but only if the containers are not up already, in which case they are linked to that (running) container.

    This means that it’s possible that you’ve spun up multiple instances of your app without thinking about it – for example, you may have a web and db container already up from a docker-compose up command, and then in a separate terminal window you run a docker-compose run web rails c. That spins up another web container to execute the command, but then links that container with the pre-launched db container.

  • There is a small but noticeable performance penalty running through both the VirtualBox VM and docker. I’ve generally noticed waiting a few extra seconds when starting a Rails environment. My overall experience has been that the penalty has not been large enough to be painful.

Try it out

Give this a shot and let me know how Docker has been working for you. What have your experiences been? What are ways in which you’ve been able to get your Docker workflow smoother? Share in the comments below.

Coming up: integration with CI and deployment.

In upcoming blog posts, we will investigate how to use the power of Docker Compose to test and build your containers in a CI-powered workflow, push to Docker registries, and deploy to production. Stay tuned!

Explorations in logic programming

| Comments

Out of Storybook, a side project I’ve been doing for a friend, I had the opportunity to model the problem domain as a constraint satisfaction problem (CSP). It goes:

A set of students are assigned a book a week. Write plan generator to create a set of student-book assignments for that week, given the following constraints:

  • All students must receive a book.
  • Each book may only be assigned to one student at a time.
  • A student may not be assigned a book s/he has received before.

Being that this was a Rails app, I put Amb, a Ruby-based CSP solver, to use. Amb is derived off Jim Weirich’s original source code, implementing a simple backtracking algorithm. (More interesting reading on the original idea behind the amb operator, proposed in a paper by LISP founder John McCarthy in 1963.)

Not having written any CSP logic since my university days, I tried to come up with a naive solution. It goes something like this:

assignment_problem_1.rblink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class AssignmentProblem
  def solve
    # Generates the cross product of sids and bids.
    spaces = student_ids.product(bag_ids)

    # Now generate combinations of those uniques
    full_solution_space = spaces.permutation(student_ids.count).to_a

    # Assign those to the CSP
    plan = solver.choose(*full_solution_space)

    solver.assert all_students_have_bags(plan)
    solver.assert assigned_bags_are_unique(plan)
    solver.assert assigned_bags_without_student_repeats(plan)

    plan
  end
end

Well this would work for 2 students and 2 bags, 3 and 3, 4 and 4, but would blow up and overflow the stack when we got to 5 students and 5 bags. Why? I was forcing Ruby to generate the full solution space before starting the CSP search problem:

1
2
3
4
5
student_ids = [:s1, :s2]
bag_ids = [:b1, :b2]
# Modeled as: the complete solution space of assignments
spaces = student_ids.product(bag_ids)
# => [[:s1, :b1], [:s1, :b2], [:s2, :b1], [:s2, :b2]]

Then I was taking that cross product and brute-force generating all possible permutations internal to that:

1
2
3
4
5
6
7
spaces.permutation(student_ids.count).to_a
# => [[[:s1, :b1], [:s1, :b2]], [[:s1, :b1], [:s2, :b1]],
#     [[:s1, :b1], [:s2, :b2]], [[:s1, :b2], [:s1, :b1]],
#     [[:s1, :b2], [:s2, :b1]], [[:s1, :b2], [:s2, :b2]],
#     [[:s2, :b1], [:s1, :b1]], [[:s2, :b1], [:s1, :b2]],
#     [[:s2, :b1], [:s2, :b2]], [[:s2, :b2], [:s1, :b1]],
#     [[:s2, :b2], [:s1, :b2]], [[:s2, :b2], [:s2, :b1]]]

The most obvious thing that stood out to me like a sore thumb was how I wasn’t simply trusting the constraint-based nature of the problem and expressing the problem in terms of the constraints, instead of attempting to imperatively generate the solution. Usage of Enumerable#permutation resulted in an O(n!) algorithm, which is unacceptable. Back to the drawing board:

assignment_problem_2.rblink
1
2
3
4
5
6
7
8
9
10
11
12
class AssignmentProblem
  # Generates tuples of student => bag assignments
  def solve
    student_ids.each do |sid|
      bid = solver.choose(*bag_ids)
      partial_plan[sid] =  bid
      solver.assert assigned_bags_are_unique(partial_plan)
      solver.assert assigned_bags_without_student_repeats(partial_plan)
    end

    partial_plan.to_a
  end

Note the main difference here is how I’ve re-tooled the solver to assign variables one at a time (in L5) and check constraints after each assignment. This simplifies the domain of the problem as a graph search and helps us more easily reason about this program.

Further explorations

Follow my csp-solvers project, where I attempt to rewrite this in Prolog and Clojure in an attempt to see how language affects how we reason about problems like these. It’s gonna be fun.

Further reading

My own robot training buddy.

| Comments

As an ultra runner, I am really into the mountains. As a software engineer, I’m really into data. So naturally, I’m interested in the intersection of both.

I’ve particularly been interested in how systems like Strava work, especially when they quantify what is known as a “Suffer Score”, a single number denoting the amount of training stress (a.k.a. suffering) you put yourself through in a workout.

How does a track workout compare to a long day on the trails? Which is tougher: a 5m tempo road run in and around my neighborhood, or a tough 2m climb into a local regional park?

Data in…

I first attacked the problem of getting data off of my phone. I record my GPX tracks in Runmeter, a fantastic iPhone application with all sorts of metrics and data export capabilities. What I wanted was a seamless way to get the data off my phone without fuss after a hard workout.

The application has a nifty feature in which it can automatically send an email to an email address after a workout is completed.

I wrote an email ingester, Velocitas, with the help of Cloudmailin, which fires off a POST request to the Node application. Velocitas does the following:

  • curls and downloads the GPX link embedded in the email.
  • Saves the GPX file to a linked Dropbox account.
  • Republishes the GPX file to a linked Strava account.

Deriving the Training Stress Score

Next up: I wanted to do a quick and dirty implementation of the (run-based) Training Stress Score. Stressfactor, a Ruby gem, is what came out of it.

It implements the rTSS as detailed in this article.

1
2
(duration_seconds * normalized_graded_pace * intensity_factor) /
(functional_threshold_pace * 3600) * 100

Stressfactor is a higher-order tookit for deriving meaning from GPX tracks, so it, at the moment, attempts to calculate the stress score and grade adjusted pace.

The data still needs validation, so I’m eager to run it on my data set of GPX tracks from the past years.

Generating reports

I’m working on this part right now — I need to nicely display a report from my workout history in Dropbox and display per-GPX. I’ve started the project — Stressreport.

Some things I’ve learned and am learning

  • The human body is complex, and cannot be easily modeled without sufficient data. That said, what I’m doing now may be sufficient for basic training data.
  • The nature of parsing and generating higher-order stats from raw data may lend itself well to experimentation with functional languages. I’m interested in trying to reimplement Stressfactor in Scala, or a functional equivalent.
  • Deploying all my apps on Heroku’s free tier may actually be an interesting start to building a microservice architecture — with the space limitations on Heroku, I’m forced to build new features on new services. Call it cheapskate architecture.

Recap: QCon SF 2014

| Comments

Blurb sent me off to QCon SF 2014 for three days.

Notes

I took a series of notes each day in attendance:

Summary

  • Big trends in continuous delivery and deployment — deploy more often, smaller feedback loops
  • A lot of emphasis on event driven architectures + microservices. Lots of emphasis on DDD as a design tool.
  • Reactive systems with functional implementations were widely discussed as a scaling tool (backpressure-sensitive) and as a coordination tool between multiple async services.
  • Big data/realtime streaming talks were interesting — my personal experience with them is limited, but it seems there is a debate over the merits of existing Lambda architecture practice.
  • A lot of talk about microservice orchestration tools — acknowledging the pain of configuration and management of many services.
  • Scala got a lotttt of attention. Probably because of its presence in bigger companies like Netflix, Twitter, LinkedIn. Wonder what smaller startups are using.
  • Web Components were a big upcoming trend in frontend technologies. Strong modularization of views + behaviors in HTML documents.

Questions

  • If I could do a startup over again, would I begin an app in Rails? Where is the sweet spot for that sort of application architecture?
  • How can I design systems such that they can be extractible into focused components/services as early as possible?
  • How can we plan for failures (fault injection)?
  • How does one implement change in software engineering organizations? Bottom-up (organic initiatives bubbling up through management) vs top-down (management/software leaders direct org to implement).
  • How are we doing with encouraging women and minorities who traditionally are underrepresented in our industry?
  • What are places in our hiring funnels that, unbeknownst to us, may be turning away or detracting women and minorities?

Conway’s Law for humans

| Comments

If you’re familiar with Conway’s Law, it states:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Or in layman’s terms, your software systems reflect the structure of the teams that create them.

Think about it — do your teams prefer to do everything themselves? Or do they ask for help from other teams? In general, as a team, we prefer to have as little dependencies as possible. The work that the engineer has to do to send an email, or wait for work from another team (like an API change or design change) is usually time-consuming and burdensome. Therefore, it is not (usually) in the team’s best interests to cross silos and ask for help from others. Your teams, if they look like this, tend to work in codebases that are generally monolithic and wholly owned by the team.

It’s not wrong, or it’s not bad, it’s just a sociological observation. This is why companies like Spotify, Netflix or Amazon have embraced Conway’s Law and changed their organizations to match the microservice-driven architecture they want. Small teams work on small codebases and are empowered to make the changes they need.

A corollary and some observations about your company culture.

Here’s a corollary, which I’ve heard in various shapes and forms. Paraphrased:

An organization’s structure tends to pattern after its leaders’ communication patterns.

I’ve been pushing an effort to unify our company’s frontend styles and UX into a unified framework in an attempt to standardize the look and feel of the site.

However, in working with our designers, I realized that they weren’t talking to each other. Designers in one department had opposing design aesthetics from designers in another. This was causing problems in the code, because the frontend framework itself was becoming fragmented. You could see it in the code. Version A of the styleguide went into Department A’s products. Version B of the styleguide went to Department B’s products.

In this case, as we kept rolling out this framework, I realized our organization had no single owner for the design language of the site. Why? I had to wonder if it had to do with some deeper communication issues between the heads of the two departments.

Code can be a good canary to organizational issues, and call out larger human issues at hand. In this case, Conway’s Law helps us root out and bring into the light structural concerns. Leaders can pay attention to these concerns and check themselves to become more open communicators.

Mocks aren’t stubs: mockist & classic testing

| Comments

With the famed “TDD is dead” debate around the Rails community largely coming to an end, I found myself referencing Martin Fowler’s article, Mocks Aren’t Stubs a good deal, trying to make sense of it in terms of how I write tests and code.

In this post I’m going to talk about mocking and stubbing and their roots, paraphrase Fowler in an attempt to explain their differences, and walk through a couple of code examples. In each case, I’m going to attempt to build this example out in Ruby and RSpec 3.

Let’s assume this implementation in our code for a BookUpdater object in Ruby. Its job is to call through its collaborating ApiClient, which wraps some aspect of an API that we want to call.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Update a book's metadata in our systems.
class BookUpdater
  attr_accessor :book, :api_client, :response

  def initialize(book, api_client)
    @book = book
    @api_client = api_client
  end

  def updated?
    !!response.success?
  end

  def update!
    response = api_client.call_update_api(book)
  end
end

What they are

Mocks

Mocks are fake objects that verify that they have received messages. In RSpec, we traditionally use the mock object utility to create these objects.

1
2
3
4
5
6
7
api_client = mock('api client')
book = Book.new

expect(api_client).to receive(:call_update_api).with(book).and_return(true)

subject = BookUpdater.new(api_client, book)
subject.list!

What’s happening here? RSpec creates a mock api_client object that will verify that, after the test case executes, it has received the :call_update_api message with the correct arguments.

The main point of this style of testing is behavior verification — that is, that your object is behaving correctly in relation with its collaborators.

Double

Let’s take a look at a double — also known as a stub. A double is a fake object that is set up to respond to a certain message with a pre-canned response, each time. Let’s take a look at how I would set up a test using doubles:

1
2
3
4
5
6
api_client = double('api client')
response = double('response', :success? => true)
book = Book.new

allow(api_client).to receive(:call_update_api).with(book).and_return(response)
expect(subject.update!).to change(subject, :updated?).from(false).to(true)

Okay, so what’s the big deal here? My test case still passes. Note that I had to change my code to focus its expectation on the subject’s state instead of the api_client.

The focus of using doubles is for state verification — that is, that so long as everybody around me is behaving according to their contracts, the test merely has to verify that internal object state changes correctly.

A third way — real objects

I won’t cover this very much in depth, but with sufficiently simple objects, one could actually instantiate real objects instead of doubles, and test an object against all its collaborators. This is, in my experience, the most common experience of working in Rails + ActiveRecord applications.

Classic vs Mockist testing: different strokes for different folks

As we saw above, the key difference between the mock and the stub (the double). The focus of the test in the mock case is on the messages being sent to the collaborators. The focus of the test when using the double is on the the subject under test (SUT).

Mocks and stubs/doubles are tools that we can use under the larger umbrellas of two TDD philosophical styles: classic vs mockist styles.

Classic TDD

  • Classic TDDists like using doubles or real objects to test collaborators.
  • From personal experience, testing classicly is oftentimes the path of least resistance. There isn’t expectation setup and verification that mockist testing requires of you.
  • Classic TDD sometimes results in creating objects that reveal state — note how the BookUpdater needed to expose an updated? method.
  • Setting up the state of the world prior to your test may be complex, requiring setting up all the objects in your universe. This can be a huge pain (has anybody ever had this problem with overcomplicated Rails models with spidery associations? Raise your hands…). Classicists may argue that the root cause here is not paying attention to your model architecture, and having too many associations is an antipattern. Alternatively, classicists oftentimes generate test factories (e.g. Rails’ FactoryGirl gem) to manage test setup.
  • Tests tend to be treatable more like black boxes, testable under isolation (due to verifications on object state) and are more resistant to refactoring.

Mockist TDD

  • Mockist TDD utilizes mocks to verify behavior between objects and collaborators.
  • It can be argued to develop “purer” objects, that are mainly concerned with objects passing messages to each other. Fowler refers to these objects as preferring role-interfaces.
  • These tests are easier to set up, as they don’t require setting up the state of the world prior to test invocation.
  • Tests tend to be more coupled to implementation, and may be more difficult to refactor due to very specific requirements for message passing between collaborators.
  • Fowler brings up a point where being a mockist means that your objects prefer to Tell Don’t Ask. A nice side effect of TDA is you generally can avoid Demeter violations.

In conclusion

In coming from a classic TDD background, I’ve oftentimes viewed mockist testing with some suspicion, particularly around how much work is involved to bring them about. Over the years, I’ve warmed up to the usage of mockist testing, but have not been diligent enough at doing pure driving TDD with mocks. In reviewing Fowler’s comments, I’m intruiged at the possibilities of mockist TDD in affecting system design, particularly in their natural inclinations toward role interfaces. I look forward to trying pure mockist TDD in a future project.

Further reading:

Running Mocha tests with ES6/AMD modules

| Comments

In one of my personal projects (Chordmeister), I’ve been trying to upgrade the code to be written in ES6 modules and transpile down to AMD modules with Square’s very excellent es6-module-transpiler project.

Since I’ve already updated an Ember app of mine to try ES6, I figured it was high time to do it on another project.

Sorry Coffeescript, but I’m moving on.

First problem: Coffeescript seems indecisive with respect to ES6 support. In order to support import or export keywords, I had to wrap the statements in backticks, making the code look like this:

1
2
3
4
5
`import ClassifiedLine from "chordmeister/classified_line"`
class Parser
  # Implementation

`export default Parser`

Except this wasn’t being picked up by es6-module-transpiler, since Coffeescript wraps the entire declaration in a closure: I was finding myself having problems compiling from Coffeescript –> ES5 JS –> ES6 JS.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
define("chordmeister/parser",
  [],
  function() {
    "use strict";
    (function() {
      // Oops, I wasn't transpiled!
      import ClassifiedLine from 'chordmeister/classified_line';
      var Parser;
      Parser = (function() {
        // Implementation
      }
      )();
      // Oops, I wasn't transpiled!
      export default Parser;
      })()
  });

So the first call: ditch Coffeescript. Write this in pure ES6.

1
2
3
4
5
6
7
8
import ClassifiedLine from 'chordmeister/classified_line';
var Parser;

Parser = (function() {
  // implementation
})();

export default Parser;

Which transpiled nicely to:

1
2
3
4
5
6
7
8
9
10
11
define("chordmeister/parser",
  ["chordmeister/classified_line","exports"],
  function(__dependency1__, __exports__) {
    "use strict";
    var ClassifiedLine = __dependency1__["default"];
    var Parser;
    Parser = (function() {
      // Implementation
    })();
    __exports__["default"] = Parser;
    });

Next up: adding AMD support in Mocha

Okay, so we need to set up a few things to get Mocha playing well with RequireJS, the AMD loader.

Our plan of attach will be to leverage the generated AMD modules and load our tests up with them. We have the benefit of being able to specifically inject dependencies into our test suite.

The tricky parts will be:

Set up the Mocha index.html runner

Install mocha, require.js, and chai via bower, then plug them into the harness:

test/index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<!doctype html>
<html>
<head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <title>Mocha Spec Runner</title>
    <link rel="stylesheet" href="../bower_components/mocha/mocha.css">
</head>
<body>
    <div id="mocha"></div>
    <script src="../bower_components/mocha/mocha.js"></script>
    <script src="../bower_components/chai/chai.js"></script>
    <script data-main="test_helper" src="../bower_components/requirejs/require.js"></script>

</body>
</html>

Note the references to data-main="test_helper", which is require.js’s way of determining its entry point after it loads.

Set up a test runner.

test/test_runner.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// Configure and set up the tests.
require.config({
  baseUrl: "../build/"
})

// Load up the files to run against
var specs = [
  'chordmeister/chord_spec.js',
  'chordmeister/song_spec.js',
  'chordmeister/parser_spec.js',
  'chordmeister/classified_line_spec.js',
];

// Start up the specs.
require(specs, function(require) {
  mocha.setup('bdd');
  expect = chai.expect;
  // Why? Some async loading condition? Is there a callback I should be hooking into?
  setTimeout(function() {
    mocha.run();
  }, 100);
});

You’ll notice that I was having synchonicity issues between spec suite load and mocha.run(). Throwing everything back a few hundred ms seemed to have done the fix.

AMD gotchas

Pay attention to the default parameter that the module exports with. This is important to remember since native ES6 will allow you to directly import it with its native syntax:

1
import Parser from "chordmeister/parser"

But if you’re using RequireJS/AMD modules, you’ll need to explicitly call out the default namespace from the required module, so like so:

1
2
3
4
require(["chordmeister/parser"], function(parser) {
  Parser = parser.default;
  new Parser() // and do stuff.
});

Let me know if you have any questions!

Implementing DDD: Chapter 2

| Comments

Chapter 2: Domains, Subdomains, and Bounded Contexts

A Domain is what a business does and the surrounding context of how it does business. It is important to model out its supported Subdomains — that is, the smaller components of the business that collaborate in real, day-to-day operations of the business. The book describes these as Core Domains.

Finally, a Bounded Context is the physical manifestation of the solution space as software — models living in a real application. Its key feature in the context of DDD is as a linguistic barrier between different domains.

In an ideal world, each Subdomain maps directly to one Bounded Context. In the real world, this is less common since we tend to build things into monolithic systems. Still — many monolithic applications have several components that could in themselves be bounded contexts.

Side note: In Rails, one could think of Engines as a Bounded Context. But that might be a blog post for another time.

It is important to get these ideas and concepts down correctly because we need correct modeling of our systems to determine what they do.

Bounded Contexts and terms

It’s not usually realistic to get the entire organization agreeing on a universal linguistic definition for every term. Instead, DDD assumes that different terms have different meanings in different contexts.

The author then dives into the an example of a book, where a book means several different things to different people in different contexts. A book is touched upon by authors, graphic designers, editors, marketing folks. In each of these contexts, the features of a book mean different things at different times. It is impossible to develop an all-knowing Book model without disagreement between stakeholders. DDD, instead, acknowledges these differences and allows stakeholders to use linguistic terms from within their unique Bounded Contexts.

Bounded Contexts may include things like:

  • Object models
  • A database schema / persistence layer
  • A SOAP or REST API
  • A user interface

Bounded context antipatterns

You may be tempted to divide up a bounded context by architectural concerns, or because you want to assign smaller tasks to developers (resource allocation). Beware that this kind of work tends to fragment the language.

DDD operates on linquistic drivers. The unity, cohesion and domain-adherence of the bounded context should be the first priority in the design.

Assigning two teams to the same bounded context results in fragmentation of the bounded context.

Ideally: we strive to assign one team to work on one Bounded Context with one Ubiquitous Language at a time.

In Rails, what are the bounded contexts? It could be the top-level Rails application, or an engine, or a gem, that define the context boundaries._

A story…

The chapter then goes on to describe their fictional team designing through three iterations off their DDD strategy:

A system in which all domains live within the same bounded context. They see the folly of this and refactor with some tactical patterns, like creating services.

This is, however, missing the point. They realized they needed to listen to business and their domain experts to find out exactly where the right places were to segregate the contexts. The team discovers that the business has a desire to go in a new direction which allows them to segregate the domain in such a way that would enable future directions for the business.

How often are we as developers in conversation with our product owners and asking where the business wants to go in the future?

Conversations with the business reveal an intention to develop an add-on product to the core product. This implies the development of two subdomains. However, further investigation reveals that the shared overlap of certain domain models (like users, roles, and access management) cannot simply be identically shared between two systems, since their linguistic meanings in the two systems differ slightly. Instead, the developers use the linguistic problem to develop a third bounded context.

The developers separate their app into three bounded contexts built as three software systems.