Write your own micro BDD test framework in one day
Behavior-driven development (BDD) is an Agile software development methodology in which an application is documented and designed around the behavior a user expects to experience when interacting with it.
BDD approach is very useful especially in the area of software testing. Writing tests that verify the behavior of the system under test not the implementation is a good practice. Tests that are just verifying implementation are useful during the development process but after that, those usually never fail.
Problem
In my current project we were replacing one system with another. The new version was supposed to be backward compatible. The system was interacting with many external services. We needed acceptance tests, that were verifying behavior of both systems and their dependencies. We started with ScalaTests specs, however the number of scenarios to be covered was big and tests were complex. We needed to verify the behavior of a system by checking if all the views and dependent systems are eventually consistent within the given scenario. One week after writing one test I needed a while to recall what the scenario is and what the given and expected values are. The values were mixed within the code that was calling different services, reading from Kafka etc. It was impossible to ask QA to reason about those test cases, not to mention to write them.
The system under test with all dependencies was running in Docker containers. Tests were using both API and DB connections to verify its state.
Solution
The solution I found was a BDD test framework — it allows to easily write multiple readable scenarios which scenarios were supposed to be just the specification without all the technical noise.
My first shot was Cucumber, I had used it in one of my previous projects. We used it to write tests for very complex domain (US education system) and verify them with the team of analytics that we got the requirements right and covered all the cases. However I find Cucumber cumbersome, the file table-like format wasn’t friendly to me (nor for my IDE), there was lots of code to write and maintain that were just Cucumber “steps”. It was flexible and generic but at the same time required the knowledge about how its internals. I was looking for something more simple.
I decided to make a spike and try to write my own micro BDD framework. My teammates were sceptical about it, they thought that it will take lots of time and we all end up maintaining it. Actually it appeared to be easier than we all expected. I decided to used HOCON to write scenarios and Typesafe Config with Ficus to load them. Then all I needed were just a couple of cases classes and the Scala test that was executing the scenario. It all took me around a day. Sounds super simple, doesn’t it?
Show me the code
Actually, I can’t, show you the actual code, since that was closed source commercial project for the fintech domain. For the purpose of this blogpost I created example domain with a similar framework. The example domain is making pizza, since everybody loves pizza. Maybe the word framework is a bit too much here, since you cannot just reuse it in different project, so pattern is the better word. Below you can see the usual Scalatest scenario. There is lots of code that creates objects, call services, asserts the results, the actual given and the expected values are hidden between the lines. Sure, we could use “fixtures”, the inputs and expected values would be easier to spot however, the scenario itself won’t tell the story. With the growing number of scenarios the file with fixture will also grow and reading it would be difficult.
In the presented solution the HOCON scenario file is super easy to read by anybody, it’s our DSL. Thanks to Typesafe Config and Ficus, deserialization of the file to Scala classes is super easy. If you follow the convention, couple of imports and ConfigFactory.load method will do the job. The scenario layout and object doesn’t need to reflect 1-to-1 how the domain is modelled in the system/API, we may simplify it to make it easier to understand and reason about. We don’t deal with “ids” with are most of the time difficult to read by humans (e.g. UUIDs), I used names/aliases and mapped them to ids. We may define fields such as “scenario-name” or “description” if we need to.
In our project, thanks to presented approach, our QA was able to write tests (she didn’t knew Scala). When testers and users were reporting that values are calculated in the other way by the new system it was super easy to verify it. We had the confidence that the system is doing what it should and if an error pops up we should be able to create the scenario for it with minimum effort.
Core ingredients:
- HOCON scenario file, that describes only one scenario at a time.
Tip: Write you HOCON file first, before writing any other test related code. Try to tell the story and focus on the given and expected values that you want to check. - Scenario class, that loads (using Ficus) the values form the scenario file to the case classes. The cases classes are reflecting the scenario objects.
Tip: If you don’t want to specify the field in every scenario, give it a default value. - BDD Test class that converts Scenario with its case classes to the API/Domain objects, executes the flow and asserts the results.
Tip: Have in mind that if tests fails we need to get meaningful error. For that purpose I recommend using Scalatest “clues”.
Make the test code simple, don’t try to cover all the edge cases when the things are going bad (e.g. service timeouts, service is down, database write fails etc). For those rare cases write plain old tests, neither the values nor the narration in those cases matter.
In the presented test case all objects are created just for test, in the actual project it required to start multiple docker containers to run tests. It would be very expensive to do it for every scenario. There are couple of solutions to handle that:
- erase the DB — this might be difficult and you need to know exactly what to erase, sometimes you need to erase data from different databases
- use different entity for every test — if data are mostly entity related (e.g user related), you may create new entity for each scenario.
- assert the expected “delta”, not the state — if system should, for example, add the new entry to orders history, you may just tests if the last history entry is yours, not that the whole history contains only one entry.
Summary
Writing you own BDD micro framework is easier than you think. Have in mind that the presented solution is problem specific, it’s the implementation of one path/flow in the application. If you have more flows/paths to test you should create separate set of configs, scenario classes and BDD Test for them. You may share some code between them but try to make them as simple as possible. Making that code general will cause that you end up with full blown and complex BDD framework that you need to support and develop. I’m a fan of writing code with easy to delete principle, my solution could be easily rewritten or deleted. All we can lose is one day of developers work. I recommend this technic/pattern to apply to acceptance test or maybe to the part of applications that has complex business/calculation logic. I don’t recommend it to the unit/integration tests, it might be an overkill. Here you will find my repository with example project on GitHub.
