This month at SoftwareMill we’ve learned (January ‘19)
Marcin, Adam and Łukasz have shared their discoveries from January:
- tips for improvements in HOCON & Lightbend Config library reusability,
- how to sort clustering keys in Cassandra,
- how to change default memory settings for Docker on macOS,
- an inspirational talk on clojure by Rich Hickey,
- how to configure RabbitMQ properly
Marcin Kubala — Guardian of Type Safety
This month I learned
I’ve been working with HOCON & Lightbend Config library from the moment I learnt Scala, quite a long time ago. I felt that sometimes I have too many repetitions in my configs and there must be a way to improve its reusability, so I started digging. Let’s take a look at what I have found so far.
“Prototyping”
I’m sure you already know how does the include statement work — usually, you put it at the beginning of the config file and benefit from other configs being merged into your document:
Includes can be nested, in this case they attach (and merge) content of another file under the surrounding config node (current config object):
Include allows you to extract a common “prototype” for your config objects into separate files.
“Inheritance”
A field in HOCON file can be defined multiple times or have more than one value. In both cases all the values will be concatenated:
All these snippets produce the same output.
Let’s focus on the 3rd example’s syntax which is a tricky one, compared to more YAML or JSON familiar constructs.
Since HOCON supports substitutions we might “extend” another config object:
Values concatenation applies for arrays and primitive values as well, although be careful when working with arrays:
If your intention is to create a two-dimensional array, you should always use comma separators to avoid accidental flattens caused by concatenation.
The rule of thumb is that whenever you need to share the piece of config among different files, you should use include statements.
If you want to provide deeper config objects’ “inheritance” (see l2child on one of the above Gists), then substitution and objects concatenation will be a way to go.
Of course, you can mix both approaches together and keep a common parent in a separate file.
This tool helped me the most
While writing about HOCON, it’s hard to not mention the HOCON plugin for IntelliJ IDEA (bundled into Scala plugin since 2019.1).
This plugin, among others, provides support for:
- jumping to included files, substituted value definitions, and even code (if the value contains an FQCN (Fully Qualified Class Name) you can go directly to its source file).
- copying references — you can put a full config path for arbitrary value in your clipboard simply by right-clicking on its key and choosing “Copy Reference” action.
- code completion.
Adam Smolarek — Senior Software Engineer
This month I learned
Cassandra clustering key
I learned that in Cassandra you can not sort by column that is not part of a cluster key. Clustering key in Cassandra is the second part of compound primary key.
For example, if we have a primary key created like this:
PRIMARY KEY ((col_a, col_b), col_c, col_d)
Partition key is constructed from coll_a and coll_b and is used to locate partition that has a record you are asking for. Clustering key is made from col_c and col_d. So you can sort by column col_c and col_d.
Default memory for Docker on macOS
I also found that default RAM for docker containers in Docker desktop is set to 2GB which is not enough if you are running Kafka with ZooKeepr and Cassandra.
To change that you can go to: Preferences > Advanced and set Memory with slider.
This I found really inspiring
I watched talk from Rich Hickey “Maybe Not” about different approach to modeling type checks in clojure. Not only different to what clojure has now with spec, but different to what other languages have. Talk is interesting even for “not clojure dev” and “statically type all the thing dev” ;)
“Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future”
I also listened to the audiobook: “Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future” which is a biography of Elon Musk approved by him. This book is in part a critique of incremental improvement in startup world where priority is high IPO for another messaging app. Book also shows how risky it is to do something bigger than messaging app, namely space rocket and electric car. It is unbelievable how many times SpaceX and Tesla were on the verge of bankruptcy and how this affected Elon’s private life.
Łukasz Lenart — Creative Software Engineer
This month I learned
In a project I’m involved in, we started using RabbitMQ to support internal communication between different microservices. At the beginning we have been using rabbitmqadmin to create all the queues at startup — this isn’t a production ready solution. But there is a way to configure RabbitMQ to fully control who changed what.
As I mentioned, we started with a simple case, right now we are extending the usage of RabbitMQ and adding support for collecting all information and events happening in different parts of our system. We decided to use an exchange of type fanout and a few dedicated queues that receive events.
The whole configuration was done via a batch file using rabbitmqctl and rabbitmqadmin. And yes, we are using Docker to configure all our services. Below is our current Dockerfile before I have introduced configuration via files:
There is a one huge drawback when you want to use those tools — a RabbitMQ server must be up & running, without this you won’t be able to connect to the server and configure anything.
To overcome this issue we simple used asleep command to wait a bit when RabbitMQ will be ready.
As you see this is a good idea if you want to play with RabbitMQ, but a totally bad idea when you want to run RabbitMQ in production.
Configuration files
RabbitMQ supports few options if you want to configure it via files. First there is a rabbitmq.conf file located in /etc/rabbitmq — it uses a new format available since RabbitMQ 3.7. If you want to use an old format, name your file rabbitmq.config and you can use an Erlang term configuration format instead.
A good idea is to start with an example config file published on Github. You can read throughout it and uncomment and change options you want.
If you need to configure some advanced feature, it’s a good idea to use another file — advanced.conf.
Also plugins allow you to use config files, e.g. if you want to use the Management Plugin (a Rest API along with a Web UI) and configure it via a file, you can define proper options in rabbitmq.conf, as specified on the plugin page.
You can also use the Management Plugin to configure queues, exchanges and policies based on a JSON file. This is the super easy option if you want to have a version controlled configuration. All you have to do is to tell the plugin to load the configuration from the JSON file:
management.load_definitions = /etc/rabbitmq/definitions.json
Add the above line to your rabbitmq.conf file as specified on the plugin page and you are done.
You will probably ask how to prepare the definitions.json file? The easies, and the most convenient way is to use the Management Plugin UI interface to setup everything you need and export the whole configuration into the file.
On the Overview page of the UI, at the bottom, you will find a button Download broker definitions that will export the whole configuration of the RabbitMQ into a JSON file. Just use the file with new options in your Dockerfile as presented below:
TL;DR: Using cli tools is good if you want to setup RabbitMQ fast and your setup isn’t too complicated. If there are more options and you want to control what and when was changed by who, using configuration files is a far better solution.
And what have you learned in January? Let us know! :)
BTW, we are always looking for outstanding professionals to join our team! Check out backend and frontend open positions!
Questions? Ask us anything about remote work, how does the cooperation with us look like, what projects do we have, or anything else - on the dedicated Slack channel 💡
