Cure your FOMO — what is Apache Ignite in 5 minutes
Every 2 weeks we discuss tech stack recommendations, software architecture insights, tools and open-source projects.
Apart from our daily work at SoftwareMill, we find time to explore a technology deeper, that we roughly knew about. The goal is to extend the horizontal bar known from the T-shaped skills concept. We meet on a bi-weekly basis and exchange basic information about a given topic — in this case it was Apache Ignite. Below you’ll find a kind-of meeting minutes.
What is Apache Ignite and what is its architecture?
According to Ignite’s home page:
Apache Ignite is a horizontally scalable, fault-tolerant distributed in-memory computing platform for building real-time applications that can process terabytes of data with in-memory speed.
Alright, so it’s a distributed system. Let’s stick with that for a bit. Apache Ignite forms a cluster and can operate in two modes: partitioned and replicated.PARTITIONED means data is spread across nodes by a key, like sharding in MongoDB, or partitioning in Kafka and Cassandra.
Partitioned caches are ideal when working with large data sets and updates are frequent.
REPLICATED is about storing all data on each node.
This mode is ideal for scenarios where cache reads are a lot more frequent than cache writes, and data sets are small.
Additionally to the PARTITIONED mode, you can setup the number of backups. This is the number of nodes backing up the primary node, but not serving any requests. Writes to backup copies can be done synchronously or asynchronously.
Another important feature is the possibility to persist data in an ACID-compliant distributed store:
Ignite native persistence is a distributed ACID and SQL-compliant disk store that transparently integrates with Ignite’s durable memory. Ignite persistence is optional and can be turned on and off. When turned off Ignite becomes a pure in-memory store.
Native here means it’s built into the project rather than an external solution Ignite would depend on.
What are the most common use-cases for Apache Ignite?
An in-memory data-grid
One or multiple underlying independent data stores are presented to the client as a unified cache layer. It allows for reads as well as propagates writes.
To run SQL or scan queries all data has to reside in the cluster in memory. This may require to allocate memory intensive machines. Also note that Ignite can read data from disk only if the data is stored in Ignite native persistence.
For example, to cope with fast writes we could use Apache Cassandra as a data store and to achieve fast random reads we could employ Apache Ignite as an in-memory data grid.
A distributed key-value store
Similar to the data grid, it’s a in-memory cache layer. It implements the JCache specification (JSR 107), enables computations on co-located data (located on the same node) and runs continuous queries notifying clients on data updates.
A good example would be using Ignite as an HTTP session store implementation.
A digital integration hub
Again similar to the data grid Ignite spans multiple data stores, that can be queried as it would be one unified shared data store.
Imagine you have a few RDBMS, like MySQL, PostgreSQL and Oracle DB and want to expose them together, as one queryable database. Watch out for table joins though — they will take place in-memory, hence all required data will be loaded into memory first which costs time and resources.
Also, all writes have to go through Apache Ignite to prevent inconsistencies between what is already in-memory and the underlying data stores.
For high-performance computing
Computations following the MapReduce paradigm can be run on co-located data. High performance is achieved since computations are done on data loaded in memory and in one node. Data shuffling over the network and disk I/O operations are not required.
A shared RDD for Apache Spark
Loading data in memory and making it available as an RDD to Spark improves the performance of Spark applications.
Where does Apache Ignite fit in the CAP theorem?
Apache Ignite, similar to other distributed systems like Apache Kafka or Cassandra, can be configured towards higher availability (AP) or to forfeit availability to ensure consistency (CP).
Although it’s not quite clear from the documentation, how configuration affects availability or consistency, I’ve found this statement on the mailing list:
When the cluster loses all copies of a partition the behavior is defined by PartitionLossPolicy. The current default is IGNORE which is indeed an AP rather than CP option. You can set it to READ_WRITE_SAFE or READ_ONLY_SAFE to get the CP behavior.
This blog post about Apache Ignite and the CAP theorem mentions inner mechanisms, which make Ignite to be considered rather a CP system by design.
For more details about CAP I can recommend a podcast with the father of the CAP theorem, Eric Brewer, on Software Engineering Radio. A distilled version is also available.
Additionally to CAP’s consistency, let’s look at consistency as found in ACID. There are three atomicity modes in which boundaries a cache operation can be performed.
Doing a TRANSACTIONAL operation …
… you can group multiple cache operations, on one or more keys, in to a single logical operation, known as a transaction. These operations will be executed without any other interleaved operations on the specified keys, and will either all succeed or all fail. There is no partial execution of the operations.
Full ACID transactions are supported via 2PC (the two-phase commit protocol), but in case where the transaction is performed only on one node, a more light-weight version — 1PC — is used.
An ATOMIC operation on the other hand executes multiple operations one at a time.
These atomicity modes are configured at the client level and can vary from call to call.
In the TRANSACTIONAL mode, two locking strategies are supported — pessimistic and optimistic locking. In pessimistic locking, timeout triggers detect deadlocks. It’s the responsibility of the client though, to always acquire locks in the same order as well as keep the number of acquired locks low and sort them by node to reduce network traffic.
Using optimistic locks, a version number is noted during reads and on writes it is compared with the current version. On version mismatch the transaction is rolled back.
Finally one interesting note: if you use distributed transactions across multiple Ignite nodes, during phase two one node may fail to commit and the transaction has to be rolled back manually. That’s the joy of 2PC.
Additionally inconsistency between the underlying data stores can happen, when one data store finished its commit and made it available to clients before a second data store finished its commit. If that second commit fails, the first one won’t be rolled back automatically.
Will it b̶l̶e̶n̶d̶ split?
A split-brain scenario happens, when a network partition occurs. You end up with two clusters and have to decide what to do. Accept requests and eventually restore consistency (resolve conflicts) across nodes? Or disable one of the clusters?
Apache Ignite has a few solutions to this problem.
One is based on TCP/IP Discovery and Baseline Topology, where in case of a partition, the two clusters won’t form one cluster again, when the partition is fixed. This will prevent data being overwritten. It’s however your responsibility to tear down the excluded nodes and start them again, so they can catch up with the survived nodes. Also in case where two disjoint clusters get updates for the same key, these colliding changes need to be resolved manually.
Another split-brain resolving strategy is based on Zookeeper Discovery. Every node is aware of the cluster topology. If a node figures out, that it cannot contact another node, it complains to Zookeeper. Zookeeper has a global overview of the topology and also data on who sees who and can do an informed decision on how to resolve a network partition by telling which nodes remain in the cluster and which are excluded.
There is also a plugin available on GitHub as well as a commercial split-brain resolver by GridGain.
Streaming capabilities
Streaming is about moving data into Ignite, performing transformations and running continuous queries against that data.
Streaming is performed in batches. They are grouped per node. This means, that a batch for one node can be sent earlier than for another, although some messages arrived later but completed the batch, where as the first one was still waiting for more messages. In other words — ordering is not preserved.
Since batches can fail and are redelivered, streaming provides an at-least once delivering guarantee. Therefore you have to choose between overwriting or ignoring values for keys, that already exist.
Also a second batch does not wait until the first one has been delivered successfully. Therefore ordering within a particular node is also not preserved.
Ingress and egress with Kafka is available via Kafka connectors.
What are Apache Ignite alternatives?
Similar to Apache Ignite, Hazelcast is also advertised as an in-memory computing platform. Likewise it integrates with SQL databases, but does not support transactions.
Redisson, a java client for Redis also features an in-memory data grid.
When it comes to analytics, similar functionality is available in Druid. Joins although supported are limited to rather smaller look-up tables.
Last but not least Infinispan is advertised as a in-memory data store with two of the primary use cases such as a cache or data grid.
All in all, Apache Ignite seems to shine, when complex SQL operations needs to be performed. Otherwise, an alternative tool may work better.
Final notes
I’ve found a quote in Martin Kleppmann’s book Designing Data‑Intensive Applications worth mentioning:
Counterintuitively, the performance advantage of in-memory databases is not due to the fact that they don’t need to read from disk. Even a disk-based storage engine may never need to read from disk if you have enough memory, because the operating system caches recently used disk blocks in memory anyway. Rather, they can be faster because they can avoid the overheads of encoding in-memory data structures in a form that can be written to disk.
