Client: Internal Project
Project Duration: 8 months (ongoing)
Goal: Being able to easily generate configurable rates of load from various data sources to various targets while measuring performance of target system.
Tech: Java, Kafka, Cassandra, HTTP, RabbitMQ, Ranger
Each time you build a system, you want to know how it will behave on production. If possible, you would like to know that before you go to the production. Usually, it is possible, but requires additional efforts. One aspect of system is its performances. We decided to simplify performance testing as much as possible.
Idea is to easily target any part of the system. Since challenge occurred for Cassandra database. We first thought of testing Cassandra. Quickly after that we realised we can abstract Cassandra to any target system and have pluggable architecture with pluggable data source and pluggable target system.
Design of Berserker, and its power is in simplicity. It only has 4 components. Data source, rate generator, target and load generator. Load generator is central component, the ‘brain’ that connects all others. Data source is pluggable part where you can specify from which system to use data (Database, Kafka, CSV file, Ranger). Target is the system you want to test. It can be anything from HTTP REST endpoints, Kafka or Cassandra to RabbitMQ. Rate generator just specifies at which rate target system should be targeted, it is highly configurable. Modular design allows for easy extension, we usually can add non-existing data source or target within a week.
Since it is simple, we used it in several performance measuring tasks, even in some cases just to populate particular system with data when performances were not of interest for us. Berserker started small, but with its good pluggable design, we we able to quickly adapt it for several use cases we had. It is growing since then and supporting more and more target systems.