R&D: Appliscale Lab in 2021

The Lab is an initiative that aims to evaluate the utility of software technologies to assess whether they are suitable for commercial applications for existing or future Appliscale customers. It also helps our engineers to stay up to date with the latest trends in the IT industry. The scope of interest encompasses language-specific software frameworks, architectural paradigms, cloud services, BigData solutions, database servers etc., in other words, anything that could be applied to solve a problem or implement a feature in a software project. The following post explains why Appliscale has founded the Lab and how our engineers and customers could benefit from it.

Online surveys, such as renowned Technology Radar, provide general insights into the adoption and maturity of software technology. The state of the technology reviews are addressed to a broad audience. They depict general trends and are independent of the project context. As valuable and insightful as these reports are, following the trends they describe without critical assessment could be counterproductive and potentially detrimental. See, for instance, the bandwagon effect. Conversely, we are interested in finding answers to more specific questions, in particular, which of the alternative software technologies should we choose for a given project. Such a decision seldom could be made without understanding the application domain and the unique project setting. The latter may include budget, time and other constraints imposed by the stakeholders.

Deciding which technologies to choose for a software project requires long term vision and expert judgment. For that reason, engineers involved in the Appliscale Lab focus on a detailed comparison between competing technologies to develop expertise that allows them to discriminate between alternative software products. The aim is to surface challenges and compromises that are consequences of adopting a particular technology.

Comparing various software technologies thoroughly would not be possible without hands-on work. The analysis may include the deployment of the solution in the cloud environment and verification of standard workflows the tool is designed for to support. Non-functional factors such as maturity, complexity, and costs associated with using the technology are evaluated too. Understandably, such an extensive investigation is a time investment. It might take up to a quarter for the Lab project to finish. The main reason for that long time frame is limited availability, as engineers who participate in the Lab on an ad-hoc and voluntary basis.

Having acquired expert knowledge and backed up with results uncovered during the Lab activities, engineers are positioned to make knowledgeable recommendations regarding the choice of technology for a software project. The Lab reduces risks on multiple frontiers. For instance, it proves that the technology can address unique projects' needs and the effective cost of using the technology is sustainable. For instance, the infrastructure, learning curve, licenses, performance bottlenecks, cost hotspots, ease of use, could be extrapolated based on the observations made during the Lab project.

Besides the business value of providing pragmatic technology recommendations supported by prior experience, the Lab initiative creates a platform for knowledge sharing and growth. Any engineers interested in staying up to date on recent developments in a given branch of technology could do so by joining the Lab project and declaring what their commitments would be. By working with their peers and meeting on a regular basis they sustain their momentum and motivation. Furthermore, from the organizational standpoint, the company strives to provide a comfortable environment, supplying project leads who drive the Lab activities and covering any additional costs (i.e., AWS infrastructure). Colleagues whose schedules are packed and preclude active and regular contributions are welcome to take the role of a mentor or a follower.

To conclude, the Lab initiative generates added value for Appliscale clients, our colleagues, and the company. We can offer clients competent software professionals who could provide up to date, knowledgable, and pragmatic advice regarding the choice of software tools and technologies. Our colleagues who participate in the Lab program are given the environment and opportunities to grow as professionals by expanding their expertise and skills. Finally, the company grows inhouse talent where engineers from the different projects could share best practices, mentor more junior colleagues, and proactively spend time awaiting their next project.

This post initiated a series of blog posts about the Appliscale Lab initiative. Subsequent posts will discuss results from the investigation that took under scrutiny BigData solutions.

The first lab (big data) project

Few projects in our company are data aggregation projects, that is transforming large quantities of data into valuable business information. To solve those challenges we use well-established technologies, proven by us and others in many projects. Like with everything in software development, big data is rapidly growing and so are the technologies around it.

Due to that, our first lab project has naturally been chosen according to one of our daily needs. We want to keep up to date with the latest developments in the big data industry, by not only reading about it but actually testing it in a real-life scenario.

We have chosen a subset of technologies that in our opinion could serve in our existing projects – as a matter of comparison to the existing implementations. After the selection phase, the technologies were divided among the teams that are focusing on the research part of the technology, deployment on the cloud, maintenance aspects of the technology, and finally running scenario-based tests on it.

The goal of the lab is to answer the question of which technology works best in provided circumstances and whether it makes sense to consider it for any of the existing or upcoming projects.

Technologies

The first lab project consists of 3 teams (9 engineers in total), each one exploring possibilities of different software solutions. Those solutions are targeted for data processing and analytics on a large scale but each of them solves those challenges in a different way. The difference lies in the high-level approach to a given problem, architecture, and way to interact with a given solution.

As a result, we have chosen the following technologies to try out:

• Apache Druid – open-source distributed data store that combines ideas from time series DBs, data warehouses, and search systems to provide a real-time analytics database with a wide range of functionalities.
• **Apache Flink **– framework and distributed processing engine for stateful computation over bounded and unbounded data streams.
• Trino – open-source distributed SQL query engine for running interactive analytic queries.

Amazon Web Services

We try to leverage AWS in a lot of our clients’ projects (take a look at some blog articles) hence we set up our lab solution on AWS to provide the necessary infrastructure and software components. As for a given moment, we could specify some of the services needed for a testing-ready stage.

• Virtual Private Cloud (VPC)
• Identity and Access Management (IAM)
• Simple Storage Service (S3)
• Relational Database Service (RDS)
• Elastic Kubernetes Service (EKS)
• Elastic MapReduce (EMR)

Additional tools

Except for services provided by Amazon the lab also includes tools for interacting and setting up infrastructure hosted inside AWS.

• Kubernetes tools – the orchestration of containers, defining resources, communicating with EKS cluster
• Terraform – defining and setting up infrastructure
• Ansible – automation of sysadmin tasks
• Apache Kafka – handling real-time data feeds

Given technology stack may change depending on needs and limitations met as lab projects continue and grow.

Technical statistics and metrics

The main concept of using technologies stated above is to leverage systems that are easy to maintain and monitor, which are also capable of working in a production environment. Additionally, proving solutions in that environment makes it possible to simulate situations happening in production environments in big data systems we have experience with.

As key metrics of each technology, we want to verify resource usage, cost generation, and ease of configuration and deployment. Those advantages however can be mutually exclusive leading to a point where we will make strategic decisions about which features are important and which can be neglected.

In order to provide an objective answer, we need a fair procedure of stating the pros and cons of a given software. Some of them, such as resource usage or cost generation can be measured by metrics provided by AWS or additional monitoring tools. Some however are very relative and depend on human opinion such as ease of use.

The lab will try to sum up each of those aspects and give an overall opinion about possible pros and cons for each of the main lab solutions. As for measurable characteristics, there are some concepts of how to gather data for analysis.

• **Resource usage **– using monitoring tools providing metrics for main characteristics such as CPU usage, memory usage, and IOPS. Except for gathering data considering physical stats, we would also want to include the usage of AWS components by each software. This can be achieved by using internal Amazon services such as CloudWatch to gather information about internal AWS usage.
• **Cost generation **– AWS provides a way to measure cost generation that clients will be charged for. For this purpose, we will properly tag each AWS resource generated for each main software. Later on, the Cost Explorer service will be used to generate estimated bills.
• **Ease of configuration **– each team is creating needed components inside Infrastructure as a Code template providing ease to read and analyze infrastructure state. This provides an opportunity to compare each environment in terms of components number and configuration procedures done during initialization.

To properly compare each software we will create testing procedures that will put the same pressure on each of them. Metrics gathered during this testing phase will then be analyzed and compared to define the characteristics of the given tool. There is also a need to analyze software during idle states periods on how costly they are when not used.

Concepts of measuring software performance can evolve as the lab project progresses.

Where are we at the moment?

At the time we publish this blog post, all out of our 3 teams taking part in this lab, have set up the infrastructure suitable for their technology (on AWS) and ingest data into Kafka. We are starting the test phases on the different data sets and are about to compare the results till the end of January.

Plans for 2022

To start is always the most difficult thing to do! Fortunately, we are close to a strong finish on our first project in the Appliscale lab and have already received multiple proposals from Appliscale’s engineers about the next editions. To share some with you – GraalVM, GraphQL, Mobile technologies are most probably going to be the topics we choose from. The planned start of the next lab is set for the beginning of Q2 2022. We are investigating the option of multiple labs running in parallel, though it depends on the teams’ sizes.

We hope the topics discussed here will match your interests and are valuable to you. If you would like to work on similar challenges as our colleagues, we are actively hiring. Feel free to check our career page.