The National Institute for Health and Welfare (THL) studies, monitors and develops measures to promote the well-being and health of the population in Finland. They gather and produce information based on research and statistics. They provide expertise and solutions, which stakeholders can use in support of decision-making and other work.
In other words, THL collects a lot of data. Online forms, integrations with other systems, research, analysis etc. All of these systems need to be up and running and all of them create logs. There are access logs, error logs, standard application logs and logs from custom applications. Some logs are even stored within databases or in some legacy systems.
In the end it’s just down to having one centralised log management system.
Thanks to the growing popularity of Elastic it is not uncommon for organisations to make own, often single use-case, implementations. As the data and value grows the conditions change which sets higher requirement on the installation. It is not uncommon for us to start working with a client when they already are in production and are having challenges due to unmanaged scaling.
At THL it was different – they immediately felt the need for a central solution and started planning it as such. This allowed us to ensure a production ready and scalable solution right from the start.
The need for a centralized log management system came due to many reasons but most importantly due to wanting to:
- Get an overview of how systems are working
- Easily spot problems and do troubleshooting
- Work around security-related matters
The solution is a somewhat typical (modern) Elastic stack installation. Data is fetched using Beats. They ship logs to a centralized Logstash installation. Logstash handles the processing of the data into suitable and agreed format and stores all data in Elasticsearch. Kibana is used for visualizing, analyzing and managing the data. THL also uses Elastic commercial features (previously known as X-pack), among others, for securing the whole environment.
Here is a typical situation: something isn’t working and THL needs to understand what and why. Using Kibana dashboards, they can now very quickly see where the problem is and when an error occurred. The dashboards welcome possibilities to drill down to root cause or just inspect usage. And all this, without needing to login into systems to localise the correct logs to look at.
Log management and Kibana also gives clear statistics on when and how systems are being used. Basic, when it comes to log management in general but for THL is a huge improvement from before. They are going from the console, grepping and tailing logs to an intuitive and easy-to-use interface where relevant information is easily accessible (see images below).
The best thing with Elastic is that it scales in many ways. Not only can one add more data and keep getting sub-second response times, but one can also do so much more with the data - for instance predictive analysis. Even if THL is currently mostly using the log data for ensuring things run smoothly, possible insights they can gain from the visualizations of the data drives discussions about using the same system for log analytics, not just log management.