Monitoring Your Microservices on AWS with Terraform and Grafana – Safety – Grape Up
Welcome again! That is the final a part of this sequence. If that is the primary you’ve ever seen, we advocate studying earlier components first. So, there’s an utility you’ve created. It consists of a number of microservices speaking to one another and a few monitoring instruments. Now, let’s encrypt this communication.
Securing backend companies
Our Java backend companies want to have the ability to expose secured endpoints and speak with one another utilizing HTTPS. For this, we are going to use keystores and truststores. Keystores are used for storing non-public keys and certificates with public keys. If our server wants to make use of HTTPS, throughout SSL handshake, the server seems to be for a personal key within the keystore and presents it’s corresponding public key and certificates to the shopper. Then the shopper seems to be for the related certificates in his truststore. If the certificates or Certificates Authorities will not be current in truststore, we’ll get an SSLHandshakeException.
For our functions we used self-signed certificates as a result of we wished to maintain safe, encrypted communication between companies and area verification(confirmed by trusted CA) was not vital on this case. Nonetheless, you should use any sort of SSL certificates for that function – one other method to go could be to make use of a free trusted certificates from https://letsencrypt.org/. It’s also possible to merely purchase a sound certificates from CA of your alternative, for the area title you personal.
We generated our keys and self-signed certificates utilizing keytool, which is a good instrument distributed with the Java SDK. With the intention to keep away from guide steps(when potential) we’re producing it utilizing terraform with local-exec provisioned:
keytool -genkey -noprompt -dname "CN=*.${var.certificate_domain}, OU=Organizational Unit, O=Group, L=Location, S=State, C=Nation" -keystore keystore.jks -alias selfsigned_foo -keyalg RSA -keysize 2048 -validity 3950 -storepass changeit
We use a so-called wildcard certificates, legitimate for all our subdomains. By subdomains, we imply our non-public subdomains created by service discovery service, utilized in service-to-service communication in non-public subnets. With wildcard we will use the identical certificates for each microservice(microservices in the long run impact can have totally different subdomains, one subdomain for every one).
With keytool we will export each key and certificates in textual content format when wanted.
keytool -export -alias selfsigned_foo -keystore keystore.jks -rfc -file cert.pem -storepass changeit
The subsequent step is to configure companies to truly use the certificates. Since on localhost we don’t wish to complicate issues with SSL, we determined to run secured connections solely within the cloud. Subsequently we created a brand new Spring profile devoted to all cloud-related configurations. One level of configuration is SSL(outlined at utility properties stage, in yml file):
server:
port: 443
ssl:
key-store: file:keystore.jks
key-store-password: changeit
key-store-type: jks
key-alias: selfsigned_foo
Utilizing utility properties for this function is certainly the best approach. Referenced keystore is added as a separate file to Docker picture.
The subsequent step is to add certificates to truststore of each service that may be known as within the cloud(with -cacerts flag we’re merely importing certificates to default Java truststore for container):
keytool -importcert -noprompt -alias selfsigned_foo -file cert.pem -cacerts -storepass changeit
We do it on Dockerfile stage, for each Java microservice working in a separate Docker container.
These 2 operations permit us to reveal our endpoints on HTTPS and to simply accept responses as trusted. We repeated the identical steps for the config server and each different microservice that we’ve in our system. Once more, a self-signed certificates was sufficient for our wants and allowed us to automate the method utilizing terraform, however this needs to be adjusted to the objective.
As soon as we’ve secured companies – what about load balancers? In addition they want to offer HTTPS endpoints. For load balancers, we created SSL certificates in AWS Certificates Supervisor – one certificates per subdomain, that we use for every one in all them.
useful resource "aws_acm_certificate" "certificates" {
domain_name = "${native.domain_name}.${var.environment_zone_name}"
validation_method = "DNS"
}
We connect a certificates (utilizing its ARN) to load the balancer listener:
useful resource "aws_lb_listener" "foo_https_listener" {
load_balancer_arn = aws_lb.foo_ecs_alb.arn
port = "443"
protocol = "HTTPS"
certificate_arn = var.certificate_arn
ssl_policy = "ELBSecurityPolicy-TLS-1-2-2017-01"
default_action {
sort = "ahead"
target_group_arn = aws_lb_target_group.foo_target_group.arn
}
}
We even have to regulate the uncovered port for the container, goal group, and safety teams(each for the load balancer and for ECS process inbound site visitors):
(...)
"portMappings": [
{
"containerPort": 443,
"hostPort": 443
}
]
(...)
(...)
load_balancer {
target_group_arn = aws_lb_target_group.foo_target_group.arn
container_name = "foo"
container_port = 443
}
(...)
useful resource "aws_lb_target_group" "foo_target_group" {
title = "foo"
port = 443
protocol = "HTTPS"
target_type = "ip"
vpc_id = var.vpc_id
health_check {
port = 443
protocol = "HTTPS"
path = "/actuator/well being"
matcher = "200"
}
depends_on = [
aws_lb.foo_ecs_alb
]
}
useful resource "aws_security_group" "foo_lb_to_ecs" {
title = "allow_lb_inbound_foo"
description = "Enable inbound Load Balancer calls"
vpc_id = var.vpc_id
ingress {
from_port = 443
protocol = "tcp"
to_port = 443
security_groups = [aws_security_group.foo_alb_sg.id]
}
}
useful resource "aws_security_group" "alb_sg" {
title = "alb-sg"
description = "Inet to ALB"
vpc_id = var.vpc_id
ingress {
protocol = "tcp"
from_port = 443
to_port = 443
cidr_blocks = [
"0.0.0.0/0"
]
}
egress {
protocol = "-1"
from_port = 0
to_port = 0
cidr_blocks = [
"0.0.0.0/0"
]
}
}
And that’s sufficient to allow full HTTPS communication on Java half – all microservices(together with config server) are speaking to one another utilizing secured connections, and any exterior request forwarded utilizing load balancer can also be acquired with HTTPS protocol.
Securing monitoring
As we want each service in our system to be secured, we have to present further configuration to monitoring companies.
For Grafana we specify protocol and certificates path in grafana.ini file, used whereas working Grafana from the official Docker picture. We will additionally allow HSTS response header:
(...)
protocol = https
(...)
cert_file = /and so on/grafana/grafana.crt
cert_key = /and so on/grafana/grafana.key
(...)
cookie_secure = true
(...)
strict_transport_security = true
(...)
For Prometheus and Loki it’s a little bit extra difficult – they each don’t present HTTPS assist out of the field. So to reveal them on HTTPS we used nginx(as reverse proxy). Nginx is run in the identical container, exposes HTTPS on particular port and redirects site visitors to explicit service working on HTTP on localhost(within the container). Subsequently communication between containers is all the time secured.
Pattern nginx configuration for Prometheus:
server {
pay attention 9090 ssl;
ssl_certificate /and so on/nginx/ssl/server.crt;
ssl_certificate_key /and so on/nginx/ssl/server.key;
server_name prometheus_ssl;
location / {
proxy_pass http://localhost:9091;
}
}
It exposes 9090 port for HTTPS and redirects all site visitors to Prometheus occasion working in a container on port 9091.
An analogous operation was configured for Loki:
server {
pay attention 3100 ssl;
ssl_certificate /and so on/nginx/ssl/server.crt;
ssl_certificate_key /and so on/nginx/ssl/server.key;
server_name loki_ssl;
location / {
proxy_pass http://localhost:3101;
}
}
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
Dockerfile’s needed to be adjusted as effectively:
FROM grafana/loki
COPY loki.yml .
COPY entrypoint.sh .
COPY loki.crt /and so on/nginx/ssl/server.crt
COPY loki.key /and so on/nginx/ssl/server.key
COPY lokihttps.conf /and so on/nginx/conf.d/default.conf
USER root
RUN apk add nginx --no-cache &&
sed -i 's/consumer nginx;//g' /and so on/nginx/nginx.conf &&
chown -R loki:loki /var/lib/nginx &&
chown -R loki:loki /var/lib/nginx/logs &&
chown -R loki:loki /var/tmp/nginx &&
chown -R loki:loki /var/log/nginx &&
chmod +x entrypoint.sh &&
mkdir -p /run/nginx &&
chown -R loki:loki /run/nginx
USER loki
ENTRYPOINT ./entrypoint.sh
EXPOSE 3100
FROM alpine
COPY prometheus.yml .
COPY entrypoint.sh .
COPY prometheus.crt /and so on/nginx/ssl/server.crt
COPY prometheus.key /and so on/nginx/ssl/server.key
COPY prometheushttps.conf /and so on/nginx/http.d/default.conf
RUN apk add prometheus nginx --no-cache &&
sed -i 's/consumer nginx;//g' /and so on/nginx/nginx.conf &&
chmod +x entrypoint.sh &&
mkdir -p /run/nginx
ENTRYPOINT ./entrypoint.sh
EXPOSE 9090
As you may see it received just a little bit extra difficult – we’ve to repeat recordsdata with keys and certificates into the Docker photos, in addition to nginx configuration. With the intention to add nginx to Loki picture, we needed to swap to the basis consumer, to have the ability to set up the applying, after which do another methods to resolve errors associated to working nginx on this explicit Linux Alpine distribution.
Accepting HTTPS connections
That’s it, for exposing our monitoring infra on HTTPS – however learn how to make it settle for different HTTPS domains? For Loki there isn’t a downside – Loki doesn’t name any service, the companies name Loki – so long as they’ve a correct certificates of their truststores – we’re good to go. What about Prometheus? This one is just a little bit extra difficult – Prometheus calls companies for his or her metrics, due to this fact it has to have the ability to name HTTPS addresses and settle for the self-signed certificates.
The right way to do it? Merely disable validation of the server certificates in Prometheus configuration for polled companies:
scrape_configs:
- job_name: 'cloud-config-server'
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
scheme: https
tls_config:
insecure_skip_verify: true
dns_sd_configs:
- names:
- '$cloud_config_server_url'
sort: 'A'
port: 443
- job_name: 'foo'
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
scheme: https
tls_config:
insecure_skip_verify: true
dns_sd_configs:
- names:
- '$foo_url
sort: 'A'
port: 443
(...)
This configuration must be repeated for all jobs outlined within the prometheus.yml file.
For Grafana we have to disable SSL verification on information sources stage:
instance: loki_datasource.yml:
apiVersion: 1
datasources:
- title: Loki
sort: loki
entry: proxy
url: http://$loki_url:3100
jsonData:
maxLines: 1000
apiVersion: 1
datasources:
- title: Loki
sort: loki
entry: proxy
url: http://$loki_url:3100
jsonData:
maxLines: 1000
tlsSkipVerify: true
The identical should be achieved for every outlined datasource.
And that’s it – each part in our system exposes its endpoints utilizing HTTPS protocol, and is ready to settle for calls from our subdomains, secured with a self-signed certificates.
Summing up the sequence
After performing all of the steps you need to be having a working utility, designed and built-in microservice structure, secured, monitored, and efficiently deployed within the AWS cloud. The chosen monitoring stack permits us to flexibly alter Grafana dashboards with related log streams to our wants.
We clearly skipped some frequent, extra fundamental components for readability functions. We assume that the reader has a minimum of a fundamental understanding of the net functions improvement course of and familiarity with AWS and terraform. Hopefully, we helped you with resolving a few of your issues. We thanks very a lot for those who’ve received this far with the lecture 🙂
