On this weblog publish, I’ll share how prospects can monitor their Azure Kubernetes Service (AKS) clusters for time drifts utilizing a customized container picture, Azure managed Prometheus and Grafana.
Azure’s underlying infrastructure makes use of Microsoft-managed Stratum 1 time servers related to GPS-based atomic clocks to make sure a extremely correct reference time. Linux VMs in Azure can synchronize both with their Azure host by way of Precision Time Protocol (PTP) gadgets like /dev/ptp0, or with exterior NTP servers over the general public web. The Azure host, being bodily nearer and extra steady, offers a lower-latency and extra dependable time supply.
On Azure, Linux VMs use chrony, a Linux time synchronization service. It offers superior efficiency beneath various community circumstances and contains superior capabilities for dealing with drift and jitter. Terminology like “Final offset” (distinction between system and reference time), “Skew” (drift fee), and “Root dispersion” (uncertainty of the time measurement) assist quantify how properly a system’s clock is aligned.
On the time of writing this text, it isn’t potential to watch clock errors on Azure Kubernetes Service nodes immediately, since node photos cannot be custom-made and are managed by Azure.
Clients could ask “How will we show our AKS workloads are protecting time precisely?” To handle this, I’ve developed an answer that consists of a customized container picture working as a DaemonSet, which generates Prometheus metrics and might be visualized on Grafana dashboards, to repeatedly monitor time drift throughout Kubernetes nodes.
This answer deploys a containerized Prometheus exporter to each node within the Azure Kubernetes Service (AKS) cluster. It exposes a metric representing the node’s time drift, permitting Prometheus to scrape the information and Azure Managed Grafana to visualise it. The design emphasizes safety and ease: the container runs as a non-root person with minimal privileges, and it securely accesses the Chrony socket on the host to extract time synchronization metrics.
As we stroll by way of the answer, it is strongly recommended that you just observe together with code on GitHub.
The customized container picture is constructed round a Python script (chrony_exporter.py) that runs the chronyc monitoring command, parses its output, and calculates a ‘clock error’ worth. This worth is calculated within the following approach:
clock_error = |last_offset| + root_dispersion + (0.5 × root_delay)This script then exports the outcome by way of a Prometheus-compatible HTTP endpoint. The one dependency it requires is the prometheus_client library, outlined within the necessities.txt file
The container is designed to run as a non-root person. The entrypoint.sh script launches the Python exporter utilizing sudo, which is the one command that this person is allowed to run with elevated privileges. This ensures that whereas root is required to question chronyc, the remainder of the container operates with a strict least-privilege mannequin:
#!/bin/bash
echo "Executing as non-root person: $(whoami)"
sudo /app/venv/bin/python /app/chrony_exporter.pyBy limiting the sudoers file to a single command, this strategy permits protected execution of privileged operations with out exposing the container to pointless threat.
The deployment is outlined as a Kubernetes DaemonSet (chrony-ds.yaml), making certain one pod runs on every AKS node. The pod has the next hardening and configuration settings:
- Runs as non-root (runAsUser: 1001, runAsNonRoot: true)
- Learn-only root filesystem to reduce tampering threat and altering of scripts
- HostPath quantity mount for /run/chrony so it could possibly question the Chrony daemon on the node
- Prometheus annotations for automated metric scraping
Instance DaemonSet snippet:
securityContext:
runAsUser: 1001
runAsGroup: 1001
runAsNonRoot: true
containers:
- title: chrony-monitor
picture:
command: ["/bin/sh", "-c", "/app/entrypoint.sh"]
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- title: chrony-socket
mountPath: /run/chrony
volumes:
- title: chrony-socket
hostPath:
path: /run/chrony
kind: Listing This setup offers the container managed entry to the Chrony Unix socket on the host whereas stopping any broader filesystem entry.
The underlying AKS node’s (Linux VM) chrony.conf file is configured to sync time from the Azure host by way of the PTP machine (/dev/ptp0). This configuration is optimized for cloud environments and contains:
- refclock PHC /dev/ptp0 for direct PTP sync
- makestep 1.0 -1 to instantly right giant drifts on startup
This ensures that point metrics replicate extremely correct native synchronization, avoiding public NTP community variability.
With these layers mixed—safe container construct, restricted execution mannequin, and Kubernetes-native deployment—you achieve a strong but minimalistic time accuracy monitoring answer tailor-made for monetary and controlled environments.
- Clone the undertaking repository:
git clone https://github.com/Azure/chrony-tracker.git - Construct the Docker picture domestically:
docker construct --platform=linux/amd64 -t chrony-tracker:1.0 . - Tag the picture to your ACR:
docker tag chrony-tracker:1.0.azurecr.io/chrony-tracker:1.0 - Push the picture to ACR:
docker push.azurecr.io/chrony-tracker:1.0 - Replace the DaemonSet YAML (chrony-ds.yaml) to make use of your ACR picture:
picture:.azurecr.io/chrony-tracker:1.0 - Apply the DaemonSet:
kubectl apply -f chrony-ds.yaml - Apply the Prometheus scrape config (ConfigMap):
kubectl apply -f ama-metrics-prometheus-config-configmap.yaml - Delete the “ama-metrics-xxx” pods from the kube-system namespace to use the brand new configurations
After these steps, your AKS nodes shall be monitored for clock drift.
As soon as the DaemonSet and ConfigMap are deployed and metrics are being scraped by Managed Prometheus, you may visualize the chrony_clock_error_ms metric in Azure Managed Grafana by following these steps:
- Open the Azure Portal and navigate to your Azure Managed Grafana useful resource.
- Choose the Grafana workspace and navigate to the Endpoint by clicking on the URL beneath Overview
- From the left-hand menu, choose Metrics after which click on on + New metric exploration
- Enter the title of the metric “chrony_clock_error_ms” beneath Search metrics and click on Choose
- You must now be capable of view the metric
- To customise it and look at all sources, click on on the Open in explorer button
To reinforce the safety of the /metrics endpoint uncovered by every pod, you may allow fundamental authentication on the exporter. This requires configuring an HTTP server contained in the container with fundamental authentication. You’ll additionally must replace your Prometheus ConfigMap to incorporate authentication credentials .
For detailed steering on securing scrape targets, discuss with the Prometheus documentation on authentication and TLS settings.
As well as it is strongly recommended to make use of Non-public hyperlink for Kubernetes monitoring with Azure Monitor and Azure managed Prometheus
If you would like to discover this answer additional or combine it into your manufacturing workloads, the next assets present invaluable steering:
Creator
Dotan Paz
Sr. Cloud Options Architect, Microsoft
