Pravesh Sudha

Monitoring Your Kubernetes Cluster: A Beginner’s Guide to Setting Up Prometheus and Grafana

Pravesh Sudha's photo
Pravesh Sudha
·

8 min read

Cover Image for Monitoring Your Kubernetes Cluster: A Beginner’s Guide to Setting Up Prometheus and Grafana

💡 Introduction

Welcome to the world of DevOps! Today, we’ll walk through setting up Prometheus and Grafana on a Kind Kubernetes cluster using Helm. But before we dive into the technical steps, let’s understand what these tools are and why they matter.

We’ll be using AWS to create an EC2 instance, which will act as our main machine for setting up the cluster. If you’re new to AWS, Kubernetes, Prometheus, or Grafana, don’t worry—I’ll break it down for you.

What Are These Tools?

  • AWS (Amazon Web Services): A cloud platform that provides services like computing, storage, and networking.

  • Kind (Kubernetes IN Docker): A tool that lets you run a lightweight Kubernetes cluster on your local machine using Docker. It’s similar to Minikube but uses container nodes instead of virtual machines.

  • Prometheus: A powerful monitoring tool that scrapes metrics from Kubernetes nodes, pods, and applications. You can also instrument your applications using Prometheus client libraries.

  • Grafana: A visualization tool that takes data from Prometheus (or other sources like Loki) and displays it in interactive dashboards.

Project Workflow

Here’s a step-by-step breakdown of what we’ll be doing:

✅ Create an EC2 instance (t2.medium)
✅ Install Docker and Kind on the instance
✅ Create a Kubernetes cluster using Kind
✅ Install and configure kubectl (Kubernetes command-line tool)
✅ Apply Kubernetes manifests for deployment
✅ Install Helm (a package manager for Kubernetes)
✅ Deploy the kube-prometheus-stack using Helm
✅ Access Prometheus on port 9090
✅ Access Grafana on port 3000

💡 Pre-requisites

Before we dive into setting up Prometheus and Grafana, let’s make sure you have the necessary requirements in place.

✨ What You’ll Need:

  • An AWS Account: Since we’ll be creating an EC2 instance to act as our main machine, you need access to AWS. If you don’t have an account yet, you can sign up at aws.amazon.com.

  • Basic Understanding of Kubernetes and Docker: While I’ll walk you through each step, having some familiarity with Kubernetes (k8s) and Docker will make things easier. If you’re new to these technologies, here’s a quick breakdown:

    • Docker: A tool for containerizing applications, allowing them to run consistently across different environments.

    • Kubernetes: An orchestration system that manages containerized applications across multiple nodes.

Once you have these prerequisites ready, we’re all set to move forward with the setup! 🚀

Step 1: Setting Up the EC2 Instance and Installing Dependencies

Now that we have our prerequisites ready, let’s kick off the project by setting up our AWS EC2 instance and installing the necessary tools.

🚀 Creating an EC2 Instance

  1. Log in to your AWS account and navigate to the EC2 Dashboard.

  2. Click on "Launch Instance" and configure the following:

    • Instance Name: Prom-Graf-project

    • OS Image: Select Ubuntu (recommended for compatibility).

    • Key Pair: Select an existing key pair or create a new one (this is needed for SSH access).

    • Storage Configuration: Allocate at least 15GB (to accommodate all the tools we’ll install).

  3. Click "Launch Instance" 🚀

🔗 Connecting to the Instance via SSH

Once your instance is running, use the SSH connect command to log in:

ssh -i /path/to/key-pair-name.pem instance-user-name@instance-public-dns-name

Make sure to replace /path/to/key-pair-name.pem with the actual path to your key pair and update the instance-user-name (usually ubuntu for Ubuntu images).

🛠️ Installing Docker on the Instance

Once inside your EC2 instance, run the following commands to update the system and install Docker:

sudo apt update
sudo apt install docker.io -y
sudo usermod -aG docker $USER && newgrp docker

This will install Docker and allow your current user to run Docker commands without sudo.

Step 2: Installing Kind and Kubectl

To simplify the setup, I have provided all necessary scripts in a GitHub repository:
🔗 GitHub Repo: K8s Kind Voting App

📥 Cloning the Repository

Once Docker is installed, clone the repo and navigate to the kind-cluster directory:

git clone https://github.com/Pravesh-Sudha/k8s-kind-voting-app.git
cd k8s-kind-voting-app/kind-cluster

🛠 Installing Kind

chmod +x install_kind.sh   # Make the script executable
./install_kind.sh          # Run the script to install Kind

🛠 Installing Kubectl and running the cluster

chmod +x install_kubectl.sh   # Make the script executable
./install_kubectl.sh          # Run the script to install kubectl

kind create cluster --config=config.yml      # Create the cluster

Step 3: Applying Kubernetes Manifests

Now that Kind and Kubectl are installed, let’s deploy our Kubernetes cluster configuration:

  1. Navigate to the k8s-specifications directory:

     cd ../k8s-specifications

  2. Apply all the Kubernetes manifests:

     kubectl apply -f .

This will set up our Kubernetes cluster with the necessary configurations.

Step 4: Installing Helm, Prometheus, and Grafana

Now that our application is running on the Kind Kubernetes cluster, it's time to install Helm, which will help us deploy Prometheus and Grafana in our cluster.

🛠 Installing Helm (Kubernetes Package Manager)

Helm is a powerful package manager for Kubernetes that simplifies the installation of complex applications. We will use Helm to install kube-prometheus-stack, which includes Prometheus, Grafana, and Alertmanager.

  1. Navigate to the commands directory (inside the cloned repo):

     cd ~/k8s-kind-voting-app/kind-cluster

  2. Run the following commands to install Helm:

     curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
 chmod 700 get_helm.sh
 ./get_helm.sh

    This script will download and install Helm 3 on your EC2 instance.

📥 Installing Prometheus & Grafana with Helm

Once Helm is installed, we can use it to install the kube-prometheus-stack. This stack includes Prometheus (for metric collection) and Grafana (for visualization).

  1. Add the required Helm repositories:

     helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
 helm repo add stable https://charts.helm.sh/stable
 helm repo update

  2. Create a separate namespace for monitoring:

     kubectl create namespace monitoring

  3. Install the kube-prometheus-stack using Helm:

     helm install kind-prometheus prometheus-community/kube-prometheus-stack --namespace monitoring \
 --set prometheus.service.nodePort=30000 \
 --set prometheus.service.type=NodePort \
 --set grafana.service.nodePort=31000 \
 --set grafana.service.type=NodePort \
 --set alertmanager.service.nodePort=32000 \
 --set alertmanager.service.type=NodePort \
 --set prometheus-node-exporter.service.nodePort=32001 \
 --set prometheus-node-exporter.service.type=NodePort

    This command deploys Prometheus, Grafana, Alertmanager, and other monitoring components into the monitoring namespace.

  4. Verify the deployment:

     kubectl get svc -n monitoring

    Ensure that all services are up and running. If some pods are still initializing, wait a few minutes and re-run the command.

🌍 Exposing Prometheus and Grafana Services

Once all pods are running, we need to expose the Prometheus and Grafana services so we can access them via our browser.

Run the following port-forwarding commands:

kubectl port-forward svc/kind-prometheus-kube-prome-prometheus -n monitoring 9090:9090 --address=0.0.0.0 &
kubectl port-forward svc/kind-prometheus-grafana -n monitoring 3000:80 --address=0.0.0.0 &

  • Prometheus will be available at http://<EC2-IP>:9090

  • Grafana will be available at http://<EC2-IP>:3000

Make sure to open port 9090, 5000, 3000 on your security groups

🔹 Use your EC2 instance's public IP to access the dashboards in your web browser.

Step 5: Running Prometheus Queries and Generating Traffic

Now that Prometheus is up and running, let’s verify that it’s correctly collecting Kubernetes cluster metrics.

🛠 Running Prometheus Queries

  1. Open Prometheus Dashboard in your browser:

     http://<EC2-IP>:9090

  2. Click on the "Graph" tab and enter the following queries one by one to check system metrics:

    ✅ CPU Usage (%)

     sum (rate (container_cpu_usage_seconds_total{namespace="default"}[1m])) / sum (machine_cpu_cores) * 100

    This query calculates the CPU usage percentage of all containers running in the default namespace.

    ✅ Memory Usage by Pod

     sum (container_memory_usage_bytes{namespace="default"}) by (pod)

    This query shows memory consumption per pod.

    ✅ Network Data Received by Pod

     sum(rate(container_network_receive_bytes_total{namespace="default"}[5m])) by (pod)

    This helps monitor incoming network traffic per pod.

    ✅ Network Data Transmitted by Pod

     sum(rate(container_network_transmit_bytes_total{namespace="default"}[5m])) by (pod)

    This checks outgoing network traffic from each pod.

  3. Since we haven’t started the application yet, most of the metrics will be static. To generate real-time data, let’s simulate some traffic.

🚀 Generating Traffic by Interacting with the Application

Now, we will expose the Vote application service and interact with it to see real-time metric updates.

  1. Expose the Vote application service:

     kubectl port-forward service/vote 5000:5000 --address=0.0.0.0 &

  2. Open your browser and navigate to:

     http://<EC2-IP>:5000

  3. Interact with the application by:

    • Casting votes

    • Refreshing the page

    • Submitting multiple requests

This will increase CPU, memory, and network usage, which will be reflected in Prometheus metrics.

  1. Go back to Prometheus and re-run the queries. You should now see dynamic changes in CPU, memory, and network metrics based on your interactions with the app!

Step 6: Visualizing Metrics in Grafana

Now that we have verified our metrics using Prometheus, it’s time to visualize them in Grafana for better insights.

🖥️ Accessing Grafana Dashboard

  1. Open your browser and navigate to:

     http://<EC2-IP>:3000

  2. Login to Grafana:

    • Username: admin

    • Password: prom-operator

(You can change the password later in the Grafana settings for better security.)

📊 Importing a Pre-Built Dashboard

Grafana allows us to import pre-configured dashboards for Prometheus metrics.

  1. Navigate to Dashboard Import:

    • Click on Dashboards → New

    • Click on Import Dashboard

  2. Enter Dashboard ID:

    • In the Import via Grafana.com section, enter the dashboard ID:

        15661

    • Set the Data Source as Prometheus

  3. Click Import, and your Grafana Dashboard will now display live metrics from Prometheus 🎉

💡 Conclusion

Monitoring is a crucial aspect of managing any Kubernetes cluster, and with Prometheus and Grafana, we have a powerful and flexible solution to track system health, application performance, and resource usage.

In this blog, we successfully:
✅ Deployed a Kind Kubernetes cluster on an AWS EC2 instance
✅ Installed and configured Prometheus for metric collection
✅ Used Grafana to visualize real-time data
✅ Explored basic PromQL queries to analyze CPU, memory, and network usage

With this setup, you now have the foundation to extend monitoring by adding custom alerts, dashboards, and integrations. You can experiment with more Grafana dashboards, enable Alertmanager to notify you of critical issues, or connect Loki for log monitoring.

By continuously monitoring your Kubernetes environment, you ensure better performance, reliability, and efficiencyfor your applications. Happy monitoring! 🚀

✨ For more informative blog, Follow me on Hashnode, X(Twitter) and LinkedIn.