In this article we are going to learn how to configure ingress controller in kubernetes so that we could run multi domain or subdomain application within same kubernetes cluster.
In this demo we are going to run application with domain “nginx.example.com” and other two subdomain called
tea.myshop.com and coffee.myshop.com in kubernetes using Nginx ingress controller. Please note that, these domains and subdomains are local domains.
To show case this demo about “How to configure ingress controller in kubernetes” I am using lxc containers install on Ubuntu(Bare-metal installation). Basically I have four containers/vms running on the lxc host as below:
root@vbhost:~# lxc list +-----------+---------+------------------------+-----------------------------------------------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +-----------+---------+------------------------+-----------------------------------------------+------------+-----------+ | haproxy | RUNNING | 10.253.121.146 (eth0) | fd42:38af:bc0d:704d:216:3eff:fefa:cb4c (eth0) | PERSISTENT | 0 | +-----------+---------+------------------------+-----------------------------------------------+------------+-----------+ | kmaster | RUNNING | 172.17.0.1 (docker0) | fd42:38af:bc0d:704d:216:3eff:fe31:7a49 (eth0) | PERSISTENT | 0 | | | | 10.253.121.39 (eth0) | | | | | | | 10.244.0.0 (flannel.1) | | | | +-----------+---------+------------------------+-----------------------------------------------+------------+-----------+ | kworker01 | RUNNING | 172.17.0.1 (docker0) | fd42:38af:bc0d:704d:216:3eff:fed4:8226 (eth0) | PERSISTENT | 0 | | | | 10.253.121.32 (eth0) | | | | | | | 10.244.1.1 (cni0) | | | | | | | 10.244.1.0 (flannel.1) | | | | +-----------+---------+------------------------+-----------------------------------------------+------------+-----------+ | kworker02 | RUNNING | 172.17.0.1 (docker0) | fd42:38af:bc0d:704d:216:3eff:fe9f:82c0 (eth0) | PERSISTENT | 0 | | | | 10.253.121.89 (eth0) | | | | | | | 10.244.2.1 (cni0) | | | | | | | 10.244.2.0 (flannel.1) | | | | +-----------+---------+------------------------+-----------------------------------------------+------------+-----------+
- So all these containers are running centos. Three of them are being used for kubernetes cluster. One will be using for haproxy which we will be utilizing as a loadbalancer to load balance the request to one of the two kubernetes worker node.
To learn more about lxc containers follow this link.
Steps:
-
Deploy kubernetes cluster:
Once you deployed three lxc containers configure one of them as kubernetes master and other two as kubernetes worker node. To cut short the configuration journey you can follow this link.
-
Deploy ha-proxy container with centos operating system:
We will be deploying one more lxc container with centos and will be configuring haproxy as per below steps:
- login to lxc container
root@vbhost:~# lxc exec haproxy bash [root@haproxy ~]#
Now install the haproxy package in the container
# yum install haproxy
Once you have installed haproxy we need change the configuration so that it will load balance the traffic between the kubernetes worker node. Open the /etc/haproxy/haproxy.cfg file and replace it with below contents:
[root@haproxy ~]# cat /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend http_front
bind *:80
stats uri /haproxy?stats
default_backend http_back
backend http_back
balance roundrobin
server kube 10.253.121.32:80
server kube 10.253.121.89:80
[root@haproxy ~]#
Make sure you change the bottom two ip address with ip address of the kubernetes worker nodes so that it could be load balanced. Here are the ip adress of my worker node which we have entered in the above haproxy configuration file. Also we are using http port 80 for our configuration.
# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kmaster Ready master 31d v1.17.1 10.253.121.39 <none> CentOS Linux 7 (Core) 4.15.0-106-generic docker://19.3.5 kworker01 Ready <none> 31d v1.17.1 10.253.121.32 <none> CentOS Linux 7 (Core) 4.15.0-106-generic docker://19.3.5 kworker02 Ready <none> 31d v1.17.1 10.253.121.89 <none> CentOS Linux 7 (Core) 4.15.0-106-generic docker://19.3.5 #
Now restart and enable the haproxy service.
# systemctl enable haproxy # systemctl restart haproxy
We are done with haproxy configuration. Now logout from it.
-
Install the Nginx-controller on kubernetes master node:
Clone the git repo and change directory to “kubernetes-ingress/deployments/”
# git clone https://github.com/nginxinc/kubernetes-ingress.git # cd kubernetes-ingress/deployments/
Create the namespace and service account.
# kubectl apply -f common/ns-and-sa.yaml
Apply the role and cluster role binding
# kubectl apply -f rbac/rbac.yaml
Make the secret
# kubectl apply -f common/default-server-secret.yaml
apply the configmap required.
# kubectl apply -f common/nginx-config.yaml
Now deploy the Ingress controller as daemon-set
# kubectl apply -f daemon-set/nginx-ingress.yaml
Now if you check the namespace and the resources within “nginx-ingress” namespace you will find similar resources being created.
# kubectl get ns NAME STATUS AGE cattle-system Active 29d default Active 31d efk Active 29d kube-node-lease Active 31d kube-public Active 31d kube-system Active 31d nginx-ingress Active 10s
# kubectl get all -n nginx-ingress NAME READY STATUS RESTARTS AGE pod/nginx-ingress-2rq6s 0/1 ContainerCreating 0 14s pod/nginx-ingress-65vnd 0/1 ContainerCreating 0 14s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/nginx-ingress 2 2 0 2 0 <none> 14s
Once they are ready we are going to deploy services.
- Deploy the Nginx service with NodePort type listening on port 80 which will run sample Nginx web-server with its default index.html file. Here is manifest file I am using for it.
# cat nginx-deploy-main.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: nginx
name: nginx-deploy-main
spec:
replicas: 1
selector:
matchLabels:
run: nginx-main
template:
metadata:
labels:
run: nginx-main
spec:
containers:
- image: nginx
name: nginx
---
apiVersion: v1
kind: Service
metadata:
name: nginx-deploy-main
spec:
type: NodePort
selector:
run: nginx-main
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 80
Deploy the Nginx service
# kubectl create -f nginx-deploy-main.yaml deployment.apps/nginx-deploy-main created service/nginx-deploy-main created
-
Deploy the tea service
Now lets deploy the another service which will be of the type NodePort with port 80. For this service I am using my own created docker image called “manmohanmirkar/mytea_image” which will simply display the message “This is Tea Shop” in the browser. This service will be get called once you try to access url “tea.myshop.com” . Sample manifest file is as below:
# cat tea.yml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: tea
name: tea-deploy
spec:
replicas: 2
selector:
matchLabels:
app: tea
template:
metadata:
labels:
app: tea
spec:
containers:
- name: tea
image: manmohanmirkar/mytea_image:latest
---
apiVersion: v1
kind: Service
metadata:
name: tea-deploy
spec:
type: NodePort
selector:
app: tea
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 80
Apply the tea service:
# kubectl create -f tea.yml deployment.apps/tea-deploy created service/tea-deploy created
-
Configure the coffee service
Deploy one more service which will be of the type NodePort with port 80. For this service I am using my own created docker image called “manmohanmirkar/mycoffee_image” which will simply display the message “This is Coffee Shop” in the browser. This service will be get called once you try to access url “coffee.myshop.com” . Sample manifest file is as below:
# cat coffe.yml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: coffee
name: coffee-deploy
spec:
replicas: 2
selector:
matchLabels:
app: coffee
template:
metadata:
labels:
app: coffee
spec:
containers:
- name: coffee
image: manmohanmirkar/mycoffee_image:latest
---
apiVersion: v1
kind: Service
metadata:
name: coffee-deploy
spec:
type: NodePort
selector:
app: coffee
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 80
Now lets apply the coffee service and deployment.
# kubectl create -f coffe.yml deployment.apps/coffee-deploy created service/coffee-deploy created
-
Deploy the Ingress resource
This is the main configuration in this article which will configure the Ingress resource. This Ingress resource will be performing the task of routing the request to respective service based on the URL. Let have look at the manifest file for the same:
# cat cafe-ingress.yml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress-resource
spec:
rules:
- host: nginx.example.com
http:
paths:
- backend:
serviceName: nginx-deploy-main
servicePort: 80
- host: tea.myshop.com
http:
paths:
- backend:
serviceName: tea-deploy
servicePort: 80
- host: coffee.myshop.com
http:
paths:
- backend:
serviceName: coffee-deploy
servicePort: 80
Basically in the above file will deploy the resource type Ingress. It has got three host based rules configured for each service. So if the host url is “nginx.example.com” then it has backend with service Name “nginx-deploy-main”. Basically it means that whenever the request with URL “nginx.example.com” made it will be get forwarded to the service “nginx-deploy-main”.
In the similar way we have two more hosts “tea.myshop.com” and “coffee.myshop.com” with backed configured as “tea-deploy” and “coffee-deploy”. This also means that whenever the request made with URL “tea.myshop.com” to the haproxy serrver it will forwarded to the service “tea-deploy”. Also in the similar way if the request with URL “coffee.myshop.com” it will be automatically get forwarded to the service “coffee-deploy” which we have already deployed.
Now lets deploy the ingress resource:
# kubectl create -f cafe-ingress.yml ingress.networking.k8s.io/my-ingress-resource created
Crosscheck the ingress resource with following commands:
# kubectl describe ing
Name: my-ingress-resource
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
nginx.example.com
nginx-deploy-main:80 (10.244.2.82:80)
tea.myshop.com
tea-deploy:80 (10.244.1.94:80,10.244.2.83:80)
coffee.myshop.com
coffee-deploy:80 (10.244.1.95:80,10.244.2.84:80)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 31s nginx-ingress-controller Configuration for default/my-ingress-resource was added or updated
Normal AddedOrUpdated 31s nginx-ingress-controller Configuration for default/my-ingress-resource was added or updated
You can able to see from the output that, if the host is “nginx.example.com” then service configured is “nginx-deploy-main” and if the host or URL is “tea.myshop.com” then service for the same is “tea-deploy” sample applicable for the coffee service with respective service name.
-
Make the DNS entries
This is the final step in which we are going to configure the DNS entries for all the three URL we have configured in the ingress resource. Basically we are going make the entries on the host machine with all the urls with ip of out haproxy container.
# lxc list haproxy +---------+---------+-----------------------+-----------------------------------------------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +---------+---------+-----------------------+-----------------------------------------------+------------+-----------+ | haproxy | RUNNING | 10.253.121.146 (eth0) | fd42:38af:bc0d:704d:216:3eff:fefa:cb4c (eth0) | PERSISTENT | 0 | +---------+---------+-----------------------+-----------------------------------------------+------------+-----------+
So ip address of haproxy is 10.253.121.146. Simply add below entries /etc/hosts of the host machine:
# cat /etc/hosts 127.0.0.1 localhost 192.168.56.5 vbhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 10.253.121.146 nginx.example.com 10.253.121.146 coffee.myshop.com 10.253.121.146 tea.myshop.com
Check the last three entries all url are pointing to haproxy which will in turn forward the request to our kubernetes worker node.
-
Testing
Now lets try to access the all URL’s one by one from the host machine.
- nginx.example.com
- tea.myshop.com
- coffee.myshop.com
So this all about the topic of How to configure ingress controller in kubernetes.
Leave a Reply