Kubernetes : Django, Redis, Frontend, Nginx (reverse proxy & ingress) setup on Bare Metal Server & CICD Guide

(0 comments)

Follow Below Steps in sequence. FYI I have created EC2 on AWS.

  1. Create ubuntu 20 LTS instance. open 22, 80, 443 port.
  2. Enable net.bridge.bridge-nf-call-iptables
sudo su
cat > /etc/sysctl.d/20-bridge-nf.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

     3. Install Docker...

mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
apt-get update
apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg2
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
echo 'deb [arch=amd64] https://download.docker.com/linux/debian stretch stable' > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y --no-install-recommends docker-ce
sudo /etc/init.d/docker restart

     4. Setup Kubernetes and set CIDR for pods

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
kubeadm init --pod-network-cidr=10.244.0.0/16

sudo su ubuntu
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
source <(kubectl completion bash)

     5. For pod networking

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl get pods --all-namespaces

    6. Untaint the master so you can run pods

kubectl taint nodes --all node-role.kubernetes.io/master-

    7.  Set up nginx-ingress

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
cat > nginx-host-networking.yaml <<EOF
spec:
template:
spec:
hostNetwork: true
EOF
kubectl -n ingress-nginx patch deployment ingress-nginx-controller --patch="$(<nginx-host-networking.yaml)"

    8.  Get or Set external IP in nginx ingress

kubectl get svc -n ingress-nginx
# if external ip not assiged. follow below after assigning Elastic IP to EC2:
kubectl edit svc ingress-nginx-controller -n ingress-nginx
...
spec:
type: LoadBalancer
externalIPs:
- <Set your Elastic IP here that assigned to EC2>

# verify assigned ip again:
kubectl get svc -n ingress-nginx

   9. Create Namespace on which It will deploy all project specific services. Here its named as "myproject"

kubectl create namespace myproject
   10. Setup Redis. create file "redis-standalone.yaml" and put below content. (Before that run "hostname" to get internal host IP) and apply with:
mkdir -p /home/ubuntu/redisdb

kubectl apply -f redis-standalone.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: redis
   namespace: "myproject"
spec:
 capacity:
   storage: 10Gi
 volumeMode: Filesystem
 accessModes:
   - ReadWriteMany
 persistentVolumeReclaimPolicy: Retain
 storageClassName: redis
 local:
   path: /home/ubuntu/redisdb
 nodeAffinity:
   required:
     nodeSelectorTerms:
     - matchExpressions:
       - key: kubernetes.io/hostname
         operator: In
         values:
           - < INTERNAL HOST IP >  # hostname of machine. type "hostname"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: redis-volume
 namespace: "myproject"
spec:
 storageClassName: redis
 accessModes:
   - ReadWriteMany
 volumeMode: Filesystem
 resources:
   requests:
     storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
 name: redis
 namespace: "myproject"
spec:
 type: ClusterIP
 ports:
   - port: 6379
     name: redis
 #clusterIP: None
 selector:
   app: redis
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: redis
 namespace: "myproject"
spec:
 selector:
   matchLabels:
     app: redis  # has to match .spec.template.metadata.labels
 serviceName: redis
 replicas: 1
 template:
   metadata:
     labels:
       app: redis  # has to match .spec.selector.matchLabels
   spec:
     containers:
       - name: redis
         image: redis:6.2.5-alpine
         imagePullPolicy: Always
         #args: ["--requirepass", "$(REDIS_PASS)"]
         args: ["--appendonly", "yes", "--save", "900", "1", "--save", "30", "1"]
         ports:
           - containerPort: 6379
             name: redis
         volumeMounts:
           - name: redisdb-data
             mountPath: /home/ubuntu/redisdb
     volumes:
     - name: redisdb-data
       persistentVolumeClaim:
         claimName: redis-volume
---

    11. Create backend secrets : create file "secret-creds.yaml" and put below contents and apply. please make sure to convert plaintext values to base64 using following commands:

echo -n ENV_VALUE | base64
if its certificate files (TLS), use below commands
cat myproject_wildcard.crt | base64
cat myproject_wildcard.pem | base64
apiVersion: v1
kind: Secret
metadata:
name: "backend-secret-creds"
namespace: "myproject"
type: kubernetes.io/generic
data:
POSTGRES_HOST: "<BASE64 OF PG HOST>"
POSTGRES_DB: "<BASE64 OF DB NAME>"
POSTGRES_USER: "<BASE64 OF PG USER>"
POSTGRES_PASSWORD: "<BASE64 OF PG PASSWORD>"
STATIC_ROOT: "<BASE64 OF /home/ubuntu/staticfiles >"
MEDIA_ROOT: "<BASE64 OF /home/ubuntu/media >"

    12.  Setup SSL secrets. create "ssl.yaml" with below content and apply:

apiVersion: v1
data:
tls.crt: < BASE64 OF TLS CRT >
tls.key: < BASE64 OF TLS KEY >
kind: Secret
metadata:
name: myproject-cert
namespace: myproject
type: kubernetes.io/tls

    13. Create docker registry secret so it can pull from AWS ECR

ECRPASS=$(aws ecr get-login-password --region < AWSREGION us-east-1 for example >)
kubectl create secret docker-registry aws-ecr-us-east-1 \
--docker-server=< AWS ECR PRIVATE REGISTRY ECR HOST > \
--docker-username=AWS \
--docker-password=$ECRPASS \
[email protected] --namespace myproject

    14. Create volumes for staticfiles and media. "staticmedia.yaml". and put below contents and apply:

mkdir -p /home/ubuntu/staticfiles

mkdir -p /home/ubuntu/media

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: staticfiles
namespace: myproject
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: staticfiles-sc
local:
path: /home/ubuntu/staticfiles
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <INTERNAL HOST IP>
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: staticfiles-volume
namespace: myproject
spec:
storageClassName: staticfiles-sc
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: media
namespace: myproject
spec:
capacity:
storage: 3Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: media-sc
local:
path: /home/ubuntu/media
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <INTERNAL IP>
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: media-volume
namespace: myproject
spec:
storageClassName: media-sc
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 3Gi


 
   15. Time to setup Docker backend deployment. create "
backend.yaml" and apply:

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: backend
 namespace: myproject
 labels:
   app: backend
spec:
 replicas: 1
 selector:
   matchLabels:
     app: backend
 strategy:
   type: RollingUpdate
   rollingUpdate:
     maxSurge: 1
     maxUnavailable: 0
 template:
   metadata:
     labels:
       app: backend
   spec:
     imagePullSecrets:
     - name: aws-ecr-us-east-1
     containers:
     - name: backend
       image: <ECR HOST>/backend:latest
       imagePullPolicy: Always
       volumeMounts:
       - name: staticfiles
         mountPath: /home/ubuntu/staticfiles
       - name: media
         mountPath: /home/ubuntu/media
       ports:
         - containerPort: 8000
           name: backend
       #resources:
       #  requests:
       #    cpu: 500m
       #    memory: "512M"
       #  limits:
       #    cpu: 1000m
       #    memory: "1024M"
       readinessProbe:
         tcpSocket:
           port: 8000
         initialDelaySeconds: 5
         periodSeconds: 10
       livenessProbe:
         tcpSocket:
           port: 8000
         initialDelaySeconds: 15
         periodSeconds: 20
       env:
         - name: POSTGRES_HOST
           valueFrom:
             secretKeyRef:
               name: backend-secret-creds
               key: POSTGRES_HOST
         - name: POSTGRES_DB
           valueFrom:
             secretKeyRef:
               name: backend-secret-creds
               key: POSTGRES_DB
         - name: POSTGRES_USER
           valueFrom:
             secretKeyRef:
               name: backend-secret-creds
               key: POSTGRES_USER
         - name: POSTGRES_PASSWORD
           valueFrom:
             secretKeyRef:
               name: backend-secret-creds
               key: POSTGRES_PASSWORD
         - name: STATIC_ROOT
           valueFrom:
             secretKeyRef:
               name: backend-secret-creds
               key: STATIC_ROOT
         - name: MEDIA_ROOT
           valueFrom:
             secretKeyRef:
               name: backend-secret-creds
               key: MEDIA_ROOT
       command: ["/code/backend_start_gunicorn_etc.sh"]
     restartPolicy: Always
     volumes:
     - name: staticfiles
       persistentVolumeClaim:
         claimName: staticfiles-volume
     - name: media
       persistentVolumeClaim:
         claimName: media-volume

    16. Create backend service "backendsvc.yaml" and apply:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: backend
  name: backend
  namespace: myproject
spec:
  ports:
    - port: 8000
  selector:
    app: backend

    17. create "frontend.yaml" and apply:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: myproject
  labels:
    app: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: frontend
    spec:
      imagePullSecrets:
      - name: aws-ecr-us-east-1
      containers:
      - name: frontend
        image: < ECR HOST >/frontend:latest
        imagePullPolicy: Always
        ports:
          - containerPort: 3000
            name: frontend
        readinessProbe:
          tcpSocket:
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 3000
          initialDelaySeconds: 15
          periodSeconds: 20
        #resources:
        #  requests:
        #    cpu: 500m
        #    memory: "512M"
        #  limits:
        #    cpu: 1000m
        #    memory: "1024M"

   18. create "frontendsvc.yaml" and apply:

---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: myproject
spec:
  selector:
    app: frontend
  ports:
  - port: 3000

   19. to create nginx specific config maps, create file "nginxconfig.yaml" with below content and apply:

"/api" and "/admin" reverse proxy to django and "/" reverse proxy to frontend service.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-main
  namespace: myproject
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    pid        /var/run/nginx.pid;

    events {
        worker_connections  1024;
    }

    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;

        log_format json_combined escape=json '{"proxy_protocol_addr": "$proxy_protocol_addr", "remote_addr": "$remote_addr", "proxy_add_x_forwarded_for": "$proxy_add_x_forwarded_for", "time_local": "$time_local", "request" : "$request", "status": "$status", "body_bytes_sent": "$body_bytes_sent", "http_referer":  "$http_referer", "http_user_agent": "$http_user_agent", "request_length" : "$request_length", "request_time": "$request_time", "upstream_addr": "$upstream_addr",  "upstream_response_length": "$upstream_response_length", "upstream_response_time": "$upstream_response_time", "upstream_status": "$upstream_status", "http_host": "$http_host", "host": "$host"}';

        access_log /dev/stdout json_combined;
        error_log /dev/stdout warn;

        sendfile        on;
        #tcp_nopush     on;

        keepalive_timeout  65;

        gzip on;
        gzip_disable "msie6";
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;

        include /etc/nginx/conf.d/*.conf;
    }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-addon
  namespace: myproject
data:
  local.conf: |
    upstream upstream_server_backend {
      server backend:8000;
    }

    upstream upstream_server_frontend {
      server frontend:3000;
    }

    server {
      listen 80;
      listen [::]:80 default_server ipv6only=on;
      server_name _;

      keepalive_timeout  65;
      client_max_body_size 100M;

      location / {
        proxy_pass http://upstream_server_frontend;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_redirect off;
        proxy_set_header  X-Forwarded-Proto  $scheme;
        proxy_read_timeout 1200;
        proxy_connect_timeout 300;
      }

      location /api {
        proxy_pass http://upstream_server_backend;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_redirect off;
        proxy_set_header  X-Forwarded-Proto  $scheme;
        proxy_read_timeout 1200;
        proxy_connect_timeout 300;
      }

      location /admin {
        proxy_pass http://upstream_server_backend;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_redirect off;
        proxy_set_header  X-Forwarded-Proto  $scheme;
        proxy_read_timeout 1200;
        proxy_connect_timeout 300;
      }

      location /media {
        alias /home/ubuntu/media;
      }

      location /static {
        alias /home/ubuntu/staticfiles;
      }

    }

   20. Create "nginxweb.yaml" deployment spec with following content and apply:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginxweb
  name: nginxweb
  namespace: myproject
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginxweb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nginxweb
    spec:
      containers:
      - image: nginx:1.19
        name: nginx-container
        ports:
        - containerPort: 80
        resources: {}
        volumeMounts:
        - name: nginx-main
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
        - name: nginx-addon
          mountPath: /etc/nginx/conf.d/
        - name: staticfiles
          mountPath: /home/ubuntu/staticfiles
        - name: media
          mountPath: /home/ubuntu/media
        - name: log
          mountPath: /var/log/nginx

      restartPolicy: Always
      volumes:
      - name: nginx-main
        configMap:
          name: nginx-main
      - name: nginx-addon
        configMap:
          name: nginx-addon
      - name: staticfiles
        persistentVolumeClaim:
          claimName: staticfiles-volume
      - name: media
        persistentVolumeClaim:
          claimName: media-volume
      - name: log
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginxweb
  name: nginxweb
  namespace: myproject
spec:
  ports:
  - name: nginxweb
    port: 80
    targetPort: 80
  selector:
    app: nginxweb

    21. create ingress.yaml with following content and apply:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/client-body-buffer-size: 2.5M
    nginx.ingress.kubernetes.io/proxy-body-size: 50M
    nginx.ingress.kubernetes.io/proxy-buffer-size: 50m
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: 1024m
  name: nginxweb
  namespace: myproject
spec:
  rules:
  - host: '*.myproject.com'
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: nginxweb
            port:
              number: 80
  tls:
  - hosts:
    - '*.myproject.com'
    secretName: myproject-cert

     22. Your environment setup is up once you point DNS records of "*.myproject.com" to Elastic IP of EC2. 
     22. For github actions based CICD, Following steps you can do to build image and update image.
           - build docker images in github actions CI and push to ECR.
           - ssh to EC2 through github actions and for CD use following commands:

# to update pod without changing to new image
kubectl -n myproject patch deployment backend  -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"$(date +%s)\"}}}}}"

# to update tag
kubectl --namespace=myproject set image deployment/backend backend=<ECR HOST>/backend:${CI_COMMIT_TAG}

# for example in github action manifests file, following section will take care of deployment. - name: Whitelisting IP uses: sohelamin/[email protected] with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: 'us-east-1' aws-security-group-id: ${{ secrets.AWS_SG_ID }} port: '22' command_timeout: 200m description: 'Github Action' - name: Deploying uses: appleboy/[email protected] with: host: ${{ secrets.AWS_EC2_HOST }} username: "ubuntu" key: ${{ secrets.AWS_EC2_PVT_KEY }} port: '22' command_timeout: 200m script: | kubectl -n myproject patch deployment backend -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"$(date +%s)\"}}}}}"

Please  Leave your comment if its usefull for you. 

.

Currently unrated

Comments

There are currently no comments

New Comment

required

required (not published)

optional

required

captcha

required