DevSecOps — Deploying WebApp on AWS EKS cluster with Github Actions
DevSecOps is an approach that combines Development, Security, and Operations to integrate security practices into the DevOps workflow. When deploying a web application on an AWS EKS (Elastic Kubernetes Service) cluster using GitHub Actions, organizations can automate and enhance the security of their deployment process.
DevSecOps pipeline diagram
Pre-requisites
Tools needed for this exercise are
- AWS account
- AWS CLI
- eksctl
- Github Account
- kubectl
The deployment process on AWS EKS with GitHub Actions generally involves the following steps:
1. Cluster Setup: Provision an EKS cluster on the AWS platform to host the web application. Configure networking, security, and resource settings according to requirements.
- Dockerfile
Set the environment variables
- Create Elastic Container Registry to hold container images and image meta data. Build application docker image and push it to registry ((This will be replaced with IaC code with terraform or pulumi)
2. Set environment variables and policies.
3. Create EKS cluster
4. Update the kubconfig
5. Check EKS cluster status
(base) skondla@skondla1-Mac:bash $ k config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop
arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop-2 arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop-2 arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop-2
gke_beautify-ocean-123456_us-central1-a_test gke_beautify-ocean-123456_us-central1-a_test gke_beautify-ocean-123456_us-central1-a_test
gke_beautify-ocean-123456_us-central1-b_staging gke_beautify-ocean-123456_us-central1-b_staging gke_beautify-ocean-123456_us-central1-b_staging
gke_beautify-ocean-123456_us-central1-c_prod gke_beautify-ocean-123456_us-central1-c_prod gke_beautify-ocean-123456_us-central1-c_prod
gke_marvalo-avenger-123456_us-east4-a_flaskapp1-demo-cluster gke_marvalo-avenger-123456_us-east4-a_flaskapp1-demo-cluster gke_marvalo-avenger-123456_us-east4-a_flaskapp1-demo-cluster
gke_marvalo-avenger-123456_us-east4-a_webapp1-demo-cluster gke_marvalo-avenger-123456_us-east4-a_webapp1-demo-cluster gke_marvalo-avenger-123456_us-east4-a_webapp1-demo-cluster
k3d-k3s-default k3d-k3s-default admin@k3d-k3s-default
skondla@flaskapp1-demo-shop-2-2.us-east-1.eksctl.io flaskapp1-demo-shop-2-2.us-east-1.eksctl.io skondla@flaskapp1-demo-shop-2-2.us-east-1.eksctl.io
* skondla@webapps-demo.us-east-1.eksctl.io webapps-demo.us-east-1.eksctl.io skondla@webapps-demo.us-east-1.eksctl.io
(base) skondla@skondla1-Mac:bash $ k get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-192-168-122-206.ec2.internal Ready <none> 10m v1.27.1-eks-2f008fe 192.168.122.206 3.234.229.120 Amazon Linux 2 5.10.179-168.710.amzn2.x86_64 containerd://1.6.19
ip-192-168-17-81.ec2.internal Ready <none> 10m v1.27.1-eks-2f008fe 192.168.17.81 22.22.38.172 Amazon Linux 2 5.10.179-168.710.amzn2.x86_64 containerd://1.6.19
ip-192-168-3-26.ec2.internal Ready <none> 10m v1.27.1-eks-2f008fe 192.168.3.26 44.87.48.24 Amazon Linux 2 5.10.179-168.710.amzn2.x86_64 containerd://1.6.19
(base) skondla@skondla1-Mac:bash $ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 28m
(base) skondla@skondla1-Mac:bash $ k get ns
NAME STATUS AGE
default Active 28m
flaskapp Active 18m
kube-node-lease Active 28m
kube-public Active 28m
kube-system Active 28m
(base) skondla@skondla1-Mac:bash $ k get pods -n kube-system
NAME READY STATUS RESTARTS AGE
aws-node-7zlkg 1/1 Running 0 26m
aws-node-jvn84 1/1 Running 0 26m
aws-node-xncj7 1/1 Running 0 26m
coredns-79df7fff65-r269m 1/1 Running 0 34m
coredns-79df7fff65-zxmp4 1/1 Running 0 34m
kube-proxy-84bg2 1/1 Running 0 26m
kube-proxy-lvwvr 1/1 Running 0 26m
kube-proxy-rrjqn 1/1 Running 0 26m2. Application Configuration: Define the desired state of the web application using Kubernetes manifests, which describe the deployment, services, and other resources necessary for the application’s operation.
Create namespace and service account
Define Application Deployment
Define Application Service
Check Deployments, and Services status
(base) skondla@skondla1-Mac:bash $ k get ns
NAME STATUS AGE
default Active 96m
flaskapp Active 86m
kube-node-lease Active 96m
kube-public Active 96m
kube-system Active 96m
(base) skondla@skondla1-Mac:bash $ k get sa -n flaskapp
NAME SECRETS AGE
default 0 86m
flaskapp1-sa 0 86m
(base) skondla@skondla1-Mac:bash $ k get deploy -n flaskapp
NAME READY UP-TO-DATE AVAILABLE AGE
flaskapp1-admin 3/3 3 3 56s
flaskapp1-user 1/3 3 1 54s
(base) skondla@skondla1-Mac:bash $ k get svc -n flaskapp
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
flaskapp1-admin-svc LoadBalancer 10.100.207.56 yodsvga458dfgabc3434fsfsdsghkgdt-123456789.us-east-1.elb.amazonaws.com 30443:30555/TCP 41m
flaskapp1-user-svc LoadBalancer 10.100.152.216 abc3434fsfsdsghkgdtyodsvga458dfg-123456789.us-east-1.elb.amazonaws.com 50443:31759/TCP 41m
(base) skondla@skondla1-Mac:bash $
(base) skondla@skondla1-Mac:bash $ k get pods -n flaskapp
NAME READY STATUS RESTARTS AGE
flaskapp1-admin-796db79ff-ffsk9 1/1 Running 0 3m4s
flaskapp1-admin-796db79ff-l5bdt 1/1 Running 0 3m4s
flaskapp1-admin-796db79ff-svx6j 1/1 Running 0 3m4s
flaskapp1-user-5887548889-jpf7x 0/1 Pending 0 3m2s
flaskapp1-user-5887548889-v2cb5 1/1 Running 0 3m2s
flaskapp1-user-5887548889-xnq58 0/1 Pending 0 3m2s
(base) skondla@skondla1-Mac:bash $ k describe pod flaskapp1-user-5887548889-jpf7x -n flaskapp
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m21s default-scheduler 0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..
Warning FailedScheduling
3. Containerization: Build a container image of the web application using a Dockerfile. Containerization ensures consistent deployment across various environments and encapsulates the application and its dependencies.
Version Control: Create a GitHub repository to store the web application’s source code. GitHub provides a centralized location for version control, code collaboration, and automated workflows.Increase worker nodes
(base) skondla@skondla1-Mac:bash $ k get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-192-168-122-206.ec2.internal Ready <none> 10m v1.27.1-eks-2f008fe 192.168.122.206 3.234.229.120 Amazon Linux 2 5.10.179-168.710.amzn2.x86_64 containerd://1.6.19
ip-192-168-17-81.ec2.internal Ready <none> 10m v1.27.1-eks-2f008fe 192.168.17.81 22.22.38.172 Amazon Linux 2 5.10.179-168.710.amzn2.x86_64 containerd://1.6.19
ip-192-168-3-26.ec2.internal Ready <none> 10m v1.27.1-eks-2f008fe 192.168.3.26 44.87.48.24 Amazon Linux 2 5.10.179-168.710.amzn2.x86_64 containerd://1.6.19
ip-192-168-8-36.ec2.internal Ready <none> 104m v1.27.1-eks-2f008fe 192.168.8.36 74.85.48.140 Amazon Linux 2 5.10.179-168.710.amzn2.x86_64 containerd://1.6.19
(base) skondla@skondla1-Mac:bash $ k get pods -n flaskapp
NAME READY STATUS RESTARTS AGE
flaskapp1-admin-796db79ff-ffsk9 1/1 Running 0 17m
flaskapp1-admin-796db79ff-l5bdt 1/1 Running 0 17m
flaskapp1-admin-796db79ff-svx6j 1/1 Running 0 17m
flaskapp1-user-5887548889-jpf7x 0/1 ContainerCreating 0 17m
flaskapp1-user-5887548889-v2cb5 1/1 Running 0 17m
flaskapp1-user-5887548889-xnq58 0/1 ContainerCreating 0 17m
(base) skondla@skondla1-Mac:bash $ k get pods -n flaskapp
NAME READY STATUS RESTARTS AGE
flaskapp1-admin-796db79ff-ffsk9 1/1 Running 0 18m
flaskapp1-admin-796db79ff-l5bdt 1/1 Running 0 18m
flaskapp1-admin-796db79ff-svx6j 1/1 Running 0 18m
flaskapp1-user-5887548889-jpf7x 1/1 Running 0 18m
flaskapp1-user-5887548889-v2cb5 1/1 Running 0 18m
flaskapp1-user-5887548889-xnq58 1/1 Running 0 18m
(base) skondla@skondla1-Mac:bash $ k get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
flaskapp flaskapp1-admin-796db79ff-8glmw 1/1 Running 0 107m 192.168.47.106 ip-192-168-19-0.ec2.internal <none> <none>
flaskapp flaskapp1-admin-796db79ff-d956f 1/1 Running 0 107m 192.168.113.110 ip-192-168-98-246.ec2.internal <none> <none>
flaskapp flaskapp1-admin-796db79ff-k6xfw 1/1 Running 0 107m 192.168.35.16 ip-192-168-61-47.ec2.internal <none> <none>
flaskapp flaskapp1-user-5887548889-9d8zd 1/1 Running 0 107m 192.168.25.97 ip-192-168-19-0.ec2.internal <none> <none>
flaskapp flaskapp1-user-5887548889-p5z8w 1/1 Running 0 107m 192.168.56.28 ip-192-168-61-47.ec2.internal <none> <none>
flaskapp flaskapp1-user-5887548889-tn9m4 1/1 Running 0 107m 192.168.64.123 ip-192-168-98-246.ec2.internal <none> <none>
kube-system aws-node-2qrnt 1/1 Running 0 108m 192.168.61.47 ip-192-168-61-47.ec2.internal <none> <none>
kube-system aws-node-cvfrj 1/1 Running 0 108m 192.168.107.105 ip-192-168-107-105.ec2.internal <none> <none>
kube-system aws-node-ddqm8 1/1 Running 0 48s 192.168.99.241 ip-192-168-99-241.ec2.internal <none> <none>
kube-system aws-node-p7hnh 1/1 Running 0 4m34s 192.168.60.206 ip-192-168-60-206.ec2.internal <none> <none>
kube-system aws-node-vlv5h 1/1 Running 0 108m 192.168.19.0 ip-192-168-19-0.ec2.internal <none> <none>
kube-system aws-node-wwgk6 1/1 Running 0 108m 192.168.98.246 ip-192-168-98-246.ec2.internal <none> <none>
kube-system coredns-79df7fff65-qhw87 1/1 Running 0 115m 192.168.106.237 ip-192-168-107-105.ec2.internal <none> <none>
kube-system coredns-79df7fff65-vblm2 1/1 Running 0 115m 192.168.106.221 ip-192-168-107-105.ec2.internal <none> <none>
kube-system kube-proxy-9wwnv 1/1 Running 0 108m 192.168.98.246 ip-192-168-98-246.ec2.internal <none> <none>
kube-system kube-proxy-cq6nq 1/1 Running 0 48s 192.168.99.241 ip-192-168-99-241.ec2.internal <none> <none>
kube-system kube-proxy-hnbdh 1/1 Running 0 108m 192.168.107.105 ip-192-168-107-105.ec2.internal <none> <none>
kube-system kube-proxy-jlgs8 1/1 Running 0 108m 192.168.61.47 ip-192-168-61-47.ec2.internal <none> <none>
kube-system kube-proxy-pmgr9 1/1 Running 0 108m 192.168.19.0 ip-192-168-19-0.ec2.internal <none> <none>
kube-system kube-proxy-v4fzn 1/1 Running 0 4m34s 192.168.60.206 ip-192-168-60-206.ec2.internal <none> <none>
webapp webapp1-demo-shop-795c7f6557-dd6sn 1/1 Running 0 16m 192.168.32.129 ip-192-168-60-206.ec2.internal <none> <none>
webapp webapp1-demo-shop-795c7f6557-zcqzj 1/1 Running 0 16m 192.168.123.226 ip-192-168-99-241.ec2.internal <none> <none>
webapp webapp1-demo-shop-795c7f6557-zvx72 1/1 Running 0 16m 192.168.17.29 ip-192-168-60-206.ec2.internal <none> <none>4. GitHub Actions: Leverage GitHub Actions, which are workflow automation scripts, to orchestrate the deployment process. Actions can be triggered by events like code pushes, pull requests, or manual triggers.
- Setup github actions trigger for example: on push
on:
push:
branches: [ "testing_aws", "master" ]2. Setup environment variables
5. Continuous Integration (CI): Establish a CI pipeline within the GitHub Actions workflow to automatically build and test the application when changes are pushed to the repository. This facilitates continuous integration and validation of the code.
Setup Build step
6. Security Scanning: Integrate security scanning tools into the CI pipeline to identify vulnerabilities, security flaws, or compliance issues. These tools can include static code analysis, dependency scanning, container image scanning, or vulnerability assessments.
Setup image Scanning step
Deployment: Use GitHub Actions to deploy the containerized web application to the EKS cluster. This involves applying the Kubernetes manifests and ensuring the desired state is achieved.
Setup deploy step
6. Continuous Delivery (CD): Implement CD practices by automating the deployment pipeline, enabling seamless delivery of new versions or updates to the web application. Strategies such as blue-green deployments or canary releases can be employed to minimize downtime and risks.
Setup application endpoints validation.
Setup notification via slack step
7. Continuous Monitoring and Logging: Set up monitoring and logging solutions to gain insights into the web application’s performance, availability, and security. Monitoring enables proactive issue identification and facilitates rapid troubleshooting.
An external observability platform to collect application, container, k8s cluster metrics, traces, logs was not set up to instrument and monitor health, performance and scaling issues in this article, however a basic native AWS cloud watch logs and dashboards will help with some insights about EKS cluster logging
Before you get to cloud watch, enable EKS cluster
8. Continuous Feedback:
Continuous Feedback in DevSecOps refers to the practice of gathering and utilizing feedback at every stage of the software development lifecycle to improve the quality, security, and performance of an application. It emphasizes the iterative and continuous improvement of processes, products, and teams based on real-time feedback and data.
Collaboration and Communication: Continuous feedback promotes effective collaboration and communication among different teams involved in software development, including developers, operations, security, and quality assurance. Feedback is shared transparently and promptly, fostering a culture of open communication and knowledge sharing.
Automated Testing and Validation: Continuous feedback relies on automated testing and validation mechanisms, such as unit testing, integration testing, and security testing, to provide immediate feedback on code quality, vulnerabilities, and compliance issues. These tests are integrated into the development process, ensuring early detection and resolution of issues.
Real-Time Monitoring and Observability: Continuous feedback involves monitoring the application and infrastructure in real-time to gather data on performance, availability, and security. Metrics, logs, and alerts are used to identify anomalies, bottlenecks, and security threats, enabling teams to address issues proactively and optimize system performance.
User Feedback and Experience: Continuous feedback incorporates user feedback and user experience data to understand the application’s usability, functionality, and performance from the end-users’ perspective. This can be obtained through surveys, user analytics, feedback forms, and usability testing. User feedback helps prioritize improvements and enhancements.
Post-Release Feedback: Continuous feedback extends beyond the development and deployment stages. It encompasses feedback obtained from users, stakeholders, and incident reports after the application is released. This feedback informs future iterations and updates, enabling organizations to address any issues or gaps that may arise in real-world usage.
Continuous Improvement: Continuous feedback drives a culture of continuous improvement. Teams actively seek feedback, analyze data, and iterate on processes, tools, and workflows to enhance efficiency, security, and quality. Lessons learned from previous iterations are incorporated into future development cycles.
The benefits of continuous feedback in DevSecOps include faster detection and resolution of issues, improved security posture, enhanced user satisfaction, increased operational efficiency, and overall improvement of software quality. By leveraging feedback throughout the software development lifecycle, organizations can continuously optimize their processes, products, and security measures, resulting in more reliable and secure applications.
By incorporating DevSecOps principles into the deployment process on an AWS EKS cluster with GitHub Actions, organizations can achieve an automated, secure, and efficient deployment process for their web applications. This approach promotes collaboration, reduces manual effort, and ensures that security is integrated throughout the software development lifecycle.
