DevSecOps — Deploying WebApp on Google Cloud GKE cluster with Github Actions
DevSecOps, a combination of Development, Security, and Operations, is an approach that integrates security practices into the DevOps workflow to ensure the continuous delivery of secure and reliable software applications. In this context, deploying a web application on a Google Cloud GKE (Google Kubernetes Engine) cluster using GitHub Actions provides an automated and streamlined process.
Note: This blog illustrates DevSecOps pipeline and does not explain Python Flask API application I wrote to automate database backups (a separate story is upcoming..) by non admin or developers. And the application code is used as an example for deploying containerized application. The pipeline uses same application to build, scan, deploy and validate service API endpoints to complete the process.
DevSecOps pipeline diagram
Pre-requisites
- Google Cloud Platform access
2. gcloud cli installation
The deployment process typically involves the following steps:
Infrastructure Setup: Create a GKE cluster on Google Cloud Platform (GCP) to host the web application. Configure necessary networking, security, and resource allocation settings. ((This will be replaced with IaC code with terraform or pulumi)
a. Setup environment variables
b. Login to Google Cloud
#Login to gcloud
gcloud auth loginc. Set Project, Compute Zone/Region
d. Enable required GCP APIs
e. Create GKE cluster
f. Check deployed GKE clusters
g. Create Artifactory repository
g. Create an IAM service account for GitHub Actions
h. Add IAM Policy binding with Service Account
i. Create a service account credentials and store it securely (Github Secrets) to use for DevSecOps pipelines
2. Containerization: Build a container image of the web application using a Dockerfile. Containerization allows for consistent deployment across different environments and ensures that the application and its dependencies are encapsulated.
Docker image manifest file with self-signed certificate
3. Application Configuration: Define the desired state of the web application using Kubernetes manifests, which describe the deployment, services, and other resources required by the application.
a. Application deployment manifest
b. Application service manifest
4. Version Control: Set up a GitHub repository to store the source code of the web application. This provides a centralized location for code collaboration, version control, and automated workflows.
Follow the link below to create and setup GitHub repo
5. GitHub Actions: Leverage GitHub Actions, which are workflow automation scripts, to orchestrate the deployment process. Actions can be triggered by events such as code pushes, pull requests, or manual triggers.
Follow the link below to get started on GitHub Actions
a. Setup environment variables
b. Checkout code
c. Test code
d. Build image
c. Scan image
d. Deploy application into Google Kubernetes Engine (GKE) Cluster
e. Validate Application Service endpoints for response
f. Send a notification to Slack channel
6. Continuous Integration (CI): Implement a CI pipeline within the GitHub Actions workflow to automatically build and test the application whenever changes are pushed to the repository. This ensures that the code is continuously integrated and validated.
on push to feature branch and master, trigger a DevSecOps workflow
a. Setup environment variables
b. Checkout code
c. Test code
d. Build image
7. Security Scanning: Incorporate security scanning tools (Trivy from Aqua) within the CI pipeline to identify vulnerabilities, security flaws, or compliance issues. This can include static code analysis, dependency scanning, container image scanning, or vulnerability assessments.
8. Deployment & Validation: Utilize GitHub Actions to deploy the containerized web application to the GKE cluster. This involves applying the Kubernetes manifests and ensuring that the desired state is achieved.
9. Test and Validation: Validate application service endpoints
10. Notification to Slack Channel:
11. Continuous Delivery (CD): Implement CD practices by automating the deployment pipeline, allowing for the seamless delivery of new versions or updates to the web application. This can include strategies such as blue-green deployments or canary releases to minimize downtime and risk.
12. Continuous Feedback
Continuous Feedback in DevSecOps refers to the practice of gathering and utilizing feedback at every stage of the software development lifecycle to improve the quality, security, and performance of an application. It emphasizes the iterative and continuous improvement of processes, products, and teams based on real-time feedback and data.
Collaboration and Communication: Continuous feedback promotes effective collaboration and communication among different teams involved in software development, including developers, operations, security, and quality assurance. Feedback is shared transparently and promptly, fostering a culture of open communication and knowledge sharing.
Automated Testing and Validation: Continuous feedback relies on automated testing and validation mechanisms, such as unit testing, integration testing, and security testing, to provide immediate feedback on code quality, vulnerabilities, and compliance issues. These tests are integrated into the development process, ensuring early detection and resolution of issues.
Real-Time Monitoring and Observability: Continuous feedback involves monitoring the application and infrastructure in real-time to gather data on performance, availability, and security. Metrics, logs, and alerts are used to identify anomalies, bottlenecks, and security threats, enabling teams to address issues proactively and optimize system performance.
User Feedback and Experience: Continuous feedback incorporates user feedback and user experience data to understand the application’s usability, functionality, and performance from the end-users’ perspective. This can be obtained through surveys, user analytics, feedback forms, and usability testing. User feedback helps prioritize improvements and enhancements.
Post-Release Feedback: Continuous feedback extends beyond the development and deployment stages. It encompasses feedback obtained from users, stakeholders, and incident reports after the application is released. This feedback informs future iterations and updates, enabling organizations to address any issues or gaps that may arise in real-world usage.
Continuous Improvement: Continuous feedback drives a culture of continuous improvement. Teams actively seek feedback, analyze data, and iterate on processes, tools, and workflows to enhance efficiency, security, and quality. Lessons learned from previous iterations are incorporated into future development cycles.
The benefits of continuous feedback in DevSecOps include faster detection and resolution of issues, improved security posture, enhanced user satisfaction, increased operational efficiency, and overall improvement of software quality. By leveraging feedback throughout the software development lifecycle, organizations can continuously optimize their processes, products, and security measures, resulting in more reliable and secure applications.
12. Monitoring and Logging: Set up monitoring and logging solutions to gather insights into the performance, availability, and security of the deployed web application. This allows for proactive identification of issues and facilitates rapid troubleshooting.
You can deploy different types of Observability tools to monitor GKE cluster and applications running on containers. Here is an high level example with GKE cluster monitoring with native Google cloud monitoring tools
13. Interacting with kubectl and Verifying Container applications
- Install kubectl
- Setup alias and auto completion — kubectl
alias k=kubectl
complete -o default -F __start_kubectl k4. Verify namespaces, deployments, services, pods, environment contexts, clusters
(base) skondla@skondla1-Mac:skondla $ k config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop
arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop-2 arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop-2 arn:aws:eks:us-east-1:123456789101:cluster/webapp1-demo-shop-2
gke_beautify-ocean-123456_us-central1-a_test gke_beautify-ocean-123456_us-central1-a_test gke_beautify-ocean-123456_us-central1-a_test
gke_beautify-ocean-123456_us-central1-b_staging gke_beautify-ocean-123456_us-central1-b_staging gke_beautify-ocean-123456_us-central1-b_staging
gke_beautify-ocean-123456_us-central1-c_prod gke_beautify-ocean-123456_us-central1-c_prod gke_beautify-ocean-123456_us-central1-c_prod
gke_marvalo-avenger-123456_us-east4-a_flaskapp1-demo-cluster gke_marvalo-avenger-123456_us-east4-a_flaskapp1-demo-cluster gke_marvalo-avenger-123456_us-east4-a_flaskapp1-demo-cluster
* gke_marvalo-avenger-123456_us-east4-a_webapp1-demo-cluster gke_marvalo-avenger-123456_us-east4-a_webapp1-demo-cluster gke_marvalo-avenger-123456_us-east4-a_webapp1-demo-cluster
k3d-k3s-default k3d-k3s-default
(base) skondla@skondla1-Mac:skondla $ k get ns
NAME STATUS AGE
default Active 6d18h
flaskapp1-namespace Active 6d17h
kube-node-lease Active 6d18h
kube-public Active 6d18h
kube-system Active 6d18h
linkerd Active 6d18h
webapp1-namespace Active 6d18h
(base) skondla@skondla1-Mac:skondla $ k get deploy -n webapp1-namespace
NAME READY UP-TO-DATE AVAILABLE AGE
webapp1-demo-shop 3/3 3 3 6d18h
(base) skondla@skondla1-Mac:skondla $ k get pods -n webapp1-namespace
NAME READY STATUS RESTARTS AGE
webapp1-demo-shop-64c59d8bd5-dx2jh 1/1 Running 0 10h
webapp1-demo-shop-64c59d8bd5-ksdd7 1/1 Running 0 10h
webapp1-demo-shop-64c59d8bd5-xxwgl 1/1 Running 0 10h
(base) skondla@skondla1-Mac:skondla $ k get svc -n webapp1-namespace
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
webapp1-service LoadBalancer 10.96.2.150 22.33.44.130 25443:31246/TCP 6d18h
(base) skondla@skondla1-Mac:skondla $ k get sa -n webapp1-namespace
NAME SECRETS AGE
default 0 6d18h
flaskapp1-serviceaccount 0 6d17h
webapp1-serviceaccount 0 6d18h
(base) skondla@skondla1-Mac:skondla $ k get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
gke-webapp1-demo-cluster-default-pool-d9ffc2ac-v0nt Ready <none> 18h v1.25.8-gke.500 10.150.15.227 22.33.44.123 Container-Optimized OS from Google 5.15.89+ containerd://1.6.18
gke-webapp1-demo-cluster-default-pool-f595ee4b-kx18 Ready <none> 10h v1.25.8-gke.500 10.150.15.233 22.44.77.134 Container-Optimized OS from Google 5.15.89+ containerd://1.6.18
(base) skondla@skondla1-Mac:skondla $ k describe pod webapp1-demo-shop-64c59d8bd5-dx2jh -n webapp1-namespace
Name: webapp1-demo-shop-64c59d8bd5-dx2jh
Namespace: webapp1-namespace
Priority: 0
Service Account: webapp1-serviceaccount
Node: gke-webapp1-demo-cluster-default-pool-d9ffc2ac-v0nt/10.150.15.227
Start Time: Sun, 18 Jun 2023 07:36:01 -0400
Labels: app=webapp1-demo-shop
pod-template-hash=64c59d8bd5
Annotations: <none>
Status: Running
IP: 10.92.1.11
IPs:
IP: 10.92.1.11
Controlled By: ReplicaSet/webapp1-demo-shop-64c59d8bd5
Containers:
webapp1-demo-shop:
Container ID: containerd://97433fa7a67eb9cc653cb94b4f93afc440d1a4d016058f54d8564038a78547f3
Image: us-east4-docker.pkg.dev/marvalo-avenger-123456/marvalo-avenger-123456/webapp1-demo-shop:fe9b5a6a90c0f026c7d080cea968c7ec15a539f1
Image ID: us-east4-docker.pkg.dev/marvalo-avenger-123456/marvalo-avenger-123456/webapp1-demo-shop@sha256:ba037763e85c0ee9c9dd5b33b9401cdc4922ae37e2c44a2a2f85df3aa7eaa620
Port: 25443/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 18 Jun 2023 07:36:36 -0400
Ready: True
Restart Count: 0
Environment:
PORT: 25443
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bsxnw (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-bsxnw:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
(base) skondla@skondla1-Mac:skondla $ k logs webapp1-demo-shop-64c59d8bd5-dx2jh -n webapp1-namespace
/app/dbWebAPI.py:5: DeprecationWarning: 'flask.escape' is deprecated and will be removed in Flask 2.4. Import 'markupsafe.escape' instead.
from flask import Flask, request, render_template, jsonify, Response, escape
* Serving Flask app 'dbWebAPI'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on all addresses (0.0.0.0)
* Running on https://127.0.0.1:25443
* Running on https://10.92.1.11:25443
Press CTRL+C to quit5. Checking WebApp service API endpoints
(base) skondla@skondla1-Mac:skondla $ curl -Lk https://`kubectl get svc -n $GKE_NAMESPACE | grep $GKE_SERVICE | awk '{print $4}'`:$GKE_APP_PORT/backup/status
<h3>Check database Backup</h3>
<form method="POST">
Snapshot Name:<br>
<input type="text" name="snapshotname" size="85" style="color:Tomato;"><br>
EndPoint Name:<br>
<input type="text" name="endpoint" size="85" style="color:Tomato;"> <br><br>
<input type="submit" value="Submit">
<input type="reset">
</form>(base) skondla@skondla1-Mac:skondla $ curl -Lk https://`kubectl get svc -n $GKE_NAMESPACE | grep $GKE_SERVICE | awk '{print $4}'`:$GKE_APP_PORT/backup/create
<h3>Create database Backup</h3>
<form method="POST">
EndPoint Name:<br>
<input type="text" name="endpoint" size="85" style="color:Tomato;"> <br><br>
<input type="submit" value="Submit">
<input type="reset">
</form>(base) skondla@skondla1-Mac:skondla $ curl -Lk https://`kubectl get svc -n $GKE_NAMESPACE | grep $GKE_SERVICE | awk '{print $4}'`:$GKE_APP_PORT/backup/delete
<h3>Delete database Backup</h3>
<form method="POST">
Snapshot Name:<br>
<input type="text" name="snapshotname" size="85" style="color:Tomato;"><br>
EndPoint Name:<br>
<input type="text" name="endpoint" size="85" style="color:Tomato;"> <br><br>
<input type="submit" value="Submit">
<input type="reset">
</form>By combining DevSecOps principles with Google Cloud’s GKE cluster and GitHub Actions, organizations can achieve a highly automated, secure, and efficient deployment process for their web applications. This approach enhances collaboration, reduces manual effort, and ensures that security is integrated throughout the software development lifecycle.
