- Taking Code changes (deltas) and automatically shepherding them through the process to become a running system, that the users of that system can actually use.
- CI/CD system is internal software that serves the purpose of managing other software (internal or external)
- CI/CD deploying to Prod e.g. GCP
- Continuously - Why? - We avoid a lot of problems when we do things continuously, or rather we get loads of problems when we don't do things continuously. Higher cost when things are done regularly
[Martin Fowler - Software Guru (Agile, Scrum CI/CD)](https://martinfowler.com/articles/continuousIntegration.html#BenefitsOfContinuousIntegration)
* _"break down the barries between customers and development"_
* _"allows your users to get new features more rapidly"_
* _"more rapid feedback on those features"_
* _"more collaborative in the development cycle"_
* _"dramatically easier to find and remove [bugs]"_
* _"dramatically less bugs, both in production and in process"_
* _"reduced risk"_
#### Relating SRE to CI/CD
Martin Fowler - _"Continuous Integration is a software development practice... [that] allows a team to develop cohesive software more rapidly"_ - Make better software - Faster!
Make Software Faster
- Efficiency
- Not just efficiency
- Changes how you interact
Make Better Software
- Feedback Loops
- Other Stuff
- Leads to "Faster"
_"I think the greatest and most wide ranging benefit of Continuous Integration is reduced risk"_
Many many smaller changes instead of single large changes
_"Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible"_
_"Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time"_
_"[A]utomatic deployment helps both speed up the process and reduce error. It's also a cheap option since it just uses the same capabilities that you use to deploy into test environments"_
Reducing Errors - Requires high quality
Purpose - Make the newly updated and validated codebase available to the users. This is automation of an action. Absolutely requires excellent automated testing in that continuous delivery stage.
Just because this newly updated and validated codebase is available to the users, this does not always mean that the updated functionality will immediately be available to those end users. I would say, in fact, it's ideal if that new functionality is not made available by the deployment... sound strange? Why is this?
Because the risk of making any change is larger when the known impact of that change is larger, then you should try to reduce the impact of the change instead. If you deployed the new code base and you know that there should be no visible impact on the users, then all of your metrics should actually show that right? The evidence of what your users are doing now should corroborate your story and not contradict it. That's just good science, and of course good science underpins good engineering.
**Feature Flags/toggles** - really valuable to this situation
Really valuable tool in toolbox. Feature flags/toggles are a powerful technique allowing teams to modify system behaviour without changing code. Now the point I'm trying to make, is if you can modify behaviour without modifying the code then you can also use these to modify code without modifying behaviour, and this is how you use them to manage risk. When you're doing something new, you build that new thing into the system, and you continuously integrate what you're doing, but to stop those unfinished changes of yours from breaking everyone else, you should put your changed functionality behind a feature flag. Make it so that the default behaviour is unchanged from how it was before you started making your changes. This means that you're not regessing any previous funtionality and you leave your feature flag defaulting to "off" until you are ready to have other people using your new thing, then you turn it on for them. And by doing this you can then manage the roll-out of the feature completely independently from the rollout of the changed code. In fact, your half finished feature will likely have been deployed several times already, but it won't have caused any problems because you managed its impact with a feature flag and each deployment would have gone through your full CI/CD pipeline to validate the quality is still there. But then, when you fully rolled out that new feature and everyone is using it and you're sure you don't want to roll it back then you can simply remove the feature flag to simplify your codebase to remove that little bit of technical debt.
_"If you deploy into production one extra automated capability you should consider is automated roll-back. Bad things do happen from time to time... Being able to automatically revert also reduces a lot of the tension of deployment, encouraging people to deploy more frequently and thus get new features out to users quickly"_
"We engineer both changes and change management systems to reduce risk, and efficiently build great software"
Running System --{Reflag}--> Released Feature --{Ask}--> Feedback --{Triage}--> Idea on Backlog --{Code (v.)}--> Code Change & Pull Request (PR) --{Approve}--> Return to "Codebase"
- Setup Cloud function, pointing to repository that we cloned making sure to specify the sub-directory where the python code is located, create the function and then call it from the cli
- One-way sync updates are normally very quick after an update to a source repo, but the sync can be forced by clicking on the gear icon and clicking 'sync from GitHub'
> Minimum scope (Source repository Writer role) to provide user permissions to clone/edit/commit a repo is by specifying the user at the repo level (clicking gear icon on repo -> Permissions). The user however cannot browse or discover the repo through the webUI, for that the user needs permissions at the project level. To discover add the user at the project level to IAM with "Source Repository Reader" role. This permissions will allow the user read permissions to every other repository within the project.
Service [cloud-run-deploy] revision [cloud-run-deploy-00001-jik] has been deployed and is serving 100 percent of traffic.
Service URL: https://cloud-run-deploy-zvr7ikogba-uc.a.run.a
```
> Use cloud run so you don't have to get into using GKE and get the container spun up as quickly as possible
- Need to give Cloud Build access to the Cloud Run service to access it on our behalf. Complete this @ **Cloud Build -> Settings -> Service Account Permissions -> Cloud Run Admin -> Enabled**
- Now the Cloud Build service account has admin access to the Cloud Run service
Up to this point we are updating source code, then manually submitting to Cloud Build; we want to automate this, and sync from source repository and use triggers
- Create a trigger to automatically start build on code commit
- Use separate build file with specific Dockerfile path
Cloud Build > Triggers > Create Trigger (Name: trigger-build, Event: Push to a branch, Source: build-demo, Branch: master, Build Configuration: Cloud Build configuration file (yaml or json), and specify location e.g. root(/cloud_build/build-run/trigger-build.yaml) of the repo) > Create
> Check the paths within each `cloudbuild.yaml` file for the `args` that are passed. When run locally, `args` path maybe simply the local directory `.` when run remotly from Cloud Build, you need to specify the path within the repo, from the root e.g. `cloud_build/build-run/.`
- Make changes to code, commit, and push, then check Cloud Build History; notice the Source has changed from `Google Cloud Storage` to `build-demo` which is the repository name
> Have a go at implementing the Terraform example: `https://cloud.google.com/solutions/managing-infrastructure-as-code`
> n.b. Been searching for Approval process within Cloud Build e.g. part of the cDelivery part of CI/CD, and it doesn't exist, you have to use a 3rd party tool e.g. spinnaker (google developed)
- Create our authentication service account if not already exists
- Grant the IAM roles necessary to that service account to the rest of our Google Cloud CI/CD services
- Go through process of creating GKE cluster
- Installing Spinnaker on the cluster
- Creating our Redis Instance and our Cloud storage buckets
- Installing all the other Spinnaker bits and pieces e.g. "halyard"
- All other config required to make spinnaker work - Google made simple via script
> After installation completes, check created resources:
- GKE Engine
- Cluster name: spinnaker-1
- 3x node cluster
- Application:
- spinnaker-1
- Compute: 3x worker nodes
- Storage
- GCS Bucket
- Memorystore
- Redis: spinnaker-1
- Cloud Source Repositories
- spinnaker-1-config // contains config files for spinnaker
- Pub/Sub
- Topics > spinnaker-1-notifications-topic
- will be used for example pipeline. Used to connect container registry to spinnaker by setting up a subscription and publisher in which spinnaker will be informed of new containers hitting container registry from which spinnaker will automatically pull the latest version of that container into a staging environment
Connect to Spinnaker - 2x options
- Forward port 8080 to tunnel to Spinnaker from the cloud shell
https://spinnaker-1.endpoints.instance-testing-6452.cloud.goog/ # Got blocked by CC Cisco Umberella rubbish
Can locate the url by Navigating to: Network Services > Load Balancing > Clicking on `*spinnaker-deck-ingress*` > Frontend HTTPS > Certificate (spinnaker-1-managed-cert) > Domain status
> Incase cloud-shell is disconnected, re-launch via: `cloudshell launch-tutorial ~/cloudshell_open/spinnaker-for-gcp/scripts/install/provision-spinnaker.md`
- Expand the "Deploy to Production" pipelines and on the first pipeline to run e.g. the piepline at bottom, next to the rubbish bin icon, click the undo arrow and accept the prompt
> Attestor image sign's the container image that is built via Cloud Build service, if you built and tried to deploy to GKE without the signing the container wouldn't deploy
- Spinnaker includes an additional internal account that spinnaker uses: `gcb-account`
- Cloud Build posts message to `cloud-builds` topic and Spinnaker subscription picks up the message
- Subscription is associated with internal account so it knows when builds are publised and can go up to Container Registry to pull new images and bring into staging pipeline
-`properties` file line 63 `export GCP_PUBSUB_SCRIPTION=`
-`setup.sh` line 256 `GCB_PUBSUB_TOPIC=projects/$PROJECT_ID/topics/cloud-builds`
- line 274 - Script creates new gcloud pub/sub.subscription
- Line 303 - `kubectl``quick-install.yaml`
- Line 153 - Config and install halyard
- Line 349 - `gcp-account` created and associated with pub/sub subscription (line 352)
> This is a modified script that comes as part of the Google Spinnaker sample application demo
_"Every time we push a new commit to our source repository Cloud Build is going to upload each of these YAML (`~/world-gift-art/spinnaker-pipeline/config/prod|staging/namespace|replicaset|service.yaml` files to our Spinnaker bucket, associated with that specific version of our image that Spinnaker is going to use to run `kubectl` commands with in order to manage our GKE cluster"_
-`cloudbuild.yaml`
- First Step in this file, uses an ubuntu Cloud Builder image and runs a script
-`mkdir config-all`
- Copy & upload all (6) the `*.yaml` files into the Spinnaker bucket, which Spinnaker will use to **create?**(no because this is completed in the steps below, maybe to initiatiate the pipelines?) Pipelines (Staging & Production) as it initiates a new namespace, replicaset and load balancer service in both piplines
-`artifacts:`
- Specify location in Cloud Storage Bucket of where to upload the `yaml` files
- Also will:
- Build a new image
- Push image to Container Registry
- Configure application within Spinnaker and Setup Production & Staging Pipelines
```
~/spin app save --application-name world-gift-art --cloud-providers kubernetes --owner-email $IAP_USE
Application save succeeded
```
- Create the 2x Pipelines under the `world-gift-art` application within Spinnaker
```
~/spin pi save -f ~/world-gift-art/spinnaker-pipeline/templates/pipelines/deploystaging.json
~/spin pi save -f ~/world-gift-art/spinnaker-pipeline/templates/pipelines/deployprod.json
```
> The tempates that were used to generate these `json` files must have been exported out of Spinnaker 1st from a Gui developed pipeline because they're way too long/complicated to be handcrafted
```
"triggers": [
{
"attributeConstraints": {
"status": "SUCCESS"
},
"enabled": true,
"expectedArtifactIds": [
"4f4d38de-80c3-4bc1-a807-c565bc4024ee"
],
"payloadConstraints": {},
"pubsubSystem": "google",
"subscriptionName": "gcb-account",
"type": "pubsub"
}
]
}
```
> Note the internal `gcb-account` that is linked with the pubsub subscription that watches the `cloud-build` topic. To inspect within Spinnaker: `Applications > world-gift-art > 'Deploy to Staging' Pipeline > Configure > 'Automated Triggers'`
- Cloud Build is building container image, which was trigger from the `git push`
- Note steps discussed earlier:
- Push Built container: `Pushing gcr.io/cicd-full-demo-f673/world-gift-art:8ea5bdc`
- Copy artifacts (`yaml` files) to GCS: `6 total artifacts uploaded to gs://spinnaker-1-uooncaqt01z9k1dntidt-1612281736/world-gift-art-manifests/8ea5bdc/
`
- Container Registry
- New image with tag & also shows the status of the vulnerability scanning
- Check Pipelines in Spinnaker:
-`Applications > world-gift-art > Pipelines > 'Deploy to Staging' is complete | 'Deploy to Production' is waiting for approval`
- To view published 'Staging' application: `Applications > world-gift-art > Infrastructure > Load Balancers > Status > Ingress (View External IP Address)`
- Test Application using `ingress` ip retrieved earlier:
- Add artwork, and upload an image
- CLick `+` button and notice link failure (Not Found) `http://<ingress_ip>/add`
- Return to the 'home' page and hover over 'Add Artwork` note link is: `http://<ingress_ip>/artworks/add`
- Therefore we have a link error that needs fixing in our codebase
- Do not promote application to Production and terminate the 'Deploy to Production' pipeline using the `x` (Cancel execution)
- Return to Cloud Shell and fix broken link
-`base.html` line 38 update to `/artworks/add`
- Commit and push
- Note application is now fixed, and approve for production
- Note images are persistent:
