12 KiB
Google Cloud CI/CD Pipeline (GCP DevOps Engineer Track Part 3)
Understanding Google Cloud CI/CD Pipelines
Big Picture
- Taking Code changes (deltas) and automatically shepherding them through the process to become a running system, that the users of that system can actually use.
- CI/CD system is internal software that serves the purpose of managing other software (internal or external)
- CI/CD deploying to Prod e.g. GCP
- Continuously - Why? - We avoid a lot of problems when we do things continuously, or rather we get loads of problems when we don't do things continuously. Higher cost when things are done regularly
- Error-Prone Humans
Costs or a software Project
43% in Testing
- Cost of removing an issue goes up dramatically as the phases of the project continue 33% Cost reduction from a good QA system 70% of outages are due to changes in a live system At least 10% (lower bound) cost saving by implementing CI/CD
Martin Fowler - Software Guru (Agile, Scrum CI/CD)
- "break down the barries between customers and development"
- "allows your users to get new features more rapidly"
- "more rapid feedback on those features"
- "more collaborative in the development cycle"
- "dramatically easier to find and remove [bugs]"
- "dramatically less bugs, both in production and in process"
- "reduced risk"
Relating SRE to CI/CD
Martin Fowler - "Continuous Integration is a software development practice... [that] allows a team to develop cohesive software more rapidly" - Make better software - Faster!
Make Software Faster
- Efficiency
- Not just efficiency
- Changes how you interact
Make Better Software
- Feedback Loops
- Other Stuff
- Leads to "Faster"
"I think the greatest and most wide ranging benefit of Continuous Integration is reduced risk"
Many many smaller changes instead of single large changes
Understanding Continuous Integration
What is CI?
"Continuous Integraion is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per dat. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible"
Purpose:
- Ensure the updated codebase is at least functinoal. Not completely broken. - This is a "Quality Bar"
Understanding Continuous Delivery
What is CD?
"Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time"
- Can be a major event
- Making a deployment a big deal is an unesscessary problem.
- Ensure the updated codebase is at least as good as it used to be - so it is releasable
- This is also a "Quality Bar"
Regression - When some functionality broke e.g. a bug specifically when something used to work but now doesn't.
CD is all about preventing regressions
If the new feature doesn't work perfectly, the worst case scenario is that at least it worked like it did before.
Understanding Continuous Deployment
What is C Deployment?
"[A]utomatic deployment helps both speed up the process and reduce error. It's also a cheap option since it just uses the same capabilities that you use to deploy into test environments"
Reducing Errors - Requires high quality
Purpose - Make the newly updated and validated codebase available to the users. This is automation of an action. Absolutely requires excellent automated testing in that continuous delivery stage.
Just because this newly update and validated codebase is available to the users, this does not always mean that the updated functionality will immediately be available to those end users. I would say, in fact, it's ideal if that new functionality is not made available by the deployment... sound strange? Why is this?
Because the risk of making any change is the larger when the known impact of that change is larger, then you should try to reduce the impact of the change instead. If you deployed the new code base and you know that there should be no visible impact on the users, then all of your metrics should actually show that right? The evidence of what your users are doing now should corroborate your story and not contradict it. That's just good science, and of course good science underpins good engineering.
Feature Flags/toggles - really valuable to this situation
Really valuable tool in toolbox. Feature flags/toggles are a powerful technique allowing teams to modify system behaviour without changing code. Now the point I'm trying to make, is if you can modify behaviour without modifying the code then you can also use these to modify code without modifying behaviour, and this is how you use them to manage risk. When you're doing something new, you build that new thing into the system, and you continuously integrate what you're doing, but to stop those unfinished changes of yours from breaking everyone else, you should put your changed functionality behind a feature flag. Make it so that the default behaviour is unchanged from how it was before you started making your changes. This means that you're not regessing any previous funtionality and you leave your feature flag defaulting to "off" until you are ready to have other people using your new thing, then you turn it on for them. And by doing this you can then manage the roll-out of the feature completely independently from the rollout of the changed code. In fact, your half finished feature will likely have been deployed several times already, but it won't have caused any problems because you managed its impact with a feature flag and each deployment would have gone through your full CI/CD pipeline to validate the quality is still there. But then, when you fully rolled out that new feature and everyone is using it and you're sure you don't want to roll it back then you can simply remove the feature flag to simplify your codebase to remove that little bit of technical debt.
"If you deploy into production one extra automated capability you should consider is automated roll-back. Bad things do happen from time to time... Being able to automatically revert also reduces a lot of the tension of deployment, encouraging people to deploy more frequently and thus get new features out to users quickly"
"We engineer both changes and change management systems to reduce risk, and efficiently build great software"
Bringing CI and CD (and CD) Together
|---- "CI" From "Codebase" to "Deployable Build" ----|
Codebase --{Integrate i.e. "Build" (v.)}--> Build (n.) / Artifacts, Containers, compiled code --{Deliver}--> Deployable Build --{Deploy}--> Running System
|---- "CD" From "Build" to "Running System" ----|
- Continuation From "Running System":
Running System --{Reflag}--> Released Feature --{Ask}--> Feedback --{Triage}--> Idea on Backlog --{Code (v.)}--> Code Change & Pull Request (PR) --{Approve}--> Return to "Codebase"
Whole system has come full circle
Continuous Integration on Google Cloud
GCP Continuous Integration Concepts
Challenges of Team-based Software Development
- Multiple Developers working on a single code base and/or related systems:
- Difficult to coordinate updates between multiple developers
- Incompatible updates from different developers = bugs and problems
- Large + infrequent updates = many bugs to fix
Solution: Continuous Integration!
- The 'CI' of CI/CD process
- DevOps culture of submitting smaller code changes, more often:
- Multiple submissions per day
- "Continuously" update code
Benefits:
- Smaller changes = fewer bugs to troubleshoot
- Fast changes = rapid fedback loop
- Rapid feedback loops = quickly resolve issues
- Result: Improved productivity and Return On Investment(ROI)!!
The CI Process - 5x Main components
- Source
- Build
- Test
- Report
- Release
CI Components - How to we make this work?
- Single Shared code repository
- Single source of truth
- Automated Build and Test tool:
- Key word: AUTOMATION
- Feedback loop for bugs
- Artifact Repository
- Release destination for built and tested code for future development
Continuous Integration Tools
| Source Control | Build & Test | Artifact Storage | |
|---|---|---|---|
| GCP | Cloud Source Repositories | Cloud Build | Container Registry |
| 3rd Party | Github, Bitbucket | CircleCI, Jenkins | docker hub, Maven etc |
Developer (Commits) > Source Control (Cloud Source Repositories) > {Trigger Build} > Build & Test (Cloud Build) > {Publish Build & Security Scan} > Artifact Management (Container Registry) > Release
Only manual step is the actual "Commit", everything else is automated through tooling
Cloud Source Repositories
Cloud Source Repository Concepts
| Fully Managed Private Git Repository | Build, test, deploy and debug code right away | | Native GCP Integration | Integrate with Cloud Build, Cloud Functions, Cloud Operations (formerly Stackdriver) and more | | Sync/mirror third party repos | One-way mirror from GitHub/Bitbucket to CSR | | Powerful code search | Google search for your code |
Development Environments and Cross-project access
- Google best practice = separate projects for each environment
- 'prod-project', 'dev-project', 'staging-project', etc
- CSR hosted in single project
- How to access single repo from multiple projects?
- Solution: grant IAM roles to members in other projects
Example: CSR in 'prod-project', accessible by 'John' in 'dev-project'
- Grant IAM access to John in either host project or host repo
- Project level = access to all repos in project
- Repo level = access to individual repo
IAM Roles
- Source repository Admin/Writer/Reader
- Admin: Full access (add roles, create/delete repo)
- Writer: Edit and commit code
- Reader: View only