Completed Part_3, now onto Part_4

.gitignore

@@ -1 +1,2 @@
-csr-demo/
+csr-demo/
+build-demo/

Part_3.md

@@ -356,65 +356,776 @@ To Summarize:
 - Assign Cloud Run Admin role to Cloud Build service account
 - Manually submit build using build file to build, push, and deploy to Cloud Run
 
+```
+gcloud services enable sourcerepo.googleapis.com
+gcloud services enable cloudbuild.googleapis.com
+gcloud services enable run.googleapis.com
+gcloud source repos create build-demo
+gcloud source repos clone build-demo
+git pull https://github.com/linuxacademy/content-gcpro-devops-engineer
+cd cloud_build/build-run/
+export PROJECT_ID=$(gcloud config list --format 'value(core.project)')
+echo $PROJECT_ID
+gcloud builds submit --tag gcr.io/$PROJECT_ID/build-run-image
+```
+
+```
+gcloud run deploy cloud-run-deploy --image gcr.io/$PROJECT_ID/build-run-image --platform managed --region us-central1 --allow-unauthenticated
+Service [cloud-run-deploy] revision [cloud-run-deploy-00001-jik] has been deployed and is serving 100 percent of traffic.
+Service URL: https://cloud-run-deploy-zvr7ikogba-uc.a.run.a
+```
+> Use cloud run so you don't have to get into using GKE and get the container spun up as quickly as possible
+
+- Need to give Cloud Build access to the Cloud Run service to access it on our behalf. Complete this @ **Cloud Build -> Settings -> Service Account Permissions -> Cloud Run Admin -> Enabled**
+  - Now the Cloud Build service account has admin access to the Cloud Run service
+
+```
+gcloud builds submit --config cloudbuild.yaml
+```
 
 #### Hands On - Automate Build with Triggers
 
+- Raw code to deployed container
+
+Up to this point we are updating source code, then manually submitting to Cloud Build; we want to automate this, and sync from source repository and use triggers
+
+- Create a trigger to automatically start build on code commit
+  - Use separate build file with specific Dockerfile path
+- Demonstrate fully automated build with triggers
+  - Only manual action is to commit code, Cloud Build handles the rest
+
+- Create triggers
+
+Cloud Build > Triggers > Create Trigger (Name: trigger-build, Event: Push to a branch, Source: build-demo, Branch: master, Build Configuration: Cloud Build configuration file (yaml or json), and specify location e.g. root(/cloud_build/build-run/trigger-build.yaml) of the repo) > Create
+
+> Check the paths within each `cloudbuild.yaml` file for the `args` that are passed. When run locally, `args` path maybe simply the local directory `.` when run remotly from Cloud Build, you need to specify the path within the repo, from the root e.g. `cloud_build/build-run/.`
+
+- Make changes to code, commit, and push, then check Cloud Build History; notice the Source has changed from `Google Cloud Storage` to `build-demo` which is the repository name
+
+```
+gcloud beta builds triggers create cloud-source-repositories \
+--repo=build-demo \
+--branch-pattern="^master$" \
+--build-config=cloud_build/build-run/trigger-build.yaml
+```
+
 #### Best Practices for Build Performance
 
+How to Build Faster
+
+- Leaner containers
+  - Building application + assembling runtime environment in Dockerfile = very large and slower container
+  - Solution:
+    - Separate app build process and runtime build via cloudbuild file e.g. Terraform runtime?
+    - When possible, use light Docker images such as Alpine
+
+- Using Cache options
+  - Cache container build artifacts in Container Registry or Cloud Storage
+  - Stores and index intermediate layers, which can be re-used in your builds
+
+- Customize virtual machine sizes for build process
+  - Cloud Build uses standard-size high managed VM's to execute builds
+  - For large builds, specify high CPU machine types
+  - 8 and 32 CPU version available
+  - `cloudbuild.yaml` syntax:
+
+```
+    options:
+      machineType: 'N1_HIGHCPU_8'
+```
+
+- Cut out unnescessary bloat
+  -  Avoid uploading unnecessary files
+  -  Include `.gcloudignore` file to reduce upload time
+  -  Typical excluded files:
+     -  `.git`
+     -  `dist`
+     -  `node_modules`
+     -  `vendor`
+     -  `*.jar`
+
 #### Hands On Lab - Establishing a CI/CD Pipeline with Google Cloud
 
 #### Hands On Lab - Triggering a CI/CD Pipeline with Google Cloud Build
 
+> Have a go at implementing the Terraform example: `https://cloud.google.com/solutions/managing-infrastructure-as-code`
+
+> n.b. Been searching for Approval process within Cloud Build e.g. part of the cDelivery part of CI/CD, and it doesn't exist, you have to use a 3rd party tool e.g. spinnaker (google developed)
+
 ### Artifact Management with Container Registry
 
 #### Container Registry and Artifact Registry
 
+> Artifact registry, announced in Spring 2020 and currently in beta, is the eventual successor to Container Registry
+
+Artifact Registry, will store containers, as well as Java and NodeJS packages
+
+Course (and exam) will focus on Container Registry, however we will make updates later if necessary
+
 #### What is Artifact Management?
 
+![CI/CD](./img/ci_cd.png)
+
+Artifact Management Services
+- Docker Hub
+- Container Registry
+- Cloud Storage
+  - Store Java Packages, binaries, tarballs
+    - Point to GCS location (gs://my-bucket)
+  - Upcoming Artifact Repository service will handle above artifacts
+- All above examples can be public or private
+
+Why use Container Registry?
+- Native integration with other GCP services
+- Better access control via GCP IAM (compared to DockerHub)
+
 #### Container Registry Concepts
 
+What is Container Registry?
+- Private GCP-native container image registry
+  - Docker and OCI image formats
+- Native integration with other GCP services
+- Behind the scenes: images stored in Cloud Storage
+  - Container Regiustry is management layer
+- 'Shift Security to the Left'
+  - Think about security early in the CI/CD process
+  - Vulnerability scanning
+  - Binary authorization
+
+Image management syntax
+- Command format when tagging, pushing or pulling images to/from Container Registry
+- `[HOSTNAME]/[PROJECT_ID]/[IMAGE]:[TAG]` (or @digest)
+- `gcr.io/my-project-id/app-image:1.1.1`
+- `gcr.io` = google container registry
+  - Defaults to US continent
+- `us.gcr.io`, `eu.gcr.io`, or `asia.gcr.io` to specify Container Registry storage location
+
+Access Control
+- No direct Container Registry roles
+- Cloud Storage access needed
+  - Push images = Storage Object Viewer (or higher)
+
+Service Account Authentication (to Cloud Storage)
+- 3rd party ('off GCP') or cross-project applications
+  - Jenkins, Spinnaker, etc not on GCP services (e.g. on-prem)
+  - Spinnaker/Jenkins/etc runningon a GKE cluster will authenticate with cluster service account
+  - Grant service account associated with application appropriate Storage IAM role
+- Default Compute Engine Service Account
+  - Grant appropriate scope to Storage
+  - Read or Read/Write scope to storage
+
 #### Container Registry Hands On
 
+To Cover:
+- Manually create and push images into Container Registry:
+- Pull images from Container Registry into Cloud Shell
+- Explore Container Registry options
+
+```
+gcloud services enable containerregistry.googleapis.com
+git clone https://github.com/linuxacademy/content-gcpro-devops-engineer
+cd content-gcpro-devops-engineer/cloud_build/build-run/
+export PROJECT_ID=$(gcloud config list --format 'value(core.project)')
+docker build -t gcr.io/$PROJECT_ID/build-run-image:v1.1 .
+docker push gcr.io/$PROJECT_ID/build-run-image:v1.1
+docker images
+docker image rm <image id>
+docker pull gcr.io/$PROJECT_ID/build-run-image:v1.1
+```
+
 #### Milestone: Changing Gears
 
 ### Continuous Deployment/Delivery Overview
 
 #### GCP Continuous Deployment Concepts
 
+![Learnt So Far](img/learnt_so_far.png)
+
+Purpose of CI/CD - Automate it!
+
+- Less toil
+- Less errors
+- Act quicker
+- Greater ROI
+
+> "Automate all the things"
+
+CI creates and stores containers
+CD deploys our containers
+
+Where do we deplpy our containers?
+
+- K8's!
+  - Full customization and control vs. other services
+- Deploy containers...
+  - ... as pods
+  - ... onto nodes
+  - ... and expose them to the world (service)
+
+- Continuous Delivery/Deployment on GCP automates container deployment to compute services (GKE)
+
 #### Importance of Deployment Automation
 
+The 'manual toil cycle' that we want to avoid:
+
+|                              |                                                             |
+| ---------------------------- | ----------------------------------------------------------- |
+| Solved by CI                 | - Update code                                               |
+|                              | - Build a docker container ('docker build')                 |
+|                              | - Push container to registry ('docker push')                |
+| Need to automate deployments | - In kubernetes, update deployment YAML file                |
+|                              | - Apply deployment YAML ('kubectl apply') - replicaSets etc |
+|                              | - Make sure nothing broke and it's working correctly        |
+|                              | - Do it all again!                                          |
+
+How do K8's deployments work?
+
+- YAML files!
+  - Deploy pods
+  - Updating pods
+  - Deployments often manage ReplicaSets, which maintain a desired number of pods
+  - Managing # of replicas is important when discussing more complex deployment models
+
+> Cloud Build did perform the 'Deployment' part of CI/CD but it's too simplifed e.g.
+
+- Fully replaces current version of pods with newer versions
+- How can we switch between versions of out app?
+- Deploy updates to small subset of users?
+- Roll back if something breaks?
+  - Specialized deployment tools are required
+
 #### Deployment Models
 
+Why do we care?
+
+- Simple deployment model:
+  - Deploy latest application (container) straight to production
+  - Completely replace previous version
+  - Cloud Build can perform simple deployments, but not more complex ones
+
+Complex Deployment Models:
+
+- Blue/Green Deployment (sometimes referred to as Red/Black)
+- Canary Deployment
+
+Blue/Green:
+- Run two identical production environments (with different versions)
+- Only one environment is live
+- Once newer version is completely deployed, switch to newest version
+
+![Blue/Green Deployment Model](./img/blue_green_deployment.png)
+
+Canary:
+- Deploy production update to small subset of users
+- Multiple live app versions, but traffic split between versions
+- Useful for 'risky' updates, exposed to small percentage of users
+- If something goes wrong, redirect all traffic to 'safe' version
+- When 'canary' update is deemed safe, direct all traffic to newer version
+
+![Canary Deployment](./img/canary.png)
+
+Manually managing deployment models? - Update those YAML files!
+
+- appv1-1.yaml
+  - replicas: 10
+  - image: app:v1.1
+- appv1-2.yaml
+  - replicas:1
+  - image: app:v1.2
+- lb-service.yaml
+  - Set selectors for each app version (blue-green deployment)
+
 #### Continuous Deployment Tools
 
+What do we need from a CD tools?
+- Automate deployment process
+- Manage blue-green deployments
+- Switch between app versions (service selectors)
+- Manage replicas between app version (canary)
+- **Provide ability to approve pushing new version to production** - Cloud Build cannot do this
+
+Authenticating CD Tools
+- Use Service Accounts for authentication
+- On premises or on GCP, service account as authentication account
+
+Tools - many choices!
+
+- Jenkins
+- Travis CI
+- Cloud Build
+- **Spinnaker**
+  - Developed by Netflix and Google
+  - Open source, multi-cloud delivery platform
+  - Automate deployments - manually approve moving staging into production
+
+To summarize:
+- Managing complex deployments requires multiple updates
+- Manually updating deployments and services = **Toil**
+- TOIL = BAD, AUTOMATION = GOOD
+- CD tools automate container deployments (even complex deployments), so we don't have to do it ourselves
+- Result: no more cycle of suffering!
+
 ### Spinnaker
 
 #### Spinnaker Concepts
 
+Section Overview:
+
+- Cover exam focused concepts for Spinnaker
+- Spinnaker's place in the CI/CD pipeline
+- Spinnaker's interaction with K8's
+- Install Spinnaker on GCP using Google-provided scripts/GitHub
+- Deploy sample CI/CD pipeline using Google-provided scripts:
+  - We will modify pipeline for out own custom application later
+
+Course focus and scope
+
+- CD tools are very complex
+- We will not go through a full deep dive of Spinnaker
+- We will cover the "need to know" requirements for:
+  - CI/CD integration
+  - Kubernetes interaction
+  - Setting up and using on GCP
+
+Why focus on Spinnaker vs. other CD tools?
+
+- Google's documentation on Dev/Ops favors Spinnaker as cloud-native CD tool
+- Spinnaker codeveloped by Google and Netflix
+- Covered on exam
+
+What is Spinnaker?
+
+- Continuous Delivery tool:
+  - Automated application management and deployment
+- Application management:
+  - View and manage GKE components
+  - Application version, load balancers, etc.
+- Application deployment:
+  - Blue/Green, Canary deployments
+
+How does Spinnaker work?
+
+- Remember those K8's YAML files?
+- Spinnaker does it for you!
+- Create YAML files
+- handles kubectl commands
+- Creates deployments, updates ReplicaSets, create load balancers with services
+- No more manual `'kubectl apply <yaml file>'`
+
+![Spinnaker in CD](img/spinnaker_place_in_cd.png)
+
 #### Hands On - Set Up Spinnaker
 
+- Spinnaker on GCP infrastructure
+- Install Spinnaker using Google-provided GitHub/scripts
+- Next lession will create sample application pipeline from this lesson
+
+Installing Spinnaker - Challenges
+
+> As of course writing? 2019? - There is no native complex Continuous Delivery tool built directly into GCP
+
+- Not a natively integrated service
+- Requires installing 3rd party software on top of K8's
+- Fortunately, Google recently supported install via maintained GutHub
+- Installed with guided Cloud Shell tutorial
+
+![Spinnaker Infrastructure Design](./img/spinnaker_infra_design.png)
+
+Getting Started:
+
+- Google's Spinnaker Guided Install Overview:
+  - https://cloud.google.com/docs/ci-cd/spinnaker/spinnaker-for-gcp
+- Cloud Marketplace solution - launches guided Cloud Shell tutorial from cloned GitHub:
+  - https://console.cloud.google.com/marketplace/details/google-cloud-platform/spinnaker
+- Important: If using Cloud playgrounds/lab, must change default GKE machine type
+
+1. Follow the marketplace (2nd) link - Clone git repo, open cloud shell and follow instructions
+
+```
+git config --global user.email "alexander.soul@computacenter.com"
+git config --global user.name "Alex Soul"
+```
+
+```
+export PROJECT_ID="instance-testing-6452"
+~/cloudshell_open/spinnaker-for-gcp/scripts/install/setup_properties.sh
+```
+
+```
+cloudshell edit \
+    ~/cloudshell_open/spinnaker-for-gcp/scripts/install/properties
+```
+
+Begin the installation:
+```
+~/cloudshell_open/spinnaker-for-gcp/scripts/install/setup.sh
+```
+
+`setup.sh`
+- Enable appropriate API's
+- Create our authentication service account if not already exists
+- Grant the IAM roles necessary to that service account to the rest of our Google Cloud CI/CD services
+- Go through process of creating GKE cluster
+- Installing Spinnaker on the cluster
+- Creating our Redis Instance and our Cloud storage buckets
+- Installing all the other Spinnaker bits and pieces e.g. "halyard"
+- All other config required to make spinnaker work - Google made simple via script
+
+> After installation completes, check created resources:
+
+- GKE Engine
+  - Cluster name: spinnaker-1
+    - 3x node cluster
+  - Application:
+    - spinnaker-1
+- Compute: 3x worker nodes
+- Storage
+  - GCS Bucket
+- Memorystore
+  - Redis: spinnaker-1
+- Cloud Source Repositories
+  - spinnaker-1-config // contains config files for spinnaker
+- Pub/Sub
+  - Topics > spinnaker-1-notifications-topic
+    - will be used for example pipeline. Used to connect container registry to spinnaker by setting up a subscription and publisher in which spinnaker will be informed of new containers hitting container registry from which spinnaker will automatically pull the latest version of that container into a staging environment
+
+Connect to Spinnaker - 2x options
+- Forward port 8080 to tunnel to Spinnaker from the cloud shell
+- Export Spinnaker publicly - 30-60 minutes using identity aware proxy
+
+- Network Services
+  - Load Balancing: ingress
+  - 
+
+clientID: 997580377062-hvnvte2r4kvaq9j3r9abklithamou0se.apps.googleusercontent.com
+secret: 5hCO0_7yyJjpQRo-9M6QIkY-
+
+https://spinnaker-1.endpoints.instance-testing-6452.cloud.goog/ # Got blocked by CC Cisco Umberella rubbish
+Can locate the url by Navigating to: Network Services > Load Balancing > Clicking on `*spinnaker-deck-ingress*` > Frontend HTTPS > Certificate (spinnaker-1-managed-cert) > Domain status
+
+> Incase cloud-shell is disconnected, re-launch via: `cloudshell launch-tutorial ~/cloudshell_open/spinnaker-for-gcp/scripts/install/provision-spinnaker.md`
+
 #### Hands On - Deploy Sample App/Pipeline
 
+![Spinnaker Pipline](./img/spinnaker-pipeline.png)
+
+`halyard` == `cli` for Spinnaker configuration
+`spin` == `cli` for Spinnaker
+`deck` == WebUI for Spinnaker
+
+Remember that any configuration changes you make locally (e.g. adding accounts) must be pushed and applied to your deployment to take effect:
+`~/cloudshell_open/spinnaker-for-gcp/scripts/manage/push_and_apply.sh`
+
+
+> Delete all resources associated with Spinnaker: `~/cloudshell_open/spinnaker-for-gcp/scripts/manage/generate_deletion_script.sh`
+
+![Spinnaker Pipeline](./img/spinnaker_pipeline_run.png)
+
+<br>
+
+![Spinnaker YAML Create & Execute](./img/spinnaker_yaml_handling.png)
+> Spinnaker created and applied appropriate Namespace, Services & ReplicaSets for the application
+
+- To preview staging application, before promotion to production:
+  - Within `deck` > Infrastructure > Load Balancers > Click `HELLOWORLDWEBAPP-STAGING` > Scroll down to `Ingress`, copy IP and paste into browser
+
+![Hello World Staging](./img/helloworld_green_deploy.png)
+
+- Approve deployment to production:
+
+![Spinnaker Approve Prod](./img/spinnaker_approve_prod.png)
+
+- Rollback Prod to Version 1
+  - Expand the "Deploy to Production" pipelines and on the first pipeline to run e.g. the piepline at bottom, next to the rubbish bin icon, click the undo arrow and accept the prompt
+
+![Spinnaker Rollback Prod](./img/spinnaker_rollback_prod.png)
+
+- Final rolled back Prod Environment, now in Green
+
+![Prod Rollback Complete](./img/prod_rolled_back_in_green.png)
+
+- Examine deployed pods
+
+![Deployed Pods](./img/spinnaker_pods_prod_staging.png)
+
 #### Milestone: Are We There, Yet?
 
 ### Securing the Deployment Pipeline
 
 #### Managing Secrets
 
+Section Overview:
+
+- Cover several security-focused concepts
+- Mostly conceptual overviews:
+  - Location in CI/CD pipeline
+  - Why you need it
+
+Problem: How to store and reference sensitive information in K8's
+
+- Pods often need to access sensitive information:
+  - Passwords
+  - SSH Keys
+  - OAuth tokens
+  - What NOT to do: store secrets directly in image code:
+  -   Very insecure
+
+Solution: Secrets Management
+
+- What is a secret?
+  - Object that contains sensitive data (password, token, etc)
+  - Not part of pods, but a pod can reference a secret
+  - Allows secure access to sensitive information w/o needing to modify deployment code CI/CD pipelines
+- Secret formats:
+  - Files in a volume
+  - Container environment variable
+  - kubelet when pulling images for you pod
+
+Secret Management Options:
+
+- Application-layer secrets encryption:
+  - Stored in etcd
+  - Encrypter by Cloud Key Management Service (KMS)
+    - GCP-native encryption key management
+  - Application access at runtime
+- Encrypt secrets in Cloud Storage using custom encryption key
+- Third-party manager (HashiCorp Vault)
+
+Primary Takeaway:
+
+- Do NOT store secrets in pods or source code
+  - Very insecure
+- Use secrets management
+
 #### Container Analysis and Vulnerability Scanning
 
+Problem: Source code used in containers may have vulnerabilities
+- Unknowingly deploying exploitable images
+- "You don't know what you don't know"
+
+Solution: Analyze containers in Container Registry for vulnerabilities
+
+How it works:
+- Enabled in Container Registry Service
+- Scans container as they are added to Container Registry, or continually scans existing images
+- Vulnerability reports accessed either via web console, `gcloud` commands, or rest API's
+
+Considerations
+- By default, continual scanning applies to images 30 days or newer, afterwards considered 'stale':
+  - Time frame can be extended
+- Currently limited to images based on Alpine, CentOS, Debian, RedHat and Ubuntu
+
+> Turn on: Container Registry > Settings > Enable Vulnerability Scanning
+
 #### Binary Authorization
 
+Summary:
+
+- Conceptual overview:
+  - Using Binary Authorization requires Anthos subscription
+- Focus on why we need it, how it's implemented, and how it related to our CI/CD pipeline
+
+Challenge: Controlling which images can run on our GKE cluster
+
+- GKE clusters can run multiple images from multiple sources
+- Images can come from a CI/CD pipeline via a specific GCR image store, but you can also deploy images from other GCR stores to the same cluster
+- Need to 'gate keep' approved images for live deployment, and block images from unapproved sources
+
+How it works
+
+- Binary authorization applied to GKE cluster
+- Apply a policy to govern allowed container images, comprised of rules and exempt images
+- Rules: Constraints that container images must pass before they can be deployed to GKE:
+  - Common rule: Digitally signed attenstation (via attestor container image)
+    - Attentation: Encrypted signature applied to approved images
+    - Can use Cloud KMS to store and use encrypted signature
+- Evalutation mode specifies the constraints
+  - Allow All/Deny All/Require Attestations
+
+![Binary Auth](./img/binary_authorization.png)
+
+<br>
+
+![Binary Auth Build YAML](./img/binary_auth_build_yaml.png)
+
+> Attestor image sign's the container image that is built via Cloud Build service, if you built and tried to deploy to GKE without the signing the container wouldn't deploy
+
 ### Full Development Pipeline
 
 #### Full CI/CD Demo - Concepts
 
+- Cover concepts for full end-to-end CI/CD pipeline
+- Install Spinnaker via Google-provided GitHub
+- Create the rest of our application pipeline
+  - One-click script provided for easy setup
+- Deploy and manage application in our pipeline:
+  - Correcting errors before they hit production
+  - Optimizing Docker images
+
+![Full CI/CD Demo](./img/full_ci_cd_pipeline_demo.png)
+
+
+Big-picture Steps:
+
+- Install Spinnaker via Google-provided scripts
+- Create rest of pipeline from single script (which we will break down in more detail)
+  - Create CSR repository using cloned GitHub
+  - Create Cloud Storage and Datastore Backends
+  - Create `cloudbuild.yaml` file with specific variables
+  - Create Kubernetes YAML files to upload to Spinnaker on each commit
+  - Create Spinnaker application and pipelines
+  - Create Cloud Build trigger
+- Enable Container Registry Vulnerability scanning
+- Push app update via commits, and watch the magic happen!
+
 #### Create Spinnaker Cluster
 
+- Install Spinnaker on a GKE cluster
+- Use same Google-provided guide as Spinnaker section
+  - With a few modifications
+- Optional content - Explore Pub-Sub setup in more details
+
+```
+https://spinnaker-1.endpoints.cicd-full-demo-f673.cloud.goog/
+```
+
+Pub/Sub
+
+- Spinnaker subscribes to `cloud-builds`
+  - Spinnaker includes an additional internal account that spinnaker uses: `gcb-account`
+- Cloud Build posts message to `cloud-builds` topic and Spinnaker subscription picks up the message
+  - Subscription is associated with internal account so it knows when builds are publised and can go up to Container Registry to pull new images and bring into staging pipeline
+
+- `properties` file line 63 `export GCP_PUBSUB_SCRIPTION=`
+- `setup.sh` line 256 `GCB_PUBSUB_TOPIC=projects/$PROJECT_ID/topics/cloud-builds`
+- line 274 - Script creates new gcloud pub/sub.subscription
+- Line 303 - `kubectl` `quick-install.yaml`
+  - Line 153 - Config and install halyard
+  - Line 349 - `gcp-account` created and associated with pub/sub subscription (line 352)
+
 #### Create App Repositories and Pipelines
 
+- Create the rest of the CI/CD pipeline:
+  - CSR, Cloud Build trigger, Cloud Builder file, custom app backends, K8's YAML files for Spinnaker, Container Registry Vulnerability scanning
+  - Setup and explore Spinnaker pipelines in further detail
+  - We will go through script actions in depth
+
+client_id: 672408868773-qi5rtd5ih8d7jeerk28dik728l45tiv6.apps.googleusercontent.com
+secret: M_RgN7kLWgT92XCDkSvtFMXb
+
+**1**  
+- 2x actions in the WebGUI
+  - Setup a region for `Datastore` backend (Cloud Firestore - legacy naming)
+    - `Datastore` > Select Datastore Mode > Choose a Database Location: `europe-west2` > Create Database
+  - Turn on Container Registry vulnerability scanning
+    - Container Registry > Settings > Enable Vulnerability Scanning
+
+**2**
+
+- Within Cloud Shell, pull down the Linux Academy Script
+```
+cd ~
+wget https://raw.githubusercontent.com/linuxacademy/content-gcpro-devops-engineer/master/scripts/create_app_and_pipelines.sh
+```
+> This is a modified script that comes as part of the Google Spinnaker sample application demo
+
+_"Every time we push a new commit to our source repository Cloud Build is going to upload each of these YAML (`~/world-gift-art/spinnaker-pipeline/config/prod|staging/namespace|replicaset|service.yaml` files to our Spinnaker bucket, associated with that specific version of our image that Spinnaker is going to use to run `kubectl` commands with in order to manage our GKE cluster"_
+
+- `cloudbuild.yaml`
+  - First Step in this file, uses an ubuntu Cloud Builder image and runs a script
+    - `mkdir config-all`
+      - Copy & upload all (6) the `*.yaml` files into the Spinnaker bucket, which Spinnaker will use to **create?**(no because this is completed in the steps below, maybe to initiatiate the pipelines?) Pipelines (Staging & Production) as it initiates a new namespace, replicaset and load balancer service in both piplines
+      - `artifacts:`
+        - Specify location in Cloud Storage Bucket of where to upload the `yaml` files
+  - Also will:
+    - Build a new image
+    - Push image to Container Registry
+
+- Configure application within Spinnaker and Setup Production & Staging Pipelines
+```
+~/spin app save --application-name world-gift-art --cloud-providers kubernetes --owner-email $IAP_USE
+Application save succeeded
+``` 
+
+- Create the 2x Pipelines under the `world-gift-art` application within Spinnaker
+```
+~/spin pi save -f ~/world-gift-art/spinnaker-pipeline/templates/pipelines/deploystaging.json
+~/spin pi save -f ~/world-gift-art/spinnaker-pipeline/templates/pipelines/deployprod.json
+```
+> The tempates that were used to generate these `json` files must have been exported out of Spinnaker 1st from a Gui developed pipeline because they're way too long/complicated to be handcrafted
+
+```
+  "triggers": [
+    {
+      "attributeConstraints": {
+        "status": "SUCCESS"
+      },
+      "enabled": true,
+      "expectedArtifactIds": [
+        "4f4d38de-80c3-4bc1-a807-c565bc4024ee"
+      ],
+      "payloadConstraints": {},
+      "pubsubSystem": "google",
+      "subscriptionName": "gcb-account",
+      "type": "pubsub"
+    }
+  ]
+}
+```
+> Note the internal `gcb-account` that is linked with the pubsub subscription that watches the `cloud-build` topic. To inspect within Spinnaker: `Applications > world-gift-art > 'Deploy to Staging' Pipeline > Configure > 'Automated Triggers'`
+
+- Create Cloud Build Trigger
+
+```
+gcloud beta builds triggers create cloud-source-repositories \
+  --repo world-gift-art \
+  --branch-pattern master \
+  --build-config spinnaker-pipeline/cloudbuild.yaml
+```
+
+CI/CD Pipeline is now built and ready to roll
+
 #### Deploy and Manage Application
 
+- Commit code changes, and let pipeline handle the rest
+- Manually approve promoting stagin pipeline to production
+
+**1**
+
+- Edit file: `/home/admin_/world-gift-art/spinnaker-pipeline/worldgiftart/templates/base.html`
+- Commit & push: `git commit -am "First real commit" ; git push`
+
+- Note:
+  - Cloud Build is building container image, which was trigger from the `git push`
+    - Note steps discussed earlier:
+      - Push Built container: `Pushing gcr.io/cicd-full-demo-f673/world-gift-art:8ea5bdc`
+      - Copy artifacts (`yaml` files) to GCS: `6 total artifacts uploaded to gs://spinnaker-1-uooncaqt01z9k1dntidt-1612281736/world-gift-art-manifests/8ea5bdc/
+`
+
+  - Container Registry
+    - New image with tag & also shows the status of the vulnerability scanning
+
+  - Check Pipelines in Spinnaker:
+    - `Applications > world-gift-art > Pipelines > 'Deploy to Staging' is complete | 'Deploy to Production' is waiting for approval`
+      - To view published 'Staging' application: `Applications > world-gift-art > Infrastructure > Load Balancers > Status > Ingress (View External IP Address)`
+
+- Test Application using `ingress` ip retrieved earlier:
+  - Add artwork, and upload an image
+  - CLick `+` button and notice link failure (Not Found) `http://<ingress_ip>/add`
+  - Return to the 'home' page and hover over 'Add Artwork` note link is: `http://<ingress_ip>/artworks/add`
+    - Therefore we have a link error that needs fixing in our codebase
+  - Do not promote application to Production and terminate the 'Deploy to Production' pipeline using the `x` (Cancel execution)
+
+![Cancel Deploy to Production Pipeline](./img/cancel_deploy_to_production.png)
+
+- Return to Cloud Shell and fix broken link
+  - `base.html` line 38 update to `/artworks/add`
+  - Commit and push
+- Note application is now fixed, and approve for production
+- Note images are persistent: 
+
+![Persistent Images using Google Datastore](./img/persistent_images_datastore_backend.png)
+
 #### Milestone: Continuity
 
 #### Hands On Lab: Create a CI/CD Pipeline on Google Cloud with Spinnaker