latest part 4

2021-02-15 17:03:31 +00:00 · 2021-02-15 17:03:31 +00:00 · e51419baa7
commit e51419baa7
parent 9a2b8efdd9
2 changed files with 252 additions and 27 deletions
--- a/Part_4.md
+++ b/Part_4.md
@ -628,29 +628,255 @@ Alerting Policy IAM Roles

 #### Section Review

+Monitoring Your Operations
+
+- Cloud Monitoring Concepts
+- Monitoring Workspaces
+- What are Metrics?
+- Exploring Workspaces and Metrics
+- Monitoring Agent
+- Monitoring API and CLI usage
+- GKE Metrics - Master to individual containers
+- Uptime Checkes
+- Establishing Human-Actionable and Automated Alerts
+
 #### Milestone: Spies Everywhere! (Check Those Vitals!)

-Hands-On Lab:
-Install and Configure Monitoring Agent with Google Cloud Monitoring
-Logging Activities
-Section Introduction
-Cloud Logging Fundamentals
-Log Types and Mechanics
-Cloud Logging Tour
-Logging Agent Concepts
-Install Logging Agent and Collect Agent Logs
-Logging Filters
-Hands-On with Advanced Filters
-VPC Flow Logs
-Firewall Logs
-VPC Flow Logs and Firewall Logs Demo
-Routing and Exporting Logs
-Export Logs to BigQuery
-Logs-Based Metrics
-Section Review
-Milestone: Let the Record Show
-Hands-On Lab:
-Install and Configure Logging Agent on Google Cloud
+#### Hands-On Lab: Install and Configure Monitoring Agent with Google Cloud Monitoring
+
+### Logging Activities
+
+#### Section Introduction
+
+Logging Activities: See next headings
+
+#### Cloud Logging Fundamentals
+
+What is Cloud Logging?
+- Cloud Operations service for storing, viewing, and interacting with logs:
+  - Reading and writing logs entries
+  - Query logs
+  - Export to other services (internal to GCP and external)
+  - Create metrics from logs
+- Interact with Logs Viewer and API
+- Multiple log types available
+- Logs used by other Cloud Operations services (debug, error reporting, etc)
+
+What is a log?
+- Record of status or event (string format)
+  - "What happened?"
+- Log Entry - individual logs in a collection
+- Log Payload - contents of the Log Entry
+  - Contains nested Fields
+
+Logs Retention Period
+- Varies by log type:
+  - Admin Activity, System Event, Access Transparency
+    - 400 days
+    - Non-configurable
+  - All other log types:
+    - 30 days by default
+    - Configurable retention period
+
+IAM Roles
+- Generic and service account varieties
+- Service Account:
+  - Logs Writer: Write logs, no view permissions
+  - Logs Bucket Writer: Write logs to logs buckets
+- Logs Viewer - View logs except Data Access/Access Transparency (known as private logs)
+- Private Logs Viewer - View all of the above
+- Logs Configuration Writer - Create logs-based metrics, buckets, views and export sinks
+  - 'Change configruations'
+- Logging Admin - Full access to all logging actions
+- Project Viewer - View all logs except Data Access/Access Transparency
+- Project Editor 0 Write, view and delete logs. Create logs-based metrics
+  - Cannot create export sinks or view Data Access/Access Transparency logs
+- Project Owner - all logging-based permissions
+
+#### Log Types and Mechanics
+
+<u>Scope of Collecting and Viewing Logs</u>
+- Scoped by project
+- View `project-1` logs in `project-1`
+- No built-in "single pane of glass"
+- Can export logs org-wide or multiple projects
+
+<u>Log Types - Primary Categories</u>
+Security Logs vs. Non-security Logs
+Always Enabled (non-configurable) vs. Mnaually Enabled (configurable):
+- Always Enabled/REquired
+  - No change
+  - 400 days retention
+- Manually Enabled logs
+  - Charged based on log amount
+  - 30 days retention (configurable)
+Above categories overlap
+
+<u>Security Logs</u>
+
+Audit logs and Access transparency logs
+- "Who did what? where? and when?"
+- Also accessible via Activity Log
+
+Admin Activity | System Event | Data Access
+
+Admin Activity
+- Records user-initiated resource configuration
+- "GCE instance created by (user)"
+- "GCS Bucket deleted by (user)"
+- Always Enabled
+
+System Event
+- Admin (non-user) initiated configuration calls
+- Always Enabled
+
+Data Access
+- Record configuration (create/modify/read) of resource data
+- "Object (x) was created in bucket (y) by (users)"
+- Must be manually enabled (except BigQuery)
+- Not applicable to public resources
+
+<u>Access Transparency Logs</u>
+- Only applicable for Enterprise or paid support plans
+- Logs og Google personnel access to your resources/data
+  - Example: Support request for VM instance
+  - Records action and access of support personnel
+- Always Enabled for applicable support plans
+
+| Log Type            | System or User configured | Records what?               | Default Setting                              |
+| ------------------- | ------------------------- | --------------------------- | -------------------------------------------- |
+| Admin Activity      | User-initiated            | Resource Configuration      | Always Enabled                               |
+| System Event        | System-initiated          | Resource Configuration      | Always Enabled                               |
+| Data Access         | User-initiated            | Resource Data Configuration | Manually Eanble                              |
+| Access Transparency | User-initiated            | Google personnel access     | Always Enabled (on applicable support plans) |
+
+<u>'Everything Else' Logs</u>
+
+Logs to Debug, Monitor and Troubleshoot:
+- Chargeable
+- User Logs - generated by software/applications
+  - Require Logging Agent
+- Platform logs - logs generated by GCP services
+  - Example: GCE startup script
+- VPC Flow Logs
+- Firewall Logs
+
+<u>Logs Pricing and Retention</u>
+
+- Always Enabled logs have no charge with 400 days retention
+  - Admin Activity, System Event, Access Transparency
+- ALL other logs are chargeable with configurable retention period (default 30 days)
+- Pricing = $0.50/GB
+
+#### Cloud Logging Tour
+
+Data access logs - Add, edit, view object in a bucket
+- Enabled through IAM > Audit Logs
+  - To enable on single service, find the service e.g. Google Cloud Storage, tick Admin Read, Data Read, Data Write
+    - Can add exempted users e.g. Admin user
+
+#### Logging Agent Concepts
+
+- Agent captures additional VM logs
+  - OS logs/events
+  - 3rd Party application logs
+- Logging agent-based on fluentd (open source data collector)
+- Only applicable to GCE and EC2 (AWS)
+  - GKE uses Cloud Operations for GKE
+
+Configuring the Agent
+- Per Google: The "out of the box" setup covers most use cases
+- Default installation/configuration covers:
+  - OS Logs
+    - Linux - syslog
+    - Windows - Event viewer
+  - Multiple 3rd party applications e.g. Apache, nginx, redis, rabbitmq, gitlab, jenkins, cassandra etc
+
+<u>Modifying Agent Logs Before Submission</u>
+- Why modify logs?
+  - Remove sensitive data
+  - Reformat log fields (e.g. conbine two fields into one)
+- Additional configuration "plug-ins" can modify records
+- `filter_record_transformer` - most common
+  - Add/modify/delete fields from logs
+
+Agent Setup Process
+- Add Repo (via provided script)
+- Update repos
+- Install Logging Agent
+- Install configuration files
+- Start the agent
+
+#### Install Logging Agent and Collect Agent Logs
+
+```
+curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
+sudo bash add-logging-agent-repo.sh
+sudo apt update
+sudo apt-get install google-fluentd
+sudo apt install -y google-fluentd-catch-all-config
+sudo service google-fluentd start
+```
+
+#### Logging Filters
+
+<u>Logs Viewer Query Interface</u>
+
+- View logsa through queries
+- Basic and Advanced query interface
+- Basic
+  - Dropdown menus - simple searches
+- Advanced
+  - View across log categories - advanced search capabilities
+
+<u>Basic and Advanced Filter Queries
+- Different query formats
+  - Search field syntax fifferent for each method
+- Basic query
+  - Not case-sensitive
+  - Built in field names for some logs
+
+<u>Advanced Filter Boolean Operators</u>
+- Group/Exclude entries
+  - AND requires all conditions are met
+  - OR requires only one condition to be met
+  - NOT excludes condition
+- Order of precendence (i.e. order of operations)
+  - NOT -> OR -> AND
+  - a OR NOT b AND NOT c OR d = (a OR (NOT B)) AND ((NOT C) OR d)
+  - AND is implied
+
+<u>Constructing Advanced Fitler Queries</u>
+- Generic text search = just type requested string
+- Searching fields
+  - Nested JSON format
+    - resource.type="gce_instance"
+    - resource.labels.zone="us-central1-a"
+- Search by set severity or greater
+  - `severity >= WARNING`
+- Filter by timestamp
+  - `timestamp>="2018-12-31T00:00:00Z" AND timestamp<="2019-01-01T00:00:00Z"`
+
+#### Hands-On with Advanced Filters
+
+#### VPC Flow Logs
+
+#### Firewall Logs
+
+#### VPC Flow Logs and Firewall Logs Demo
+
+#### Routing and Exporting Logs
+
+#### Export Logs to BigQuery
+
+#### Logs-Based Metrics
+
+#### Section Review
+
+#### Milestone: Let the Record Show
+
+#### Hands-On Lab: Install and Configure Logging Agent on Google Cloud
+
 SRE and Alerting Policies
 SLOs and Alerting Strategy
 Service Monitoring
@ -664,8 +890,7 @@ Setting Up Your App with Cloud Profiler
 Analyzing Cloud Profiler Data
 Section Review
 Milestone: It All Adds Up!
-Hands-On Lab:
-Discovering Latency with Google Cloud Trace
+Hands-On Lab: Discovering Latency with Google Cloud Trace
 Identifying Application Errors with Debug/Error Reporting
 Section Introduction
 Troubleshooting with Cloud Debugger
@ -673,8 +898,7 @@ Establishing Error Reporting for Your App
 Managing Errors and Handling Notifications
 Section Review
 Milestone: Come Together - Reprise (Debug Is De Solution)
-Hands-On Lab:
-Correcting Code with Cloud Debugger
+Hands-On Lab: Correcting Code with Cloud Debugger
 Course Conclusion
 Milestone: Are We There, Yet?
 landscape
--- a/qwiklabs_1.md
+++ b/qwiklabs_1.md
@ -91,6 +91,7 @@ How do you set SLO's for your SLI's?
 - Example: Say our target SLO is 99% of requests will be served within 300ms in the last four weeks, when we measure our SLI we see that only 95% of requests were served within 300ms in the past four weeks thereby missing our target SLO


-
-
+Edge Cases
+- e.g. Black Friday
+-