From e51419baa7bd9b7275f78e9e1ccf900ad18cf0a3 Mon Sep 17 00:00:00 2001 From: Alex Soul Date: Mon, 15 Feb 2021 17:03:31 +0000 Subject: [PATCH] latest part 4 --- Part_4.md | 274 +++++++++++++++++++++++++++++++++++++++++++++----- qwiklabs_1.md | 5 +- 2 files changed, 252 insertions(+), 27 deletions(-) diff --git a/Part_4.md b/Part_4.md index 61273b8..dc70eb6 100644 --- a/Part_4.md +++ b/Part_4.md @@ -628,29 +628,255 @@ Alerting Policy IAM Roles #### Section Review +Monitoring Your Operations + +- Cloud Monitoring Concepts +- Monitoring Workspaces +- What are Metrics? +- Exploring Workspaces and Metrics +- Monitoring Agent +- Monitoring API and CLI usage +- GKE Metrics - Master to individual containers +- Uptime Checkes +- Establishing Human-Actionable and Automated Alerts + #### Milestone: Spies Everywhere! (Check Those Vitals!) -Hands-On Lab: -Install and Configure Monitoring Agent with Google Cloud Monitoring -Logging Activities -Section Introduction -Cloud Logging Fundamentals -Log Types and Mechanics -Cloud Logging Tour -Logging Agent Concepts -Install Logging Agent and Collect Agent Logs -Logging Filters -Hands-On with Advanced Filters -VPC Flow Logs -Firewall Logs -VPC Flow Logs and Firewall Logs Demo -Routing and Exporting Logs -Export Logs to BigQuery -Logs-Based Metrics -Section Review -Milestone: Let the Record Show -Hands-On Lab: -Install and Configure Logging Agent on Google Cloud +#### Hands-On Lab: Install and Configure Monitoring Agent with Google Cloud Monitoring + +### Logging Activities + +#### Section Introduction + +Logging Activities: See next headings + +#### Cloud Logging Fundamentals + +What is Cloud Logging? +- Cloud Operations service for storing, viewing, and interacting with logs: + - Reading and writing logs entries + - Query logs + - Export to other services (internal to GCP and external) + - Create metrics from logs +- Interact with Logs Viewer and API +- Multiple log types available +- Logs used by other Cloud Operations services (debug, error reporting, etc) + +What is a log? +- Record of status or event (string format) + - "What happened?" +- Log Entry - individual logs in a collection +- Log Payload - contents of the Log Entry + - Contains nested Fields + +Logs Retention Period +- Varies by log type: + - Admin Activity, System Event, Access Transparency + - 400 days + - Non-configurable + - All other log types: + - 30 days by default + - Configurable retention period + +IAM Roles +- Generic and service account varieties +- Service Account: + - Logs Writer: Write logs, no view permissions + - Logs Bucket Writer: Write logs to logs buckets +- Logs Viewer - View logs except Data Access/Access Transparency (known as private logs) +- Private Logs Viewer - View all of the above +- Logs Configuration Writer - Create logs-based metrics, buckets, views and export sinks + - 'Change configruations' +- Logging Admin - Full access to all logging actions +- Project Viewer - View all logs except Data Access/Access Transparency +- Project Editor 0 Write, view and delete logs. Create logs-based metrics + - Cannot create export sinks or view Data Access/Access Transparency logs +- Project Owner - all logging-based permissions + +#### Log Types and Mechanics + +Scope of Collecting and Viewing Logs +- Scoped by project +- View `project-1` logs in `project-1` +- No built-in "single pane of glass" +- Can export logs org-wide or multiple projects + +Log Types - Primary Categories +Security Logs vs. Non-security Logs +Always Enabled (non-configurable) vs. Mnaually Enabled (configurable): +- Always Enabled/REquired + - No change + - 400 days retention +- Manually Enabled logs + - Charged based on log amount + - 30 days retention (configurable) +Above categories overlap + +Security Logs + +Audit logs and Access transparency logs +- "Who did what? where? and when?" +- Also accessible via Activity Log + +Admin Activity | System Event | Data Access + +Admin Activity +- Records user-initiated resource configuration +- "GCE instance created by (user)" +- "GCS Bucket deleted by (user)" +- Always Enabled + +System Event +- Admin (non-user) initiated configuration calls +- Always Enabled + +Data Access +- Record configuration (create/modify/read) of resource data +- "Object (x) was created in bucket (y) by (users)" +- Must be manually enabled (except BigQuery) +- Not applicable to public resources + +Access Transparency Logs +- Only applicable for Enterprise or paid support plans +- Logs og Google personnel access to your resources/data + - Example: Support request for VM instance + - Records action and access of support personnel +- Always Enabled for applicable support plans + +| Log Type | System or User configured | Records what? | Default Setting | +| ------------------- | ------------------------- | --------------------------- | -------------------------------------------- | +| Admin Activity | User-initiated | Resource Configuration | Always Enabled | +| System Event | System-initiated | Resource Configuration | Always Enabled | +| Data Access | User-initiated | Resource Data Configuration | Manually Eanble | +| Access Transparency | User-initiated | Google personnel access | Always Enabled (on applicable support plans) | + +'Everything Else' Logs + +Logs to Debug, Monitor and Troubleshoot: +- Chargeable +- User Logs - generated by software/applications + - Require Logging Agent +- Platform logs - logs generated by GCP services + - Example: GCE startup script +- VPC Flow Logs +- Firewall Logs + +Logs Pricing and Retention + +- Always Enabled logs have no charge with 400 days retention + - Admin Activity, System Event, Access Transparency +- ALL other logs are chargeable with configurable retention period (default 30 days) +- Pricing = $0.50/GB + +#### Cloud Logging Tour + +Data access logs - Add, edit, view object in a bucket +- Enabled through IAM > Audit Logs + - To enable on single service, find the service e.g. Google Cloud Storage, tick Admin Read, Data Read, Data Write + - Can add exempted users e.g. Admin user + +#### Logging Agent Concepts + +- Agent captures additional VM logs + - OS logs/events + - 3rd Party application logs +- Logging agent-based on fluentd (open source data collector) +- Only applicable to GCE and EC2 (AWS) + - GKE uses Cloud Operations for GKE + +Configuring the Agent +- Per Google: The "out of the box" setup covers most use cases +- Default installation/configuration covers: + - OS Logs + - Linux - syslog + - Windows - Event viewer + - Multiple 3rd party applications e.g. Apache, nginx, redis, rabbitmq, gitlab, jenkins, cassandra etc + +Modifying Agent Logs Before Submission +- Why modify logs? + - Remove sensitive data + - Reformat log fields (e.g. conbine two fields into one) +- Additional configuration "plug-ins" can modify records +- `filter_record_transformer` - most common + - Add/modify/delete fields from logs + +Agent Setup Process +- Add Repo (via provided script) +- Update repos +- Install Logging Agent +- Install configuration files +- Start the agent + +#### Install Logging Agent and Collect Agent Logs + +``` +curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh +sudo bash add-logging-agent-repo.sh +sudo apt update +sudo apt-get install google-fluentd +sudo apt install -y google-fluentd-catch-all-config +sudo service google-fluentd start +``` + +#### Logging Filters + +Logs Viewer Query Interface + +- View logsa through queries +- Basic and Advanced query interface +- Basic + - Dropdown menus - simple searches +- Advanced + - View across log categories - advanced search capabilities + +Basic and Advanced Filter Queries +- Different query formats + - Search field syntax fifferent for each method +- Basic query + - Not case-sensitive + - Built in field names for some logs + +Advanced Filter Boolean Operators +- Group/Exclude entries + - AND requires all conditions are met + - OR requires only one condition to be met + - NOT excludes condition +- Order of precendence (i.e. order of operations) + - NOT -> OR -> AND + - a OR NOT b AND NOT c OR d = (a OR (NOT B)) AND ((NOT C) OR d) + - AND is implied + +Constructing Advanced Fitler Queries +- Generic text search = just type requested string +- Searching fields + - Nested JSON format + - resource.type="gce_instance" + - resource.labels.zone="us-central1-a" +- Search by set severity or greater + - `severity >= WARNING` +- Filter by timestamp + - `timestamp>="2018-12-31T00:00:00Z" AND timestamp<="2019-01-01T00:00:00Z"` + +#### Hands-On with Advanced Filters + +#### VPC Flow Logs + +#### Firewall Logs + +#### VPC Flow Logs and Firewall Logs Demo + +#### Routing and Exporting Logs + +#### Export Logs to BigQuery + +#### Logs-Based Metrics + +#### Section Review + +#### Milestone: Let the Record Show + +#### Hands-On Lab: Install and Configure Logging Agent on Google Cloud + SRE and Alerting Policies SLOs and Alerting Strategy Service Monitoring @@ -664,8 +890,7 @@ Setting Up Your App with Cloud Profiler Analyzing Cloud Profiler Data Section Review Milestone: It All Adds Up! -Hands-On Lab: -Discovering Latency with Google Cloud Trace +Hands-On Lab: Discovering Latency with Google Cloud Trace Identifying Application Errors with Debug/Error Reporting Section Introduction Troubleshooting with Cloud Debugger @@ -673,8 +898,7 @@ Establishing Error Reporting for Your App Managing Errors and Handling Notifications Section Review Milestone: Come Together - Reprise (Debug Is De Solution) -Hands-On Lab: -Correcting Code with Cloud Debugger +Hands-On Lab: Correcting Code with Cloud Debugger Course Conclusion Milestone: Are We There, Yet? landscape diff --git a/qwiklabs_1.md b/qwiklabs_1.md index ba66bc5..c6342db 100644 --- a/qwiklabs_1.md +++ b/qwiklabs_1.md @@ -91,6 +91,7 @@ How do you set SLO's for your SLI's? - Example: Say our target SLO is 99% of requests will be served within 300ms in the last four weeks, when we measure our SLI we see that only 95% of requests were served within 300ms in the past four weeks thereby missing our target SLO - - +Edge Cases +- e.g. Black Friday +-