latest part 4
This commit is contained in:
Alex Soul
parent
9a2b8efdd9
commit
e51419baa7
274
Part_4.md
274
Part_4.md
@ -628,29 +628,255 @@ Alerting Policy IAM Roles
|
|||||||
|
|
||||||
#### Section Review
|
#### Section Review
|
||||||
|
|
||||||
|
Monitoring Your Operations
|
||||||
|
|
||||||
|
- Cloud Monitoring Concepts
|
||||||
|
- Monitoring Workspaces
|
||||||
|
- What are Metrics?
|
||||||
|
- Exploring Workspaces and Metrics
|
||||||
|
- Monitoring Agent
|
||||||
|
- Monitoring API and CLI usage
|
||||||
|
- GKE Metrics - Master to individual containers
|
||||||
|
- Uptime Checkes
|
||||||
|
- Establishing Human-Actionable and Automated Alerts
|
||||||
|
|
||||||
#### Milestone: Spies Everywhere! (Check Those Vitals!)
|
#### Milestone: Spies Everywhere! (Check Those Vitals!)
|
||||||
|
|
||||||
Hands-On Lab:
|
#### Hands-On Lab: Install and Configure Monitoring Agent with Google Cloud Monitoring
|
||||||
Install and Configure Monitoring Agent with Google Cloud Monitoring
|
|
||||||
Logging Activities
|
### Logging Activities
|
||||||
Section Introduction
|
|
||||||
Cloud Logging Fundamentals
|
#### Section Introduction
|
||||||
Log Types and Mechanics
|
|
||||||
Cloud Logging Tour
|
Logging Activities: See next headings
|
||||||
Logging Agent Concepts
|
|
||||||
Install Logging Agent and Collect Agent Logs
|
#### Cloud Logging Fundamentals
|
||||||
Logging Filters
|
|
||||||
Hands-On with Advanced Filters
|
What is Cloud Logging?
|
||||||
VPC Flow Logs
|
- Cloud Operations service for storing, viewing, and interacting with logs:
|
||||||
Firewall Logs
|
- Reading and writing logs entries
|
||||||
VPC Flow Logs and Firewall Logs Demo
|
- Query logs
|
||||||
Routing and Exporting Logs
|
- Export to other services (internal to GCP and external)
|
||||||
Export Logs to BigQuery
|
- Create metrics from logs
|
||||||
Logs-Based Metrics
|
- Interact with Logs Viewer and API
|
||||||
Section Review
|
- Multiple log types available
|
||||||
Milestone: Let the Record Show
|
- Logs used by other Cloud Operations services (debug, error reporting, etc)
|
||||||
Hands-On Lab:
|
|
||||||
Install and Configure Logging Agent on Google Cloud
|
What is a log?
|
||||||
|
- Record of status or event (string format)
|
||||||
|
- "What happened?"
|
||||||
|
- Log Entry - individual logs in a collection
|
||||||
|
- Log Payload - contents of the Log Entry
|
||||||
|
- Contains nested Fields
|
||||||
|
|
||||||
|
Logs Retention Period
|
||||||
|
- Varies by log type:
|
||||||
|
- Admin Activity, System Event, Access Transparency
|
||||||
|
- 400 days
|
||||||
|
- Non-configurable
|
||||||
|
- All other log types:
|
||||||
|
- 30 days by default
|
||||||
|
- Configurable retention period
|
||||||
|
|
||||||
|
IAM Roles
|
||||||
|
- Generic and service account varieties
|
||||||
|
- Service Account:
|
||||||
|
- Logs Writer: Write logs, no view permissions
|
||||||
|
- Logs Bucket Writer: Write logs to logs buckets
|
||||||
|
- Logs Viewer - View logs except Data Access/Access Transparency (known as private logs)
|
||||||
|
- Private Logs Viewer - View all of the above
|
||||||
|
- Logs Configuration Writer - Create logs-based metrics, buckets, views and export sinks
|
||||||
|
- 'Change configruations'
|
||||||
|
- Logging Admin - Full access to all logging actions
|
||||||
|
- Project Viewer - View all logs except Data Access/Access Transparency
|
||||||
|
- Project Editor 0 Write, view and delete logs. Create logs-based metrics
|
||||||
|
- Cannot create export sinks or view Data Access/Access Transparency logs
|
||||||
|
- Project Owner - all logging-based permissions
|
||||||
|
|
||||||
|
#### Log Types and Mechanics
|
||||||
|
|
||||||
|
<u>Scope of Collecting and Viewing Logs</u>
|
||||||
|
- Scoped by project
|
||||||
|
- View `project-1` logs in `project-1`
|
||||||
|
- No built-in "single pane of glass"
|
||||||
|
- Can export logs org-wide or multiple projects
|
||||||
|
|
||||||
|
<u>Log Types - Primary Categories</u>
|
||||||
|
Security Logs vs. Non-security Logs
|
||||||
|
Always Enabled (non-configurable) vs. Mnaually Enabled (configurable):
|
||||||
|
- Always Enabled/REquired
|
||||||
|
- No change
|
||||||
|
- 400 days retention
|
||||||
|
- Manually Enabled logs
|
||||||
|
- Charged based on log amount
|
||||||
|
- 30 days retention (configurable)
|
||||||
|
Above categories overlap
|
||||||
|
|
||||||
|
<u>Security Logs</u>
|
||||||
|
|
||||||
|
Audit logs and Access transparency logs
|
||||||
|
- "Who did what? where? and when?"
|
||||||
|
- Also accessible via Activity Log
|
||||||
|
|
||||||
|
Admin Activity | System Event | Data Access
|
||||||
|
|
||||||
|
Admin Activity
|
||||||
|
- Records user-initiated resource configuration
|
||||||
|
- "GCE instance created by (user)"
|
||||||
|
- "GCS Bucket deleted by (user)"
|
||||||
|
- Always Enabled
|
||||||
|
|
||||||
|
System Event
|
||||||
|
- Admin (non-user) initiated configuration calls
|
||||||
|
- Always Enabled
|
||||||
|
|
||||||
|
Data Access
|
||||||
|
- Record configuration (create/modify/read) of resource data
|
||||||
|
- "Object (x) was created in bucket (y) by (users)"
|
||||||
|
- Must be manually enabled (except BigQuery)
|
||||||
|
- Not applicable to public resources
|
||||||
|
|
||||||
|
<u>Access Transparency Logs</u>
|
||||||
|
- Only applicable for Enterprise or paid support plans
|
||||||
|
- Logs og Google personnel access to your resources/data
|
||||||
|
- Example: Support request for VM instance
|
||||||
|
- Records action and access of support personnel
|
||||||
|
- Always Enabled for applicable support plans
|
||||||
|
|
||||||
|
| Log Type | System or User configured | Records what? | Default Setting |
|
||||||
|
| ------------------- | ------------------------- | --------------------------- | -------------------------------------------- |
|
||||||
|
| Admin Activity | User-initiated | Resource Configuration | Always Enabled |
|
||||||
|
| System Event | System-initiated | Resource Configuration | Always Enabled |
|
||||||
|
| Data Access | User-initiated | Resource Data Configuration | Manually Eanble |
|
||||||
|
| Access Transparency | User-initiated | Google personnel access | Always Enabled (on applicable support plans) |
|
||||||
|
|
||||||
|
<u>'Everything Else' Logs</u>
|
||||||
|
|
||||||
|
Logs to Debug, Monitor and Troubleshoot:
|
||||||
|
- Chargeable
|
||||||
|
- User Logs - generated by software/applications
|
||||||
|
- Require Logging Agent
|
||||||
|
- Platform logs - logs generated by GCP services
|
||||||
|
- Example: GCE startup script
|
||||||
|
- VPC Flow Logs
|
||||||
|
- Firewall Logs
|
||||||
|
|
||||||
|
<u>Logs Pricing and Retention</u>
|
||||||
|
|
||||||
|
- Always Enabled logs have no charge with 400 days retention
|
||||||
|
- Admin Activity, System Event, Access Transparency
|
||||||
|
- ALL other logs are chargeable with configurable retention period (default 30 days)
|
||||||
|
- Pricing = $0.50/GB
|
||||||
|
|
||||||
|
#### Cloud Logging Tour
|
||||||
|
|
||||||
|
Data access logs - Add, edit, view object in a bucket
|
||||||
|
- Enabled through IAM > Audit Logs
|
||||||
|
- To enable on single service, find the service e.g. Google Cloud Storage, tick Admin Read, Data Read, Data Write
|
||||||
|
- Can add exempted users e.g. Admin user
|
||||||
|
|
||||||
|
#### Logging Agent Concepts
|
||||||
|
|
||||||
|
- Agent captures additional VM logs
|
||||||
|
- OS logs/events
|
||||||
|
- 3rd Party application logs
|
||||||
|
- Logging agent-based on fluentd (open source data collector)
|
||||||
|
- Only applicable to GCE and EC2 (AWS)
|
||||||
|
- GKE uses Cloud Operations for GKE
|
||||||
|
|
||||||
|
Configuring the Agent
|
||||||
|
- Per Google: The "out of the box" setup covers most use cases
|
||||||
|
- Default installation/configuration covers:
|
||||||
|
- OS Logs
|
||||||
|
- Linux - syslog
|
||||||
|
- Windows - Event viewer
|
||||||
|
- Multiple 3rd party applications e.g. Apache, nginx, redis, rabbitmq, gitlab, jenkins, cassandra etc
|
||||||
|
|
||||||
|
<u>Modifying Agent Logs Before Submission</u>
|
||||||
|
- Why modify logs?
|
||||||
|
- Remove sensitive data
|
||||||
|
- Reformat log fields (e.g. conbine two fields into one)
|
||||||
|
- Additional configuration "plug-ins" can modify records
|
||||||
|
- `filter_record_transformer` - most common
|
||||||
|
- Add/modify/delete fields from logs
|
||||||
|
|
||||||
|
Agent Setup Process
|
||||||
|
- Add Repo (via provided script)
|
||||||
|
- Update repos
|
||||||
|
- Install Logging Agent
|
||||||
|
- Install configuration files
|
||||||
|
- Start the agent
|
||||||
|
|
||||||
|
#### Install Logging Agent and Collect Agent Logs
|
||||||
|
|
||||||
|
```
|
||||||
|
curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
|
||||||
|
sudo bash add-logging-agent-repo.sh
|
||||||
|
sudo apt update
|
||||||
|
sudo apt-get install google-fluentd
|
||||||
|
sudo apt install -y google-fluentd-catch-all-config
|
||||||
|
sudo service google-fluentd start
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Logging Filters
|
||||||
|
|
||||||
|
<u>Logs Viewer Query Interface</u>
|
||||||
|
|
||||||
|
- View logsa through queries
|
||||||
|
- Basic and Advanced query interface
|
||||||
|
- Basic
|
||||||
|
- Dropdown menus - simple searches
|
||||||
|
- Advanced
|
||||||
|
- View across log categories - advanced search capabilities
|
||||||
|
|
||||||
|
<u>Basic and Advanced Filter Queries
|
||||||
|
- Different query formats
|
||||||
|
- Search field syntax fifferent for each method
|
||||||
|
- Basic query
|
||||||
|
- Not case-sensitive
|
||||||
|
- Built in field names for some logs
|
||||||
|
|
||||||
|
<u>Advanced Filter Boolean Operators</u>
|
||||||
|
- Group/Exclude entries
|
||||||
|
- AND requires all conditions are met
|
||||||
|
- OR requires only one condition to be met
|
||||||
|
- NOT excludes condition
|
||||||
|
- Order of precendence (i.e. order of operations)
|
||||||
|
- NOT -> OR -> AND
|
||||||
|
- a OR NOT b AND NOT c OR d = (a OR (NOT B)) AND ((NOT C) OR d)
|
||||||
|
- AND is implied
|
||||||
|
|
||||||
|
<u>Constructing Advanced Fitler Queries</u>
|
||||||
|
- Generic text search = just type requested string
|
||||||
|
- Searching fields
|
||||||
|
- Nested JSON format
|
||||||
|
- resource.type="gce_instance"
|
||||||
|
- resource.labels.zone="us-central1-a"
|
||||||
|
- Search by set severity or greater
|
||||||
|
- `severity >= WARNING`
|
||||||
|
- Filter by timestamp
|
||||||
|
- `timestamp>="2018-12-31T00:00:00Z" AND timestamp<="2019-01-01T00:00:00Z"`
|
||||||
|
|
||||||
|
#### Hands-On with Advanced Filters
|
||||||
|
|
||||||
|
#### VPC Flow Logs
|
||||||
|
|
||||||
|
#### Firewall Logs
|
||||||
|
|
||||||
|
#### VPC Flow Logs and Firewall Logs Demo
|
||||||
|
|
||||||
|
#### Routing and Exporting Logs
|
||||||
|
|
||||||
|
#### Export Logs to BigQuery
|
||||||
|
|
||||||
|
#### Logs-Based Metrics
|
||||||
|
|
||||||
|
#### Section Review
|
||||||
|
|
||||||
|
#### Milestone: Let the Record Show
|
||||||
|
|
||||||
|
#### Hands-On Lab: Install and Configure Logging Agent on Google Cloud
|
||||||
|
|
||||||
SRE and Alerting Policies
|
SRE and Alerting Policies
|
||||||
SLOs and Alerting Strategy
|
SLOs and Alerting Strategy
|
||||||
Service Monitoring
|
Service Monitoring
|
||||||
@ -664,8 +890,7 @@ Setting Up Your App with Cloud Profiler
|
|||||||
Analyzing Cloud Profiler Data
|
Analyzing Cloud Profiler Data
|
||||||
Section Review
|
Section Review
|
||||||
Milestone: It All Adds Up!
|
Milestone: It All Adds Up!
|
||||||
Hands-On Lab:
|
Hands-On Lab: Discovering Latency with Google Cloud Trace
|
||||||
Discovering Latency with Google Cloud Trace
|
|
||||||
Identifying Application Errors with Debug/Error Reporting
|
Identifying Application Errors with Debug/Error Reporting
|
||||||
Section Introduction
|
Section Introduction
|
||||||
Troubleshooting with Cloud Debugger
|
Troubleshooting with Cloud Debugger
|
||||||
@ -673,8 +898,7 @@ Establishing Error Reporting for Your App
|
|||||||
Managing Errors and Handling Notifications
|
Managing Errors and Handling Notifications
|
||||||
Section Review
|
Section Review
|
||||||
Milestone: Come Together - Reprise (Debug Is De Solution)
|
Milestone: Come Together - Reprise (Debug Is De Solution)
|
||||||
Hands-On Lab:
|
Hands-On Lab: Correcting Code with Cloud Debugger
|
||||||
Correcting Code with Cloud Debugger
|
|
||||||
Course Conclusion
|
Course Conclusion
|
||||||
Milestone: Are We There, Yet?
|
Milestone: Are We There, Yet?
|
||||||
landscape
|
landscape
|
||||||
|
|||||||
@ -91,6 +91,7 @@ How do you set SLO's for your SLI's?
|
|||||||
- Example: Say our target SLO is 99% of requests will be served within 300ms in the last four weeks, when we measure our SLI we see that only 95% of requests were served within 300ms in the past four weeks thereby missing our target SLO
|
- Example: Say our target SLO is 99% of requests will be served within 300ms in the last four weeks, when we measure our SLI we see that only 95% of requests were served within 300ms in the past four weeks thereby missing our target SLO
|
||||||
|
|
||||||
|
|
||||||
|
Edge Cases
|
||||||
|
- e.g. Black Friday
|
||||||
|
-
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user