Can We Upload Exisiting Log Files to Stackdriver

This page explains how Cloud Logging processes log entries, and describes the key components of Logging routing and storage.

At a high level, this is how Deject Logging routes and stores log entries:

Ingesting and routing logs with the Log Router

The following sections explain how logs are ingested past Logging and routed through the Log Router using sinks.

Log Router

A log entry is sent to the Google Cloud resource specified in its logName field during its entries.write call.

Cloud Logging receives log entries through the Cloud Logging API where they pass through the Log Router. The sinks in the Log Router bank check each log entry against the existing inclusion filter and exclusion filters that decide which destinations, including Cloud Logging buckets, that the log entry should be sent to. Y'all can use combinations of sinks to route logs to multiple destinations.

To reliably route logs, the Log Router also stores the logs temporarily (not depicted in the epitome), which buffers against temporary disruptions on any sink. Note that the Log Router's temporary storage is distinct from the longer term storage provided by Logging buckets.

Sinks

Sinks control how Cloud Logging routes logs. Using sinks, you tin route some or all of your logs to supported destinations. Some of the reasons that you might want to control how your logs are routed include the post-obit:

• To store logs that are unlikely to be read but that must be retained for compliance purposes.
• To organize your logs in buckets in a format that is useful to you.
• To use big-information analysis tools on your logs.
• To stream your logs to other applications, other repositories, or tertiary parties.

Sinks belong to a given Google Deject resource: Cloud projects, billing accounts, folders, and organizations. When the resource receives a log entry, it routes the log entry co-ordinate to the sinks contained by that resources and, if enabled, any ancestral sinks belonging under the resources hierarchy. The log entry is sent to the destination associated with each matching sink.

Deject Logging provides two predefined sinks for each Deject project, billing account, folder, and organization: _Required and _Default. All logs that are generated in a resource are automatically candy through these two sinks and then are stored either in the correspondingly named _Required or _Default buckets.

Sinks act independently of each other. Regardless of how the predefined sinks process your log entries, yous tin can create your own sinks to route some or all of your logs to various supported destinations or to exclude them from existence stored by Cloud Logging.

The routing behavior for each sink is controlled by configuring the inclusion filter and exclusion filters for that sink. Depending on the sink'southward configuration, every log entry received by Deject Logging falls into one or more of these categories:

• Stored in Cloud Logging and not routed elsewhere.

• Stored in Cloud Logging and routed to a supported destination.

• Not stored in Cloud Logging but routed to a supported destination.

• Neither stored in Deject Logging nor routed elsewhere.

You lot usually create sinks at the Cloud project level, simply if you lot want to combine and road logs from the resources contained by a Google Cloud organisation or folder, you can create aggregated sinks.

Y'all tin't route log entries that Logging received earlier your sink was created considering routing happens as logs pass through the Logging API, and new routing rules merely apply to logs written after those rules have been created. If you need to road log entries retroactively, see Re-create logs.

Inclusion filters

For whatsoever new sink, if you don't specify filters, all logs lucifer and are routed to the sink'due south destination. You tin configure the sink to select specific logs by setting an inclusion filter. You lot can also set one or more exclusion filters to exclude logs from the sink'due south destination.

When y'all configure sinks, you create inclusion filters by using the Logging query language. Sinks can also contain multiple exclusion filters.

Every log entry received by Logging is routed based on these filtering rules:

• The sink'south exclusion filters override any of its defined inclusion filters. If a log matches any exclusion filter in the sink, then it doesn't match the sink regardless of any inclusion filters defined. The log entry isn't routed to that sink's destination.

• If the sink doesn't incorporate an inclusion filter, then the following happens:

  • If the log entry matches any exclusion filter, it isn't routed to the sink'southward destination.
  • If the log entry doesn't match any exclusion filter, it is routed to the sink's destination. An empty inclusion filter selects all logs.

• If the sink contains an inclusion filter, then the post-obit happens:

  • If the log entry matches the inclusion filter, it is routed to the sink's destination.
  • If the log entry doesn't match the inclusion filter, information technology isn't routed to the sink's destination.

Exclusion filters

When you create a sink, you tin fix multiple exclusion filters, letting you lot exclude matching log entries from being routed to the sink's destination or from beingness ingested by Deject Logging. You create exclusion filters past using the Logging query language.

Logs are excluded after they are received by the Logging API. Therefore, excluding logs doesn't reduce the number of entries.write API calls.

Excluded log entries aren't bachelor in the Logs Explorer or Deject Debugger.

Log entries that aren't routed to at least one log bucket, either explicitly with exclusion filters or considering they don't lucifer whatever sinks with a Logging storage destination, are also excluded from Error Reporting.

User-defined log-based metrics are computed from log entries in both included and excluded logs. For more data, see Monitor your logs.

Supported destinations

You lot can utilise the Log Router to route sure logs to supported destinations in any Cloud project. Logging supports the following sink destinations:

• Cloud Storage: JSON files stored in Deject Storage buckets; provides cheap, long-term storage.
• BigQuery: Tables created in BigQuery datasets; provides big information analysis capabilities.
• Pub/Sub: JSON-formatted letters delivered to Pub/Sub topics; supports third-party integrations, such every bit Splunk, with Logging.
• Cloud Logging: Log entries held in log buckets; provides storage in Cloud Logging with customizable retention periods.

For more than information on routing logs to supported destinations, see Configure sinks.

Storing, viewing, and managing logs

The following section details how logs are stored in Cloud Logging, and how you tin can view and manage them.

Log buckets

Cloud Logging uses log buckets as containers in your Google Cloud projects, billing accounts, folders, and organizations to store and organize your logs data. The logs that yous store in Cloud Logging are indexed, optimized, and delivered to let you lot clarify your logs in real fourth dimension. Cloud Logging buckets are different storage entities than the similarly named Cloud Storage buckets.

For each Cloud project, billing account, folder, and organization, Logging automatically creates two log buckets: _Required and _Default. Logging automatically creates sinks named _Required and _Default that, in the default configuration, route logs to the correspondingly named buckets.

Yous can disable the Default sink, which routes logs to the _Default log bucket. To change the beliefs of Default sinks created for any new Cloud projects or folders created in your system, you lot can configure default settings for your organization.

Y'all can't change routing rules for the _Required bucket.

Additionally, you can create user-defined buckets for any Cloud project.

You create sinks to road all, or just a subset, of your logs to any log bucket. This flexibility allows yous to cull the Cloud project in which your logs are stored and what other logs are stored with them.

For more information, see Configure log buckets.

_Required log bucket

Cloud Logging automatically routes the following types of logs to the _Required bucket:

• Admin Activity audit logs
• System Event audit logs
• Access Transparency logs

Deject Logging retains the logs in this bucket for 400 days; you lot tin't change this retention period.

You can't modify or delete the _Required saucepan. You can't disable the _Required sink, which routes logs to the _Required bucket.

Neither ingestion pricing nor storage pricing applies to the logs data stored in the _Required log saucepan.

_Default log bucket

Any log entry that isn't ingested past the _Required bucket is routed by the _Default sink to the _Default saucepan, unless you lot disable or otherwise edit the _Default sink. For instructions on modifying sinks, run across Manage sinks.

You can't delete the _Default bucket.

Logs held in the _Default bucket are retained for xxx days, unless y'all configure custom retention for the bucket.

Cloud Logging pricing applies to the logs information held in the _Default bucket.

User-defined log buckets

You lot can also create user-divers log buckets in any Cloud project. By applying sinks to your user-defined log buckets, yous tin route any subset of your logs to any log saucepan, letting you cull which Deject project your logs are stored in and which other logs are stored with them.

For example, for any log generated in Project-A, you tin configure a sink to route that log to user-defined buckets in Project-A or Project-B.

Cloud Logging pricing applies to the logs information held in this bucket, regardless of the log type.

You tin can configure custom retention for the bucket.

For information on managing your user-defined log buckets, including deleting or updating them, encounter Configure and manage log buckets.

Regionalization

Log buckets are regional resource. The infrastructure that stores, indexes, and searches your logs is located in a specific geographical location. Google manages that infrastructure so that your applications are available redundantly beyond the zones inside that region.

When you create a log bucket or set an organisation-level regional policy, yous tin can choose to store your logs in any of the following regions:

Continent	Regions
Asia	asia-east1 asia-east2 asia-northeast1 asia-northeast2 asia-northeast3 asia-south1 asia-south2 asia-southeast1 asia-southeast2
Commonwealth of australia	australia-southeast1 australia-southeast2
Europe	europe-central2 europe-north1 europe-west1 europe-west2 europe-west3 europe-west4 europe-west6
North America	northamerica-northeast1 northamerica-northeast2 the states-central1 us-east1 u.s.a.-east4 us-west1 us-west2 the states-west3 u.s.-west4
South America	southamerica-east1

In addition to these regions, y'all tin set up the location to global, which means that you don't need to specify where your logs are physically stored.

If y'all want to automatically apply a item storage region to the _Default and _Required buckets created in your organization, you tin can configure a default resources location.

For more information on logs data location, run into Data regionality for Deject Logging.

Organization policy

You can create an organization policy to ensure that your organization meets your compliance and regulatory needs. Using an arrangement policy, you can specify in which regions your organization can create new log buckets. You can besides restrict your organization from creating new log buckets in specified regions.

Cloud Logging doesn't enforce your newly created organization policy on existing log buckets; it only enforces the policy on new log buckets.

For information on creating a location-based organization policy, see Restrict resource locations.

In addition, you can configure a default resources location to cull which storage region to utilise to the _Default and _Required buckets created in your organisation.

Retention

Deject Logging retains logs according to retention rules applying to the log bucket type where the logs are held.

You tin configure Cloud Logging to retain logs between 1 day and 3650 days. Custom retention rules utilise to all the logs in a bucket, regardless of the log blazon or whether that log has been copied from some other location.

For information on setting retention rules for a log saucepan, see Configure custom retention.

Log views

Log views let yous control who has access to the logs within your log buckets.

Cloud Logging automatically creates the _AllLogs view for every saucepan, which shows all logs. Cloud Logging also creates a view for the _Default saucepan chosen _Default, which shows all logs except Information Access audit logs.

Because log buckets can incorporate logs from multiple Cloud projects, you might want to command which Cloud projects different users can view logs from. You can create custom log views, which give you more granular access command for those buckets.

For more than data, run into Manage log views.

Using logs in the Google Cloud ecosystem

The following department provides information on using logs in the broader Google Cloud.

Log-based metrics

Log-based metrics are Cloud Monitoring metrics that are based on the content of log entries. If Cloud Logging receives a log entry for a Cloud project that matches the filters of one of the Cloud project's metrics, and then that log entry is reflected in the metric data.

Sink exclusion filters apply to system-divers user metrics, which count only logs that are included for ingestion by the Cloud projection.

Sink exclusion filters don't utilise to user-defined log-based metrics. Fifty-fifty if you exclude logs from being ingested by Cloud Logging API and the logs aren't stored in any Logging buckets, you could see those logs counted in these metrics.

Log-based metrics apply at the Deject project level. These metrics are calculated past the Log Router and employ to logs just in the Deject project in which they're received.

For more data, meet Log-based metrics overview.

Finding logs in supported destinations

To learn about the format of routed log entries and how the logs are organized in destinations, run across View logs in sink destinations.

Common employ cases

To address common use cases for routing and storing logs, see the following documents and tutorials:

• Amass your organisation's log into a central logs bucket.

• Regionalize your Google Deject project'southward logs using logs buckets.

• Configure multi-tenant logging for Google Kubernetes Engine (GKE) clusters.

• Design patterns for routing logging information

Compliance needs

For best practices on using routing for data governance, see the following documents:

• Enable customer-managed encryption keys for Log Router.

• Logs data: A pace by step guide for overcoming mutual compliance challenges

• Data governance: Principles for securing and managing logs

Access control with IAM

For information on how y'all use Identity and Access Management (IAM) roles and permissions to control access to Deject Logging data, see the Access command with IAM.

Pricing

To understand ingestion and storage pricing, meet the Cloud Logging pricing information.

Deject Logging doesn't charge to route logs, but destination charges might utilise. For details, review the appropriate service's pricing details:

• Cloud Storage pricing
• BigQuery pricing
• Pub/Sub pricing
• Cloud Logging pricing

Note besides that if you send and so exclude your Virtual Private Deject flow logs from Cloud Logging, VPC flow log generation charges apply in improver to the destination charges.

What'south side by side

To help you route and store Cloud Logging information, see the following documents:

• To create sinks to route logs to supported destinations, run across Configure sinks.

• For routing and sinks troubleshooting information, encounter Troubleshoot routing and sinks.

bartonanclund.blogspot.com

Source: https://cloud.google.com/logging/docs/routing/overview

Share :