GCP Audit Logs

Audit logs are boring. At least we hope they’re boring. If your audit logs are exciting, you are likely having a bad day. But audit logging of some sort is often a good idea, and many of us forget to set it up and verify that we understand the data on a regular basis. This post walks through setting up and using the audit logging capabilities of GCP. Operations and site reliability are not my areas of expertise. Also, I am not a security professional or a compliance lawyer. Please consider this blog post just one of the many things you should be doing regarding logging and monitoring when preparing your projects for the big scary internet.

Terminology

Before I dive into the how-to, I want to refine my terminology a bit. There are many things called audit logs. In this post, I’m focusing on two categories of logs: Admin Activity Logs and Data Access Logs. Admin Activity Logs record the actions of admins on your project. These may include spinning up new instances, altering the project metadata, enabling APIs, or deploying an application. Having good admin audit logs allows you to retroactively figure out how a given change was made and often who made it. Data Access Logs are what you’d expect, they record when users access data. Google Cloud Platform has Data Access Logs for Cloud Storage, Cloud Dataproc, Deployment Manager, Cloud SQL, Compute Engine, and several other products.

Most people have some admin activity logging set up. Admin activity logs, like all logs, are only as useful as what you are logging. I hope no one is still doing this, but ten years ago it was pretty common practice to have a shared admin account used by multiple people. I worked on a system like this, and when configs were changed in production, we couldn’t track down the person responsible since all the logs showed up as root.

Data Access Logs are less commonly used for a couple of reasons. First off, a lot of the data we store we want folks to access, and we don’t need to know who. Things like images, web assets, public data sets, and scripts are in this category. Second, data access logs can be extremely verbose. Knowing that someone listed the contents of your backups bucket is noise if your only goal is to find out when someone accesses “my-super-secret-file.txt.” And while noisy logs make some people feel safer, I find they make my job harder.

Admin Activity Logs

On GCP most admin activity is logged by default. Here is a list of all the products that have admin activity logs and what release stage the logging support is at.

There are a couple of places that you can view the recent admin actions. The first is on the activity tab of the GCP Console homepage. This view allows you to see admin actions for multiple projects in an organization. You can filter by the type of log or the service (GKE, App Engine, BigTable) and the type of event you are interested in. You can also filter logs by the actor if that is helpful.

You can also view logs in Stackdriver Logging where there are additional querying and filtering capabilities. Stackdriver Logging also allows you to set up alerts on your logs by creating a metric. I did a longer blog post about this earlier this year. There’s also a Cloud Minute video that demonstrates the steps to create a log-based alert.

Data Access Logs

Data Access logs on GCP are not enabled by default (BigQuery is an exception to this). If you want to set up Data Access logs you must explicitly turn them on. All the documentation on enabling data access logs is here, but this post will walk you through the basics.

Data Access logs are configured as part of the IAM policy for a project or organization. You also cannot turn them on or off in the UI; you must use the ‘gcloud’ command line tool to modify the IAM policy. Finally, be careful when editing the IAM policy. If you configured the IAM policy incorrectly, you could remove all access to a project or organization. One of my co-workers found this out by accident and rendered her project unusable.

I find it easiest to use Cloud Shell to make changes like this. You can use your machine, but Cloud Shell has an up-to-date version of gcloud installed plus my favorite text editor.

The first step is to pull down your existing IAM policy. You can do this with the following command:

gcloud projects get-iam-policy [PROJECT_ID] > policy.yaml

Now you need to edit that file and add the data access log configuration at the top of the file. To enable all data access logs, add the following to the top of your file.

auditConfigs:
- auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_WRITE
  - logType: DATA_READ
  service: allServices

If you have a publicly readable Cloud Storage bucket, it might make sense to enable data access logs just for writes. This would let you see when the bucket was updated but would eliminate log spam from millions of users. In that case, you would add the following to your policy file:

auditConfigs:
- auditLogConfigs:
  - logType: DATA_WRITE
  service: storage.googleapis.com

Once you have a policy file, you can apply it using the gcloud projects set-iam-policy [PROJECT_ID] ./policy.yaml. This will apply your new policy and turn on audit logging for your project.

Learn More

Audit logging is a complex topic. There are many opinions on how to do it “properly.” What works for one project may not work for another. To learn more about the audit logging support that GCP offers, please check out the audit logging page on the GCP website.