Skip to content

Export Audit Logs#

Introduction#

Audit logs can be exported to your storage of preference. In case audit logs have not been configured yet in your Hopsworks cluster, please see Access Audit Logs.

Note

As an example, in this guide we will show how to export audit logs to BigQuery using the bq command-line tool.

Prerequisites#

In order to export audit logs you need SSH access to the Hopsworks cluster.

Step 1: Create a BigQuery Table#

Create a dataset and a table in BigQuery.

The table schema is shown below.

fullname          mode      type        description
pathInfo          NULLABLE  STRING  
methodName        NULLABLE  STRING  
caller            NULLABLE  RECORD  
dateTime          NULLABLE  TIMESTAMP   bq-datetime
userAgent         NULLABLE  STRING  
clientIp          NULLABLE  STRING  
outcome           NULLABLE  STRING  
parameters        NULLABLE  STRING  
className         NULLABLE  STRING  
caller.userId     NULLABLE  STRING  
caller.email      NULLABLE  STRING  
caller.username   NULLABLE  STRING

Step 2: Export Audit Logs to the BigQuery Table#

Audit logs can be exported in different formats. For instance, to export audit logs in JSON format set audit_log_file_type=io.hops.hopsworks.audit.helper.JSONLogFormatter.

Info

For more information on how to configure the audit log file type see the audit_log_file_type configuration variable in Audit logs.

To export the audit logs to the BigQuery table created in the previous step, run the following command.

bq load --project_id <projectId> \
        --source_format=NEWLINE_DELIMITED_JSON \
        <DATASET.TABLE> \
        /srv/hops/domains/domain1/logs/audit/server_audit_log0.log

Tip

This command can be configured to run periodically on a given schedule by setting up a cronjob.

Conclusion#

In this guide you showed how you can export audit logs in your Hopsworks cluster to a BigQuery table.