Skip to main content

Export Teleport Audit Events to Splunk

Report an Issue

Teleport's Event Handler plugin receives audit events from the Teleport Auth Service and forwards them to your log management solution, letting you perform historical analysis, detect unusual behavior, and form a better understanding of how users interact with your Teleport cluster.

In this guide, we will show you how to configure the Teleport Event Handler plugin to send your Teleport audit events to Splunk.

How it works​

In this setup, the Teleport Event Handler plugin receives audit events from the Teleport Auth Service over a gRPC channel and sends them to a local Fluentd instance as HTTP requests. The Fluentd instance forwards these requests to the Splunk HTTP Event Collector (HEC), which in turn sends them to Splunk Cloud Platform or Splunk Enterprise for visualization and alerting.

Prerequisites​

  • A running Teleport cluster. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl and tsh clients.

    Installing tctl and tsh clients

    1. Determine the version of your Teleport cluster. The tctl and tsh clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at /v1/webapi/find and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service:

      TELEPORT_DOMAIN=teleport.example.com:443
      TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')"

    2. Follow the instructions for your platform to install tctl and tsh clients:

      Download the signed macOS .pkg installer for Teleport, which includes the tctl and tsh clients:

      curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg

      In Finder double-click the pkg file to begin installation.

      danger

      Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security.

  • Recommended: Configure Machine & Workload Identity to provide short-lived Teleport credentials to the plugin. Before following this guide, follow a Machine & Workload Identity deployment guide to run the tbot binary on your infrastructure.
  • Splunk Cloud Platform or Splunk Enterprise v9.0.1 or above.
  • Fluentd version v1.12.4 or greater. The Teleport Event Handler will create a new fluent.conf file you can integrate into an existing Fluentd system, or use with a fresh setup.
  • A server, virtual machine, Kubernetes cluster, or Docker environment to run the Teleport Event Handler plugin.
  • On Splunk Enterprise, port 8088 should be open to traffic from the host running the Teleport Event Handler and Fluentd instance.
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email@example.com to your Teleport username: 
    tsh login --proxy=teleport.example.com --user=email@example.com
    tctl status
    Cluster  teleport.example.com
    Version  19.0.0-dev
    CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Step 1/5. Set up the Event Handler plugin​

The Event Handler plugin is a binary that runs independently of your Teleport cluster. It authenticates to your Teleport cluster and the Fluentd instance using mutual TLS. In this section, you will install the Teleport Event Handler plugin and generate credentials that the plugin will use for authentication.

Install the Event Handler plugin​

Follow the instructions for your environment to install the Teleport Event Handler plugin:

The Event Handler plugin is provided in amd64 and arm64 binaries for downloading. Replace ARCH with your required version.

curl -L -O https://cdn.teleport.dev/teleport-event-handler-v19.0.0-dev-linux-ARCH-bin.tar.gz
tar -zxvf teleport-event-handler-v19.0.0-dev-linux-ARCH-bin.tar.gz
sudo ./teleport-event-handler/install

Generate a starter config file​

Generate a configuration file with placeholder values for the Teleport Event Handler plugin. Later in this guide, we will edit the configuration file for your environment.

Run the configure command to generate a sample configuration. Replace mytenant.teleport.sh with the DNS name of your Teleport Enterprise Cloud tenant:

teleport-event-handler configure . mytenant.teleport.sh:443

You'll see the following output:

Teleport event handler 19.0.0-dev

[1] mTLS Fluentd certificates generated and saved to ca.crt, ca.key, server.crt, server.key, client.crt, client.key
[2] Generated sample teleport-event-handler role and user file teleport-event-handler-role.yaml
[3] Generated sample fluentd configuration file fluent.conf
[4] Generated plugin configuration file teleport-event-handler.toml
[5] Generated plugin helm configuration file teleport-plugin-event-handler-values.yaml

The plugin generates several setup files:

ls -l
-rw------- 1 bob bob     1038 Jul  1 11:14 ca.crt
-rw------- 1 bob bob     1679 Jul  1 11:14 ca.key
-rw------- 1 bob bob     1042 Jul  1 11:14 client.crt
-rw------- 1 bob bob     1679 Jul  1 11:14 client.key
-rw------- 1 bob bob      541 Jul  1 11:14 fluent.conf
-rw------- 1 bob bob     1078 Jul  1 11:14 server.crt
-rw------- 1 bob bob     1766 Jul  1 11:14 server.key
-rw------- 1 bob bob      260 Jul  1 11:14 teleport-event-handler-role.yaml
-rw------- 1 bob bob      343 Jul  1 11:14 teleport-event-handler.toml
-rw------- 1 bob bob      343 Jul  1 11:14 teleport-plugin-event-handler-values.yaml
File(s)Purpose
ca.crt and ca.keySelf-signed CA certificate and private key for Fluentd
server.crt and server.keyFluentd server certificate and key
client.crt and client.keyFluentd client certificate and key, all signed by the generated CA
teleport-event-handler-role.yamluser and role resource definitions for Teleport's event handler
teleport-event-handler.tomlExample event handler configuration
teleport-plugin-event-handler-values.yamlExample event handler helm configuration
fluent.confFluentd plugin configuration
Running the Event Handler separately from the log forwarder

This guide assumes that you are running the Event Handler on the same host or Kubernetes pod as your log forwarder. If you are not, you will need to instruct the Event Handler to generate mTLS certificates for subjects besides localhost. To do this, use the --cn and --dns-names flags of the teleport-event-handler configure command.

For example, if your log forwarder is addressable at forwarder.example.com and the Event Handler at handler.example.com, you would run the following configure command:

teleport-event-handler configure --cn=handler.example.com --dns-names=forwarder.example.com

The command generates client and server certificates with the subjects set to the value of --cn.

The --dns-names flag accepts a comma-separated list of DNS names. It will append subject alternative names (SANs) to the server certificate (the one you will provide to your log forwarder) for each DNS name in the list.

Define RBAC resources​

The teleport-event-handler configure command generated a file called teleport-event-handler-role.yaml. This file defines a teleport-event-handler role and a user with read-only access to the event API:

kind: role
metadata:
  name: teleport-event-handler
spec:
  allow:
    rules:
      - resources: ['event', 'session']
        verbs: ['list','read']
version: v5
---
kind: user
metadata:
  name: teleport-event-handler
spec:
  roles: ['teleport-event-handler']
version: v2

Move this file to your workstation (or recreate it by pasting the snippet above) and use tctl on your workstation to create the role and the user:

tctl create -f teleport-event-handler-role.yaml
user "teleport-event-handler" has been created
role 'teleport-event-handler' has been created
tip

You can also create and edit roles using the Web UI. Go to Access -> Roles and click Create New Role or pick an existing role to edit.

Enable issuing of credentials for the Event Handler role​

With the role created, you now need to allow the Machine & Workload Identity bot to produce credentials for this role.

This can be done with tctl, replacing my-bot with the name of your bot:

tctl bots update my-bot --add-roles teleport-event-handler

Export the plugin identity​

Give the plugin access to a Teleport identity file. We recommend using Machine ID for this in order to produce short-lived identity files that are less dangerous if exfiltrated, though in demo deployments, you can generate longer-lived identity files with tctl:

Configure tbot with an output that will produce the credentials needed by the plugin. As the plugin will be accessing the Teleport API, the correct output type to use is identity.

For this guide, the directory destination will be used. This will write these credentials to a specified directory on disk. Ensure that this directory can be written to by the Linux user that tbot runs as, and that it can be read by the Linux user that the plugin will run as.

Modify your tbot configuration to add an identity output.

If running tbot on a Linux server, use the directory output to write identity files to the /opt/machine-id directory:

services:
- type: identity
  destination:
    type: directory
    # For this guide, /opt/machine-id is used as the destination directory.
    # You may wish to customize this. Multiple outputs cannot share the same
    # destination.
    path: /opt/machine-id

If running tbot on Kubernetes, write the identity file to Kubernetes secret instead:

services:
  - type: identity
    destination:
      type: kubernetes_secret
      name: teleport-event-handler-identity

If operating tbot as a background service, restart it. If running tbot in one-shot mode, execute it now.

You should now see an identity file under /opt/machine-id or a Kubernetes secret named teleport-event-handler-identity. This contains the private key and signed certificates needed by the plugin to authenticate with the Teleport Auth Service.

Step 2/5. Configure Splunk​

In this section, you will configure Splunk to ingest and index Teleport audit events, and create a token for the HEC to authenticate to Splunk. All steps within this section take place within the Splunk web interface.

Create an index for Teleport audit events​

  1. Visit the home page of the Splunk UI and navigate to Settings > Indexes. Click New Index. Name your index teleport-audit-logs and assign the Index Data Type field to Events.
  2. Fill in the values of the remaining fields, Max raw data size and Searchable retention (days) based on the needs of your organization.
  3. Click Save.

[Optional] Create a source type for Teleport audit events​

By default, Splunk's _json source type expects the time field to be in a different format than that of Teleport audit events. This means that the event will show up in Splunk at the time it was ingested, not at the time it was generated. Adjust the format of the time field in Splunk so Teleport audit events appear as expected.

  1. Navigate to Settings -> Source Types.
  2. Find the _json source type and click Clone.
  3. Name the new source type _json-gotime.
  4. Under Timestamp click Advanced.
  5. Input %Y-%m-%dT%H:%M:%S.%3NZ and click Save.

Create a token for the HTTP Event Collector​

  1. Visit the home page of the Splunk UI.
  2. Navigate to Settings > Data inputs.
  3. In the Local inputs table, find the HTTP Event Collector row and click Add new.
  4. Enter a name you can use to recognize the token later so you can manage it, e.g., Teleport Audit Events.
  5. Click Next.
  6. In the Input Settings view, next to the Source type field, click Select.
  7. In the Select Source Type dropdown menu, click Structured, then choose the _json-gotime type you created earlier. If you skipped that optional step, choose _json. Splunk will index incoming logs as JSON, which is the format the Event Handler uses to send logs to Splunk.
  8. In the Index section, select the teleport-audit-logs index you created earlier.
  9. Click Review, then view the summary and click Submit.
  10. Copy the Token Value field and assign TOKEN to it so you can use it later in this guide.

Step 3/5. Connect Fluentd to the Splunk HTTP Event Collector​

While Splunk used to maintain a dedicated output plugin at https://github.com/splunk/fluent-plugin-splunk-hec, this has been deprecated. We can still connect Fluentd to the HEC using Fluentd's built-in HTTP output plugin.

In Step 1, you generated a configuration file for Fluentd at fluent.conf. Edit your fluent.conf file as follows:

  1. Edit the <parse> section with the following section:

        <parse>
      @type json
      json_parser oj

      # This time format is used by the plugin. This field is required.
      time_type string
      time_format %Y-%m-%dT%H:%M:%S
      keep_time_key true
    </parse>

    This will preserve the time field in the JSON when it is sent to Splunk. Some Fluentd output plugins require having a parsed time available.

  2. Assign https://splunk-hec.example.com:8088/services/collector/raw to your Splunk HTTP Event Collector endpoint.

    For Splunk Enterprise, this will be of the format:

    • https://<host>:8088/services/collector/raw

    For Splunk Cloud, this will be of the format:

    • AWS: https://http-inputs-<host>.splunkcloud.com:443/services/collector/raw
    • Google Cloud / Azure: https://http-inputs.<host>.splunkcloud.com:443/services/collector/raw
    • Free Trial: https://<host>.splunkcloud.com:8088/services/collector/raw

    For more information on Splunk HEC URIs, see Splunk HTTP Event Collector .

  3. Edit the <match test.log> section with the following section.

     <match test.log>
   @type http
   endpoint splunk-endpoint
   headers {"Authorization": "Splunk TOKEN"}
   # tls_verify_mode none
   <buffer>
     flush_interval 2s
   </buffer>
 </match>

    This configures Fluentd to authenticate to the HEC using the token you generated earlier. If using Splunk Cloud Free Trial, uncomment tls_verify_mode none to allow self-signed certificates.

  4. Test your changes by running Fluentd:

    docker run -u $(id -u ${USER}):$(id -g ${USER}) -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluent/fluentd:edge

Step 4/5. Run the Teleport Event Handler plugin​

Now that you have configured your Fluentd instance to receive logs via HTTP and forward them to Splunk, you will modify the Event Handler configuration and run the Event Handler to test your configuration.

Configure the Teleport Event Handler​

In this section, you will configure the Teleport Event Handler for your environment.

Earlier, we generated a file called teleport-event-handler.toml to configure the Fluentd event handler. This file includes setting similar to the following:

storage = "./storage"
timeout = "10s"
batch = 20
# concurrency is the number of concurrent sessions to process. By default, this is set to 5.
concurrency = 5
# The window size configures the duration of the time window for the event handler
# to request events from Teleport. By default, this is set to 24 hours.
# Reduce the window size if the events backend cannot manage the event volume
# for the default window size.
# The window size should be specified as a duration string, parsed by Go's time.ParseDuration.
window-size = "24h"
# types is a comma-separated list of event types to search when forwarding audit
# events. For example, to limit forwarded events to user logins
# and new Access Requests, you can assign this field to
# "user.login,access_request.create".
types = ""
# skip-event-types is a comma-separated list of audit log event types to skip.
# For example, to forward all audit events except for new app deletion events,
# you can include the following assignment:
# skip-event-types = ["app.delete"]
skip-event-types = []
# skip-session-types is a comma-separated list of session recording event types to skip.
# For example, to forward all session events except for malformed SQL packet
# events, you can include the following assignment:
# skip-session-types = ["db.session.malformed_packet"]
skip-session-types = []

[forward.fluentd]
ca = "/home/bob/event-handler/ca.crt"
cert = "/home/bob/event-handler/client.crt"
key = "/home/bob/event-handler/client.key"
url = "https://fluentd.example.com:8888/test.log"
session-url = "https://fluentd.example.com:8888/session"

[teleport]
addr = "teleport.example.com:443"
identity = "identity"

Modify the configuration to replace fluentd.example.com with the domain name of your Fluentd deployment.

Next, modify the configuration file as follows:

addr: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account (e.g., teleport.example.com:443 or mytenant.teleport.sh:443).

identity: Fill this in with the path to the identity file you exported earlier.

client_key, client_crt, root_cas: Comment these out, since we are not using them in this configuration.

If you are providing credentials to the Event Handler using a tbot binary that runs on a Linux server, make sure the value of identity in the Event Handler configuration is the same as the path of the identity file you configured tbot to generate, /opt/machine-id/identity.

Start the Teleport Event Handler​

Start the Teleport Event Handler by following the instructions below.

Copy the teleport-event-handler.toml file to /etc on your Linux server. Update the settings within the toml file to match your environment. Make sure to use absolute paths on settings such as identity and storage. Files and directories in use should only be accessible to the system user executing the teleport-event-handler service such as /var/lib/teleport-event-handler.

Next, create a systemd service definition at the path /usr/lib/systemd/system/teleport-event-handler.service with the following content:

[Unit]
Description=Teleport Event Handler
After=network.target

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/teleport-event-handler start --config=/etc/teleport-event-handler.toml --teleport-refresh-enabled=true
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-event-handler.pid

[Install]
WantedBy=multi-user.target

If you are not using Machine & Workload Identity to provide short-lived credentials to the Event Handler, you can remove the --teleport-refresh-enabled true flag.

Enable and start the plugin:

sudo systemctl enable teleport-event-handler
sudo systemctl start teleport-event-handler
Choose when to start exporting events

You can configure when you would like the Teleport Event Handler to begin exporting events when you run the start command. This example will start exporting from May 5th, 2021:

teleport-event-handler start --config /etc/teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"

You can only determine the start time once, when first running the Teleport Event Handler. If you want to change the time frame later, remove the plugin state directory that you specified in the storage field of the handler's configuration file.

Once the Teleport Event Handler starts, you will see notifications about scanned and forwarded events:

sudo journalctl -u teleport-event-handler
DEBU   Event sent id:f19cf375-4da6-4338-bfdc-e38334c60fd1 index:0 ts:2022-09-2118:51:04.849 +0000 UTC type:cert.create event-handler/app.go:140...

Step 5/5. Visualize your audit events in Splunk​

Since our setup forwards audit events to Splunk in the structured JSON format, Splunk automatically indexes them, so fields will be available immediately for use in visualizations. You can use these fields to create dashboards that track the way users are interacting with your Teleport cluster.

For example, from the Splunk UI home page, navigate to Search & Reporting > Dashboards > Create New Dashboard. Enter "Teleport Audit Log Types" for the title of your dashboard and click Classic Dashboards. Click Create then, in the Edit Dashboard view, click Add Panel.

In the Add Panel sidebar, click New > Column Chart. For the Search String field, enter the following:

index="teleport-audit-logs" | timechart count by event

Once you click Add to Dashboard you will see a count of Teleport event types over time, which gives you a general sense of how users are interacting with Teleport:

Troubleshooting connection issues​

If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that:

  • The certificate the Teleport Event Handler is using to connect to your Teleport cluster is not past its expiration date. This is the value of the --ttl flag in the tctl auth sign command, which is 12 hours by default.
  • Ensure that in your Teleport Event Handler configuration file (teleport-event-handler.toml), you have provided the correct host and port for the Teleport Proxy Service.

Next steps​

Now that you are exporting your audit events to Splunk, consult our audit event reference so you can plan visualizations and alerts.

To see all of the options you can set in the values file for the teleport-plugin-event-handler Helm chart, consult our reference guide.