Home Assistant and Splunk inside Docker

During last weekend I joined up with a friend of mine for a cup of coffee at my favorite coffee shop – Narcoffee over a discussion on how we’re doing with our continuous home automation projects.

He’s using an Intel NUC on which he runs Home Assistant in Docker, plus some other containers which include InfluxDB and… what a surprise? Splunk!

Yes, Splunk can run as a container in Docker. How cool is that? The repository is even present on Docker Hub.

It occurred to me that this was gonna help me professionally since in a couple of months I plan to steer my career towards Splunk.

Unfortunately Splunk runs only on amd64 CPU architecture so running a container on my Raspberry PI 3 was out of the question. I don’t intend to buy a NUC or other similar products so my only option at this time was to use a cloud instance.

I really like DigitalOcean. It gives me a vibe when I use their web user interface. Simple and powerful. Did I mention I find it cheap?

Only 5$ a month for 1vCPU, 1GB RAM and 25GB SSD instance. They can also host your DNS zone and this is cool since you can assign A or AAAA records directly to a droplet (the name they use for a cloud instance). They also offer floating IP addresses which you can use in case you want a fixed IP address and they assign a PTR record to it. Did I mention that’s for FREE?

You should also check their community guides. So inspiring!

Now let’s get down to business! For the Splunk instance I’ve chosen the Container Distribution: Fedora Atomic. You can go for anything since Docker can be installed on any distribution.

While it’s not mandatory to have your own DNS zone and to actually host it on DigitalOcean your life will become easier by doing so.

The other option, is to create a Duck DNS account and integrate that into your droplet (simple cron with a curl). Then you’ll use something like splunk.duck-dns.org as a FQDN to access the Splunk WUI. Having a FQDN will allow you to generate the Let’s Encrypt certificate for Splunk which will allow communication from Home Assistant to Splunk via the HTTP Event Collector over the Internet (Yes, with TLS).

In my case I’ve used my own domain which is hosted on the DigitalOcean DNS servers. I am going to use the fictional splunk.example.com as a FQDN in this article.

After the droplet is created connect to it via SSH and run the following commands to fetch the two Docker images:

docker pull splunk/splunk
docker pull certbot/dns-digitalocean

Create the following paths on the Droplet:

mkdir -p /opt/splunk/var
mkdir -p /opt/splunk/etc
mkdir -p /opt/letsencrypt/etc/letsencrypt
mkdir -p /opt/letsencrypt/var/lib/letsencrypt

Create a Read/Write API key in DigitalOcean using their WUI.

Create a digitalocean.ini file inside /opt/letsencrypt/etc/letsencrypt directory:

touch /opt/letsencrypt/etc/letsencrypt/digitalocean.ini
chmod 600 /opt/letsencrypt/etc/letsencrypt/digitalocean.ini

The file should contain the DigitalOcean API key created before:

# DigitalOcean API credentials used by Certbot
dns_digitalocean_token = 0000111122223333444455556666777788889999aaaabbbbccccddddeeeeffff

Assign the correct SELinux type for those folders you’ve just created:

chcon -R -u system_u -r object_r -t container_file_t /opt/splunk
chcon -R -u system_u -r object_r -t container_file_t /opt/letsencrypt

We’ll now generate the SSL certificate for splunk.example.com by using certbot with the domain validation method. They guys from certbot have created a plugin which simplifies this process and have provided a Docker image on the Hub – certbot/dns-digitalocean.

docker run -it --rm --name certbot -v "/opt/letsencrypt/etc/letsencrypt:/etc/letsencrypt" -v "/opt/letsencrypt/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-digitalocean certonly

Answer the questions and when asked for the file which contains the API key type in:

/etc/letsencrypt/digitalocean.ini

We’ll need just two files which we’ll use in Splunk:

/opt/letsencrypt/etc/letsencrypt/archive/splunk.example.com/fullchain1.pem
/opt/letsencrypt/etc/letsencrypt/archive/splunk.example.com/privkey1.pem

Install docker-compose and then reboot your droplet:

rpm-ostree install docker-compose
systemctl reboot

Create docker-compose.yml with the following content:

---
version: '3'
services:
  splunk:
    container_name: splunk
    domainname: example.com
    hostname: splunk
    image: splunk/splunk
    ports:
      - "8000:8000"
      - "8088:8088"
    volumes:
      - "/opt/splunk/var:/opt/splunk/var"
      - "/opt/splunk/etc:/opt/splunk/etc"
    environment:
      SPLUNK_START_ARGS: --accept-license --answer-yes
    restart: unless-stopped

We’re now ready to start our Splunk container. Run the following in the same directory as docker-compose.yml file:

docker-compose up -d

You can now access your Splunk WUI at http://splunk.example.com:8000. The initial credentials are admin/changeme and you will be required to change the password.

First stop is to activate HTTPS for the WUI. Open the Splunk WUI and navigate to Settings -> Server Settings -> General Settings and enable SSL.

Restart Splunk by navigating to Settings -> Server Controls -> Restart Splunk.

After a while you will be able to access the WUI at https://splunk.example.com:8000

Let’s install our signed certificate into Splunk. Run the following commands:

cp /opt/splunk/etc/auth/splunkweb/cert.pem /opt/splunk/etc/auth/splunkweb/cert.pem.orig
cp /opt/splunk/etc/auth/splunkweb/privkey.pem /opt/splunk/etc/auth/splunkweb/privkey.pem.orig
cp /opt/letsencrypt/etc/letsencrypt/live/splunk.example.com/fullchain.pem /opt/splunk/etc/auth/splunkweb/cert.pem
cp /opt/letsencrypt/etc/letsencrypt/live/splunk.example.com/privkey.pem /opt/splunk/etc/auth/splunkweb/privkey.pem

Restart Splunk by navigating to Settings -> Server Controls -> Restart Splunk.

The Home Assistant component for Splunk offers the option to use SSL with the Splunk HEC but after testing it I figured the certificate needed to be signed by a known trusted CA since it was doing validation on it. One option would have been to add the Splunk CA certificate to Home Assistant CA trust store but I didn’t want to have my own hass.io build so I decided to replace the certificates into Splunk with the same ones I’m using for accessing the WUI. Same FQDN, right?

At first I didn’t find specific documentation on how to do it so I’ve tried to hack my way in.

First enable the HEC by making Splunk listen on 8088. Navigate to Settings -> Data Inputs -> HTTP Event Collector and click on Global Settings and Enable HEC.

I’ve connected to TCP:8088 using openssl:

openssl s_client -connect splunk.example.com:8088

This will show you the installed certificate.

Navigate to your droplet inside this directory and grep for a section from the certificate the previous command:

cd /opt/splunk/etc/auth
grep -E "pattern" -r .

This will show you that the certificate is installed in the server.pem file.

The server.pem file is structured as followed:

The certificate we already have is in the correct format and the Let’s Encrypt CA certificate as well.
We just need to put the password on our private key and recode it from PKCS#5 to PKCS#8.

cd /opt/splunk/etc/auth
openssl rsa -des3 -in splunkweb/privkey.pem -out privkey_enc.pem -passout pass:password
openssl pkcs8 -in privkey_enc.pem -topk8 -v2 des3 -out privkey_enc_pkcs8.pem -passin pass:password -passout pass:password
cp server.pem server.pem.orig

Open splunkweb/cert.pem and split the two certificates. The first one is the splunk.example.com certificate and the second one is the CA certificate.

Create server.pem file by having the first certificate at the top of the file, followed by the contents of privkey_enc_pkcs8.pem and finishing with the second certificate. Fix the SELinux types:

chcon -R -u system_u -r object_r -t container_file_t /opt/splunk/etc/auth

Restart Splunk by navigating to Settings -> Server Controls -> Restart Splunk.

Validate that the certificate can be successfully verified:

openssl s_client -connect splunk.example.com:8088

You should see a line which says:

Verify return code: 0 (ok)

Download the Home Assistant Splunk App

Install the App into Splunk by clicking on the Splunk icon and then on the Manage Apps icon.

Click on Install app from file and upload the zip file containing the Home Assistant App.

Let’s now create the HEC entry for the Home Assistant. Navigate to Settings -> Add Data. Then choose monitor(files and ports on this Splunk indexer).

Go for HTTP Event Collector and give the entry a name (eg. Home Assistant).

In the next screen, for Source Type choose Select and go for Structured -> homeassistant_event.

For the App context choose HomeAssistant (homeassistant).

For the Index choose homeassistant.

Click on Review and then Submit.

Make a note on the token value. You can find the token afterwards also by navigating to Settings -> Data Inputs -> HTTP Event Collector.

Now to the Home Assistant configuration. Open your configuration.yaml and add this section:

splunk:
  token: YOUR_TOKEN
  ssl: true
  host: splunk.example.com

Restart Home Assistant and soon you’ll have events coming into Splunk.

Best Practices:

Don’t use the default ports in Splunk. This mean you should change at least in our case the WUI port 8000 and the HEC port 8088.
Even better if you have a static IP, setup an iptables rule which allows communication on 8088 only from your public Home Assistant IP address.

Hints:

The free Splunk license has some limitations. The first limitation I’ve hit was that I couldn’t send Email alerts.

You can request a free Splunk Developer license which unlocks the features.

I recommend Mailgun as an Email service if you don’t want to run your own Email server and don’t have a requirement for your own domain mailbox. You can setup a catch-all route in mailgun which redirects all Emails to your gmail account for example.

IoT Blog

Home Assistant and Splunk inside Docker

Best Practices:

Hints:

Leave a Reply Cancel reply