Deploy the Air-Gapped Everactive Appliance on Linux
The Everactive Appliance provides an on-prem version of our services for sensor data acquisition and processing.
Hardware Requirements
• Linux Kernel 5.15.X (Tested on Ubuntu Server 22.04 and Debian 12)
• 4GB of RAM, 2 CPUs
• 25 GB of Disk space
Installation and Configuration
Prerequisites
Docker
Intall Docker following the instructions from official docker docs: https://docs.docker.com/engine/install/ubuntu/
Taskfile
We include a copy of Taskfile in the release to use the
Download and Extract the Everactive Appliance release
$ wget '{PRESIGNED URL}' -O export-v1.0.0.tar.gz
$ tar -zxvf export-v1.0.0.tar.gz
Load the docker images
The tarball contains the complete container images we need to load in docker.
$ cd export
export$ sudo bin/task load
Taskfile
We include a copy of the taskfile tool and scripts to simplify the configuration operations.
Example Output
export$ sudo bin/task load
task: [run:_load] docker load -i images/airgapped-database.tar
task: [run:_load] docker load -i images/airgapped-portainer.tar
task: [run:_load] docker load -i images/airgapped-api.tar
task: [run:_load] docker load -i images/airgapped-broker.tar
task: [run:_load] docker load -i images/airgapped-grafana.tar
task: [run:_load] docker load -i images/airgapped-proxy.tar
task: [run:_load] docker load -i images/airgapped-prometheus.tar
task: [run:_load] docker load -i images/airgapped-cdc.tar
task: [run:_load] docker load -i images/airgapped-sink.tar
task: [run:_load] docker load -i images/airgapped-tools.tar
42ebc17ba669: Loading layer [=> ] 32.77kB/1.47MB
42ebc17ba669: Loading layer [==================================================>] 1.47MB/1.47MB
42ebc17ba669: Loading layer [==================================================>] 1.47MB/1.47MB
...
14009dfb3416: Loading layer [===============> ] 26.18MB/83.86MB
0620092853f5: Loading layer [==================================================>] 807.4kB/807.4kB
14009dfb3416: Loading layer [==================================================>] 83.86MB/83.86MB
b2139c584a93: Loading layer [==================================================>] 3.584kB/3.584kB
3cc6e2e29424: Loading layer [==================================================>] 4.096kB/4.096kB
05ecc919987a: Loading layer [==================================================>] 2.56kB/2.56kB
Customize the Environment
We need to configure the virtual machine's external hostname or IP address and the passwords for the different services.
Set the following environment variables using the corresponding values (you just need to edit the first two). In this example, the virtual machine has the IP address 172.17.0.1.
export PROXY_EXTERNAL_HOST=172.17.0.1
export PASSWORD=airgap_password
export POSTGRES_PASSWORD=$PASSWORD
export BROKER_PASSWORD=$PASSWORD
export BROKER_EXTERNAL_HOST=$PROXY_EXTERNAL_HOST
export GF_SECURITY_ADMIN_PASSWORD=$PASSWORD
export PROXY_PASSWORD=$PASSWORD
export PORTAINER_PASSWORD=$PASSWORD
Now run the customization task to prepare the configuration files. Make sure to use the -E flag to preserve the environment variables you just set.
export$ sudo -E bin/task customize
You can take a look at the file .env in the current folder and check how the different settings will be applied to the services. It is also a good place
Example Output
export$ sudo bin/task customize
task: [run:customize] sed -i "s/POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD=${POSTGRES_PASSWORD}/g" .env
sed -i "s/SINK_TIMESCALE_PASSWORD=.*/SINK_TIMESCALE_PASSWORD=${POSTGRES_PASSWORD}/g" .env
sed -i "s/BROKER_PASSWORD=.*/BROKER_PASSWORD=${BROKER_PASSWORD}/g" .env
sed -i "s/SINK_RABBITMQ_AMQP_PASSWORD=.*/SINK_RABBITMQ_AMQP_PASSWORD=${BROKER_PASSWORD}/g" .env
sed -i "s/SINK_RABBITMQ_ADMIN_PASSWORD=.*/SINK_RABBITMQ_ADMIN_PASSWORD=${BROKER_PASSWORD}/g" .env
sed -i "s/BROKER_EXTERNAL_HOST=.*/BROKER_EXTERNAL_HOST=${BROKER_EXTERNAL_HOST}/g" .env
sed -i "s/PROXY_EXTERNAL_HOST=.*/PROXY_EXTERNAL_HOST=${PROXY_EXTERNAL_HOST}/g" .env
sed -i "s/GF_SECURITY_ADMIN_PASSWORD=.*/GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD}/g" .env
sed -i "s/PROXY_PASSWORD=.*/PROXY_PASSWORD=${PROXY_PASSWORD}/g" .env
Bootstrap the self-signed certificates
To enable TLS in the appliance services, we need to create self-signed certificates and keys. With this command, we bootstrap a CA and the derivative certificates inside certificates/certs:
export$ sudo bin/task certs
Useful certificates
To connect to MQTT from outside the installed instance we recommend downloading the following files for connecting locally to MQTT connections over TLS:
• certificates/certs/airgap-broker.key
• certificates/certs/airgap-broker.crt
• certificates/certs/EveractiveCA.crt
Example Output
export$ sudo bin/task certs
task: [run:certs] bash certificates/bootstrap.sh
Create Certificates
airgap-broker
Initializing CA
Created CA/certs/EveractiveCA.key
Created CA/certs/EveractiveCA.crt
Created CA/certs/EveractiveCA.crl
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = EveractiveCA
Validity
Not Before: Jul 21 14:49:21 2023 GMT
Not After : Jul 21 14:59:20 2123 GMT
Subject: CN = EveractiveCA
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Create Broker Certificates: airgap-broker, 172.17.0.1
Created CA/certs/airgap-broker.key
Created CA/certs/airgap-broker.csr
Created CA/certs/airgap-broker.crt from CA/certs/airgap-broker.csr signed by CA/certs/EveractiveCA.key
Create Proxy Certificates: airgap-proxy, 172.17.0.1
Created CA/certs/airgap-proxy.key
Created CA/certs/airgap-proxy.csr
Created CA/certs/airgap-proxy.crt from CA/certs/airgap-proxy.csr signed by CA/certs/EveractiveCA.key
Initial Gateway and Sensor Configuration
When the Data API container starts up, it will attempt to read a file in the root of the export directory called config.yml. This file contains the configuration for one or more sensors and gateways. Everactive support will provide you with this file in a separate download.
Example config.yml file
gateways:
- enableWakeup: true
evernet1Settings:
enabled: true
syncTime: 0
wakeupCode: "7C6EA12C"
wakeupOffset: 60
wakeupPeriod: 60
evernet2Settings:
beaconOffset: 400
broadcastPeriod: 60
enabled: true
panId: 25
name: "3AG32B5"
serialNumber: "3AG32B5"
sensors:
- encryptionKey: a-secret-value
macAddress: 01:02:03:43:AA:BC:DE:FF
manufacturingFwVersion: 1.0.1
manufacturingPartNumber: evr-01-23456
manufacturingSerialNumber: evrac01020343AABCDEFF
type: VibrationalElectrical
Start the Docker Stack
You are now ready to start the Docker Compose stack. Run the following command.
export$ sudo bin/task up
Example Output
bin/task up
task: [run:up] docker-compose -f docker-compose.yml up -d
[+] Running 12/12
✔ Network export_everactivenet Created 0.1s
✔ Volume "export_airgap-prometheus-data" Created 0.0s
✔ Volume "export_airgap-postgres-data" Created 0.0s
✔ Volume "export_airgap-rabbitmq-data" Created 0.0s
✔ Volume "export_airgap-cdc-data" Created 0.0s
✔ Container airgap-database Started 1.1s
✔ Container airgap-broker Started 1.4s
✔ Container airgap-proxy Started 1.3s
✔ Container airgap-prometheus Started 1.1s
✔ Container airgap-cdc Started 2.2s
✔ Container airgap-api Started 2.3s
✔ Container airgap-sink Started 2.4s
Validate the install
Once started, you can run the built-in validation report. This will run a suite of test to make sure the stack is functioning correctly.
✅ - Success: Test passed
🟠 - Warning: The test did not pass, but it’s not broken - See the troubleshooting guide
🛑 - Error: The test failed; something is wrong - See the troubleshooting guide
export$ sudo bin/task validate
Example Output
export$ sudo bin/task validate
task: [run:validate] docker run --rm -v $PWD/validate:/opt/validate -w /opt/validate docker.io/library/airgapped-tools:latest /bin/bash -c "ansible-playbook playbook.yaml --extra-vars 'api_url=https://$PROXY_EXTERNAL_HOST'"
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [Diagnostics] *************************************************************
TASK [Get API Health] **********************************************************
ok: [localhost]
TASK [Assert API is health] ****************************************************
ok: [localhost] => {
"changed": false,
"msg": "✅ API is healthy"
}
TASK [Get API Sensor Data] *****************************************************
ok: [localhost]
TASK [Assert sensors API response success] *************************************
ok: [localhost] => {
"changed": false,
"msg": "✅ API has returned a successful response"
}
TASK [Assert sensors API has data] *********************************************
ok: [localhost] => {
"assertion": "sensors.json.data | length > 0",
"changed": false,
"evaluated_to": false,
"failed_when_result": false,
"msg": "🟠 API no sensors data returned"
}
TASK [Get API Gateway Data] ****************************************************
ok: [localhost]
TASK [Assert gateways API response success] ************************************
ok: [localhost] => {
"changed": false,
"msg": "✅ API has returned a successful response"
}
TASK [Assert gateways API returns data] ****************************************
ok: [localhost] => {
"assertion": "gateways.json.data | length > 0",
"changed": false,
"evaluated_to": false,
"failed_when_result": false,
"msg": "🟠 API no gateways data returned"
}
PLAY RECAP *********************************************************************
localhost : ok=8 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Services
Data API
The Everactive API can be accessed via REST calls and using basic authentication (ask Everactive support for credentials) with any HTTP Client. Test the API Service by calling the health endpoint:
curl --request GET \
--url http://HOST/data/v1/health \
--header 'accept: application/json'
API Documentation
Metrics
Prometheus
The Everactive appliance includes a Prometheus server that collects metrics from the different services. You can access the Prometheus UI at: https://HOST/prometheus/graph
• username: airgap_user
• password: airgap_password
All metrics are exposed through the Prometheus Federation endpoint to integrate other systems like Grafana or DataDog.
curl -k -G --data-urlencode 'match\[]={job=~".+"}' \
-u airgap_user:airgap_password \
https://HOST/prometheus/federate
Troubleshooting
ISSUE: sudo bin/task validate report returned 🛑 API is not healthy
SOLUTION: This means that the API is not healthy. Typically this means that the API can not connect to the database. Use the command docker ps | grep airgap-database to check the status of the database service. Use the command docker logs airgap-database and look for errors. If you are not sure how to proceed, send a full dump of the logs docker logs > dump.txt to Everactive support.
ISSUE: sudo bin/task validate report returned 🟠 API no gateways data returned
SOLUTION: This is just a warning to let you know no gateways are reporting yet. Once a Gateway(s) is correctly configured, try rerunning the sudo bin/task validate task. You should now see ✅ API has returned sensor data. If you still do not get sensor data contact Everactive support.
ISSUE: sudo bin/task validate report returned 🟠 API no sensors data returned
SOLUTION: This is just a warning to let you know no sensors are reporting yet. Once Gateway(s) and sensors are configured, try rerunning the sudo bin/task validate task. You should now see ✅ API has returned gateway data. If you still do not get sensor data contact Everactive support.
Upgrading
Before beginning the upgrade process always double check the UPGRADE.md and release notes in the CHANGELOG.md file.
When a new release tar.gz is available, download it to your machine. Unpack it in a location other than the current install location.
wget '{PRESIGNED URL}' -O export-new-version.tar.gz
tar -zxvf export-new-version.tar.gz
cd export-new-version
Review the UPGRADE.md for any specific instructions about upgrading to a specific version.
Run the upgrade task and set the INSTALL_DIR to the path where the current version is installed. This operation will install the new docker images and necessary files.
sudo bin/task upgrade INSTALL_DIR=/path/to/current/export
Finally, run the following commands in the /path/to/current/export to reload the services.
cd /path/to/current/export
sudo bin/task reload
Updated about 1 year ago