Exec as root user in Kubernetes

Hi πŸ‘‹,

In this short tutorial I will show you a way of getting a root shell in containers running inside a modern Kubernetes cluster.

Prerequisites:

  • Root access to the cluster node in which the container is running.

Problem Statement

We wan’t root access into a running container, exec gives us non-root user.

➜  Downloads k get pods
NAME                     READY   STATUS    RESTARTS   AGE
my-release-cassandra-0   1/1     Running   0          2m9s

➜  Downloads k exec -it pod/my-release-cassandra-0 -- /bin/bash
I have no name!@my-release-cassandra-0:/$ whoami
whoami: cannot find name for user ID 1001
I have no name!@my-release-cassandra-0:/$ touch test
touch: cannot touch 'test': Permission denied
I have no name!@my-release-cassandra-0:/$ 

Solution

To obtain root access. First grab the Container ID from inside the pod.

k describe pod my-release-cassandra-0
Containers:
  cassandra:
    Container ID:  containerd://8fa7af3900d556aa8a91b1ac4cbe46335e8df233f8645b0a2329b2f0e6d76177
    Image:         docker.io/bitnami/cassandra:4.0.7-debian-11-r0

Then if it the id starts with containerd:// run the following command on the node the pod is running:

sudo runc --root /run/containerd/runc/k8s.io/ exec -t -u 0 8fa7af3900d556aa8a91b1ac4cbe46335e8df233f8645b0a2329b2f0e6d76177 /bin/bash

You should get a root shell into the Cassandra container:

root@my-release-cassandra-0:/# whoami
root
root@my-release-cassandra-0:/# touch test
root@my-release-cassandra-0:/# ls
bin	 boot  docker-entrypoint-initdb.d  etc	 lib	media  opt   root  run.sh  srv	test  usr
bitnami  dev   entrypoint.sh		   home  lib64	mnt    proc  run   sbin    sys	tmp   var

Thanks for reading and happy cloud surfing! πŸ„

Apache Flink Checkpoints on S3 and S3 compatible storage

Hello,

Recently someone working at Yahoo emailed me regarding an old thread I’ve started on the Apache Flink user mailing list. I’ve replied to the e-mail but also decided to turn the reply into a blog post, because it might help other people as well.

Email

Hi,

I was able to get it working after tinkering with it. The issue was mainly a miscommunication, we didn’t formally know which authentication method we were using in AWS. We we’re using only s3://

Here are our configuration options:

On S3 compatible storage:

fs.s3a.access.key: ""
fs.s3a.secret.key: ""
fs.s3a.connection.ssl.enabled: "false"
fs.s3a.endpoint: "ceph-mcr-1.xxx.xxx.xxx:xxx"
fs.s3a.list.version: "1"
s3.path.style.access: "true"
containerized.master.env.ENABLE_BUILT_IN_PLUGINS: "flink-s3-fs-hadoop-1.13.2.jar"
containerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS: "flink-s3-fs-hadoop-1.13.2.jar"
state.backend: "rocksdb"
state.backend.incremental: "true"
state.checkpoints.dir: "s3://bucket-name/checkpoints/$cluster_name$/"
state.savepoints.dir: "s3://bucket-name/savepoints/$cluster_name$/"

On S3 with AWS:

fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider",
containerized.master.env.ENABLE_BUILT_IN_PLUGINS: "flink-s3-fs-hadoop-1.13.2.jar",
containerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS: "flink-s3-fs-hadoop-1.13.2.jar",
state.backend: "rocksdb",
state.backend.incremental: "true",
state.checkpoints.dir: "s3://xxx/checkpoints/$cluster_name$/",
state.savepoints.dir: "s3://xxx/savepoints/$cluster_name$/"

fs.s3a.aws.credentials.provider was the authentication method (credentials provider) that we were missing, it’s not found in the Hadoop plugin docs[2] but it’s found in AWSJavaSDK docs[3][4]. AWS mounts secrets inside Flink pods so using this provider should make it work without further configuration.

Note that flink-s3-fs-hadoop-1.13.2.jar needs to be adapted to your Flink version. $cluster_name should also be substituted with your cluster/deployment name.

That’s pretty much it, I’m also attaching the Flink S3 docs[1] to the email. Thanks for reaching out! Hope you’ll figure it out!

Best,

Denis Nutiu

docs/deployment/filesystems/s3/[1]
hadoop-aws/tools/hadoop-aws/index.html#S3A[2]
amazonaws/auth/AWSCredentialsProvider.html[3]
amazonaws/auth/WebIdentityTokenCredentialsProvider.html[4]

As a side note, if you’re using the Flink Operator to deploy your Flink job you can set environment variables in the pod template file instead of flink-config.yaml.

Thanks for reading!

Container log monitoring on Microk8s with Loki, Grafana and Promtail

Hi πŸ‘‹

This is a short tutorial describing how to monitor your Kubernetes cluster container logs using Loki stack. But why? Because it is easier to view, filter your logs in Grafana and to store them persistently in Loki rather than viewing them in a terminal.

Let’s get started! Assuming you already have Microk8s installed, enable the following addons:

You can enable an add-on by running microk8s enable. Ex: microk8s enable dns

addons:
  enabled:
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    metrics-server    # K8s Metrics Server for API access to service metrics
    storage               # Storage class; allocates storage from host directory

Note: Microk8s comes with a bundled kubectl and helm3. Just run microk8s kubectl or microk8s helm3. If you want to use your host kubectl you can configure it via: microk8s config > ~/.kube/config.

Warning: Be extra careful when running the microk8s config > ~/.kube/config command because it will overwrite the old config file.

Then proceed by installing Loki. Loki will store all the logs using object storage. This is efficient but the trade-off is that you can’t do complex aggregations and searches against your data. We are going to install Loki for exploration purposes but if you’re looking for a production ready version, check out the loki distributed helm chart.

Run the following helm commands to install Loki. You may want to install helm or use microk8s helm3 command.

helm repo add grafana https://grafana.github.io/helm-charts

helm install loki grafana/loki -

You should get the following pods and services by running kubectl get pods and kubectl get services:

NAME                        READY   STATUS        RESTARTS   AGE
loki-0                      1/1     Running       0          9m8s

NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes      ClusterIP   10.152.183.1     <none>        443/TCP    54m
loki-headless   ClusterIP   None             <none>        3100/TCP   9m23s
loki            ClusterIP   10.152.183.187   <none>        3100/TCP   9m23s

Now, we can safely install Promtail. Promtail will import all the container logs into Loki and it should work auto-magically by auto-discovering all the pods that are running inside your cluster.

To let Promtail know about our existing Loki’s address, we can give it the service URL: http://loki-headless.default.svc.cluster.local:3100/loki/api/v1/push.

helm install promtail grafana/promtail --set config.lokiAddress=http://loki-headless.default.svc.cluster.local:3100/loki/api/v1/push

Finally, we need to visualize the logs using Grafana. Install it by running the helm command and then, edit the service to change its type from ClusterIP to NodePort.

Changing the service type to NodePort will allow you to visit Grafana in your browser without the need of adding an ingester.

❗❗To use vscode as the default editor export the following environment variable: KUBE_EDITOR=code -w

helm install grafana grafana/grafana

 kubectl edit service/grafana
# Change metadata.spec.type to NodePort
# Grab the service's port using kubectl get services and look for 32204:
# grafana                         NodePort    10.152.183.84    <none>        80:32204/TCP   6d

Note: If you’re on Windows to access the service you will need to run kubectl cluster-info and use the IP address of the cluster. On Linux you should be able to access http://localhost:32204.

kubectl cluster-info
Kubernetes control plane is running at https://172.20.138.170:16443

To access Grafana visit: http://172.20.138.170:32204 where 32204 is the service’s NodePort.

Grab your Grafana admin password by following the instructions from the helm notes. The notes are displayed after Grafana has been installed. If you don’t have base64 on your OS check out CyberChef, it can decode base64 text.


After you’ve successfully logged in, head to Settings -> DataSources and add the Loki data source.

Head back to the Explore menu and display Loki’s logs using the Loki data source in Grafana. You can click log browser to view all available values for the app label.

Promtail should now import logs into Loki and create labels dynamically for each newly created container. If you followed along, congratulations!

Thanks for reading and happy hacking! πŸ”§

Home automation with Home Assistant on Raspberry PI – Getting Started

Hi πŸ‘‹

The purpose of this article is to get you started quickly with a Home Assistant on a Raspberry Pi. It’s a simple walkthrough on how to install Home Assistant and configure it so it will boot with your PI.

I will use my old Raspberry PI V3 board.

Flashing the Raspberry PI OS

You will need a microSD card of reasonable size, I’m using a 16GB one and a USB Adapter to connect it with my PC.

Head over to Raspberry Pi OS website and download your preferred image, for my Home Assistant I’ve chosen Raspberry Pi OS with desktop and recommended software. After the download is completed, unzip the file and prepare to flash it.

To flash the OS image on the SD card I will use a program called balenaEtcher.

Download it, select your OS image, select the SD card, and hit flash.

After SD card flashing finishes, it is time to setup the Wi-Fi connection. If you’re using an ethernet cable you can skip this step, however, remember to enable SSH.

Setting up the Wi-Fi and enabling SSH

Unplug the SD card from the computer and plug it back. You should see two new drives D: and E:

  1. Open your favorite text editor and create an empty file called ssh in drive E:. This will enable SSH access.
  2. Create a new file called wpa_supplicant.conf using your text editor and paste the following contents in it:
country=us
update_config=1
ctrl_interface=/var/run/wpa_supplicant

network={
   scan_ssid=1
   ssid="YOUR_WIFI_SSID"
   psk="YOUR_WIFI_PASSWORD"
}

Don’t forget to replace YOUR_WIFI_SSID and YOUR_WIFI_PASSWORD with the corresponding values regarding your Wi-Fi network.

Eject the SD card from your computer and plug it into the PI. At boot, the PI should automatically connect to your Wi-Fi network.

Installing Home Assistant Core

Find your Raspberry PI’s IP address and connect to it via ssh. You can run the command ssh pi@192.168.0.XXX. The password for the pi user should be raspberry.

After getting a shell, follow the instructions for installing Home Assistant from the official website.

Ensure that you run each command on its own line. Don’t directly copy the entire code block, copy each line individually.

Starting Home Assistant on boot

If you can access the Home Assistant web GUI using http://192.168.0.XXX:8123 then the next step would be to create a new systemd service so that some assistant starts at boot. Please replace XXX with your Raspberry PI’s IP address.

To create a new service:

  1. Start a new shell on the Raspberry or ensure that you’re using the pi user. We will execute commands with sudo.
  2. Use sudo nano /etc/systemd/system/hass.service to create a new file and paste the following contents into it:
[Unit]
Description=HomeAssistant Service
After=network.target

[Service]
User=homeassistant
WorkingDirectory=/home/homeassistant
Environment="PATH=$PATH:/srv/homeassistant/bin"
ExecStart=/srv/homeassistant/bin/hass

[Install]
WantedBy=multi-user.target

Stop hass command if it’s running and enables the service by executing:

sudo systemctl start hass.service
sudo systemctl enable hass.service
sudo systemctl status hass.service

If the service is running normally, everything is set up. You can safely reboot your PI and the Home Assistant service will run after boot.

Configuring Home Assistant

When visiting the Home Assistant’s web interface for the first time, you will be prompted to create a new user. You may also download the Home Assistant application for your mobile device if you wish to track things like battery, storage, steps, location and so on, in Home Assistant.

In future articles I will show you how to configure the BME680 enviromental sensor and how to activate the Apple Homekit integration. Until then, have fun exploring Home Assistant docs.

Things to do further:

Unattended Upgrades – Enable unattended upgrades for your Raspbian OS. Ensures that your OS’s is always patched and up to date.

UFW – Secure your Home Assistant server with the uncomplicated firewall.

Change default passwords or disable SSH login via password.

Thanks for reading and happy automations! πŸ“š