Go Pattern: Sorting a slice on multiple keys

Hi πŸ‘‹

In this article I want to highlight a simple pattern for sorting a slice in Go on multiple keys.

Given the following structure, let’s say we want to sort it in ascending order after Version, Generation and Time.

type TheStruct struct {
	Generation int
	Time       int
	Version    int
}

The way we sort slices in Go is by using the sort interface or one of the sort.Slice functions. To sort the slice after the above criteria we’ll call slice.Sort with the following function.

	sort.Slice(structs, func(i, j int) bool {
		iv, jv := structs[i], structs[j]
		switch {
		case iv.Version != jv.Version:
			return iv.Version < jv.Version
		case iv.Generation != jv.Generation:
			return iv.Generation < jv.Generation
		default:
			return iv.Time < jv.Time
		}
	})

The slice will be sorted after the following fields: Version, Generation and Time. The trick is the switch statement and the case expression case iv.Version != jv.Version followed by the statement return iv.Version < jv.Version.

You can use this pattern whenever you want to sort slices over multiple fields in Go.

Thanks for reading! 🍻

Source Code

package main

import (
	"fmt"
	"sort"
)

type TheStruct struct {
	Generation int
	Time       int
	Version    int
}

func main() {
	var structs = []TheStruct{
		{
			Generation: 1,
			Time:       150,
			Version:    0,
		},
		{
			Generation: 1,
			Time:       200,
			Version:    0,
		},
		{
			Generation: 1,
			Time:       200,
			Version:    2,
		},
		{
			Generation: 1,
			Time:       500,
			Version:    0,
		},
		{
			Generation: 1,
			Time:       100,
			Version:    0,
		},
		{
			Generation: 1,
			Time:       400,
			Version:    0,
		},
		{
			Generation: 2,
			Time:       400,
			Version:    0,
		},
		{
			Generation: 2,
			Time:       100,
			Version:    2,
		},
		{
			Generation: 1,
			Time:       300,
			Version:    0,
		},
	}

	fmt.Printf("%v\n", structs)

	sort.Slice(structs, func(i, j int) bool {
		iv, jv := structs[i], structs[j]
		switch {
		case iv.Version != jv.Version:
			return iv.Version < jv.Version
		case iv.Generation != jv.Generation:
			return iv.Generation < jv.Generation
		default:
			return iv.Time < jv.Time
		}
	})
	fmt.Printf("%v\n", structs)

}

Output

[{1 150 0} {1 200 0} {1 200 2} {1 500 0} {1 100 0} {1 400 0} {2 400 0} {2 100 2} {1 300 0}]
[{1 100 0} {1 150 0} {1 200 0} {1 300 0} {1 400 0} {1 500 0} {2 400 0} {1 200 2} {2 100 2}]

Also, special thanks to RP.πŸ’–

How I got my PR merged into Apache Flink

Hi πŸ‘‹

This is a short story on how I got my pull request merged into Apache Flink.

It started with the need to set CPU and Memory limits to Flink jobs running under Kubernetes.

The first thing I did was to join the user mailing list and ask around if someone has encountered the issue and if there’s a solution to it. The people from the mailing list were very friendly and they pointed me to an existing ticket on the Flink jira board, which was exactly what I needed.

To speed things up, I decided to implement the ticket by myself. I wrote on the mailing list that I want to implement FLINK-15648 and started signing the Apache individual contributor license agreement.

After sending the signed document via email, I cloned the Flink project from GitHub and imported it into my IntelliJ IDE. Flink has some great documentation on how to setup your IDE and import the project.

Lastly, I’ve implemented the feature and submitted the PR flink/pull/17098. The first time I forgot to generate the code docs and I’ve got a CI error. After the error was fixed, the PR was merged. It did not speed things up as I initially thought since it was merged into Flink 1.15. Nonetheless, It was a smooth and fun process and the code review that I’ve received was also well done.

I hope your experience contributing to open-source software will be as fun as mine was.

Thanks for reading and happy hacking!

How to install aΒ specific Python version on Linux

Hello, πŸ‘‹

In this article I will show you how to install Python versions on Linux using the following methods: compiling from source, dead snakes ppa and pyenv.

To make things easier, if you want to follow along in an environment that you can break, you can create a local Kubernetes cluster using Minikube.

Next, I’m going to use the following yaml file to create an Ubuntu pod:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  labels:
    app: ubuntu
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always

Save the above yaml in a file ubuntu_pod.yaml and run:

kubectl apply -f ./ubuntu_pod.yaml

To get a shell on the Ubuntu pod, run:

kubectl exec -it ubuntu -- /bin/bash

To start from scratch, simply delete the pod with kubectl delete pod/ubuntu and then recreate it.

Compiling Python from source

Before compiling Python, you will need to setup the build environment, thankfully, it is straightforward.

Pyenv has great instructions on it: https://github.com/pyenv/pyenv/wiki#suggested-build-environment.

On Ubuntu, to build Python, install the following packages:

apt-get update; apt-get install make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Then, search the desired python version here and, for example to install Python 3.9, run:

wget https://www.python.org/ftp/python/3.9.9/Python-3.9.9.tgz
tar -xzf Python-3.9.9.tgz
cd Python-3.9.9

Then, run configure:

./configure --enable-optimizations

And finally run make install if you want to replace the default Python installation or make altinstall to install python under the binary name of python3.9

make altinstall

To test the installation run:

python3.9 --version
Python 3.9.9

pip3.9 --version
pip 21.2.4 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)

Installing Python via a third party PPA deadsnakes

To install Python using the deadsnakes ppa run:

apt-get update
apt-get install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt-get update
apt install python3.9 python3-pip

Then, to test the installation run:

root@ubuntu:/# python3.9 --version
Python 3.9.10

root@ubuntu:/# python3.9 -m pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.9)

Installing Python via Pyenv

I already written an article on how to install Python using Pyenv, check it out if you wish.

https://nuculabs.wordpress.com//2020/06/27/pyenv-for-linux-users/

Thanks for reading! πŸ“š

Pytest Fixtures and Yield

Hi πŸ‘‹

In this short article I want to explain the use of the yield keyword in pytest fixtures.

What is pytest?

Pytest is a complex python framework used for writing tests. It has lots of advanced features and it supports plugins. Many projects prefer pytest in addition to Python’s unitttest library.

What is a fixture?

A test fixture is a piece of code that fixes some common functionality that is required for writing the unit tests. This functionality can be

  • a connection to the database
  • a testing http server or client
  • creation of a complex object

You can read more about test fixtures on Wikipedia.

What does yield keyword do?

In Python, the yield keyword is used for writing generator functions, in pytest, the yield can be used to finalize (clean-up) after the fixture code is executed. Pytest’s documentation states the following.

β€œYield” fixtures yield instead of return. With these fixtures, we can run some code and pass an object back to the requesting fixture/test, just like with the other fixtures.

https://docs.pytest.org/en/6.2.x/fixture.html

An example code could be the following:

@pytest.fixture()
def my_object_fixture():
    print("1. fixture code.")
    yield MyObjectThatRequiresCleanUp()
    print("4. fixture code after yield.")

Running a sample test which utilizes the fixture will output:

collected 1 item                                                                                                                                                                       

tests\test_my_object.py 1. fixture code.
2. Initializing MyObjectThatRequiresCleanUp
3. test code.
.4. fixture code after yield.

Running the same test but now with the fixture my_object_fixture2, will output:

tests\test_my_object.py 1. fixture code.
2. Initializing MyObjectThatRequiresCleanUp
2.1 Entering
3. test code.
.3.1 Exiting
Clean exit
4. fixture code after yield.

I hope I could successfully ilustrate with these examples the order in which the testing and fixture code is run.

To run the tests, I’ve used pytest --capture=tee-sys . in the project root. The file contents are attached to the end of this article. The --capture parameter is used to capture and print the tests stdout. Pytest will only output stdout of failed tests and since our example test always passes this parameter was required.

Conclusion

Pytest is a python testing framework that contains lots of features and scales well with large projects.

Test fixtures is a piece of code for fixing the test environment, for example a database connection or an object that requires a specific set of parameters when built. Instead of duplicating code, fixing the object’s creation into a fixture makes the tests easier to maintain and write.

yield is a python keyword and when it is used in conjunction with pytest fixtures it gives you a nice pythonic way of cleaning up the fixtures.

Thanks for reading! πŸ“š


Contents of the Pytest fixtures placed in tests/__init__.py

import pytest

from my_object import MyObjectThatRequiresCleanUp


@pytest.fixture()
def my_object_fixture():
    print("1. fixture code.")
    yield MyObjectThatRequiresCleanUp()
    print("4. fixture code after yield.")


@pytest.fixture()
def my_object_fixture2():
    print("1. fixture code.")
    with MyObjectThatRequiresCleanUp() as obj:
        yield obj
    print("4. fixture code after yield.")

Contents of my_object.py

class MyObjectThatRequiresCleanUp:
    def __init__(self):
        print("2. Initializing MyObjectThatRequiresCleanUp")

    def __enter__(self):
        print("2.1 Entering")
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        print("3.1 Exiting")
        if exc_type is None:
            print("Clean exit")
        else:
            print("Exception occurred: {}".format(exc_type))

Contents of test_my_object.py placed in tests/test_my_object.py

from tests import my_object_fixture


def test_my_object(my_object_fixture):
    print("3. test code.")