Using confluent-kafka-go on MacOS M1

Hello,

TLDR;

brew install librdkafka openssl@3 pkg-config
export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl@3/lib/pkgconfig"
go test -tags dynamic ./...

I’ve been transition from a Linux machine to a MacOS M1 machine at work and when I ran tests for a Golang project, I noticed that the test failed on modules depending on librdkafka.

Initially I’ve had problems with Kafka on MacOS M1 on Docker, since I was using an older image version that didn’t have any arm64 build, updating to images to version 7.2.1[1] fixed the issues.

This time however the problems was with my Golang dependencies and not the Docker containers. Running the tests resulted in:

go test ./...
[...]
ld: warning: ignoring file /Users/dnutiu/go/pkg/mod/github.com/confluentinc/confluent-kafka-go@v1.8.2/kafka/librdkafka_vendor/librdkafka_darwin.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
Undefined symbols for architecture arm64:

Undefined symbols for architecture arm64” My guess is that the published confluent-kafka-go package does not contain (yet) symbols for arm64, to fix the issues you can use the module with another librdkafka.

When installing librdkafka formula from Homebrew the library is built for arm64 architecture. To install run:

brew install librdkafka openssl@3 pkg-config

Next, we’ll use a tool called pkg-config to tell librdkafka where to find the other libraries, since it depends on openssl we need to export PKG_CONFIG_PATH. To grab the value run:

brew info openssl
==> openssl@3: stable 3.0.5 (bottled) [keg-only]
Cryptography and SSL/TLS Toolkit
https://openssl.org/
/opt/homebrew/Cellar/openssl@3/3.0.5 (6,444 files, 27.9MB)
  Poured from bottle on 2022-08-31 at 14:10:49
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/openssl@3.rb
License: Apache-2.0
==> Dependencies
Required: ca-certificates ✘
==> Caveats
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
  /opt/homebrew/etc/openssl@3/certs

and run
  /opt/homebrew/opt/openssl@3/bin/c_rehash

openssl@3 is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS provides LibreSSL.

If you need to have openssl@3 first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/openssl@3/bin:$PATH"' >> ~/.zshrc

For compilers to find openssl@3 you may need to set:
  export LDFLAGS="-L/opt/homebrew/opt/openssl@3/lib"
  export CPPFLAGS="-I/opt/homebrew/opt/openssl@3/include"

For pkg-config to find openssl@3 you may need to set:
  export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl@3/lib/pkgconfig"

Check that everything works by running:

pkg-config --libs --cflags rdkafka

-I/opt/homebrew/Cellar/openssl@3/3.0.5/include -I/opt/homebrew/Cellar/librdkafka/1.9.2/include -I/opt/homebrew/Cellar/zstd/1.5.2/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -L/opt/homebrew/Cellar/librdkafka/1.9.2/lib -lrdkafka

Now, all you need to do is run your tests with the -tags dynamic flag, it will instruct confluent-kafka-go to use the librdkafka library that we’ve built from source.

Thanks for reading and happy hacking!🫢

DeMorgan’s Law

DeMorgan’s law is a simple law that I learned at UPT during one of my hardware classes. While it is useful in hardware it, it is also useful when writing programs.

If you have a condition like not (A and B), you can rewrite it to !A or !B.

if __name__ == '__main__':
    a = True
    b = True

    if not (a and b):
        print("True")
    else:
        print("False")

    if not a or not b:
        print("True")
    else:
        print("False")

Object Pool Pattern

Hi πŸ‘‹

In this article we’ll talk about the Object Pool pattern in Golang.

The Object Pool pattern is a design pattern used in situations when constructing objects is a costly operation, for example building an HTTPClient or DatabaseClient object can take some time.

By having a pool of resources, the resources are requested from the pool when needed and then returned when not needed so they can be reused later.

Programs can benefit from this pattern because once the object is constructed when you need it again, you’ll just grab an instance instead of constructing it again from scratch.

In Golang this pattern is easily implemented with sync.Pool. Given a struct Resource struct, to implement an object pool we’ll need to pass the NewResource function to the pool.

To track how many active instances, we have of the object Resource, we use the counter variable.

Resource

var logger = log.Default()
var counter = 0
 
type Resource struct {
    id string
}
 
func NewResource() *Resource {
    logger.Printf("NewResource called")
    counter += 1
    return &Resource{id: fmt.Sprintf("Resource-%d", counter)}
}
 
func (r *Resource) doWork() {
    logger.Printf("%s doing work", r.id)
}
 

Let’s demo sync.Pool!

Demo 1️⃣

In the first demo, we get the resource from the pool, do some work and then put it back. By doing this one step at the time in the end we’ll end with just one Resource instance.

func demo1() {
	println("demo1")
	theResourcePool := sync.Pool{New: func() any {
		return NewResource()
	}}

	for i := 0; i < 10; i++ {
		item := theResourcePool.Get().(*Resource)
		item.doWork()
		theResourcePool.Put(item)
	}

	println("done", counter)
}

Output

demo1
2022/08/17 22:38:59 NewResource called
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
2022/08/17 22:38:59 Resource-1 doing work
done 1

Resource-1 is the only instance that does work.

Demo 2️⃣

In demo2 we spawn 10 goroutines, that use the pool. Since all goroutines start roughly at the same time and require a resource to doWork, in the end the pool will have 10 Resource instances.

func demo2() {
	println("demo2")
	wg := sync.WaitGroup{}
	theResourcePool := sync.Pool{New: func() any {
		return NewResource()
	}}

	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			item := theResourcePool.Get().(*Resource)
			item.doWork()
			theResourcePool.Put(item)
		}()

	}
	wg.Wait()

	println("done", counter)
}

Output

demo2
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 Resource-3 doing work
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 Resource-4 doing work
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 Resource-5 doing work
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 Resource-6 doing work
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 Resource-7 doing work
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 Resource-8 doing work
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 NewResource called
2022/08/17 22:41:12 Resource-1 doing work
2022/08/17 22:41:12 Resource-2 doing work
2022/08/17 22:41:12 Resource-9 doing work
2022/08/17 22:41:12 Resource-10 doing work
done 10

Demo 3️⃣

In demo3 doing the same thing we did in demo2 with some random sleeps in between, some goroutines are faster and others are slower. The faster goroutines will also return the resource faster to the pool and slower goroutines which start at a later time will reuse the resource instead of creating a new one.

func demo3() {
	println("demo2")
	wg := sync.WaitGroup{}
	theResourcePool := sync.Pool{New: func() any {
		return NewResource()
	}}

	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			time.Sleep(time.Duration(rand.Intn(900)+100) * time.Millisecond)
			item := theResourcePool.Get().(*Resource)
			item.doWork()
			time.Sleep(time.Duration(rand.Intn(100)+100) * time.Millisecond)
			theResourcePool.Put(item)
		}()

	}
	wg.Wait()

	println("done", counter)
}

Output

demo2
2022/08/17 22:42:35 NewResource called
2022/08/17 22:42:35 Resource-1 doing work
2022/08/17 22:42:35 NewResource called
2022/08/17 22:42:35 Resource-2 doing work
2022/08/17 22:42:35 NewResource called
2022/08/17 22:42:35 Resource-3 doing work
2022/08/17 22:42:36 Resource-1 doing work
2022/08/17 22:42:36 Resource-2 doing work
2022/08/17 22:42:36 Resource-3 doing work
2022/08/17 22:42:36 Resource-1 doing work
2022/08/17 22:42:36 NewResource called
2022/08/17 22:42:36 Resource-4 doing work
2022/08/17 22:42:36 NewResource called
2022/08/17 22:42:36 Resource-5 doing work
2022/08/17 22:42:36 Resource-2 doing work
done 5

Only 5 Resource instances have been created at this time.

Conclusion

The object pool pattern is a great pattern when you need to reuse an instance of an object. Constructing the object every time can be slow.

In Go we have sync.pool which implements the Object Pool pattern for us, we just need to give it a New function that returns a pointer.

Thanks for reading! πŸ“š

References

Full Code

package main

import (
	"fmt"
	"log"
	"math/rand"
	"sync"
	"time"
)

var logger = log.Default()
var counter = 0

type Resource struct {
	id string
}

func NewResource() *Resource {
	logger.Printf("NewResource called")
	counter += 1
	return &Resource{id: fmt.Sprintf("Resource-%d", counter)}
}

func (r *Resource) doWork() {
	logger.Printf("%s doing work", r.id)
}

func demo1() {
	println("demo1")
	theResourcePool := sync.Pool{New: func() any {
		return NewResource()
	}}

	for i := 0; i < 10; i++ {
		item := theResourcePool.Get().(*Resource)
		item.doWork()
		theResourcePool.Put(item)
	}

	println("done", counter)
}

func demo2() {
	println("demo2")
	wg := sync.WaitGroup{}
	theResourcePool := sync.Pool{New: func() any {
		return NewResource()
	}}

	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			item := theResourcePool.Get().(*Resource)
			item.doWork()
			theResourcePool.Put(item)
		}()

	}
	wg.Wait()

	println("done", counter)
}

func demo3() {
	println("demo2")
	wg := sync.WaitGroup{}
	theResourcePool := sync.Pool{New: func() any {
		return NewResource()
	}}

	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			time.Sleep(time.Duration(rand.Intn(900)+100) * time.Millisecond)
			item := theResourcePool.Get().(*Resource)
			item.doWork()
			time.Sleep(time.Duration(rand.Intn(100)+100) * time.Millisecond)
			theResourcePool.Put(item)
		}()

	}
	wg.Wait()

	println("done", counter)
}

func main() {
	demo1()
	//demo2()
	//demo3()
}

Go Pattern: Sorting a slice on multiple keys

Hi πŸ‘‹

In this article I want to highlight a simple pattern for sorting a slice in Go on multiple keys.

Given the following structure, let’s say we want to sort it in ascending order after Version, Generation and Time.

type TheStruct struct {
	Generation int
	Time       int
	Version    int
}

The way we sort slices in Go is by using the sort interface or one of the sort.Slice functions. To sort the slice after the above criteria we’ll call slice.Sort with the following function.

	sort.Slice(structs, func(i, j int) bool {
		iv, jv := structs[i], structs[j]
		switch {
		case iv.Version != jv.Version:
			return iv.Version < jv.Version
		case iv.Generation != jv.Generation:
			return iv.Generation < jv.Generation
		default:
			return iv.Time < jv.Time
		}
	})

The slice will be sorted after the following fields: Version, Generation and Time. The trick is the switch statement and the case expression case iv.Version != jv.Version followed by the statement return iv.Version < jv.Version.

You can use this pattern whenever you want to sort slices over multiple fields in Go.

Thanks for reading! 🍻

Source Code

package main

import (
	"fmt"
	"sort"
)

type TheStruct struct {
	Generation int
	Time       int
	Version    int
}

func main() {
	var structs = []TheStruct{
		{
			Generation: 1,
			Time:       150,
			Version:    0,
		},
		{
			Generation: 1,
			Time:       200,
			Version:    0,
		},
		{
			Generation: 1,
			Time:       200,
			Version:    2,
		},
		{
			Generation: 1,
			Time:       500,
			Version:    0,
		},
		{
			Generation: 1,
			Time:       100,
			Version:    0,
		},
		{
			Generation: 1,
			Time:       400,
			Version:    0,
		},
		{
			Generation: 2,
			Time:       400,
			Version:    0,
		},
		{
			Generation: 2,
			Time:       100,
			Version:    2,
		},
		{
			Generation: 1,
			Time:       300,
			Version:    0,
		},
	}

	fmt.Printf("%v\n", structs)

	sort.Slice(structs, func(i, j int) bool {
		iv, jv := structs[i], structs[j]
		switch {
		case iv.Version != jv.Version:
			return iv.Version < jv.Version
		case iv.Generation != jv.Generation:
			return iv.Generation < jv.Generation
		default:
			return iv.Time < jv.Time
		}
	})
	fmt.Printf("%v\n", structs)

}

Output

[{1 150 0} {1 200 0} {1 200 2} {1 500 0} {1 100 0} {1 400 0} {2 400 0} {2 100 2} {1 300 0}]
[{1 100 0} {1 150 0} {1 200 0} {1 300 0} {1 400 0} {1 500 0} {2 400 0} {1 200 2} {2 100 2}]

Also, special thanks to RP.πŸ’–