A Critical Review of Kotlin/Native

Charlotte Skye

2020/12/17

Categories: kotlin review Tags: kotlin

(Note: I am not a mobile developer. K/N is heavily oriented towards mobile developers. I do not care. This post is about desktop development.)

Kotlin/Native is the version of Kotlin that compiles to a single native executable for a desktop platform, as part of Kotlin/Multiplatform. Unfortunately, you can’t do anything with it except play around with collections or print to the console without dropping down to calling glibc or Win32 which is very gross and unergonomic and full of pitfalls.

Instead, I’ve spent the last few months writing a standard library for K/N that bridges the gap between raw C calls and a Kotlin API, called Tinlok. This post is my review of the experience of using Kotlin/Native.

tl;dr Kotlin/Native is very promising, but has some critical issues that make it unsuited for anything beyond toys or demos.

Lifetimes of Unmanaged Objects

In most garbage collected languages, objects allocated in a scope get automatically freed and resources returned to the OS once the scope exits. Kotlin/Native is GC’d, and obeys this for Kotlin objects:

public fun example() {
    val list = mutableListOf(1, 2, 3)
}

When you call example(), the list is allocated in memory. When example() returns, the list is deallocated. Pretty simple! The list is an example of a managed object, as it is managed by the language runtime.

Of course, in most languages, allocating and deallocating memory is not very useful by itself (outside of certain esoteric situations). Usually, you want to do some I/O, such as on files or the network.

Creating Unmanaged Objects

Python is a widely used garbage collected language, which uses calls to the underlying operating system to perform tasks, such as opening a file. Let’s do a comparison between how Kotlin/Native handles things, and how Python handles things. For the sake of comparison, I’m ignoring the Python open function (as we will write our own for demonstration instead).

First, the naiive Kotlin/Native version:

fun readFile() {
    val fd: Int = platform.posix.open("/proc/version", O_RDONLY)
    val buf = ByteArray(2048)
    buf.usePinned {
        platform.posix.read(fd, buf.addressOf(0), 256)
    }
    val version = buf.toKString()
    println("Our Linux version is ${version}")
}

Pretty gross. Let’s look at the Python version, using similarly low-level APIs:

import os

def read_file() -> None:
    fd = os.open("/proc/version", os.O_RDONLY)
    version = os.read(f, 256) 
    print(f"Our Linux version is {version}")

Cleaning Up Unmanaged Objects

In both of these, the version string is managed by the garbage collector, so they will be automatically freed when the function exits. However, in both of these, the file descriptor fd is managed by the operating system; it’s just a regular integer. The GC isn’t aware of this. So when the function exits, the file referred to will remain open forever. This is called an unmanaged object, and is created by calling out to a C function (C FFI will be expanded upon later). Luckily, we can fix this.

In Kotlin/Native:

fun readFile() {
    val fd: Int = platform.posix.open("/proc/version", O_RDONLY)
    val buf = ByteArray(2048)
    buf.usePinned {
        platform.posix.read(fd, buf.addressOf(0), 256)
    }
    val version = buf.toKString()
    println("Our Linux version is ${version}")
    platform.posix.close(fd)
}

In Python:

import os

def read_file() -> None:
    fd = os.open("/proc/version", os.O_RDONLY)
    version = os.read(f, 256) 
    print(f"Our Linux version is {version}")
    os.close(fd)

Now the fd is always closed when the function returns. Unless…

Exceptions and Unmanaged Objects

/// example.kt:
// Uh oh!
internal fun println(message: String) = throw Exception("Evil!")

/// example.py:
def print(*args, **kwargs):
    raise Exception("Evil!")

Somebody’s snuck into our codebase and replaced the definition of our print functions. Both K/N and Python have exceptions, which can be thrown anywhere, disrupting control flow. Now, our close() functions will never get called, as the function returns early. Luckily, there’s some solutions (and, no, “don’t use exceptions” is not a solution. Go away.)

In Python, we can use a context manager. This consists of a function called when entering the context manager, and a function called when exiting the context manager. This acts as a more ergonomic way of writing try/finally cleanup statements, and are used in nearly all Python code which is handling some closeable resource. The Python standard library provides the open function for files which is what people normally use, but let’s write our own for demonstration. We will also add a helper function to clean up our API.

class FileWrapper(object):
    def __init__(self, path: str):
        self._fd = os.open(path)

    def __enter__(self):
        pass

    def read(self, count: int = 256) -> bytes:
        return os.read(self._fd, count)

    def __exit__(self, *args):
        if self._fd:
            os.close(self._fd)

        ## Re-raise exceptions that may have happened.
        return False

def read_file() -> None:
    with FileWrapper("/proc/version") as f:
        version = f.read(256)
        print(f"Our Linux version is {version}")

Now, when the evil print raises, the file will still be closed no matter what.

Kotlin doesn’t really have the same thing, but we can emulate it using lambdas, in a similar manner to java.util.AutoCloseable:

public class FileWrapper(path: String) {
    private val fd = platform.posix.open(path)

    public fun read(count: Int = 256): ByteArray {
        val buf = ByteArray(count)
        buf.usePinned {
            platform.posix.read(fd, buf.addressOf(0), 256)
        }
        return buf
    }

    public fun close() {
        platform.posix.close(fd)
    }
}

public inline fun <R> FileWrapper.use(
    path: String,
    block: (FileWrapper) -> R
): R {
    val wrapper = FileWrapper(path)
    return try {
        block(wrapper)
    } finally {
        wrapper.close()
    }
}

public fun readFile() {
    val wrapper = FileWrapper("/proc/version")
    wrapper.use {
        val buf = it.read()
        val version = buf.toKString()
        println("Our Linux version is ${version}")
    }
}

Now, if an exception happens in our evil println(), the file will always be close. Simple. Kind of.

API Cleanliness

There is a problem with this approach! If you’re an application writer, fine, you can just make sure you don’t get any exceptions thrown (similar vibe to “just write better C code”). If you’re a library developer, you can’t ensure your objects are properly closed because there’s nothing in the compiler stopping somebody from just using your object, unless you design it specifically. For example, your wrapper class can have a private constructor, then you must invoke a function with a lambda to get an instance:

// FileWrapper now only has a private constructor, so can't be made externally.
public class FileWrapper private constructor(path: String) { ... }

public inline fun <R> openFile(
    path: String, block: (FileWrapper) -> R
): R {
    val wrapper = FileWrapper(path)
    return wrapper.use(block)
}

This looks pretty horrible if you have multiple objects you need to close.

public fun doesSomething() {
    openResourceOne { one ->
        openResourceTwo { two ->
            openResourceThree { three ->
                // Getting dangerously close to that right margin!
            }
        }
    }
}

It also lets you shoot yourself in the foot pretty spectacularly:

public fun terrible() {
    val file = openFile("...") { it -> it }  // Oops! file is closed at the lambda exit!
    file.read(/*...*/)  // At best, this fails, at worst you now fucked up some other fd's state
}

Alternatively, you can mandate a DeferScope or a similar construct to be passed to all your constructors, which will register an auto-closer.

public class FileWrapper(scope: DeferScope, path: String) {
    init {
        scope.defer { close() }
    }
    // ...
}

public fun ugly() = memScoped {
    val file = FileWrapper(this, "/proc/version")
    val file2 = FileWrapper(this, "/proc/cpuinfo")
    // etc, need to pass ``this`` each time
}

(DeferScope and defer are not referred to anywhere outside of API docs.)

A developer, when faced with these design choices, will go “fuck you” and completely ignore your safe API (Evidence: Look at any Rust vs C/++ discussion online).

An escape hatch?

All hope is not lost! Usually, we have an escape hatch we can use to automatically close our resources when a wrapper object is garbage collected. This is usually named something like a Finalizer. Many languages have this:

Let’s modify our original Python FileWrapper to use __del__ to ensure that the file descriptor is ALWAYS closed.

class FileWrapper(object):
    def __init__(self, path: str):
        self._fd = os.open(path)

    def __enter__(self):
        pass

    def read(self, count: int = 256) -> bytes:
        return os.read(self._fd, count)

    def __exit__(self, *args):
        if self._fd:
            os.close(self._fd)

        ## Re-raise exceptions that may have happened.
        return False

    def __del__(self):
        os.close(self._fd)

Simple. Whilst you should never rely on __del__ for cleanup, it’s good to use it in the case that a developer forgets to use with.

Now let’s modify our Kotlin file wrapper to clean up in case the lambda isn’t used:

public class FileWrapper() {
    // Uhh...
}

Kotlin/Native has no concept of this.

Implications

If you want to use an unmanaged external resource, such as an externally allocated library struct, or a file descriptor, or literally anything, then either:

  1. You wrap it all in a DeferScope nested hell
  2. You wrap it all in a lambda nested hell
  3. You wrap it all in a try/finally nested hell
  4. You give up and tell your developers to just Write Better Code.

(I believe that most developers would go for option 4). This makes K/N unsafe by design; sure, you can work around it with lambda hell, but that is a failure of language or runtime design to require a workaround to avoid resource leaks.

How could this be fixed?

Kotlin/Native needs some sort of finalizer. Given the memory model, such a finalizer would have stronger guarantees than most implementations; objects are frozen and immutable and as such a second reference can’t be created (easily). Something akin to the Cleaner API from Java would suffice, or even just a Finalizable interface that can be implemented.

FFI / C Interop

In the previous section, certain actions were performed by calling out into the C standard library, such as the open and read functions; these are an example of C FFI calls.

Kotlin/Native, being a compiled language, supports calling C functions through its C FFI, which it calls cinterops. The gist of it is:

  1. Write a .def file that contains the list of C header files.
  2. Run the cinterop tool (usually via Gradle).
  3. Call the generated stub functions in your Kotlin code.

This happens automatically, and 95% of the time is pretty good. It’s one of the strong points of Kotlin/Native, even if the complex class hierachy takes a little while to understand properly. But that other 5% is pretty damned annoying.

Library Paths, Includes, Linking, and Static Linking

In a typical compilation workflow, using a C function is achieved by first compiling your code into object files, including the definition of said function. This function need not be implemented yet if it is contained in an external file. These definitions are stored in header files (file.h), which are included in a source file using the C preprocessor, and the compiler will emit unresolved symbols that need to be resolved with the real functions (rather than the definitions).

Kotlin/Native works similarly; you define either a list of headers or the declarations themselves in a .def file. The cinterop tool processes this, and produces a stub library that only contains the definitions and no implementations. The Kotlin/Native compiler then uses these definitions when compiling your code to produce object files, ready to be linked.

The second stage of compiling is called the link stage. In this stage, the linker takes all the libraries you want to link your application or library against, looks at the unresolved symbols in your object files and produces a final application or binary with the locations of the real symbols there instead. Pretty simple (not really)!

In order to do this, however, you need two things:

  1. The header files containing the definitions
  2. The library files containing the implementations

Finding these files is an unsolved problem in computer science, and has lead to thousands of competing implementations to solve this.

Finding Files

In the simplest case, a .def file can just list the headers to generate stubs for.

package = external.openssl
headers = openssl/bio.h openssl/err.h openssl/pem.h openssl/ssl.h openssl/x509.h openssl/x509v3.h 
          openssl/pem.h openssl/asn1.h openssl/bn.h
headerFilter = openssl/*

So, where are these header files located? You can instruct the compiler to look in certain locations with the compilerOpts directive. This is useful for external libraries that are commonly installed, such as util-linux, or OpenSSL, or similar.

compilerOpts.linux_arm64 = -I/usr/include

This has a chance to immediately break your Kotlin/Native install:

Please don’t do this, as it is generally harmful, because Kotlin/Native uses its own toolchain (see above), and by adding /usr/include you get two toolchains mixed in your include path.

(Curiously, the docs suggest using -I/usr/local/include anyway.)

Okay. Fine. So you can’t rely on your OS packages. Very cool! Definitely not inconvenient. Instead, you have to either vendor your headers directly, or download them at build time:

val downloadWin64Binaries by tasks.registering(Download::class) {
    src("https://github.com/glfw/glfw/releases/download/$glfwVersion/glfw-$glfwVersion.bin.WIN64.zip")
    dest(downloadsDir.resolve("glfw.bin.WIN64.zip"))

    overwrite(false)
}
val unzipWin64Binaries by tasks.registering(Copy::class) {
    from(downloadWin64Binaries.map { zipTree(it.dest) }) {
        eachFile {
            relativePath = RelativePath(true, *relativePath.segments.drop(1).toTypedArray())
        }
        include("glfw-*/include/**", "glfw-*/lib-mingw-w64/**")

        includeEmptyDirs = false
    }
    into(glfwWin64Dir)
}

(credit: KGL)

This sucks! You have to add this custom logic to basically every buildscript that requires an external library that isn’t one header file large. This also quite obviously won’t work for dynamic libraries, as Gradle buildscripts don’t carry across publications.

Static Linking

As established, using /usr/include is a no-go. Using /usr/lib is probably a similar situation which means dynamic linking is out of the equation. This means either statically linking your library, or translating the library into Kotlin and including it. Let’s look at static linking.

To statically link a library into your project, you can use the staticLibraries and libraryPaths directives in a .def file to include the .a file directly into your final klib or application. This works fine if your OS provides static libraries built for the package you need; if it doesn’t (e.g. on Arch) you will have to distribute the .a file yourself. However, for whatever reason libraryPaths only takes absolute paths; relative paths, such as to your build directory, do not work. Instead, you have to drop down to Gradle configuration:

            val sourceSet = defaultSourceSetName
            val libPath = "src/$sourceSet/$LIB_NAME.a"
            val staticLib = project.file(libPath)
            val path = staticLib.absolutePath

            kotlinOptions {
                freeCompilerArgs = listOf("-include-binary", path)
            }

This is undocumented except for in the konanc help string.

Attack of the 219 Foot GNU C Library

This is a minor side point, but one that bit me several times for months. When statically linking a library (or, when your library is linked), the K/N toolchain uses its own glibc and other system libraries to link against. This is a compile-time only thing, as the final library dynamically loads your own libraries, anyway.

However, the libraries in the system root are VERY outdated:

$ exa -l libc.so.6
lrwxrwxrwx 12 char 11 Nov  2016 libc.so.6 -> libc-2.19.so
$ bin/x86_64-unknown-linux-gnu-gcc --version
x86_64-unknown-linux-gnu-gcc (crosstool-NG crosstool-ng-1.22.0) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.

When compiling static libraries for usage in a K/N project, keep in mind to use the konanc GCC instead of your native GCC to ensure feature detection of e.g. getrandom in autoconf scripts works correctly for the old toolchain.

How could this be fixed?

Integrate with something akin to vcpkg or Conan. This would avoid having to download library headers manually, avoid finicky OS package manager locations, and avoid having to distribute your own static libraries for every dependency.

Writing FFI Code

Let’s get to actually writing some code that uses C interops. First, some basic by-value parameters:

val floored = platform.posix.floor(1.5)

Simple enough. This is the simplest scenario for FFI; you pass a number by value and get a number by value in return. Of course, most FFI code doesn’t just operate on numbers; it uses pointers. Let’s add a pointer to the mix, by allocating something:

memScoped {
    val byte = alloc<ByteVar>()
}

This allocates a single one-byte variable on the heap. ([Primitive]Var is the C FFI version of regular primitive types). You can get or set the value of this ByteVar with the value property.

In order to get the actual pointer to this variable, you use the ptr extension on the type. So now you have a CPointer<ByteVar>, which acts as a pointer to the number on the heap. This CPointer type is what you usually want to pass around to things.

This is not very useful, however. A single byte? Very few functions take a single uint8_t. Let’s allocate an array instead, initialised to be all-zero. This corresponds to a char* array, or a C string.

memScoped {
    val buf = allocArray<ByteVar>(256)
    platform.posix.memset(buf, 0, 256)
}

Cool, now we can pass things around to this array. Let’s set some values on it, like a few characters of a string:

buf[0] = 109
buf[1] = 111
buf[2] = 111
buf[3] = 110

Now let’s check the length of this string, as an example:

val length = strlen(buf)

Uh oh. This won’t compile, because strlen expects a kotlin.String. Wait, what? Kotlin strings are Pascal-strings, not C-strings, how does that work?

Automatic String Conversion

Kotlin/Native supports something called automatic string conversion, where parameters or return values of type const char* are automatically converted into a kotlin.String. This means you can pass strings directly to functions expecting const char* without needing a manual conversion step.

This is an opt-out mechanism, so all functions will get the string conversion unless explicitly told not to in the .def file. Making it be opt-out makes a lot of wrong assumptions because const char* is not always a valid UTF-8 string like Kotlin assumes; it can be treated as just an opaque set of bytes. In theory, as an opt-in only mechanism, it would be a good feature.

strlen() and friends

Yeah, this one is pretty simple. strlen gets the length of a zero-terminated string. Kotlin strings are pascal strings. There’s no usecase ever where I want to strlen() a kotlin.String, but there are usecases where I want to strlen a C string.

socket options on Windows

With WinSock2, getsockopt and setsockopt use const char* as the option type. Whilst that may be a stupid design on the behalf of Windows, automatic string conversion means you simply can’t call these two functions from Kotlin/Native.

Linux File API

One of the assumptions made for automatic string conversion is that C strings are in UTF-8. When it comes to Linux filesystems this is simply not true; any sequence of bytes can be a filename, and likewise any sequence of bytes can be passed to the C functions. Automatic string conversion means you simply can’t interact with any files with invalid Unicode in their name. This has a pretty good chance of breaking applications e.g. you can’t do a recursive delete on a folder with bad files, thus throwing random exceptions.

Always On The Heap

Another minor point is that things nearly always have to be allocated directly on the heap (in some form). If you are familiar with C, you may be familiar with allocating a local variable then passing it as a pointer, usually for usage with size parameters (this pattern is more common in the Win32 APIs, for size parameters). With K/N, you have to allocate directly on the heap instead:

memScoped {
    val readCnt = alloc<UIntVar>()
    val res = ReadFile(hnd, addr, size, readCnt.ptr, null)
}

It would be nice to be able to pass a var of a primitive type to the function instead.

Pinning

Using C-allocated buffers is unmanaged (see above for the pitfalls) and is worse than being able to use a native ByteArray that is fully managed and can act as a collection. Let’s do it:

val buf = ByteArray(256)
recv(fd, buf, 256)  // Nope!

That won’t work. These functions expect a pointer, not an array. Luckily, we can get a pointer reference:

val buf = ByteArray(256)
recv(fd, buf.refTo(0), 256)

This may corrupt memory, as Kotlin is free to swap out that ByteArray from underneath you, and recv will gladly trash memory. I’ve certainly had a memory corruption bug where I passed a pointer like this and had a system call trash my memory. The solution you are meant to use is in fact the usePinned helper:

val buf = ByteArray(256)
buf.usePinned {
    recv(fd, it.addressOf(0), 256)
}

Keen eyes will notice I used this earlier on. Even keener eyes will notice that requiring this is mostly undocumented, except for a passing mention in the C interop docs that don’t really explain why or when to use pinning.

HMPP And Commonizer

HMPP (Hierarchal Multiplatform Projects) is a way of sharing code effectively between different related targets. An HMPP project can have targets for Linux on different architectures share a common Linux code source set, for example. This avoids having to write the same code over and over again for platform-specific APIs. In theory.

The commonizer is a feature added with Kotlin 1.4 that will “commonize” cinterop libraries shared by a hierarchal source set. For example, with a HMPP setup where linuxX64 and linuxArm64 both depend on a linuxMain source set, the posix and linux platform libraries are processed and a commonized form is added to the linuxMain source set. This also avoids having to write the same code that uses functionality common between platforms, e.g. BSD socket code which is supported by both Linux and Windows (via WinSock2).

In practice, however, it falls apart quite quickly.

Missing Platform Libraries

This is my personal number one gripe with Kotlin/Native because it is so trivially stupid I don’t see why it’s even a problem.

K/N comes in prebuilt variants for each platform, which the Kotlin/Multiplatform plugin downloads at the first run. On Linux AMD64, this contains the platform libraries for Linux AMD64, Linux ARM-HPF and AArch64, Linux MIPS32 and MIPSEL32, and Android ARM32 and AArch64. Note, the lack of MinGW platform libraries. On Windows AMD64, the build contains the platform libraries for Linux AMD64, Linux ARM-HPF and AArch64, Android AMD64 and X86 (?), and Android ARM32 and AArch64.

The commonizer for platform libraries runs based on what is included in this prebuilt distribution. If you have a common sourceset for both Windows and Linux, when building under Linux the commonizer will NOT produce a common platform library for said common sourceset.

However, if you download a copy of the Windows prebuilt compiler, and take the mingwX64 platform library, copy it into your Linux klib directory, delete the commonized files, and force the commonizer to re-run, a commonized platform library WILL be generated. It even works this way on Windows! Your IDE will work fine, and the program will be compiled. I see literally no reason why the platform library is missing on Linux builds. It also kneecaps your ability to write code for Windows properly, since you get zero help from your IDE in the Windows sourceset until you add the missing library.

No commonizing on user libraries

It is now 3am. You have designed your API interfaces to account for a lack of destructors. You have written an unholy Gradle buildscript to download and link against the library you’re writing bindings for. You have copied the platform libraries into your local compiler. You are finally ready to start writing your implementation code.

The joke is on you here, because you’re gonna have to amend that buildscript some more to copy over the bindings for each platform, because the commonizer doesn’t work for user-provided libraries. In my opinion this makes it borderline useless. The bindings are going to be the exact same for each platform in nearly every case. I understand this is a “planned feature”, but it’s still incredibly frustrating to have to add yet another set of files to the copyCommonCBindings Gradle task.

Expect/Actual: Expect Greatness, Actual Meh

(That subtitle was hyperbolic. I just wanted to get an expect/actual joke in there.)

Kotlin/Multiplatform provides the expect and actual constructs to make creating multiplatform software easier. In a common sourceset, you use expect to define some structure that will be present in the library or application, and then in a platform sourceset you use actual to implement the function/class/etc. This works out well in most cases, and is a mostly well designed feature, but it still has some very annoying rough edges.

Expect Interfaces

When defining an actual interface, all the properties and functions inside the actual interface need to be re-defined, even if you don’t implement them. I don’t see any reason why you need to do this; you can’t meaningfully change the empty declarations between expect and actual.

Actual Typealiases

A common pattern is to use an actual typealias. This breaks for unsigned types as they are marked as inline. Just a minor annoyance that I have actually come across.

Conclusion

Despite the issues mentioned above, I think Kotlin/Native is a promising language/runtime. I like Kotlin, I think it’s a great language, and I like native binaries (who doesn’t?). I hope that the Kotlin team fixes these issues so that K/N can be the best it can be. I’m definitely going to continue writing my libraries so that they are ready to be used when K/N is usable. However, K/N is currently not suitable for any serious work because of the finalizer flaw, and is often frustrating due to a subpar experience when it comes to handling external C libraries.

>> Home