Moving to Kotlin Multiplatform

[JohnLCaron]

We are trying out using okio library to move to Kotlin Multiplatform. Most of the interest in KMP is for mobile development like KMP Compose, and most examples are for UI programming. Still rather immature for building native code libraries.

Also trying com.fleeksoft.charset for a port of Java Charsets.

Eventually we may be able to move to kotlinx.io, currently in alpha:

"A multiplatform Kotlin library providing basic IO primitives. kotlinx-io is based on Okio but does not preserve backward compatibility with it."

[JohnLCaron]

As of version 0.3 (6/10/25), we have moved our IO to okio (but not yet KMP).

It seems that theres some slowdown, see clibs/testStatus.txt. There may be an extra data copy.

1. Okio doesnt have the same buffering strategy as RandomAccessFile.
2. One has to copy to/from a Buffer, rather than from a ByteArray. This may be a necessary abstraction for MP.

To be investigated

[JohnLCaron]

It looks like okio uses JVM 1.8 bytecode. Dunno if thats a performance problem.

tasks.withType().configureEach {
compilerOptions {
jvmTarget = JvmTarget.JVM_1_8
freeCompilerArgs.add("-Xjvm-default=all")
}
}

[JohnLCaron]

Investigating kotlinx a bit more. 

started 10/4/17, 1 FTE+ until 3/28/19 (Sergey Maskov)

Ilya Ryzhenkov. took over, posetd a rather ambitious README posted 4/29/19:

# Structure

NOTE: This part represents the target vision of the package. Most modules are still unavailable.

`kotlinx-io` package has many modules and you can use as much as you need. 

* `core` – defines all the low-level functionality for working with binary data and basic text.
* `async` – (unavailable) implements asynchronous versions of input and output data streams.
* `platform` – (unavailable) provides low-level platform facilities such as reading or writing from/to a file.
* `sockets` – (unavailable) provides low-level functionality for interacting with network.
* `cryptography` – (unavailable) provides encryption & decryption functionality.
* `compression` – (unavailable) provides compression & decompression functionality.
* `files` – (unavailable) provides advanced file system functionality such as working with paths and directories. 

# Core 

This module provides few core I/O primitives that are used across other modules and can be used to define
custom binary inputs and outputs, as well as processing raw memory.

* `Buffer` – represents a continuous memory block of specific size. 
Provides direct positional read and write operations for primitives and arrays of primitives. 
* `Input` – represents a source of bytes. Provides sequential reading functionality and a special `preview` mode
for processing bytes without discarding them. 
* `Output` – represents a destination for bytes. Provides sequential writing functionality. 
* `Bytes` – represents binary data of arbitrary size, potentially spanned across several buffers.
Can be built using using `Output` and can be read using `Input`.

It also has basic facilities for working with text, with UTF-8 implemented efficiently in core, 
and `Charsets` giving access to platform-dependent functionality for converting text into bytes and back.

## Buffers

Buffer is direct representation of memory on the target platform implemented using efficient platform-dependent 
mechanisms. A buffer of arbitrary size can be allocated and released using `PlatformBufferAllocator`. 
It is user's responsibility to release an allocated buffer. 

All operations with a `Buffer` are performed in network byte order (Big-Endian). 
There are helper functions `reverseByteOrder` defined for all primitive types to reverse the byte order when it is needed. 

## Inputs

An `Input` is a high-performance buffered entity for reading data from an underlying source.  
It is an abstract class with only few abstract methods and a plentiful of convenience built around them. 
There are functions to read primitives, arrays of primitives, higher-level extension methods for reading UTF-8 text,
text encoded with a custom `Charset`, and more. One can define any other read methods using extensions and provided primitives.

`Input` design doesn't provide facilities for direct manipulation of the current reading position, 
but instead it has the `preview` mechanism which we believe is a lot safer, efficient and enough for most look-ahead
scenarios.

Preview operation instructs `Input` to start accumulating buffers instead of discarding them when they are exhausted,
thus making it possible to revert to the initial position without performing additional I/O operations. 


    input.readLong() // (0) reads long value and discards bytes
    input.preview {  // (1) begins preview operation and stops discarding bytes
        readShort() // (2) read short value and keep the bytes
    } // completes preview operation and rewinds the input to the state (1) 
    input.readShort()  // (3) reads short value from (2) again


Note that `preview` function provides another, nested `Input` to the lambda as a receiver 
which should be used for all preview reads. 
Implementation can choose to alter original `Input` state or create a new instance, 
so one should always be using the instance provided as a receiver to `preview`. 

Preview operations can be nested, each keeping its own state and position, thus making it possible to compose
operations on Inputs.

## Outputs

An `Output` is a high-performance buffered entity for writing data to an underlying destination. 
Like `Input`, it provides all the primitive operations as well as a number of convenience functions for text output.

Similarly, `Output` doesn't provide a mechanism to rewind backwards and update data, but using `Bytes` one can easily
implement complex scenarios such as writing a size before a block, calculating hashes and so on. 

## Bytes

A `Bytes` type is useful for transferring data between various endpoints, accumulating data in memory or sending 
repetitive bytes to different outputs. 

`Bytes` can be produced by building function `buildBytes { … }` where lambda has an `Output` as a receiver, 
thus making it possible to conveniently generate content, or use in any I/O operations or custom user's functions. 


    val bytes = buildBytes {
        writeLong(0x0001020304050607)
        writeShort(0x0809)
    }


When you have a `Bytes` instance, you can know the number of bytes stored, and can obtain an `Input` to read these bytes.
Creating an `Input` is a zero-copy operation, underlying mechanics simply reuses buffers for reading data.

    val input = bytes.input()
    input.readLong() 


Writing such an instance into `Output` is also zero-copy operation, since implementation will send existing buffers
to the underlying destination. 


    output.writeBytes(bytes)


Combining these features makes it possible to write domain-specific functions for complex data writing:


fun Output.writeWithSizeAndHash(writer: Output.()->Unit) {
    val bytes = buildBytes(writer)
    writeInt(bytes.size)
    writeBytes(bytes)
    val hash = bytes.input().calculateHash()
    writeLong(hash)
}


## Text

[TBD] Efficient UTF-8 and platform-dependent Charsets

## Pools

[TBD] Allocating and releasing a buffer each time one is needed can be inefficient, 
so the package provides facilities for buffer pools.

# Async

[TBD] `InputChannel` and `OutputChannel` as an asynchronous (suspending) versions of `Input` and `Output`
 

# Platform

[TBD]
* `FileInput` and `FileOutput` with a very limited set of operations such as `open`. No paths, no directories, no access control.
* `Process` type for launching an external processes, and interacting with their inputs and outputs.
* `Environment` type for interacting with environment variables.

sporadic work, thsn marked obsolete in 5/25/22:

The library was an experiment for providing a cross-platform multiplatform IO primitives
and its development and maintenance was effectively postponed in 2019.

Its current API is frozen at version 0.1.16 and won't receive any further API updates, 
improvements or bugfixes.

### Future replacement

Future replacement is to be discussed, currently we do recomment using [Okio](https://github.com/square/okio/)
for multiplatform IO primitives.

[JohnLCaron]

10/19/22, project forked and remade by Vsevolod Tolstopyatov probably using the recent code from okio

5/2/23 Filipp Zhinkin: changed Readme to ~ current, remove obsolete status.

6/27/23 Filipp Zhinkin

A trimmed-down API (#136)

What changed compared to the previous API version:
- Removed byte strings and ByteString related APIs
- Moved some methods that were previously implemented as member functions to extensions, provided basic implementation
- Get rid of hash functions and select API
- Get rid of unsafe cursors
- Updated documentation
- Changed Buffer::toString behavior
- Improved test coverage
- Changed naming of read/write methods, explicitly documented exception policies
- Added extensions for unsigned types
- Made code-point-related extensions internal
- On JVM, Buffer no longer implements NIO interfaces
- Buffer has default equals/hashCode

A few new build features were added:
- Enable Dokka
- Enable Kover for core-io
- Enabled explicit API mode

9/14/23 new section of README:

**kotlinx-io** is built around `Buffer` - a mutable sequence of bytes.

`Buffer` works like a queue, allowing to read data from its head or to write data to its tail.
`Buffer` provides functions to read and write data of different built-in types, and to copy data to or from other `Buffer`s.
Depending on the target platform, extension functions allowing data exchange with platform-specific types are also available.

A `Buffer` consists of segments organized as a linked list: segments allow reducing memory allocations during the buffer's expansion and copy,
with the latter achieved by delegating or sharing the ownership over the underlying buffer's segments with other buffers.

**kotlinx-io** provides interfaces representing data sources and destinations - `Source` and `Sink`,
and in addition to the *mutable* `Buffer` the library also provides an *immutable* sequence of bytes - `ByteString`.

An experimental filesystem support is shipped under the `kotlinx.io.files` package,
which includes the `FileSystem` interface and its default implementation - `SystemFileSystem`.

`FileSystem` provides basic operations for working with files and directories, which are represented by yet another class under the same package - `Path`.

work by Filipp Zhinkin gets more consistent, 8/2/24 Jake Wharton (square) starts contributing.
gets more sporadic at end of 2024 through 2/25.
Almost nothing since 2/26/25.

[JohnLCaron]

Okio seems fine. benchmark.