Bubbler

English | 简体中文

Bubbler is a proto generator optimized for IoT devices. It compiles the .bb proto file and generates the output in the specified target language.

Bubbler's proto is powerful and can be non-byte-aligned, which is useful for IoT devices with limited resources. Explained below.

Also, you may need syntax highlighting for .bb files, see bubbler-vscode, or install it from VSCode Marketplace.

Warning: Bubbler is still in development and is not ready for production use.

Installation

git clone https://github.com/xaxys/bubbler.git
cd bubbler
make

Usage

bubbler [options] <input file>

Options

• -t <target>: Target language
• -o <output>: Output Path
• -rmpath <path[,path...]>: Remove Path Prefix (Remove the path prefix of the output file path when generating files) This option is usually used when generating Go target. For example, if a .bb file has go_package option set to github.com/xaxys/bubbler/proto/rpc, the generated file will be generated in the output/github.com/xaxys/bubbler/proto/rpc directory. If you want to remove the path prefix github.com/xaxys/bubbler/proto/rpc, you can set this option to github.com/xaxys/bubbler/proto. Then the generated file will be generated in the output/rpc directory.
• -inner: Generate Inner Class (Nested Struct)
• -single: Generate Single File (Combine all definitions into one file, instead of one generated file per source file)
• -minimal: Generate Minimal Code (Usually without default getter/setter methods)
• -decnum: Force Generate Decimal Format for Constant Value (Translate 0xFF to 255, 0b1111 to 15, etc.)
• -memcpy: Enable memory copy for fields (Duplicate content of string and bytes fields when decoding, instead of directly referencing the original buffer)
• -signext <method>: Sign Extension Method used for Integer Field (Options: shift, arith)

Examples

bubbler -t c -minimal -o output/ example.bb
bubbler -t c -single -o gen.hpp example.bb
bubbler -t py -decnum -signext=arith -o output example.bb
bubbler -t go -rmpath github.com/xaxys/bubbler/proto -o output example.bb

Target Languages

Run bubbler to see the list of supported target languages.

Targets:
  c
  cpp
  csharp [cs]
  commonjs [cjs]
  go
  java
  python [py]

When selecting the target language, you can use the aliases inside []. For example, python can be abbreviated as py.

• dump: Output the parse tree (intermediate representation) of the .bb file.

• c: C language, output one .bb.h file and one .bb.c file for each .bb file.

  • With -single: Output one file that includes all definitions for all .bb files. The output file name (including the extension) is determined by the -o option.
  • With -minimal: No generation of getter/setter methods for fields.
  • With -memcpy: Use malloc to heap-allocate memory for string and bytes fields, and copy the content from the original buffer.
  • Without -memcpy: Pointer reference to the original buffer for string and bytes fields. Zero-copy and zero-heap-allocate.

• cpp: C++ language, output one .bb.hpp file and one .bb.cpp file for each .bb file. The folder structure will not be affected by the cpp_namespace option.

  • With -single: Output one file that includes all definitions for all .bb files. The output file name (including the extension) is determined by the -o option.
  • With -minimal: No generation of getter/setter methods for fields.
  • With -memcpy: Use std::shared_ptr<uint8_t[]> to heap-allocate memory for bytes fields, and copy the content from the original buffer. string fields will always use std::string and be copied every time.
  • Without -memcpy: Use std::shared_ptr<uint8_t[]> with null deleter to reference the original buffer for bytes fields. string fields will always use std::string and be copied every time.

• csharp: C# language, output one .cs file for each structure defined in each .bb file. The folder structure will not be affected by the csharp_namespace option.

  • With -single: Output one file that includes all definitions for all .bb files. The output file name (including the extension) is determined by the -o option.
  • With -memcpy: Use byte[] as the type for bytes fields. Encode and decode methods will only be compatible with byte[] parameters. Older .NET Framework versions should use this option.
  • Without -memcpy: Use Memory<byte> as the type for bytes fields. Encode and decode methods will be compatible with byte[], Memory<byte> and Span<byte> (encode only) parameters. The System.Memory package is required for this case.

• commonjs: CommonJS module, output one .bb.js file for each .bb file. (Please note that BigInt is used for int64 and uint64 fields, which is not supported in some environments.)

  • With -single: Output one file that includes all definitions for all .bb files. The output file name (including the extension) is determined by the -o option.
  • Force enabled: -memcpy.

• go: Go language, output one .bb.go file for each .bb file. The folder structure will be affected by the go_package option. (i.e., github.com/xaxys/bubbler will generate in the github.com/xaxys/bubbler directory)

  • With -single: Output one file that includes all definitions for all .bb files. The output file name (including the extension) is determined by the -o option. The package name is determined by the package statement of the input .bb file.
  • With -memcpy: Make a copy of the bytes field when decoding. The string field will always be copied.
  • Without -memcpy: A slice of the original buffer will be assigned to the bytes field. The string field will always be copied.

• java: Java language, output one .java file for each structure defined in each .bb file. The folder structure will be affected by the java_package option. (i.e., com.example.rovlink will generate in the com/example/rovlink directory)

  • Force enabled: -memcpy.

• python: Python language, output one _bb.py file for each .bb file.

  • With -single: Output one file that includes all definitions for all .bb files. The output file name (including the extension) is determined by the -o option.
  • Force enabled: -memcpy.

Protocol Syntax

Bubbler uses a concise syntax to define data structures and enumeration types.

See examples in the example directory.

Package Statements

Use the package keyword to define the package name. For example:

package com.example.rovlink;

The package name is used to generate the output file name. For example, if the package name is com.example.rovlink, the output file name is rovlink.xxx and is placed in the ${Output Path}/com/example/ directory.

Only one package statement is allowed in a .bb file, and it can not be duplicated globally.

Option Statements

Use the option keyword to define options. For example:

option omit_empty = true;
option go_package = "example.com/rovlink";
option cpp_namespace = "com::example::rovlink";
option csharp_namespace = "Example.Rovlink";
option java_package = "com.example.rovlink";

The option statement cannot be duplicated in a .bb file.

Warning will be reported if an option is unknown.

Supported Options

omit_empty

If omit_empty is set to true, the generated code will not generate files without typedefs.

package all;

option omit_empty = true;

import "rovlink.bb";
import "control.bb";
import "excomponent.bb";
import "excontrol.bb";
import "exdata.bb";
import "host.bb";
import "mode.bb";
import "sensor.bb";

In this example, the omit_empty option is set to true, and this .bb file will not generate an all.xxx file.

You can use this option to generate multiple .bb files at once, without writing an external script to do multiple bubbler calls.

go_package

If go_package is set, the generated code will use the specified package name in the generated Go code.

cpp_namespace

If cpp_namespace is set, the generated code will use the specified namespace in the generated C++ code.

csharp_namespace

If csharp_namespace is set, the generated code will use the specified namespace in the generated C# code. The folder structure will not be affected.

java_package

If java_package is set, the generated code will use the specified package name in the generated Java code. The generated folder structure will be based on the package name.

Import Statements

Use the import keyword to import other Bubbler protocol files. For example:

import "control.bb";
import "a.bb";

Enumeration Types

Use the enum keyword to define enumeration types. The definition of an enumeration type includes the enumeration name and enumeration values. For example:

enum FrameType[1] {
    SENSOR_PRESS = 0x00,
    SENSOR_HUMID = 0x01,
    CURRENT_SERVO_A = 0xA0,
    CURRENT_SERVO_B = 0xA1,
};

### Variable-sized Types

Bubbler supports `string` and `bytes` types for variable-sized data.

#### String

`string` type is used for text strings. In binary form, the text encoding of a string field is UTF-8. Since a string shouldn't contain any EOF character, the end of a string is defined by `\0` in data stream. So the size of a string field is `str.utf8_length + 1`.

#### Bytes

`bytes` type is used for store binary data in any form. In binary form, bytes data is store in `length` + `data` two part.

`length` part denotes the num of bytes contained in the bytes field. It is composed by several size unit, each unit occupied 1 byte as below:

```text
  0   1   2   3   4   5   6   7
+---+---+---+---+---+---+---+---+
| C |       7-bit Size          |
+---+---+---+---+---+---+---+---+
  C = Continue Flag

Continue Flag indicates if there is another size unit after this unit. If Continue Flag is 0, it is the last size unit.

7-bit spaces stores the length in little-endian.

Therefore, 1 size unit can represent 0-127 bytes data; 2 size units can represent 128-16383 bytes data; ...

data part contains the raw binary data in continues form.

Note: Variable-sized types make the structure size dynamic.

Enum Value as Constant

You can use enum values defined in previous enum types as constant values for fields or other enum values.

enum FrameType[1] {
    FRAME_DATA = 0x01,
};

struct DataFrame {
    FrameType opcode = FRAME_DATA;
    bytes data;
};

In this example, FrameType is an enumeration type with four enumeration values: SENSOR_PRESS, SENSOR_HUMID, CURRENT_SERVO_A, and CURRENT_SERVO_B.

Enumeration values cannot be negative (tentatively), and if the value is not filled in, the default value of the enumeration value is the previous enumeration value plus 1.

The number in the square brackets after the enumeration type name indicates the width of the enumeration type, for example, [1] indicates 1 byte. You can also use the # symbol to represent bytes and bits, for example, #1 represents 1 bit, #2 represents 2 bits. You can also use them in combination, for example, 1#4 represents 1 byte 4 bits, that is, 12 bits.

Recommended to use PascalCase for enumeration type names. But only capitialization of the first letter is mandatory.

Recommended to use ALLCAP_CASE for enumeration values. But only capitialization of the first letter is mandatory.

Data Structures

Use the struct keyword to define data structures. The definition of a data structure includes the structure name and a series of fields. For example:

struct Frame[20] {
    FrameType opcode;
    struct SomeEmbed[1] {
        bool valid[#1];
        bool error[#1];
        uint8 source[#3];
        uint8 target[#3];
    };
    uint8<18> payload;
};

In this example, Frame is a data structure with three fields: opcode, SomeEmbed, and payload. opcode is of type FrameType, SomeEmbed is an anonymous embedded data structure, and payload is of type uint8.

Please note that Bubbler does not have the concept of scope (to accommodate the C language), so the names Frame and SomeEmbed as data structure names are not allowed to be duplicated globally, even if SomeEmbed is an anonymous embedded data structure.

Recommended to use PascalCase for data structure names. But only capitialization of the first letter is mandatory.

Recommended to use snake_case for field names. But only uncaptialization of the first letter is mandatory.

Field Types

The Bubbler protocol supports four types of fields: regular fields, anonymous embedded fields, constant fields, and empty fields.

• Regular fields: Consist of a type name, field name, and field width (optional).
• Anonymous embedded fields: An anonymous field, which can be a struct definition or a defined struct name, its internal subfields will be promoted and expanded into the parent structure.
• Constant fields: A field with a fixed value, its value is determined at the time of definition and cannot be modified. The field name is optional. If there is a field name, the corresponding field will be generated. When encoding, the value of the constant field will be ignored. When decoding, the value of the constant field will be checked. If it does not match, an error will be reported.
• Empty fields: A field without a name and type, only width, used for placeholders.

Regular Fields

Regular fields consist of a type name, field name, and field width. For example:

struct Frame {
    RovlinkFrameType opcode;
};

In this example, opcode is a regular field, its type is RovlinkFrameType.

The field width is optional. If the width is not filled in, the field width is the width of the type.

The field width can be less than the width of the type, for example:

struct Frame[20] {
    int64 my_int48[6];
};

In this example, my_int48 is a 6-byte field, its type is int64, but its width is 6 bytes, so it will only occupy 6 bytes of space when encoding.

However, for fields of struct type, the field width must be equal to the width of the type

Anonymous Embedded Fields

Anonymous embedded fields are nameless data structures that can contain multiple subfields. For example:

struct Frame {
    int64 my_int48[6];
    struct SomeEmbed[1] {
        bool valid[#1];
        bool error[#1];
        uint8 source[#3];
        uint8 target[#3];
    };
};

In this example, SomeEmbed is an anonymous embedded field, it contains four subfields: valid, error, source, and target.

The subfields of the anonymous embedded field will be promoted and expanded into the parent structure. The generated structure is as follows:

struct Frame {
    int64_t my_int48;
    bool valid;
    bool error;
    uint8_t source;
    uint8_t target;
};

Anonymous embedded fields can also be a defined data structure, for example:

struct AnotherTest {
    int8<2> arr;
}

struct Frame {
    int64 my_int48[6];
    AnotherTest;
    uint8<18> payload;
};

In this way, the generated structure is as follows:

struct Frame {
    int64_t my_int48;
    int8_t arr[2];
    uint8_t payload;
};

Constant Fields

Constant fields are fields with a fixed value, its value is determined at the time of definition and cannot be modified. For example:

struct Frame {
    uint8 FRAME_HEADER = 0xAA;
};

In this example, FRAME_HEADER is a constant field with a value of 0xAA.

Or you can use an enum value defined in a previous enum type as a constant value:

enum FrameType[1] {
    FRAME_KEEPALIVE = 0x00,
    FRAME_DATA = 0x01,
};

struct Frame {
    FrameType opcode = FRAME_DATA;
    bytes data;
};

The value of the constant field will be ignored during encoding and checked during decoding. If it does not match, an error will be reported.

Empty Fields

Empty fields are fields without a name and type, they only have a width. Empty fields are often used for padding or aligning data structures. For example:

struct Frame {
    void [#2];
};

In this example, void [#2] is an empty field that occupies 2 bits of space.

Field Options

Field options are used to specify additional attributes of a field. For example, you can use the order option to specify the byte order of an array:

struct AnotherTest {
    int8<2> arr [order = "big"];
}

In this example, the byte order of the arr field is set to big-endian.

Note: The setting of endianness is also effective for floating-point types. However, currently, floating-point values are always interpreted in little-endian order, with the most significant bit storing the sign bit, followed by the exponent bits, and finally the fraction bits.

Custom getter/setter

You can define custom getter and setter methods for a field to perform specific operations when reading or writing field values. For example:

struct SensorTemperatureData {
    uint16 temperature[2] {
        get temperature_display(float64): value / 10 - 40;
        set temperature_display(float64): value == 0 ? 0 : (value + 40) * 10;
        set another_custom_setter(uint8): value == 0 ? 0 : (value + 40) * 10;
    };
}

In this example, the temperature field has a custom getter method and two custom setter methods.

The custom getter named temperature_display returns afloat64 type and calculates the result based on value / 10 - 40. Here,value is filled with the field value and is of type uint16.

The custom setter named temperature_display accepts a float64 type parameter and calculates the result based on value == 0 ? 0 : (value + 40) * 10 to set the field value. Here, value is filled with the parameter value and is of type float64.

The custom setter named another_custom_setter accepts a uint8 type parameter and calculates the result based on value == 0 ? 0 : (value + 40) * 10 to set the field value. Here, value is filled with the parameter value and is of type uint8.

Please note that the custom getter and setter method names cannot be the same as any field names, and getter and setter methods with the same name must return and accept the same type.

Recommended to use snake_case for getter/setter names. But only uncaptialization of the first letter is mandatory.

Generated Code API

The generated code for each language provides a consistent API for encoding and decoding.

C

// Encode struct to buffer. Returns number of bytes written.
int64_t <StructName>_encode(<StructName>*ptr, void* data);

// Decode struct from buffer. Returns number of bytes read, or -1 on error.
int64_t <StructName>_decode(<StructName>*ptr, void* data);

// Calculate the size of the encoded data.
int64_t <StructName>_encode_size(<StructName>* ptr);

C++

// Encode to buffer. Returns number of bytes written.
int64_t encode(void* data);

// Decode from buffer. Returns number of bytes read, or -1 on error.
int64_t decode(void* data);

// Calculate the size of the encoded data.
int64_t encode_size();

Go

// Encode to buffer. Returns number of bytes written.
func (s *StructName) Encode(data []byte) int64

// Decode from buffer. Returns number of bytes read, or -1 on error.
func (s *StructName) Decode(data []byte) int64

// Calculate the size of the encoded data.
func (s *StructName) EncodeSize() int64

Java

// Encode to buffer. Returns number of bytes written.
public long encode(byte[] data);

// Decode from buffer. Returns number of bytes read, or -1 on error.
public long decode(byte[] data);

// Calculate the size of the encoded data.
public long encodeSize();

Python

# Encode to buffer. Returns number of bytes written

def encode(self, data) -> int:

# Decode from buffer. Returns number of bytes read, or -1 on error

def decode(self, data) -> int:

# Calculate the size of the encoded data

def encode_size(self) -> int:

C#

// Encode to buffer. Returns number of bytes written.
public long Encode(byte[] data);
// or with -memcpy=false
public long Encode(Span<byte> data);

// Decode from buffer. Returns number of bytes read, or -1 on error.
public long Decode(byte[] data);
// or with -memcpy=false
public long Decode(ReadOnlySpan<byte> data);

// Calculate the size of the encoded data.
public long EncodeSize();

CommonJS

// Encode to buffer. Returns number of bytes written.
encode(data);

// Decode from buffer. Returns number of bytes read, or -1 on error.
decode(data);

// Calculate the size of the encoded data.
encode_size();

Contributing

Contributions to Bubbler are welcome.

License

MIT License

Related Repositories

• CoralReefPlayer - CoralReefPlayer, a low-latency streaming media player.
• OpenFinNAV - FinNAV, a flight control firmware library for underwater robots (ROV/AUV).