Enumerate weirdnesses in the protocol

[chrivers]

For beginners, there are quite a few weird surprises. We should make a list, for reference.

Here's the ones I can think of right now:

Enums differ in size for no good reason

1. BeamFrequency:
   PlayerShip object:
     beam_frequency: enum<u8, BeamFrequency>
   ClientPacket::SetBeamFreq:
     beam_frequency: enum<u32, BeamFrequency>

2. MainScreenView:
   PlayerShip object:
     main_screen_view: enum<u8, MainScreenView>
   ClientPacket::SetMainScreen:
       main_screen_view: enum<u32, MainScreenView>

Enums are sometimes used in "nullable" form (offset by 1)

For example, in GameMasterMessage the console_type field can be 0, which means no console. If it's greater than 0, subtract 1 from the value to get the normal ConsoleType enum value

Some packets are (at least originally) intended for sendings integers, but contain non-integers

For example, the client packet ConvertTorpedo has major type valueFourInts, but really only contains one f32. .. sigh

Bools are always either 1 bytes or 4.. unless they're 2

PlayerShip.shields_up is a 2-byte bool, and it seems to be the only one in the entire protocol

Strings often contain junk data

It's very common for strings read from the network to contain random junk bytes after the \u0000 (UTF-16 NULL) character.

WeaponsConsole object updates can contain junk data

The official server sometimes sends WeaponsConsole updates that contain fields in the bit mask that are sent on the wire, but (seemingly) contain just junk. The reader has to check the tube_status fields, to know which tube data can be trusted.

All arrays are either fixed-size or terminated by an integer.. except when they aren't

The server packet GameOverReasons contain an array of strings. This packet seems to end at the end of the packet frame, instead of being terminated by an integer. This is the only known known instance of this behaviour.

All strings are utf16-le.. except the welcome message

Every string in the protocol is encoded as utf16-le (no bytemark), except for the string in the server packet Welcome. That one is ASCII.. because.. reasons

All non-ObjectUpdate packets always contain all fields... except IncomingAudio

For some reason, IncomingAudio seems to be the only packet there the number of valid fields depend on the contents of the packet. If this was the first field, this could be a case of a missed subtype field, but it's the second field... double sigh

[NoseyNick]

Enums differ in size for no good reason:

3. ConsoleType is Int (u32) in SetConsole, Byte (u8) in ConsoleStatus
    ... and then the "nullable" u32 +1 thing

I implement Artemis::ConsoleType as an Int (u32), but create a subclass Artemis::ConsoleType::C which is the same except u8 storage format, and a subclass Artemis::ConsoleType::GM which has the offset-1 thing :-/

Bools are always either 1 bytes or 4.. unless they're 2

There are a few other u16 "unknowns" in the doc, no idea if any of those are bool16s

All strings are utf16-le.. except the welcome message

Well it IS officially called plainTextGreeting :-p

All non-ObjectUpdate packets always contain all fields... except IncomingAudio

Does something like this work?

parser AudioCommand (u32)
  0x01: struct<emptystruct>
  0x02: struct<AudioStrings>
struct emptystruct
struct AudioStrings
  Title: string
  File: string

[NoseyNick]

All arrays are either fixed-size or terminated by an integer.. except when they aren't

Proposal for discussion: There's no such thing as a fixed sized array. ALL arrays are terminated by a predetermined integer OR end of packet.

ObjectUpdate packet is 0x00000000-terminated

EngGridUpdate has an array of system_grid_status terminated by 0xff, and damcon_team_status terminated by 0xfe

FighterBayStatus is terminated by 0x00000000

GameOverStats is an array of struct Statistic {u8=0, i32=value, string=label} terminated by 0xce

GameOverReason already caused offence by being variable length, terminated by end of packet, I think?

Object EngineeringConsole contains 8 heats, 8 energies, 8 coolants, but they are not arrays. If they WERE arrays, you'd need to clarify that the bits in the bitmask correspond to individual elements IN the array, NOT 1 bit for the whole array. If they are 8 separate heats, they are 8 separate bits in the bitmask for 8 separate attributes in the object.

Object NpcShip, similarly, has 8 separate types of damage, and 5 separate types of beam resistance.

object WeaponsConsole, heck someone already unwound one of the "arrays" into separate u8 for homing, nukes, mines, emps, plasma. The tubes are 6 separate times, 6 separate statuses, 6 separate ordnance_types

packet ServerPacket AllShipSettings is an end-of-packet-terminated array
ConsoleStatus console_status similarly end-of-packet-terminated.

Version packet contains version_major, version_minor, version_patch, not version sizedarray<u32, 3> ... or heck, if you MUST consider it to be an array, that one is end-of-packet terminated too.

"Discuss" :-)

[chrivers]

Enums differ in size for no good reason:

1. ConsoleType is Int (u32) in SetConsole, Byte (u8) in ConsoleStatus
   ... and then the "nullable" u32 +1 thing
   I implement Artemis::ConsoleType as an Int (u32), but create a subclass Artemis::ConsoleType::C which is the same except u8 storage format, and a subclass Artemis::ConsoleType::GM which has the offset-1 thing :-/

Bools are always either 1 bytes or 4.. unless they're 2

There are a few other u16 "unknowns" in the doc, no idea if any of those are bool16s

Could be, certainly. I guess we'll have to wait and see what Project Holodeck uncovers for us :)

I have a sneaking suspicion it's really just a u8 that's suffering from some kind of alignment weirdness.. but perhaps time will tell

All strings are utf16-le.. except the welcome message

Well it IS officially called plainTextGreeting :-p

True, I grant you that ;-)

All non-ObjectUpdate packets always contain all fields... except IncomingAudio

Does something like this work?

parser AudioCommand (u32)
0x01: struct
0x02: struct
struct emptystruct
struct AudioStrings
Title: string
File: string

Unfortunately not. If it was just a normal subpacket type, that would be fine. But this here would require us to multiplex on a value that is not the first in the packet, which breaks every assumption about subtype multiplexing so far. But something like this might work:
..in pseudo-new syntax, because it's nice ;-)

# in, say, client.stf
struct IncomingAudio
    id: u32
    data: client.AudioPacket

struct AudioPacketIncoming
    title: string
    file: string

struct AudioPacketPlaying

parser AudioPacket(read=u32)
    enums.AudioMode::incoming: AudioPacketIncoming
    enums.AudioMode::Playing: AudioPacketPlaying

This allows us to put the "id" in before the multiplexing field. Is that the same as your idea?

[NoseyNick]

Is that the same as your idea?

I think that's roughly what I meant, yes, except you used enums.AudioMode::incoming to be slightly clearer than 0x2
I'm still fairly convinced all "objects" are multiplexing on a bitfield, and all "parsers" are multiplexing on an enum :-p

[chrivers]

Proposal for discussion: There's no such thing as a fixed sized array. ALL arrays are terminated by a predetermined integer OR end of packet.

Oh, curious! I like this!

ObjectUpdate packet is 0x00000000-terminated

Yes

EngGridUpdate has an array of system_grid_status terminated by 0xff, and damcon_team_status terminated by 0xfe

Check

FighterBayStatus is terminated by 0x00000000

Right

GameOverStats is an array of struct Statistic {u8=0, i32=value, string=label} terminated by 0xce

Roger

GameOverReason already caused offence by being variable length, terminated by end of packet, I think?

Indeed

Object EngineeringConsole contains 8 heats, 8 energies, 8 coolants, but they are not arrays. If they WERE arrays, you'd need to clarify that the bits in the bitmask correspond to individual elements IN the array, NOT 1 bit for the whole array. If they are 8 separate heats, they are 8 separate bits in the bitmask for 8 separate attributes in the object.

I didn't originally plan for them to be arrays, actually. I wrote them as "maps", from "ShipSystem" -> value. Since we're doing a complete enumeration, the ShipSystem value is not actually saved.

All maps that I have found, use a bit for each element. I think that's a fairly reasonable spec for it.

Object NpcShip, similarly, has 8 separate types of damage, and 5 separate types of beam resistance.

Agreed - also maps :)

object WeaponsConsole, heck someone already unwound one of the "arrays" into separate u8 for homing, nukes, mines, emps, plasma. The tubes are 6 separate times, 6 separate statuses, 6 separate ordnance_types

Yeah, that was me. But the principle is the same again - map<weaponstype, u8>

packet ServerPacket AllShipSettings is an end-of-packet-terminated array
ConsoleStatus console_status similarly end-of-packet-terminated.

Hmm, I suppose... but I'm a little skeptical. It's kind of hard to say how one should react to a AllShipSettings packet, unless it contains exactly 8 items? If it contains less than 8, I suppose we could just update some of the ships. But should we do that, or assume it's invalid?

And what if it contains 9 or more ships?

I think it's pretty reasonable to say that a valid AllShipSettings packet contains exactly 8 ships, since that's what the authentic game always(?) seems to do.

ConsoleStatus is probably more reasonably described as a map over the ConsoleStatus type. Then it's automatically expected to be larger if we expand that type.

Version packet contains version_major, version_minor, version_patch, not version sizedarray<u32, 3> ... or heck, if you MUST consider it to be an array, that one is end-of-packet terminated too.

Wait, I disagree on this one. If this packet was expanded in the future, I don't think it would be a good or safe assumption to think the version number got longer and more complicated.

Things that are end-of-packet terminated are dynamic in nature - such as the GameOverReasons.

However, I concede that the sizedarray spec for it was a cut corner, and is nicer as 3 named fields, now that we know how their intention :)

"Discuss" :-)

Don't mind if I do ;-)

[chrivers]

Is that the same as your idea?
I think that's roughly what I meant, yes, except you used enums.AudioMode::incoming to be slightly clearer than 0x2

Ah, I'd be ok with literal consts too. The main trouble is the missing id:u32 in the beginning. That one really does throw a wrench in the machinery. If it wasn't for that, it'd just be a simple submessage type.

I'm still fairly convinced all "objects" are multiplexing on a bitfield, and all "parsers" are multiplexing on an enum :-p

Depending on how we wing the final format, this should be pretty clear.. but perhaps we don't need in-language support to enforce this status? :)