Validation error for logical type local-timestamp-millis in avro message because of strict date-time format json-validation on frontend

[Alexxx221]

Issue submitter TODO list

• I've looked up my issue in FAQ
• I've searched for an already existing issues here
• I've tried running main-labeled docker image and the issue still persists there
• I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

We are using the Schema Registry and Avro message format.

When sending a message with Produce Message button and value containing an avro logical type local-timestamp-millis field in format YYY-MM-DDThh:mm:ss.sss (without timezone) I get a validation error with the following text about incorrect date-time format:

Content/properties/txCreateDate/oneOf/0/type - must be null
Content/properties/txCreateDate/oneOf/1/properties/long/format - must match format "date-time"
Content/properties/txCreateDate/oneOf - must match exactly one schema in oneOf

And this is wrong, because according to a avro specification quote The local-timestamp-{millis,micros,nanos} logical type represents a timestamp in a local timezone, regardless of what specific time zone is considered local. So i guess string representation of this date would be exactly YYY-MM-DDThh:mm:ss.sss (without timezone).

You can try to pass frontend validation by setting timezone like this YYY-MM-DDThh:mm:ss.sssZ or YYY-MM-DDThh:mm:ss.sss+00:00, but then you will get 500 backend error with stacktrace:

	at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2111)
	Original Stack Trace:
		at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2111)
		at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:2010)
		at java.base/java.time.LocalDateTime.parse(LocalDateTime.java:494)
		at java.base/java.time.LocalDateTime.parse(LocalDateTime.java:479)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion$LogicalTypeConversion.lambda$static$14(JsonAvroConversion.java:470)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion.lambda$processLogicalType$4(JsonAvroConversion.java:298)
		at java.base/java.util.Optional.map(Optional.java:260)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion.processLogicalType(JsonAvroConversion.java:298)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion.convert(JsonAvroConversion.java:140)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion.convert(JsonAvroConversion.java:113)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion.convert(JsonAvroConversion.java:69)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion.convert(JsonAvroConversion.java:120)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion.convert(JsonAvroConversion.java:69)
		at io.kafbat.ui.util.jsonschema.JsonAvroConversion.convertJsonToAvro(JsonAvroConversion.java:59)
		at io.kafbat.ui.serdes.builtin.sr.Serialize.serializeAvro(Serialize.java:101)
		at io.kafbat.ui.serdes.builtin.sr.SchemaRegistrySerde.lambda$serializer$15(SchemaRegistrySerde.java:273)
		at io.kafbat.ui.serdes.SerdeInstance.lambda$serializer$3(SerdeInstance.java:83)
		at io.kafbat.ui.serdes.SerdeInstance.wrapWithClassloader(SerdeInstance.java:34)
		at io.kafbat.ui.serdes.SerdeInstance.lambda$serializer$4(SerdeInstance.java:83)
		at io.kafbat.ui.serdes.ProducerRecordCreator.create(ProducerRecordCreator.java:27)
		at io.kafbat.ui.service.MessagesService.sendMessageImpl(MessagesService.java:178)
		at io.kafbat.ui.service.MessagesService.lambda$sendMessage$9(MessagesService.java:159)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
		at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:181)
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
		at java.base/java.lang.Thread.run(Thread.java:1583)

And this is completely logic, LocalDateTime class is not for dates with time zones.

As a result, it is currently impossible to send messages with local-timestamp-millis fields.

Expected behavior

Json validation of date as string in format YYY-MM-DDThh:mm:ss.sss for avro date local-timestamp-millis should pass on frontend.

Your installation details

1. 1.4.2

Steps to reproduce

1. Create avro schema for topic with field in value {"logicalType": "local-timestamp-millis", "type": "long"}
2. Send message in topic with this field. For example 2052-07-06T02:57:22.228

Screenshots

No response

Logs

No response

Additional context

Here is an example of how such a field is formed for the json schema, what is used for validation and how it looks in our avro schema.
Schema registry:

{
    "default": null,
    "name": "txCreateDate",
    "type": [
        "null",
        {
            "logicalType": "local-timestamp-millis",
            "type": "long"
        }
    ]
}

Json schema:

{
    "txCreateDate": {
        "oneOf": [
            {
                "type": "null"
            },
            {
                "type": "object",
                "properties": {
                    "long": {
                        "type": "string",
                        "format": "date-time"
                    }
                }
            }
        ]
    }
}

Important thing is "format": "date-time". If you refer to the documentation of the validation library, then you can see the following date-time: date-time (time-zone is mandatory).

I assume that the reason for the problem lies in this question.
When upgrading kafbat-ui from version 1.1 to 1.2, the package ajv-formats' was updated": "2.1.1" -> " ajv-formats": "3.0.1", which is why the validation of the time zone has become strict.
This is written in the v3.0.0-rc.0 release of ajv-formats package.
Quote: Formats time and date-time now require timezone, as per JSON Schema specification / RFC3339.

The simplest solution, as I understand it, would be to change the private static final String DATE_TIME = "date-time" to iso-date-time in JsonAvroConversion.java, because quote: that have optional timezone, they can be used for backward compatibility.

For now, we are rolled back to 1.1. On this version validation is correct.

[github-actions]

Hi Alexxx221! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues.
Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.