Release Version 0.4: Enhanced STJ Format with New Validation and Features

[yaniv-golan]

Release Version 0.4: Enhanced STJ Format with New Validation and Features

This update introduces significant enhancements to the specification, schema, tools, and documentation to improve the flexibility, interoperability, and robustness of the STJ format.

What's New

1. Addition of word_timing_mode

• New Field: Introduced the word_timing_mode field in the segments object to indicate the completeness of word-level timing data.
  • Allowed Values:
    • "complete": All words in the text are included in the words array.
    • "partial": Only some words are included in the words array.
    • "none": No word-level timing data is provided.
• Purpose: Helps applications understand how to process the words array within a segment, especially when dealing with incomplete or absent word-level timing information.

2. Updated JSON Schema (stj-schema.json)

• Schema Enhancements:
  • Added the word_timing_mode field to the schema.
  • Implemented stricter validation rules to enforce new requirements.
• Validation Improvements:
  • Segments must not overlap in time and should be ordered sequentially.
  • Words must be within their parent segment's start and end times.
  • Confidence scores must be within the range [0.0, 1.0].
  • Language codes are validated against ISO 639-1 and ISO 639-3 standards.
  • Zero-duration segments or words (start equals end) must include the appropriate duration flag set to "zero" in additional_info.

3. Enhanced Validators

Python Validator (stj_validator.py)

• New Validation Logic:
  • Checks for overlapping segments and words.
  • Validates word_timing_mode consistency with the presence and completeness of the words array.
  • Ensures text consistency between the text field and the concatenated words array when word_timing_mode is "complete".
  • Validates language codes using the iso639-lang library (version 2.4.2).
• Error Reporting: Improved error messages to provide detailed information about validation failures.

JavaScript Validator (stj-validator.js)

• New Validation Logic:
  • Similar enhancements as the Python validator.
  • Uses the iso-639-1 package to validate language codes.
• Error Handling: Provides clear and descriptive error messages to aid in debugging.

4. Updated Conversion Tools

• Python and JavaScript Conversion Scripts (stj_to_srt, stj_to_vtt, stj_to_ass):
  • Modified to handle the word_timing_mode field appropriately.
  • For segments with word_timing_mode set to "complete", the text is reconstructed from the words array.
  • Ensures compatibility with subtitle formats that may not support word-level timing.

5. Comprehensive Test Coverage

• New Test Cases:
  • Added tests to cover the new validation rules.
  • Included test STJ files that intentionally violate specific validation rules to ensure validators correctly identify errors.
• Test Files: (currently used only by the JavaScript tests suite)
  • overlapping_segments.stj.json
  • invalid_language.stj.json
  • invalid_word_timing_mode.stj.json
  • zero_duration_word_without_flag.stj.json
  • invalid_confidence_scores.stj.json
  • invalid_speaker_id.stj.json
  • word_outside_segment_timings.stj.json
  • words_overlap_or_out_of_order.stj.json

6. Documentation Updates

• Specification Document (stj-specification.md):
  • Updated to reflect the changes in version 0.4.
  • Detailed explanation of the word_timing_mode field and new validation requirements.
• README and Guides:
  • Updated version numbers and dates.
  • Revised examples and usage instructions to align with the new specification.
• API Reference and Best Practices:
  • Included information on handling the new fields and adhering to the enhanced validation rules.

Breaking Changes

• Stricter Validation: Files that were valid under version 0.3 may fail validation under version 0.4 due to the new requirements.
• Action Required: Users should review their STJ files and update them to comply with the new specification.
  • Ensure that word_timing_mode is correctly set in segments.
  • Verify that segments and words do not overlap and are within the correct time ranges.
  • Update any zero-duration segments or words to include the appropriate duration flag.

How to Upgrade

1. Update Your STJ Files:

   • Add the word_timing_mode field to segments as appropriate.
   • Ensure that your STJ files adhere to the new validation rules.

2. Update Tools and Dependencies:

   • Python:
     • Install the updated stj_validator.py and conversion scripts from the repository.
     • Install iso639-lang version 2.4.2:

       pip install iso639-lang==2.4.2

   • JavaScript:
     • Install the updated stj-validator.js and conversion scripts.
     • Install iso-639-1 package:

       npm install iso-639-1

3. Run Validation:

   • Use the updated validators to check your STJ files against the new schema.

     # Python
     python stj_validator.py path/to/your_file.stj.json path/to/stj-schema.json
     
     # JavaScript
     node stj-validator.js path/to/your_file.stj.json path/to/stj-schema.json

4. Update Integrations:

   • If you have custom integrations or tools that consume STJ files, ensure they are updated to handle the new fields and validation logic.

Acknowledgments

This release was triggered by mluggy comments on word-by-word caption tools expectations. Thanks for spotting this!

Feedback and Contributions

If you encounter any issues, have suggestions, or would like to contribute to the project, please:

• Submit Issues: Report bugs or request features via the GitHub Issues page.
• Contribute Code: Fork the repository and submit a pull request with your enhancements.
• Join the Discussion: Engage with the community to share ideas and best practices.

Links

• Specification Document: stj-specification.md
• JSON Schema: stj-schema.json
• Examples: examples directory
• Tools and Scripts:
  • Python: tools/python
  • JavaScript: tools/javascript

---

This discussion was created from the release Release Version 0.4: Enhanced STJ Format with New Validation and Features.