Toml encoder #2552

mikefarah · 2025-12-19T23:11:21Z

No description provided.

Copilot

Pull request overview

This PR implements a TOML encoder for yq, adding the ability to encode data structures to TOML format with support for colorization, comment preservation, and roundtrip conversion. The encoder handles TOML-specific constructs including inline tables, table sections, arrays of tables, and maintains proper comment formatting.

Key changes:

New TOML encoder with comprehensive support for TOML syntax including inline tables, table sections, and array-of-tables
Syntax colorization support for TOML output using the fatih/color library
Enhanced TOML decoder to preserve and handle comments (head, line, and standalone comments)
Roundtrip test coverage ensuring TOML can be parsed and re-encoded with fidelity

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/yqlib/encoder_toml.go	Implements the complete TOML encoder with colorization support, handling various TOML constructs (inline tables, table sections, arrays)
pkg/yqlib/decoder_toml.go	Enhanced decoder to track and preserve comments for roundtrip encoding
pkg/yqlib/toml.go	Defines TOML preferences structure with colorization configuration
pkg/yqlib/format.go	Updates TOML format to use the new encoder with preferences
pkg/yqlib/candidate_node.go	Preserves EncodeSeparate flag for format-specific encoding hints
cmd/utils.go	Integrates TOML color preferences with global color configuration
pkg/yqlib/toml_test.go	Adds comprehensive roundtrip test scenarios for various TOML constructs
pkg/yqlib/doc/usage/toml.md	Documents the roundtrip scenarios with examples
pkg/yqlib/doc/usage/hcl.md	Corrects "colorization" to "colorisation" for consistency
pkg/yqlib/doc/usage/headers/hcl.md	Corrects "colorization" to "colorisation" for consistency
examples/sample.toml	Updates sample file to a more comprehensive TOML example
project-words.txt	Adds "colorisation" to accepted project words
agents.md	Adds general development rules and guidelines

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pkg/yqlib/encoder_toml.go

Copilot · 2025-12-19T23:17:25Z

pkg/yqlib/encoder_toml.go

+		line = strings.TrimSpace(line)
+		if !strings.HasPrefix(line, "#") {
+			line = "# " + line
+		}
+		if _, err := w.Write([]byte(line + "\n")); err != nil {


The writeComment function trims whitespace from each comment line, which could remove intentional indentation. While TOML comments typically don't use indentation, if a multi-line comment contains ASCII art, formatted text, or intentionally indented content, this trimming would alter the comment's appearance. Consider preserving the original line content after the '#' character.

Suggested change

line = strings.TrimSpace(line)

if !strings.HasPrefix(line, "#") {

line = "# " + line

}

if _, err := w.Write([]byte(line + "\n")); err != nil {

trimmed := strings.TrimSpace(line)

// Preserve intentionally indented content after an existing '#' while

// still normalizing lines that are not yet comments.

if trimmed == "" {

// Keep behavior of emitting a blank comment line

if _, err := w.Write([]byte("# \n")); err != nil {

return err

}

continue

}

if idx := strings.Index(line, "#"); idx != -1 {

// Line already contains a '#': preserve everything from the first

// '#' onward, including any indentation or formatting after it.

commentPortion := line[idx:]

if _, err := w.Write([]byte(commentPortion + "\n")); err != nil {

return err

}

continue

}

// Line does not contain '#': keep previous behavior of trimming and

// prefixing with "# ".

lineToWrite := "# " + trimmed

if _, err := w.Write([]byte(lineToWrite + "\n")); err != nil {

ccoVeille · 2025-12-19T23:16:09Z

cmd/utils.go

 	yqlib.ConfiguredYamlPreferences.ColorsEnabled = colorsEnabled
 	yqlib.ConfiguredJSONPreferences.ColorsEnabled = colorsEnabled
 	yqlib.ConfiguredHclPreferences.ColorsEnabled = colorsEnabled
+	yqlib.ConfiguredTomlPreferences.ColorsEnabled = colorsEnabled


I know the others are inconsistent, but I would use

Suggested change

yqlib.ConfiguredTomlPreferences.ColorsEnabled = colorsEnabled

yqlib.ConfiguredTOMLPreferences.ColorsEnabled = colorsEnabled

pkg/yqlib/candidate_node.go

ccoVeille · 2025-12-19T23:19:05Z

pkg/yqlib/doc/usage/hcl.md

 - Comments (leading, head, and line comments)
 - Nested structures (maps and lists)
- Syntax colorization when enabled
+- Syntax colorisation when enabled


You may configure misspell to use UK English via your .golangci.yaml file

https://golangci-lint.run/docs/linters/configuration/#misspell

Follow up PR for anyone interested

Switch to uk #2557

pkg/yqlib/encoder_toml.go

ccoVeille · 2025-12-19T23:23:40Z

pkg/yqlib/encoder_toml.go

+			key := path[len(path)-1]
+			for _, it := range node.Content {
+				// [[key]] then body
+				if _, err := w.Write([]byte("[[" + key + "]]\n")); err != nil {


Here I'm playing the devil advocate role.

Are we sure the key cannot contain [[ ? And so it would require some escaping.

Not sure - going to leave that for a another PR 😅

Follow up PR for anyone interested

Fix string escape handling in TOML colorization #2556

ccoVeille · 2025-12-19T23:25:41Z

pkg/yqlib/encoder_toml.go

+}
+
+// colorizeToml applies syntax highlighting to TOML output using fatih/color
+func (te *tomlEncoder) colorizeToml(input []byte) []byte {


Same remark here TOML is an acronym

Suggested change

func (te *tomlEncoder) colorizeToml(input []byte) []byte {

func (te *tomlEncoder) colorizeTOML(input []byte) []byte {

Copilot · 2025-12-20T04:04:26Z

@mikefarah I've opened a new pull request, #2554, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: ccoVeille <[email protected]>

Co-authored-by: mikefarah <[email protected]>

# Conflicts: # pkg/yqlib/toml_test.go

Copilot · 2025-12-20T04:37:42Z

@mikefarah I've opened a new pull request, #2556, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: ccoVeille <[email protected]>

mikefarah · 2025-12-20T04:41:17Z

Thanks @ccoVeille !

Co-authored-by: mikefarah <[email protected]>

mikefarah added 7 commits December 14, 2025 19:03

wip toml encoder

5f9bf8d

Handles comments!

3a53238

Colours!

1338b52

wip

4e9d5e8

Merge branch 'master' into toml_encoder

aa85852

Comments!

161be10

spelling

b974d97

mikefarah requested a review from Copilot December 19, 2025 23:11

Copilot started reviewing on behalf of mikefarah December 19, 2025 23:11 View session

Copilot AI reviewed Dec 19, 2025

View reviewed changes

ccoVeille reviewed Dec 19, 2025

View reviewed changes

Initial plan

c1b81f1

Copilot AI mentioned this pull request Dec 20, 2025

Fix TOML colouriser incorrectly highlighting inline arrays as table sections #2554

Merged

4 tasks

mikefarah and others added 7 commits December 20, 2025 15:05

Update pkg/yqlib/candidate_node.go

b8d90fd

Co-authored-by: ccoVeille <[email protected]>

Update pkg/yqlib/encoder_toml.go

4d620bf

Co-authored-by: ccoVeille <[email protected]>

Add test case and fix colorization bug for inline arrays in TOML

aa5134e

Co-authored-by: mikefarah <[email protected]>

Fixing number color issue

0914121

Convert to UK English spelling (colourization, coloured)

c132c32

Co-authored-by: mikefarah <[email protected]>

Merge remote-tracking branch 'origin/pr/2552' into copilot/sub-pr-2552

1de4ec5

# Conflicts: # pkg/yqlib/toml_test.go

Formatting

56eb365

Copilot AI mentioned this pull request Dec 20, 2025

Fix string escape handling in TOML colorization #2556

Merged

6 tasks

Update pkg/yqlib/encoder_toml.go

b7cbe59

Co-authored-by: ccoVeille <[email protected]>

Copilot AI and others added 4 commits December 20, 2025 15:55

Add test for string escape bug and implement fix

7f60daa

Co-authored-by: mikefarah <[email protected]>

Fix spelling: use British English Colourization

5d6c204

Co-authored-by: mikefarah <[email protected]>

Merge branch 'master' into toml_encoder

7198d16

whitespace

207bec6

mikefarah merged commit 207bec6 into master Dec 20, 2025
4 checks passed

-		line = strings.TrimSpace(line)
-		if !strings.HasPrefix(line, "#") {
-			line = "# " + line
-		}
-		if _, err := w.Write([]byte(line + "\n")); err != nil {
+		trimmed := strings.TrimSpace(line)
+		// Preserve intentionally indented content after an existing '#' while
+		// still normalizing lines that are not yet comments.
+		if trimmed == "" {
+			// Keep behavior of emitting a blank comment line
+			if _, err := w.Write([]byte("# \n")); err != nil {
+				return err
+			}
+			continue
+		}
+		if idx := strings.Index(line, "#"); idx != -1 {
+			// Line already contains a '#': preserve everything from the first
+			// '#' onward, including any indentation or formatting after it.
+			commentPortion := line[idx:]
+			if _, err := w.Write([]byte(commentPortion + "\n")); err != nil {
+				return err
+			}
+			continue
+		}
+		// Line does not contain '#': keep previous behavior of trimming and
+		// prefixing with "# ".
+		lineToWrite := "# " + trimmed
+		if _, err := w.Write([]byte(lineToWrite + "\n")); err != nil {

	yqlib.ConfiguredTomlPreferences.ColorsEnabled = colorsEnabled
	yqlib.ConfiguredTOMLPreferences.ColorsEnabled = colorsEnabled

	func (te *tomlEncoder) colorizeToml(input []byte) []byte {
	func (te *tomlEncoder) colorizeTOML(input []byte) []byte {

Uh oh!

Toml encoder #2552

Toml encoder #2552

Conversation

mikefarah commented Dec 19, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

ccoVeille Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ccoVeille Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

ccoVeille Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ccoVeille Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

mikefarah Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

ccoVeille Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ccoVeille Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Dec 20, 2025

Uh oh!

Copilot AI commented Dec 20, 2025

Uh oh!

mikefarah commented Dec 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ccoVeille Dec 20, 2025 •

edited

Loading