Proposal: adding metadata to dialog files

[JarbasAl]

Problems being solved

• 1 - in some languages TTS utterances grammatical gender may depend on the gender of the person listening (the user)
  • this can later be detected in STT audio via speaker-recognition
  • gender recognition from audio is possible but not accurate, it is best to tie it to a enrolled speaker if possible
• 2 - in some languages TTS utterances grammatical gender may depend on the gender of the speaker (the assistant)
  • eg. in portuguese you there is no gender neutral way to say "thank you", you say "obrigado"/"obrigada"
  • this would be a setting of the voice assistant
• 3 - personality settings
  • "increase sarcasm by 20%"

Dialog Json

New file format, .jsonl, where each line in the file contains a json of the format

{"utterance": "XXX", "attitude": "sarcastic", "assistant_gender": "male", "listener_gender": "male", "weight": 0.3}

jsonl format info: https://jsonlines.org

Personality Settings

in mycroft.conf users can tweak personality

weights reflect how likely a dialog is to be selected

"persona": {
    "gender": "male",
    "attitudes": {
        "normal": 10,
        "flippant": 8.0,
        "funny": 8.0,
        "irritable": 0
      }
}

How it works

• 1 - load .jsonl file if it exists, else old .dialog file
• 2 - filter samples per assistant gender
• 3 - select an attitude based on weights defined in mycroft.conf
• 4 - filter samples per attitudes
• 5 - select based on weights of .jsonl file

references

previous discussions

• Consolidated proposal for adding metadata to dialog files MycroftAI/mycroft-core#2436
• ChanceNCounter/mycroft-core@faa1d2d
• grammatical gender support MycroftAI/mycroft-core#1874

[builderjer]

I like the idea, but with more and more metadata added, does it increase the overhead? or not enough to matter? I am not exactly clear how much processing power the metadata from here, and from utterances and ... actually take.

I know with c++ pointers it wouldn't be noticeable at all, but with python, I just don't know.

[ChanceNCounter]

We can revisit everything about this if we want, so leave nothing on the table. I am in favor of replacing certain components with compiled funbits, but that’s getting into a broader discussion that I was hoping would wait until the upstream/downstream situation is settled in writing.

Edit: the dialog renderer would be a good place for me to put my money where my mouth is in that regard

[NeonDaniel]

I would propose refactoring this to keep it in sync with the dialog:

dialog.yaml

- name: hello
  dialog:
  - hi {{name}}
  - hello, {{name}}
  persona: {}
- name: goodbye
...

In the "old style", this would have been hello.dialog and goodbye.dialog. In either case, I prefer yaml files for user-specified resources like this to enforce readability for the next person and avoid troubleshooting punctuation

[JarbasAl]

this is jsonl, each line is valid json, doesnt really work well with yaml IMHO...

https://jsonlines.org/

I also quite like that we get a direct translation into a python dict, im not a fan of yaml in general, dont want to rehash this talk but more on yaml vs json has been said in linked issue above

[ChanceNCounter]

For the record, all JSON is valid YAML. The reverse is not true. If we support YAML, anybody who prefers JSON can just keep using it.

[JarbasAl]

updated initial comment to clarify what is being proposed is jsonL format, not json

[NeonDaniel]

this is jsonl, each line is valid json, doesnt really work well with yaml IMHO...

https://jsonlines.org/

I also quite like that we get a direct translation into a python dict, im not a fan of yaml in general, dont want to rehash this talk but more on yaml vs json has been said in linked issue above

I had missed that, but still not sure I like json as a format for things one might edit/override in a terminal/text editor (as you said, we've been down that road before).

On the other hand, this is probably sufficiently complex enough that we could justify a utility to generate the output so a skill author doesn't miss any params and to guide them through providing enough alternative utterances for different personas

[ChanceNCounter]

I agree with Daniel above, on all counts. I also think a utility could work just as well for a YAML/JSON solution, however, which would be as thoroughly accessible as I can imagine.