Sample command

[thangld201]

Hi @Helw150 , thanks for the wonderful works !

Could you update sample scripts to reproduce the experiments please (e.g. sh files) ?

[Helw150]

Thanks for the kind words, though I can't take much credit as it was @liuyanchen1015 who led this project!

You can find the script which runs the Roberta experiments in ./MultiDialect/run.py and script which runs the FLAN-T5 experiments in ./MultiTask/run_t5.py.

The datasets all are loaded from HuggingFace where they were preprocessed. Is there a particular issue running this?

[thangld201]

Thanks for your swift response!

To clarify, I see the python files but there is no specification about arguments, so I am not sure which argument to set. For example:

1. The default argument in the MultiDialect/run.py for num_transformation_rules was None. In the DADA paper, it was only mentioned that "we trained nearly 200 feature adapters" so it was confusing (missing concrete number), because in Multi-Value there were 235 transformation rules.

2. About Sec 5.1, how do you match which linguistic feature to which dialect ? Do you rely on Table 7-17 in the MultiValue paper and manually match, or is there a dictionary file somewhere (I did not see it in the multivalue repo, but could also be due to me missing something, would be great if you help point this out!) ?

3. Do you have the codes for the experiments in Sec 5.2 ? As I cannot find them in the repo. The descriptions are not very clear to me (so many 'average' :(( ) so I am not sure how to implement correctly...

cc @liuyanchen1015 @Helw150

[Helw150]

1. The default argument of None is correct! If provided, that argument truncates the complete list of transformation rules. Otherwise, the list is left untruncated and the full list of features from MultiDialect/transformation_rules_list.json is used. You can see where this is implemented on lines 56-59 of MultiDialect/run.py.

2. An implementation of each dialect is in the Dialects.py file in the Multi-VALUE repo. To use them, you import the class corresponding to the dialect you are interested in e.g.

from src.Dialects import HongKongDialect
hke = HongKongDialect()
print(hke)
print(hke.transform("I talked with them yesterday"))
print(hke.executed_rules())

The mapping between dialects and features are sourced from e-WAVE. You can find that untransformed data in a machine readable format in resources/ewave.csv in the Multi-VALUE repo.

3. I don't have the code for that data visualization, but I'll ask Yanchen to add it to the repo if that's helpful as I think he should still have the code!

An attempt at clarifying the idea of the experiment which I hope helps in the meantime. Let's say we have an adapter A trained on syntactic feature F. We'd hope that adapter A should have a larger activation when feature F is present than it does in general. Otherwise, adapter A is getting used even on data it wasn't trained to handle.

We want to distinguish average activation in these two cases, so this gives us two averages: the average activation of A when feature F is present and the average activation of A across all data (including when feature F isn't present).

The difference between these two gives us a sense of the correlation between adapter activation and the actual presence of the types of syntax that adapter was trained to handle!