Experiments

Implemention of A/B testing to perform staged rollouts of new functionality (see implementation). Please check wiki for more details on A/B experiments. Few key specs include but are not limited to,

• Experiment groups are logged in the beginning just when extension loads
• If user is in experiment, ensure that experiments are used in the first session itself
• If they users have opted out of telemetry, then they are opted out of AB testing and downloading the configuration
• If fetch for new experiment fails, keep previous definition around (check the error is logged)
• Check if experiment is enabled and then do things based on that, for eg. If InExperiment (“LS – enabled”) and jedi is using default configuration, then Extension activates with language server
• Logic for checking if you are in an experiment: In Experiment(foo) =HASH>=min and < max. To verify this, you must know the value of HASH (depends on machineId and experiment salt). See below for how to get that.
• Currently we have two experiments set up,
• • LS - enabled, LS - control
• • AlwaysDisplayTestExplorer - enabled, AlwaysDisplayTestExplorer - control

How to setup:

• Edit experiments.json to contain,

[
    {
        "name": "LS - control",
        "salt": "LS",
        "min": 0,
        "max": 10
    },
    {
        "name": "LS - enabled",
        "salt": "LS",
        "min": 85,
        "max": 100
    },
    {
        "name": "AlwaysDisplayTestExplorer - enabled",
        "salt": "LS",
        "min": 70,
        "max": 90
    },
    {
        "name": "AlwaysDisplayTestExplorer - control",
        "salt": "LS",
        "min": 40,
        "max": 50
    }
]

• To get HASH, set breakpoint at line 154 in src\client\common\experiments.ts and activate the extension. Once you hit the breakpoint, check the value of hash%100 variable.
• Edit experiments.json file to suit your needs. For eg. to make sure you are in experiment xxx, adjust min and max to make sure your HASH lies between min and max.
• Experiments are downloaded using URL specified in constant configUri (Present in src\client\common\experiments.ts). For validating purposes, edit the constant to contain an invalid URI, which makes sure we always use experiments.json to get exps.