Proposal for layer options (settings file)

[MathiasMagnus]

In the spirit of "not invented here", because OpenCL Layers are following the success of Vulkan layers, it makes sense to follow the same format and locations for setting settings on how layers should behave when loaded.

File structure example

# Comments start with #
# settings (by convention) are prefixed by layer name
layer.setting1 = 10
# Empty lines are ignored, trailing and leading whitespace is trimmed

layer.setting2 = true # Line comments

In general this is a direct translation of vk_layer_settings.txt

$VARIABLE is used for environment variables.

Search locations for the file

The order of precedence through which config files are found are as follows:

1. If $OPENCL_LAYERS_SETTINGS_PATH is set to a path to a directory, use the file cl_layer_settings.txt file from it. Otherwise if set it should be set to the file name to read the settings from.
2. If $OPENCL_LAYERS_SETTINGS_PATH is not set search for platform specific directories listed below.
3. If no settings are found in any of the platform specific locations, then fall back to cl_layer_settings.txt in the current working directory.

Platform specific locations

Linux and MacOS

• $XDG_DATA_HOME/settings.d/opencl/cl_layer_settings.txt
  • if $XDG_DATA_HOME is not set try with $HOME/.local/share

Windows

• If {HKEY_LOCAL_MACHINE}\Software\Khronos\OpenCL\Settings has key with zero value, the the name of
  the key is used as filename.
• Non admin only: HKEY_CURRENT_USER instead of HKEY_LOCAL_MACHINE

Android

The vulkan validation layers use the same locations as in linux, but environment variables are
replaced by properties.

[Kerilk]

Sounds good to me.

Just a few comments/thoughts:

• We should have a defined naming convention to map layer options to environment variables
• Do we need a way to communicate those options from the loader to the layers? Or are layers responsible for fetching the file and finding their respective options (in this case we may want to provide helpers)? The more I think about it the more is favor the later, but this is not an easy choice.
• We should add new cl_layer_info enum values:
  • CL_LAYER_NAME, this one is straightforward
  • CL_LAYER_OPTIONS, this one format is to be determined. Returning a pointer to a null terminated array of versioned structs filled with pointers to strings describing the option name and possible values comes to mind but I have not thought this one through.
• We should add a mechanism to list the layers that would be loaded by the loader, their name and available options, and the value they would have (default or set by configuration file or environment)

Am I missing anything?

[Snektron]

We should have a defined naming convention to map layer options to environment variables

It seems that #6 already includes this. It uses a simple scheme where <layer_name>.<setting_name> is translated to CL_<LAYER_NAME>_<SETTING_NAME>, which should be a decent enough 1:1 mapping. Perhaps using OPENCL_ as prefix would be better though, just so that it is similar to the already existing OPENCL_LAYERS and OPENCL_LAYERS_SETTING_PATH.

Do we need a way to communicate those options from the loader to the layers? Or are layers responsible for fetching the file and finding their respective options (in this case we may want to provide helpers)? The more I think about it the more is favor the later, but this is not an easy choice.

The simplest solution is to just let every layer handle this themselves. This is also the solution that was chosen for the Vulkan layers, since the LunarG layers and Khronos layers contain their own implementation of the setting loader.

We should add new cl_layer_info enum values:

• CL_LAYER_NAME, this one is straightforward
• CL_LAYER_OPTIONS, this one format is to be determined. Returning a pointer to a null terminated array of versioned structs filled with pointers to strings describing the option name and possible values comes to mind but I have not thought this one through.

We should add a mechanism to list the layers that would be loaded by the loader, their name and available options, and the value they would have (default or set by configuration file or environment)

I think there are a few approaches possible here:

• A standalone program similar to clinfo, which simply queries clGetLayerInfo.
• A separate layer which does the same. This has the added caveat that it would print its "own" information, but it can be used in conjunction with a users existing program (they would only need to append libPrintLayerInfo.so to their OPENCL_LAYERS variable).
• An environment variable which instructs layers to print settings and options. This would make each layer responsible for printing its own information.

The first option seems the most flexible to me. Additionally, if CL_LAYER_OPTIONS is some kind of dynamic string array rather than a structure with standardized fields for every option, it would allow other people to implement new layers without having the standard modified.

[Kerilk]

@Snektron Thanks for the feedback:

We should have a defined naming convention to map layer options to environment variables

It seems that #6 already includes this. It uses a simple scheme where <layer_name>.<setting_name> is translated to CL_<LAYER_NAME>_<SETTING_NAME>, which should be a decent enough 1:1 mapping. Perhaps using OPENCL_ as prefix would be better though, just so that it is similar to the already existing OPENCL_LAYERS and OPENCL_LAYERS_SETTING_PATH.

I am terrible at naming things, but you may be correct that OPENCL_ might be a better prefix.

Do we need a way to communicate those options from the loader to the layers? Or are layers responsible for fetching the file and finding their respective options (in this case we may want to provide helpers)? The more I think about it the more is favor the later, but this is not an easy choice.

The simplest solution is to just let every layer handle this themselves. This is also the solution that was chosen for the Vulkan layers, since the LunarG layers and Khronos layers contain their own implementation of the setting loader.

I agree, this is simpler, and more flexible.

We should add new cl_layer_info enum values:

• CL_LAYER_NAME, this one is straightforward
• CL_LAYER_OPTIONS, this one format is to be determined. Returning a pointer to a null terminated array of versioned structs filled with pointers to strings describing the option name and possible values comes to mind but I have not thought this one through.

We should add a mechanism to list the layers that would be loaded by the loader, their name and available options, and the value they would have (default or set by configuration file or environment)

I think there are a few approaches possible here:

* A standalone program similar to clinfo, which simply queries `clGetLayerInfo`.

* A separate layer which does the same. This has the added caveat that it would print its "own" information, but it can be used in conjunction with a users existing program (they would only need to append `libPrintLayerInfo.so` to their `OPENCL_LAYERS` variable).

* An environment variable which instructs layers to print settings and options. This would make each layer responsible for printing its own information.

The first option seems the most flexible to me. Additionally, if CL_LAYER_OPTIONS is some kind of dynamic string array rather than a structure with standardized fields for every option, it would allow other people to implement new layers without having the standard modified.

I think we agree on this as well. The only constraint here is that this tool is hosted and distributed with the icd loader, so that code can be shared between the two, ensuring that both behave identical WRT layers loading.

[Kerilk]

Hello,

I am drafting the additional queries to the clGetLayerInfo of the cl_loader_layers extension.
For the options reporting mechanism I had 2 variants in mind:

1. Using 'null' terminated (empty terminated would be more exact) array of structures pointing to static strings (my initial idea):

typedef struct {
  const char *name; // name == null is list terminator
  const char *value; // Option current value after reading configuration files and environment variables
  const char *default_value;
  const char *description;
} cl_layer_option;

This departs quite a bit from the way OpenCL usually handle info queries, where usually a pointer to the drivers internal data is never returned (as far as I can tell). We could make it so that the buffer required from the user would hold both the list and the strings. This is not an especially difficult scheme to implement. We also may want to add more entries to the structure to avoid having to extend it later on.

2. Using several queries to return buffers of null terminated strings:

• CL_LAYER_NUM_OPTIONS
• CL_LAYER_OPTION_NAMES
• CL_LAYER_OPTION_VALUES
• CL_LAYER_OPTION_DEFAULT_VALUES
• CL_LAYER_OPTION_DESCRIPTIONS

This would for instance return "optiona\0optionb\0" for CL_LAYER_OPTION_NAMES if optiona and optionb are supported. This is easier on the layer implementer and easier to extend, but a bit more cumbersome for the user to parse than 1.

@Snektron where you thinking of something else?

[Snektron]

That seems about what I had in mind.

This departs quite a bit from the way OpenCL usually handle info queries, where usually a pointer to the drivers internal data is never returned (as far as I can tell). We could make it so that the buffer required from the user would hold both the list and the strings. This is not an especially difficult scheme to implement. We also may want to add more entries to the structure to avoid having to extend it later on.

The closest analogue in the current spec are queries that return a cl_name_version[]. The cl_name_version structure contains the string name, which is just an array with a maximum size. This would limit the maximum size that a particular option can be, and would also require the user to allocate a relatively large amount of memory (for say 64 bytes per entry like in cl_name_version would yield 64*4=256 bytes per entry). On the other hand this yields an interface that is close to that of existing OpenCL functions, and I also doubt how important it is to save that memory here.

[Kerilk]

The closest analogue in the current spec are queries that return a cl_name_version[]. The cl_name_version structure contains the string name, which is just an array with a maximum size. This would limit the maximum size that a particular option can be, and would also require the user to allocate a relatively large amount of memory (for say 64 bytes per entry like in cl_name_version would yield 64*4=256 bytes per entry). On the other hand this yields an interface that is close to that of existing OpenCL functions, and I also doubt how important it is to save that memory here.

Indeed. I am not worried about memory footprint here, but I am worried that for the description (or the value if it is an absolute path), whichever size we choose end-up being too small. To ease the discussion I'll be doing a prototype of the layer utility tool we mentioned previously, as well as extend the sample print layer from the OpenCL-ICD-Loader to test it.