Adopting ISO C++ sender/receiver model

[AerialMantis]

Note: this is still an early draft to begin the discussion, much of this will need to be further defined.

Overview

This discussion proposes changes to the SYCL interface which reflect a new design for the SYCL execution model based on the proposed ISO C++ sender/receiver model (https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2300r7.html).

The motivation for this is:

• To solve existing issues with the SYCL 2020 execution model.
• To provide a way for users to control eager vs lazy execution.
• To achieve a closer alignment with the direction of ISO C++.

If we want to achieve these goals it will require a change to the SYCL interface, but we should ensure this change is minimal to avoid big breaking changes for users, and we should do it just once if possible.

Motivation

All command functions such as submit or shortcut functions like parallel_for return an event which allows users to create dependency chains and synchronize with submitted commands. However, the event class in SYCL conflates usage for dependencies and usage for synchronizing with submitted commands.

Furthermore, the SYCL execution model currently does not specify whether commands submitted to the runtime are executed eagerly or lazily. The user has no control over this and as a result must assume that they could be executed lazily and synchronize with the commands (via event wait, queue wait, or buffer destruction) in order to ensure the work has been enqueued to the backend.

This is evident by looking at two use cases.

In the first case, where users require low latency execution, i.e. commands being executed as soon after submission as possible, the SYCL specification cannot guarantee this. If the event is implemented as eager then the command is executed immediately, so no synchronization is required to ensure the command is executing. However, if the event is implemented as lazy then the command will not execute immediately, and the only way to ensure this is to synchronize with the event or queue or the destruction of a buffer which is being written to by the command.

In the second case, where users wish to create task graphs, i.e. commands are collected and batched all together and the creation of native event objects can be elided. If the event is implemented as eager then the backend must provide a native backend event object for every command, even if it's never used, creating a performance overhead. If the event is implemented as lazy then the creation of native backend event objects can be elided, however, the event class still requires synchronizations and query capabilities.

Considered Solutions

There are a few different options which have been considered to solve this:

• Introduce a property for course-grained events; synchronization on an event may also synchronize with earlier events (https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/extensions.md#hipsycl_ext_coarse_grained_events).
  • This breaks current user expectations of events.
• Introduce a property for discarding events; synchronization on an event may fail (https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/supported/sycl_ext_oneapi_discard_queue_events.asciidoc).
  • This causes errors if used incorrectly.
• Introduce new interfaces for queue/handle which either return an event or don't ([SYCL][DOC] Add sycl_ext_oneapi_enqueue_functions #11491).
  • Causes bifurcation in the SYCL interface

Alternative Approach Using Sender/Receiver

Another option is to adopt the sender/receiver model of execution proposed in P2300 for ISO C++.

The sender/receiver model works by composing senders which are like std::future but lazy, starting with a sender produced by scheduling on a particular execution resource and then adding commands via sender algorithms, which can then be synchronized with.

In the sender/receiver model:

• A scheduler represents where work is executed.
• A sender represents the work to be executed.
• A receiver represents the completion of and synchronization with the work.

Receivers are not typically exposed to users.

To introduce the sender/receiver model in SYCL we would need to do the following:

• Introduce SYCL sender types (compile-time & polymorphic).
• Introduce a SYCL scheduler type and a get_scheduler() member function on queue for retrieving a scheduler.
• Introduce implementations of std algorithms, starting with the basics such as schedule and ensure_started, further algorithms could be added incrementally.
• Introduce new versions of SYCL command functions as sender-based algorithms for submit and the shortcut functions such as parallel_for, memcpy, etc.

Benefits of Using Sender/Receiver

There are a number of benefits of using the sender/receiver model.

• SYCL 2020 doesn't specify whether commands are executed eagerly or lazily, so users must assume they could be lazy, therefore there is no change to the programming paradigm.
• Adopting lazy execution by default will make the task graph use case more efficient by default.
• Defining the execution as lazy with algorithms for forcing eagerness resolves confusion in the low latency use case.
• Users can still use senders as they did with events before, discarding them if they want.
• All options for synchronization still apply (buffer destruction, event wait, queue wait).
• There are use cases for eager/hybrid execution such as low latency applications, we can allow eager execution as a scheduling hint, via a property.
• Allows SYCL runtime to more easily batch commands.
• Allows users to more easily reflect intent for eagerness.
• Avoids bifurcating the command function interfaces.

Status of Senders/Receivers in ISO C++

The senders/receiver paper (P2300) is in review by the C++ committee, and it is hoped that it will be approved for C++26, however, if it isn't then it could be later before it's available in C++.

Example

Below is an example of SYCL using the sender/receiver model.

queue q;
 
auto sch = q.get_scheduler();
 
auto s0 = schedule(sch);
auto s1 = submit(s0, [&](handler &cgh){
  cgh.parallel_for(range, kernel1);
});

auto s2 = q.submit(s1, [&](handler &cgh){
  cgh.parallel_for(range, kernel2);
});

this_thread::sync_wait(s2);

The above example can also be reprsented differently using the sender/receiver pipe syntax, here this is using the parallel_for shortcut function.

queue q;
 
auto s = schedule(q.get_scheduler())
  | parallel_for(range, kernel1)
  | parallel_for(range, kernel2);

this_thread::sync_wait(s);

Proposal

Execution Stream

Introduce a new type; execution_stream which represents the smallest granularity of execution resource for a backend, that is a single stream of execution and executes in order.

SYCL Scheduler

Introduce a new type; scheduler which conforms to the scheduler concept of P2300 and is associated with a SYCL queue and device to which work will be enqueued.

Add a new member functions to the queue and execution_stream classes; queue::get_scheduler and execution_stream::get_scheduler() which return a scheduler.

SYCL Senders

Introduce the concept of sender in SYCL as an object which represents a series of commands which have not yet been enqueued to the backend. Specify that there are two kinds of sender:

• Unique implementation-defined sender types which are return from command functions such as submit.
• A polymorphic type-erased sender type which the implementation-defined sender types can be converted to.

Introduce the concept of implementation-defined sender types which conforms to the sender concept of P2300 and represents zero or more commands such as kernel launches or copies which have not yet been enqueued to the backend.

Introduce a new type sender which provides a polymorphic wrapper to any implementation-defined sender type, providing a conversion operator for those types.

SYCL Sender Factories

Introduce the schedule factory function as follows:

<impl-defined-sender> schedule(scheduler);

Returns an empty implementation-defined sender which represents the transition to the target device with the sender's queue or execution_stream.

SYCL Sender Algorithms

Introduce all existing command functions (member functions of queue) in SYCL 2020 as algorithms, including submit, single_task, parallel_for, memcpy, memset, fill, copy, prefetch and mem_advise as follows:

template <typename Sender, template-args>
impl-defined-sender function(Sender, args);

Each function would behave as currently defined in SYCL 2020, but returning an implementation-defined sender representing the specified command as to be executed but not yet enqueued to the backend.

SYCL Sender Adapters

Introduce the ensure_started adapter function as follows:

<impl-defined-sender> ensure_started(scheduler);

Takes a SYCL sender type, ensures that the commands it represents are enqueued to the backend and then returns a new implementation-defined sender type.

SYCL Sender Consumers

Introduce the sync_wait consumer function as follows:

namespace this_thread {
  void sync_wait(scheduler);
}

Takes a SYCL sender type, ensures that the commands it represents are enqueued to the backend and then synchronizes with the execution resource executing those commands before returning.

Note: in P2300 this_thread::sync_wait returns a std::optional<std::tuple<>>, it needs to be considered whether SYCL should implement this as in P2300 or returns void.