FEAT: manual control of completion events generation [DRAFT]

[grom72]

FEAT: manual control of completion events generation

Rationale

Each time rpma_conn_wait() is called, it arms CQ to trigger a new event as soon as the next work completion is ready.

rpma/src/conn.c

Line 275 in e711430

errno = ibv_req_notify_cq(ev_cq, 0 /* all completions */);

That may cause unnecessary event generation if an application can not consume completions fast enough.
To avoid that two improvements are needed:

• block the next event triggering setup inside rpma_conn_wait()
• provide an API to arm CQ to generate a new event with the next work completion prepared

Description

The new semantic of the rpma_conn_wait() can be as follow:
librpma.h:

...
#define RPMA_CONN_POSTPONE_NOTIFY_REQ 1
...

int
rpma_conn_wait(struct rpma_conn *conn, int flags, struct rpma_cq **cq, bool *is_rcq)
{
...
	if (!(flags & RPMA_CONN_POSTPONE_NOTIFY_REQ)) {
		errno = ibv_req_notify_cq(ev_cq, 0 /* all completions */);
		if (errno) {
			*cq = NULL;
			RPMA_LOG_ERROR_WITH_ERRNO(errno, "ibv_req_notify_cq()");
			return RPMA_E_PROVIDER;
		}
	}
}

and a new API call to trigger events on completion:

int
rpma_cq_wait_req(rpma_cq *cq, int flags) {
	if (cq == NULL)
		return RPMA_E_INVAL;
	errno = ibv_req_notify_cq(cq->cq, flags /* all completions */);
	if (errno) {
		RPMA_LOG_ERROR_WITH_ERRNO(errno, "ibv_req_notify_cq()");
		return RPMA_E_PROVIDER;
	}
}

With such an approach we are able to support the following pattern of completions processing:

	while (1) {
		if ((ret = rpma_conn_wait(conn, &cq, NULL, RPMA_CONN_POSTPONE_NOTIFY_REQ)))
			return ret;
		if ((ret = rpma_cq_get_wc(cq, MAX_N_WC, wc, &num_got))) {
			if (ret == RPMA_E_NO_COMPLETION) {
				rpma_cq_wait_req(cq, 0);
				continue;
			}
			return ret;
		}
		process_comp(wc, num_got);
		if ((ret = rpma_cq_wait_req(cq, 0)))
			return ret;
		if ((ret = rpma_cq_get_wc(cq, MAX_N_WC, wc, &num_got))) {
			if (ret == RPMA_E_NO_COMPLETION) {
				continue;
			}
			return ret;
		}
	}