SEF Library
Design Environment
The Software-Enabled Flash™️ (SEF) Library runs on a Linux™️ host. It only supports user mode and does not support forked processes. The SEF Library API is defined by SEFAPI.h and implemented in libsef.so. It converts interface calls into NVMe SEF command set commands and submits them to the SEF kernel driver.
Design Strategy
The SEF Library makes every effort to execute what it's asked to do. It does not enforce any policies. Limits are the responsibility of the caller. I/O requests that are larger than the device's max data transfer size are split up by the library in a way to simulate a single device request. For split writes and copies, requests for flash addresses are serialized to keep the addresses sequential when possible. The write commands and split reads run concurrently with no limit placed on how many requests are generated. Large I/Os and large I/O depths can overwhelm the device queues, causing the driver to reject requests. These are automatically retried after a delay. This makes the API simple to use but can lead to significant latency and may render weighted fair queueing I/O policies ineffective.
Threading Model
A threading model was chosen that prevents lost I/O completion and deadlock. I/O uring is used to send all asynchronous requests to the SEF driver. The submitting and completion of asynchronous SEF driver requests is handled by an internal, statically sized thread pool based on the number of CPUs. When possible, synchronous requests use ioctl() issued from the caller's thread. Pair commands, which can't use ioctl(), use a dedicated thread. This design allows the lifetime of the caller's thread to be independent of the I/O it issues and prevents deadlock when a synchronous API call is made from an asynchronous I/O completion callback. New I/O issued from a completion routine will be submitted using the completion thread, avoiding the overhead of switching to an I/O thread. There are two additional threads started by the SEF Library to handle notifications. One handles udev events sent by the SEF driver, and the other monitors closing Super Blocks and issues a close notification once all write I/O has completed.
Use of the SEF API will never internally deadlock, but issuing a synchronous request or waiting for a resource owned by another completion may add latency to any I/O issued by the blocked thread. Deadlock may still occur externally if a completion routine blocks on resources that require another completion routine to execute.