Skip to main content

Persistence

The SEF Reference FTL uses the persistence layer to store the flash translation layer’s operating metadata, such as the lookup table, and Super Block information to the flash memory. The persistence layer manages that in two parts: Data Management and Data Tree storage.

Data Management​

The data management code, which is located in persistence.c, is used to manage the data to be stored, stored data, and the root pointer.

Data to Be Stored​

After the persistence layer is initialized, individual components of the Reference FTL can queue data to be flushed. The data is stored and identified by a unique key and index. While queuing the data, the caller may provide a callback function that will be called when the data is flushed to the flash memory or when the persistence cleanup is called. This callback function may be used to clean up allocated memory or perform other follow-up actions. Moreover, given the return value of isFlushed, appropriate action may be taken considering whether the data was flushed.

Queuing the data does not ensure the data is stored to the flash memory. In order to store the data on the flash memory, the queued data should be flushed by calling PDLFlushToFlash(). The caller is responsible for freeing the data previously stored on the flash memory. In other words, the previously written data should be cleared manually by calling PDLFreeFlash() to avoid leaks.

Stored Data​

In order to keep track of the flushed data, a metadata table is used. The metadata table is an array of stored objects and their unique keys, unique indexes, sizes, and CRC-32. The array of metadata is flushed at the time of storage and is used to identify and read the stored data.

As part of persistence initialization, the persistence layer uses the root pointers to locate the metadata table and find out about the stored data. A cyclic redundancy check (CRC-32) is used in order to verify the stored metadata. Moreover, given that multiple code bases may use the persistence layer independent of the Reference FTL, each code base should use a Globally Unique Identifier (GUID). The GUID is checked at the time of persistence initialization.

To read the stored data, the caller uses a unique key and index. The stored data is also verified using the CRC-32.

In order to separate the user data from the metadata stored by persistence, the persistence layer stores the data on isolated Super Blocks. The persistence layer manually allocates Super Blocks that are only used by persistence to store the queued data.

Root Pointer Management​

The root pointers are used to keep track of the persisted data. They are used as pointers into the device where the metadata table is stored. If the root pointer is null, it is assumed that the device is empty and no persistence data has been flushed; however, a null root pointer could also indicate that the device is not empty and the persistence layer was not able to store the metadata or update the root pointer. In order to avoid ambiguity, the caller should mark the root pointer as dirty after data has been written to the device.

Given that the Reference FTL may experience unexpected shutdowns, the root pointer is an important tool for verifying the freshness of the read data. To detect an unclean shutdown, the root pointer is checked at initialization time. A dirty root pointer denotes that the software experienced an unexpected shutdown.

The root pointer is set to clean either automatically, after data is flushed to the flash memory, or manually, by being marked as clean.

The root pointer is marked as dirty by creating a Flash Address that has its QoS Domain ID set to zero. The root pointer can be marked as clean and reconstructed by setting the QoS Domain ID of the Flash Address back to the QoS Domain ID of the loaded QoS Domain. The Reference FTL uses this mechanism in order to detect an unexpected shutdown.

In summary, the root pointer is checked when the Reference FTL is initialized, and a dirty root pointer denotes the occurrence of an unexpected shutdown. If the root pointer is clean, the Reference FTL marks the root pointer as dirty when the first write is performed. When the Reference FTL goes through the expected shutdown process, the data is flushed and a clean root pointer is set, denoting a clean shutdown.

Data Tree Object​

The Data Tree Object (DTO) is a self-contained data structure used to store contiguous data. The need for a DTO becomes apparent given SEF’s support for various defect-management methods that do not guarantee contiguous data placement. The kFragmented method, for example, may not store data contiguously.

The DTO root is always 1 ADU in size and stores the flash addresses of the next tree nodes. For example, if an ADU is 4096 bytes and the DTO node structure is 16 bytes, then 510 flash addresses (8 bytes each) can be used to store either the locations of the next layer’s nodes or the location of data. Given this structure, the root node can store the flash addresses for up to 2 megabytes of data.

Figure 5 illustrates the structure of an example DTO. In this example, just by having a two-layer tree, persistence is able to keep track of an object with a maximum size of 1065 megabytes.

Figure 5: Data Tree Object Structure​

Data Tree Object Structure

The DTO offers functions to read, write, free, and get the size of an entire tree in order to simplify use of the given tree. Moreover, the host can get the Super Blocks used by a given tree. Given the structure of the DTO, it must be read as a whole and not partially.