ESP32 Storage programming

Понятие хранения в программировании хранения включает в себя методы хранения данных для последующего использования. ESP32 имеет ОЗУ, но когда ESP32 выключен, содержимое ОЗУ теряется. Таким образом, нам нужен механизм, чтобы сделать это хранилище более постоянным. ESP32 обычно имеет доступ к флэш-памяти, которая электрически подключается через специальную шину SPI. Обычно размер флеш-памяти составляет 4 Мбайт. Мы можем получить доступ к флэш-памяти через SPI Flash API.

Partition table

Архитектура ESP32 представляет собой концепцию, называемую таблицей разделов, которая в основном представляет собой «карту» или «Макет» того, что содержится во флэш-памяти. Таблица разделов находится в 0x8000 во флэш-памяти и имеет длину 0xC00 байт, что обеспечивает пространство для около 95 отдельных таблиц записей. Каждая запись в таблице имеет структуру записи, которая логически содержит:

• type – The type of the partition. One of:
  • data
  • app
• subtype – The sub-type of the partition. One of:
  • nvs – Used for non volatile storage.
  • phy
  • factory
  • coredump – Used to hold core dumps.
  • ota
  • fat – Used for the FAT file system.
• address – The offset address (flash) of the start of the partition.
• size – The size of the partition in bytes.
• label – An optional null terminated string (max 16 characters + NULL)
• encrypted – Is the partition encrypted.

The partition table is read-only to our applications and can be accessed with a rich API provided by the ESP-IDF. The table is written into flash storage by the flash tool.

The offset size is optional. Blank offsets will be placed contiguously after the previous data. Offsets are 64K aligned.

A tool called "gen_esp32part.py" is available as part of the tooling to build binary representations of the table. We can build a binary table from a comma separated value file using: $ gen_esp32part.py –verify input_partitions.csv binary_partitions.bin We can convert a binary file back to a CSV using: $ gen_esp32part.py --verify binary_partitions.bin input_partitions.csv and we can list the content of a binary file using: $ gen_esp32part.py binary_partitions.bin

The partition table used by your application is defined by make menuconfig in the

Partition Table entry:

Within that section we have three possibilities: If we select to use a custom partition table, there are further options: • Custom partition CSV file – The Comma Separated Values file that contains the text description of our partition definitions. • Factory app partition offset – The location of our factory application. I recommend starting with the ESP-IDF supplied partition table as a skeleton. Take a copy of this and place in your project. The supplied file can be found at: <ESP-IDF>/components/partition_table/partitions_singleapp.csv and copy this to the file: paritions.csv See also: • Partition API • esp_vfs_fat_spiflash_mount • esp_vfs_fat_register

Non Volatile Storage

Non volatile storage is memory that can be written to such that after a power off or restart, the same data can be read from it again without loss. It is preserved over a restart. Within this data we can store configuration and operational values for our applications. For example, we might store the network SSID and password such that when the device is restarted, it know which network to connect to and the password to present.

The storage is partitioned into named areas. For a given named area, we can then write name/value pairs to the storage and also read name/value pairs. There are getter/setter functions for most data types including signed and unsigned integers, strings and blobs of data.

A named area is opened for access with a call to nvs_open(). The name of the area is passed in as a parameter. We are returned a logical "handle" that we can subsequently use to refer to this storage area. Once we have a handle, we can get/set items of named data. The data items are referenced by a key name … effectively turning the storage area into a hash map. If we change data by performing a set function, this does not automatically cause the data to be written to the nonvolatile storage. Instead, the storage is updated when we call nvs_commit(). It is up to the internal implementation as to when the actual update is performed and it could happen prior to nvs_commit(). The contract is that when we return from nvs_commit() then we are assured that all updates have been processed. When we have done all our sets and gets, we should call nvs_close() to declare that we are not going to work with storage any more at this time so that the run-time can clean up any resources it may have opened.

The details of the algorithms used to manage NVS are exposed in the documentation. The high level intent of NVS is to store simple strings and integers and other flags as opposed to be a rich "file system" like structure. There is currently no de-fragmentation performed on the storage.

See also: • nvs_open • nvs_commit • nvs_close

Virtual File System

The Virtual File System (VFS) is the architecture provided by the ESP-IDF that gives us the capability of saving and loading data from our applications using a file system I/O. The VFS isn't tied to any one particular technology but is instead an architectural abstraction used to provide the I/O interface to a variety of different implementations. The key to the VFS is a data type called esp_vfs_t. This structure contains the following: • fd_offset – • flags – Operational flags. Use ESP_VFS_FLAG_DEFAULT. • close/close_p – Close a previously opened file. • closedir/closedir_p – Close a previously opened directory. • fstat/fstat_p – Get stats/details of a file. • link/link_p – Create a new link to a file. • lseek/lseek_p – Change the data pointer within a file. • mkdir/mkdir_p – Create a new directory entry. • open/open_p – Open a named file. • opendir/opendir_p – Open a directory for reading. • read/read_p – Read the contents of a file. • readdir/readdir_p – Read a record from a directory. • rename/rename_p – Rename a file. • rmdir/rmdir_p – Delete a directory entry. • seekdir/seekdir_p – Set the position of the next readdir(). • stat/stat_p – Get stats/details of a file. • telldir/telldir_p – Return the current direction stream. • unlink/unlink_p – Remove a file. • write/write_p – Write into a file.

After populating the structure, we need to register our new virtual file system with a call to esp_vfs_register(). We need to be cognizant that the intended caller of file I/O expects a POSIX like environment.

See also: • esp_vfs_register • Virtual filesystem componen t

VFS Implementations

Since the VFS provides an architectural model, we need to consider actual implementations of it. As of 2016-11, none are yet available. The first anticipated implementations will be file systems stored in flash. These will provide persistent storage of data through a file API. Potential implementations will include FAT or SPIFFS.

We can also produce our own specialized implementations. One interesting idea is to allow the ESP32 to be a network client of external file systems. Possibilities include: • NFS • SSH • FTP • TFTP • HTTP servers • Google Drive • Other cloud based systems

It may seem strange to have a network device access data through a file mechanism only to have it then farm out the requests as another network call … however there may be benefits. The ESP32 could cache the received data either in RAM or local flash and only perform external network requests if the requested data is not available elsewhere.

When working with file I/O, we can use the streams file mechanisms imported via "stdio.h" or use the lower level file I/O imported through "fcntl.h". See also: • VFS mapping to SPIFFS

FATFS File System

The FatFs file system is an implementation of the FAT/exFAT file system as found in earlier PC operating systems such as MS-DOS and early Windows (before FAT32 and NTFS). The implementation is open source and is supplied "pre-ported" to the ESP32 as part of the ESP-IDF distribution. The ESP-IDF mapping for the FATFS maps the file system to the posix IO functions. This means that we don't need to learn any special APIs in order to read and write files. We can use open(), close(), read(), write() and the other methods exposed through Virtual File System. Before we can use these APIs, we need to perform some preliminary setup. 1. Call esp_vfs_fat_register 2. Call ff_diskio_register 3. Call f_mount

To unregister 1. Close all open files 2. Call f_mount with NULL 3. Call ff_diskio_register with NULL 4. Call esp_vfs_fat_unregister

By default, the filenames are constrained to the old 8.3 format (short names), however, should we choose, we can enable long file name control in the make menuconfig settings.

See also: • FatFS – Generic FAT File System Module • Virtual File System • FatFs file system

Spiffs File System

The SPI Flash File System (SPIFFS) is a file system mechanism intended for embedded devices. To configure SPIFFs we need to determine some numbers. First is the physical page size. Next comes the physical block size. Next we decide on the logical block size. This will be some integer multiplier of the physical block size.

The whole SPIFFS file system must be a multiple of the logical block size. Next comes the logical page size which is some multiplier of the logical block size. A common ESP32 sizing is 64K for the logical block size and 256 for the logical page size.

To be clear a 1 block is n x pages.

When a SPIFFS API call is made, a zero or positive response indicates success while a value < 0 indicates an error. The nature of the error can be retrieved through the SPIFFS_errno() call. The SPIFFS implementation does not directly access the flash memory. Instead, a functional area called a hardware abstraction layer ("hal") provides this service. A SPIFFS integration requires that three functions be created that have the following signatures: s32_t (*spiffs_read)(u32_t addr, u32_t size, u8_t *dst) s32_t (*spiffs_write)(u32_t addr, u32_t size, u8_t *src) s32_t (*spiffs_erase)(u32_t addr, u32_t size) If they succeed, the return code should be SPIFFS_OK (0). On an ESP32, these will map to the SPI flash APIs. To use a SPIFFS file system, we must perform a call to SPIFFS_mount(). This takes as input a configuration structure that tells SPIFFS how much flash is available and a variety of other properties. In addition, some working storage must be allocated for various internal operations. These sizes can be tuned. Here is an example of configuration for mounting a file systems:

1. define LOG_PAGE_SIZE 256

static uint8_t spiffs_work_buf[LOG_PAGE_SIZE*2]; static uint8_t spiffs_fds[32*sizeof(uint32_t)]; static uint8_t spiffs_cache_buf[(LOG_PAGE_SIZE+32)*4]; spiffs fs; spiffs_config cfg; cfg.phys_size = 512*1024; // use 512K cfg.phys_addr = 2*1024*1024 - cfg.phys_size; // start spiffs at 2MB - 512K cfg.phys_erase_block = 65536; // according to datasheet cfg.log_block_size = 65536; // let us not complicate things cfg.log_page_size = LOG_PAGE_SIZE; // as we said cfg.hal_read_f = esp32_spi_flash_read; cfg.hal_write_f = esp32_spi_flash_write; cfg.hal_erase_f = esp32_spi_flash_erase; int res = SPIFFS_mount(&fs, &cfg, spiffs_work_buf, spiffs_fds, sizeof(spiffs_fds), spiffs_cache_buf, sizeof(spiffs_cache_buf), 0); Once we have mounted the file system, we can then open a file, write content into it and close it. For example: char *fileName = "/f1/my_file"; spiffs_file fd = SPIFFS_open(&fs, fileName, SPIFFS_CREAT | SPIFFS_TRUNC | SPIFFS_RDWR, 0); SPIFFS_write(&fs, fd, (u8_t *)"Hello world", 12); SPIFFS_close(&fs, fd); Similarly, if we wish to read data from the file we can perform the following: char buf[12]; spiffs_file fd = SPIFFS_open(&fs, fileName, SPIFFS_RDWR, 0); SPIFFS_read(&fs, fd, (u8_t *)buf, 12); SPIFFS_close(&fs, fd); Using the The SPIFFS file system could be hierarchical in nature such that it contains both directories and files but it seems that in reality it is not. There is only one directory called the root. The root directory is "/". To determine the members of a directory, we can open a directory for reading with the SPIFFS_opendir() API and, when we are finished, close the reading operation with a SPIFFS_closedir() API call. We can walk through the directory entries with calls to SPIFFS_readdir(). For example: spiffs_DIR spiffsDir; SPIFFS_opendir(&fs, "/", &spiffsDir); struct spiffs_dirent spiffsDirEnt; while(SPIFFS_readdir(&spiffsDir, &spiffsDirEnt) != 0) { printf("Got a directory entry: %s\n", spiffsDirEnt.name); } SPIFFS_closedir(&spiffsDir); To make this clear, in Linux, if we created "/a/b/c.txt" this would normally create a file called "c.txt" in a directory called "b" in a directory called "c". In SPIFFS, this actually creates a single file called "/a/b/c.txt" where the "/" characters are merely part of the file name. When we perform SPIFFS_opendir(), there isn't actually a directory structure but just one single flat list of ALL files which may or may not have "slashes" in their names. To create a file, we can use the SPIFFS_open() API by supplying a SPIFFS_CREAT flag. See also: • SPIFFs API • Github: pellepl/spiffs • Github: igrr/mkspiffs – The mkspiffs tool. • SPI Flash • Virtual File System mapping to SPIFFS

Building SPIFFs for the ESP32

Under the heading of "let's build on each other", an excellent job has been done of porting SPIFFs to the ESP32 by the LUA team (Jaume Olivé Petrus). The source code can be found: https://github.com/whitecatboard/Lua-RTOS-ESP32/tree/master/components/spiffs. They have packaged it as an ESP-IDF component. mkspiffs tool In addition to the fantastic SPIFFs library, there is also a tool called "mkspiffs" that can take a directory structure on your file system and build a SPIFFs image from it that can then be loaded into flash memory to provide pre-loaded data. One can download the Git repository for mkspiffs and compile it. I found no issues and it compiled at first go. The syntax is: mkspiffs { -c <packdir> | -u <destdir>|-l|-i} \ -b <number> -p <number> -s <number> One of: • -c <directory to pack> • -u <dest to unpack into> • -l – list content • -i – visualize content and • -b <number> – Block size in bytes (for example 65536) • -p <number> – Page size in bytes (for example 256) • -s <number> – fs image size in bytes. Page 401 Visualizing an image file shows results such as: 0 idid___________ era_cnt: 0 1 _______________ era_cnt: 0 2 _______________ era_cnt: 0 3 _______________ era_cnt: 0 4 _______________ era_cnt: 0 5 _______________ era_cnt: 0 6 _______________ era_cnt: 0 7 _______________ era_cnt: 0 8 _______________ era_cnt: 0 9 _______________ era_cnt: 0 10 _______________ era_cnt: 0 11 _______________ era_cnt: 0 12 _______________ era_cnt: 0 13 _______________ era_cnt: 0 14 _______________ era_cnt: 0 15 _______________ era_cnt: 0 era_cnt_max: 1 last_errno: 0 blocks: 16 free_blocks: 15 page_alloc: 4 page_delet: 0 used: 1004 of 52961 total: 52961 used: 1004 Once we have an image file, we can the load it to flash with: esptool.py --chip esp32 --port "/dev/ttyUSB0" --baud 115200 write_flash -z --flash_mode "dio" --flash_freq "40m" <address> <file> See also: • Github: igrr/mkspiffs – The mkspiffs tool.

The ESP File System – EspFs

Part of the Github project known as "Spritetm/libesphttpd" is a module called "espfs" which is the "ESP File System". What this module does is allow one to make an image from a set of files on your PC and store that combined image in flash memory. From there, a set of APIs are provided to read and access those files and their content. It is vital to note that the data in these files is read-only. There is no API to update the content of the files. Only the data that is initially written to flash is available to be read. As part of the project there is a utility called "mkespfsimage" that takes as input a set of file names and streams as output the image data that should be flashed. For example: find | ./mkespfsimage [-c compressor] [-l compression_level] > out.espfs (Note that the project has compression capabilities that I am ignoring at this point).

Once the data is in flash, we can then use the APIs supplied by the component to perform the underlying data access.

They are: • EspFsInitResult espFsInit(void *flashAddress) • int espFsFlags(EspFsFile *fh) • EspFsFile *espFsOpen(char *fileName) • int espFsRead(EspFsFile *fh, char *buff, int len) • void espFsClose(EspFsFile *fh) An attempt to port the code to utilize ESP32 technologies was undertaken and can be found here: https://github.com/nkolban/esp32-snippets/tree/master/filesystems/espfs

This adds a new function called: • int espFsAccess(EspFsFile *fh, void **buf, size_t *len) This function returns a pointer to the whole content of the file which is stored in buf. The length of the file is stored in len and also returned from the function as a whole. The data is accessed directly from flash without any RAM copies. In addition, the function called: • EspFsInitResult espFsInit(void *flashAddress, size_t size) was augmented to include the size of the flash storage to map. Here is an example application: ESP_LOGD(tag, "Flash address is 0x%x", (int)flashAddress); if (espFsInit(flashAddress, 64*1024) != ESPFS_INIT_RESULT_OK) { ESP_LOGD(tag, "Failed to initialize espfs"); return; } EspFsFile *fh = espFsOpen("files/test3.txt"); if (fh != NULL) { int sizeRead = 0; char buff[5*1024]; sizeRead = espFsRead(fh, buff, sizeof(buff)); ESP_LOGD(tag, "Result: %.*s", sizeRead, buff); size_t fileSize; char *data; sizeRead = espFsAccess(fh, (void **)&data, &fileSize); ESP_LOGD(tag, "Result from access: %.*s", fileSize, data); espFsClose(fh); } SD, MMC and SDIO interfacing Secure Digital (SD) is a standard for removable media. These devices are also known as "flash cards" or "SD cards". The idea is that an SD card contains data that can be both read and written. The SD cards store the data as raw memory and it is common to create a file system that lives on top of the data. The FAT16 and FAT32 file system formats are commonly used. SD cards come in a variety of physical dimensions and with a variety of capacities and speeds. For the physical dimensions there are three distinct types known as "SD", "miniSD" and "microSD" ranging from largest to smallest. For capacity, there are again three distinct types known as "SD", "SDHC" and "SDXC". SD SDHC SDXC Capacity x <= 2GB 2GB <= x <=32GB 32GB <= x <= 2TB File system FAT12, FAT16 FAT32 exFAT For our story, we will ignore SDXC. An additional characteristic of SD cards is their rated speed. The common speeds are: Class 2 2MB/s Class 4 4MB/s Class 6 6MB/s Class 10 10MB/s The SD specification is large and comprehensive. If we were to try and implement the SD specification ourselves we would be delving down into a whole host of puzzles. As such, it is common to leverage pre-existing implementations of the specification and, thankfully, the ESP-IDF provides us with exactly that. There is also an excellent example application provide in the examples/storage/sd_card directory of the ESP-IDF. The SD card can be used to hold data but can not be used to hold instruction code for execution. As such, the SD card shouldn't be considered as an alternative to the flash memory accessible via SPI for code storage. The SD card should be used to store application data that can be read by or written by running applications. See also: Page 404 • Wikipedia: Multi Media Card • Wikipedia: Secure Digita l • SD Association Charting data Consider some of the sensors