ESP32 is a powerful 32-bit controller that runs at speeds of 160 ~ 240 MHz. Designed for IoT, the controller features on-chip Wi-Fi (40 MHz), Bluetooth 4.0, Bluetooth Low Energy, CAN, and an Ethernet interface. With 34 GPIO, 18 analog pins, 16 software PWM channels, 4 SPI ports, 2 I2C ports, 2 I2S ports and 2 UART ports, the ESP32 offers endless possibilities for embedded design. Being such a capable chip, the ESP32 can be deployed to power a variety of IoT network nodes.
Different IoT devices have different hardware requirements. Managing data right at the heart of a “thing” is a crucial aspect of embedded hardware. Once you select ESP32 for your IoT project, you don't need to worry about computing resources. ESP32 is more than enough to run a networked embedded application. Memory and RAM are areas where microcontrollers often have their natural limits, but the ESP32's RAM (512 KB) is sufficient for any embedded or IoT task. The ESP32 also has 4 to 16 MB flash, of which 512 bytes are used to store data while the rest are used to store user code.
However, the 512 bytes (ROM) available for data may be insufficient for many IoT projects, especially those that involve the use of audio or video files or that require longer integrated data logging. For such projects, we need to explore other storage media that can be interfaced with ESP32 to extend memory. Fortunately, ESP32 can be loaded with different file systems that remove the restriction of using just flat files. In this article, we will explore different storage devices that can interface with the ESP32 and investigate different possible file systems on the ESP32 to enable proper data storage and management.
ESP32 flash memory
ESP32 comes with on-chip or onboard flash memory of 4 MB, 8 MB or 16 MB. Most of this memory space is reserved for storing the bootloader, radio driver, and application code. As noted earlier, only 512 bytes of flash memory are available to store user data. Flash memory is partitioned into three slots for non-OTA configuration: NVS library, PHY_init, and factory. The NVS library partition is the only space where user application parameters can be saved. The factory partition stores the bootloader, while the PHY partition stores the radio driver for Bluetooth and Wi-Fi, responsible for implementing the physical network layer. With OTA (Over-The-Air Updates) configuration, there are three additional partitions, of which one “OTAdata” is used to store the OTA application and two partitions are used to store OTA updates.
Flash memory in the ESP32 can only store scalar user data initiated through variables or data structures in the code. These scalar values are stored in the NVS library partition as application parameters. Flash memory is a few MB in size, which is only enough for cost-sensitive applications that take up little memory space. Some parts of flash memory can be partitioned to store user data such as text, images and other files only after mounting a file system (SPIFFS or FAT) on it. Otherwise, flash memory can only store user-generated binary code and application parameters. The SPI flash on the ESP32 can use 2-line or 4-line mode of operation. The data rate can be set to 40 MHz or 80 MHz.
Extending memory with SD cards
Although flash memory can be mounted and partitioned to store data such as images and other files, it is often insufficient to store an appropriate volume of data. For example, audio and video files can be several megabytes in size. These file types can never be stored in flash; the best alternative is to use SD cards.
SD cards from 4GB to 32GB tend to work well with ESP32, where the maximum size of a file can be up to 4GB. There is plenty of space to store audio, video and database files. The ESP32 can transmit audio/video, capture images and maintain large data records by connecting an SD card of sufficient size. There are two interfaces on the ESP32 for accessing SD cards – SDIO and SPI. If the SDIO interface is used to connect an SD card reader with ESP32, it occupies six GPIOs fixed on the board. The SPI interface can be multiplexed with any GPIO and requires three or four pins. It is possible to connect more than one SD card to the ESP32 by configuring additional pins as chip selection. The SDIO interface has a default frequency of 20 MHz that can be configured to 40 MHz for line 1/4/8 modes. The frequency of the SPI interface is configurable through user code. On ESP32-S2, only the SPI interface is available for interfacing SD cards. The SD card must have a FAT file system to work with ESP32.
Extending memory with eMMC cards
eMMC cards are similar to SD cards and use the same driver as SD cards. It is important to note that eMMC memory cards are not supported on the ESP32-S2. With other ESP32 models, eMMC cards can be interfaced to store audio, video and database files. These memory cards can be connected to the ESP32 using the SDIO interface only. The default data rate is 20 MHz, which can be set to 40 MHz with 8-line mode and 4-line DDR mode. eMMC cards are often interfaced with ESP32 chips in wearables like smartwatches. In such applications, the eMMC card stores system applications containing graphics and/or database files.
Extending memory with EEPROM
In low-power applications such as IIoT, image, audio or video files do not need to be stored. The controller needs some extra space to store configuration parameters, keep logs, or restore data before retransmission. In these applications, SD or eMMC cards can be overkill, draining the battery and unnecessarily multiplying the cost of the device. For these situations, extra space of 1 MB or 16 MB can be added to the device through the EEPROM interface with ESP32. The ESP32 supports Atmel's AT24C01/02 EEPROM chips and can be connected to the ESP32 via an I2C interface. ESP32 can also read/write EEPROM at a data rate of 100 to ~400KHz and access data via direct addressing.
File system supported on ESP32
A typical ESP32 setup is simple – until you have to deal with flat files. However, when it is necessary to store audio, video, database or other complex files on the ESP32 board, complications arise. Microcontrollers are designed to store simple data and rarely need to operate on complex file types. Fortunately, the ESP32 can be loaded with different file systems to store and handle complex file types. The following file systems are supported on ESP32.
- NVS Library
- SKEWERS
- FAT
- Virtual file system
NVS library file system
One of the default partitions on ESP32 flash is the NVS library. The partition can be used to store application parameters. Application data stored in NVS is not lost when the ESP32 is turned off, restarted, or encounters a power failure. The partition needs to be activated using the NVS library in user code, which then mounts the NVS library file system to the respective partition. The file system allows you to store key-value pairs where the keys can be ASCII strings and the values can be integers, strings, or binary large objects (BLOBs) such as character arrays. The NVS partition is not yet suitable for storing large strings. Depending on the standard or custom partition table applied, the partition size is limited to KB or MB.
SPIFFS file system
Serial Peripheral Interface Flash File System (SPIFFS) is the default file system supported by the ESP32. SPIFFS is a lightweight microcontroller file system with SPI NOR flash of less than 128 MB. The file system requires less RAM compared to the FAT file system. It supports wear leveling, data consistency, and POSIX support. The file system can be mounted on the ESP32 or unified into a virtual file system on the ESP32. The SPIFFS file system is suitable for partitioning flash memory to store images, audio, video and other files.
FAT file system
The SPIFFS file system does not come with directory support because it is not platform independent. An organized directory structure is often necessary in many situations, such as when the ESP32 hosts a small web server with some HTML, CSS, and image files. The other situation could be when external memory devices such as SD cards or eMMC cards interface with the ESP32 and have files stored not directly in the root but in directories. In these cases, the ESP32 can be mounted with the FAT-Fs file system. FAT-Fs act as a platform-independent layer and provide a unified interface to access all available memory devices such as external flash chips, SD cards, and eMMC.
The file system supports a directory structure and allows data encryption. FAT-Fs supports gigabyte memory devices, while SPIFFS is only suitable for flash memories up to 128 MB. Wear leveling is optional on FAT-Fs. The FAT-Fs file system on the ESP32 should be used to store audio, video, images, and other types of files when the files need to be stored on large-capacity external memory devices or require a directory structure for storage.
Virtual file system
The ESP32 can also be loaded with Virtual File System. It unifies the SPIFFS and FAT-Fs file system on the ESP32 and provides a file-like interface for external drivers. Virtual file system is useful when SPIFFS and FAT file systems are required on ESP32 for certain reasons.
Conclusion
Managing user and network data can be challenging with microcontrollers. ESP32 is no exception. ESP32 is a powerful Wi-Fi development platform for building IoT applications of all sizes. Fortunately, the ESP32's memory can be extended with external flash/EEPROM, SD cards, and eMMC. External flash chips or EEPROM are suitable when a small increase in storage capacity is needed. If your application requires the storage of large files, such as audio, video, or database files, you must use SD or eMMC cards.
With the appropriate selection of on-board or on-device storage, the next important step in data management on the ESP32 is to select a suitable file system. If an application only requires the storage of important key-value parameters that should not be lost during shutdown or power failure, the NVS library file system is sufficient. For example, IP-level parameters for connecting the ESP32 to an existing network can be stored in the NVS partition as key-value pairs in the NVS library file system. If the application requires storing audio logs, video logs, or small-sized data, SPIFFS is more suitable because it will not consume RAM. For applications that require storing large files, organizing files in a directory structure, or encrypting data, the FAT file system on the ESP32 is the choice. For some rare circumstances, both SPIFFS and FAT file system may be required on ESP32, after which ESP32 can be loaded with virtual file system to extract the benefits of FAT and SPIFFS file system features. Although the ESP32 supports storing audio and video files, the maximum data rate of 40 MHz may still be a restriction.
