Sign language is a system of communication through visual gestures. However, not everyone understands sign language. Therefore, in this project we will build a device that converts hand gestures into speech.
What is needed: flexible sensors, a Mini Player DF and a microcontroller that has analog input and supports serial communication through a universal asynchronous receiver/transmitter (UART) — we use ESP32.
The ESP32 offers multiple analog input channels, which accommodate enough flexible sensors to interact with a person's hands and fingers (to relay sign language gestures). However, for this project, we will use a single flexible sensor, which is enough to gesture for food, water or medicine.
You can add more flexible sensors to your device, including enough to fully convert American Sign Language to audible speech.
Required components
- ESP32 x1
- DF Mini Player x1
- MicroSD Card x1
- Speaker x1
- Flexible sensor x1
- 330Ω resistance x1
- Connecting wires
- Breadboard or Perfboard
Circuit Connections
DOIT ESP32 DevKit Version 1 is used for this project. The pin diagram of this ESP board is below.
DF Mini Player is used to play the audible sentences converted from sign language. It's a small music playback system that plays audio files from a microSD card. Supported file formats include MP3, WAV and WMA. The module has a microSD card slot and built-in amplifier and a 3.5mm audio jack for interfacing speakers or headphones. It has UART and USB interfaces to communicate with a microcontroller.
Here is the DF Mini Player pin diagram:
The module has the following pin configurations:
Connections:
- To connect the DF Mini Player to the ESP32, connect the Mini Player's VCC and GND pins to the ESP32's 3V3 and ground pins, respectively.
- Then connect the RX and TX pins of the DF Mini Player to the GPIO26 and GPIO27 pins of the ESP32.
- Then connect a speaker (<3W) to the DF Mini Player's SPK1 and SPK2 pins.
To play DF Mini Player, audio files must be loaded onto a MicroSD card. We will play three audio instructions from the device. These instructions can be converted from text to speech and downloaded as MP3 files from ttsmp3.com.
Audio instructions are attached as MP3 files below.
- The MP3 file for the “Please give me water” instruction as 1.MP3:
- The MP3 file for the “Please give me food” instruction as 2.MP3:
- The MP3 file for the instruction “Please give me medicine” as 3.MP3:
Load these files onto a MicroSD card and insert the card into the DF Mini Player slot.
A flexible sensor is used to detect hand gestures (sign language). This sensor is connected to a voltage divider circuit with analog input on GPIO36 pin.
The device circuit diagram:
The flexible sensor can be attached to a glove or attached to a person's fingers with tape so that it bends with the finger as they use gestures to communicate with sign language.
Required libraries
To build this project, you will need to install two libraries in the Arduino IDE – espSoftwareSerial and DFRobotDFMiniPlayer. The espSoftwareSerial is required to use the software serial on the ESP32. DFRobotDFMiniPlayer is used to work with DF Mini Player.
Arduino Sketch
How it works
The flexible sensor detects hand gestures as they communicate through sign language. When the sensor is bent, its resistance changes, so the voltage output of the voltage divider circuit also changes.
The analog output of the flexible sensor is detected by the ESP32 and used to determine the instruction/command. DF Mini Player plays MP3 files from MicroSD card based on instructions communicated via ESP32 over UART.
The code
The sketch begins by importing SoftwareSerial.h and DFRobotDFPlayerMini.h. Variables are declared for the DF Mini Player RX and TX pin connections, and a software serial object is instantiated.
Next, a DF Mini Player object is instantiated and the variables for connecting the analog input to the flex sensor are declared. A variable to store the analog value is defined and initialized to zero.
In the setup function, the baud rate for serial and software serial communication is set to 9600 bps. DF Mini Player starts up and its volume is set to 20.
In the loop function, the ESP32 reads the analog input from the flex sensor. According to the analog input value, the MP3 file of the detected gesture is played by calling the player.play method.
Results