Augmented Reality-based sensor data display

Augmented Reality and Mixed Reality are the future of human interaction with the real world. Augmented Reality is a technology that allows you to project live images, videos or media into the real-world environment along with the real objects visible through a camera or wearable holographic device. Technology allows you to interact with the real-world environment while manipulating it through computer-generated graphics and virtual interfaces. It is a way of modifying a computer's view of the real-world environment.

In this project, Augmented Reality is used to monitor sensor data provided through an IoT board. The IoT board used in the project is the Particle Photon. The Photon is installed in a remote location with multiple sensors (DHT 11 Temperature and Humidity Sensor, LDR Sensor, IR Proximity Sensor and IR Sensor) connected to it. The board is connected to the cloud server and sends sensor data over the air to an augmented reality application. The app projects sensor data while the Android phone's camera focuses on a target image.

Fig. 1: Image showing augmented reality app displaying sensor data

Particle Photon is an Arduino compatible IoT board. To write program code for any Photon, the developer needs to create an account on the Particle website and register the Photon board with his user account. The program code can then be written in the Web IDE on the Particle website and transferred to a registered IOT board over the Internet. If the selected Cluster, Photon here, is turned on and connected to the Particle cloud service, the code is written to the selected card over the air via an internet connection, and the card begins to operate according to the transferred code.

The augmented reality application used in the project is built using the Vuforia augmented reality software development kit and Unity Editor. The development platform allows you to compile a readily available database, upload target images, and create UI elements in the Unity Editor. The user interface can be easily linked to the downloaded database by dragging and dropping UI elements. The app can then link to the Photon card's data logging URL to capture real-time data. The platform allows you to compile the project into a cross-platform mobile application for use on any smartphone.

Required components –

IoT Board Circuit:

1. Photon Particle.

2. DHT11 sensor.

3. LDR sensor.

4. Any Wi-Fi connection.

5. Infrared sensor

6. Transistor BC547

7. 1K ohm resistor

8. 5mm LED

9. Proximity sensor.

Mobile side:

1. Any Android phone

Block diagram –

Fig. 2: Block diagram of particle photon-based augmented reality sensor data monitor

Circuit Connections –

In the project, the circuit is built by connecting several sensors to the Particle Photon. The circuit has the following components connected to the Photon in the manner mentioned below –

DHT-11 Temperature and Humidity Sensor – The DHT-11 sensor reads the ambient temperature and humidity and relays the data to the microcontroller as digital data. The data pin of the DHT11 temperature and humidity sensor is connected to the A0 pin of the Particle Photon, and the VCC and ground are connected to the common VCC and ground.

LDR sensor – The LDR is used to detect light intensity. The sensor is connected to Particle Photon pin A1. The sensor is connected to a potential divider circuit. The LDR provides an analog voltage, which is converted to a digital readout by the built-in ADC.

IR Sensor – IR sensor is used to detect any obstacle. The sensor is connected to pin A2 of the Particle Photon. The sensor is connected to a potential divider circuit. The IR sensor provides an analog voltage, which is converted into a digital reading by the built-in ADC.

IR Proximity Sensor – The IR proximity sensor detects the distance to an obstacle. The sensor module has three pins – VCC, Ground and Output. VCC and ground are connected to the common VCC and ground. The output pin of the sensor module is connected to the D0 pin of the Particle Photon. The sensor output is an analog voltage, which varies from 3.1 V at 10 cm to 0.4 V at 80 cm away.

Power supply – In the circuit, Photon and other ICs need a regulated DC of 5V for their functioning. An 18V battery is used as the primary power source. The battery power is regulated to 5V and 12V using 7805 and 7812 ICs. Pin 1 of both voltage regulator ICs is connected to the battery anode and pin 2 of both ICs is connected to ground. The respective voltage outputs are taken from pin 3 of the respective voltage regulator ICs. An LED along with a 10K Ω pull-up resistor is also connected between the common ground and the output pin to get a visual cue of power continuity.

How the circuit works –

Once the program code is transferred to Particle Photon, it starts operating accordingly. The Particle Photon must be connected to any internet hotspot via Wi-Fi. The Arduino-compatible code in the Photon reads data from the sensors. Four sensors connected to the board – DHT 11 temperature and humidity sensor, LDR sensor, IR sensor and IR proximity sensor.

The DHT11 Temperature and Humidity Sensor is a digital sensor with an integrated capacitive humidity sensor and a thermistor. It relays real-time temperature and humidity readings every 2 seconds. The sensor operates with a power supply of 3.5 to 5.5 V and can read temperatures between 0° C and 50° C and relative humidity between 20% and 95%. The DHT 11 Sensor sends data in digital format to a controller pin in a one-wire protocol, which must be implemented on the firmware side. First, the data pin is configured for input and a start signal is sent to it. The start signal comprises a LOW for 18 milliseconds followed by a HIGH for 20 to 40 microseconds followed by a LOW again for 80 microseconds and a HIGH for 80 microseconds. After sending the start signal, the pin is configured for digital output and the 40-bit data comprising the temperature and humidity reading is locked. The first two bytes of the 5-byte data are an integer part and a decimal part of the relative humidity reading, respectively. The third and fourth bytes are an integer and decimal part of the temperature reading, and the last is a checksum byte. The one-wire protocol is implemented in the firmware using an open source library available for Photon.

The other sensors emit an analog voltage on their respective controller pins. Analog voltage is read and digitized using an integrated ADC channel. The analogRead function is used to read the analog voltage at the controller pin.

Sensor values ​​are read and stored in different variables. The formatted strings are then passed to the Photon data logging URL using the particle library publish method. The library is automatically imported and compiled by the Particle Web IDE and does not need to be imported explicitly.

The augmented reality application is linked to the data logging URL. It fetches the data from the URL and displays it on the application UI buttons. The buttons are projected when the camera is focused on the target image.

Check out the Photon code to learn how the Arduino-compatible code reads data from sensors and publishes it to the linked data logging URL. Also, check out how the AR app is created using the Vuforia SDK and linked to the registered Photon board.

Programming guide –

The AR application was created using the Vuforia SDK and involves the use of the Unity Editor. Developing Augmented Reality applications using Vuforia SDK involves the following steps –

– Open the Vuforia Developer Portal and register an account.

– Verify the email ID and log in to the registered user account.

– Tap “Develop” in the navigation bar and click “Destination Manager”.

– Click “Add Database” and enter a unique database name in the pop-up window. Keep the “device” option selected and click “Create”

button.

– Click the database name in the database list on the updated page and click the “Add target” button. Select “Single Image” and browse to the target image on your computer. Now the target image will be loaded and shown in the target list.

– Click the “Download Database” button and select “Unity Editor” as the development platform in the pop-up window. Click “Download”

button. The database will be compiled and downloaded to your computer.

– Click “Downloads” in the navigation bar and download Vuforia SDK for Unity.

– Open the Unity Editor SDK. Create a new project and save it in a folder on your computer.

– In the Unity Editor SDK, “ARCamera” must be selected under “Hierarchy” in the left sidebar. Click on “Assets” -> “Import Package” ->

“Custom Package” and browse the Vuforia Package. Once the package is imported, follow again “Assets” -> “Import Package” -> “Custom

Package” and browse to the destination image. Similarly, import the previously downloaded database.

– Copy the license key and paste it into “App License Key” under “Vuforia Behavior” in the sidebar. Check “Load database ”

and “Enable” in “Database Load Behavior”.

– Select “Image Target” under “Hierarchy” in the left sidebar and select the database name under “Image Target Behavior” in the right sidebar. The target image will be displayed on the plane.

– Right click on the “Hierarchy” window and select “UI” -> “Screen”. Select “World Space” in “Render Mode” in the “Canvas” option in the

right sidebar and double-click “Screen” under Hierarchy” in the left sidebar.

– Under “Rect Transform” in the right sidebar, enter an appropriate position, rotation and scale values ​​and click “Add Component”.

Select the image.” Click on the target image from the database and set “Texture Mode” to “Sprite”. Click “Apply” and drag the image from

Database in “Project” to “Source Image” in “Canvas Renderer”. Change the color and transparency level.

. Right-click “Button” and click “UI” -> “Text” to add text to the button. Select the transformation, font, and paragraph properties as desired. Add more text to the button.

– Duplicate button by right-clicking on it in the “Hierarchy” window and selecting “Duplicate”. Create five more buttons and change their text.

– Right click on the “Hierarchy” window and click on “Create Empty”. Rename the folder to “Code Container” and add database scripts to

the folder.

– Drag and drop buttons and texts from “Image Target” in the “Hierarchy” window to the database in the right sidebar.

– Log in to the Particle Web IDE and click on the Dashboard icon and click on “Logs”. Click the registration URL icon. Copy and paste the URL into “Photon

Particle URL” under “Read Stream” in the right sidebar of the selected database.

– Compile the project to create the apk file.

The Photon code is as follows –

First, the DHT sensor library is imported. Particle Web IDE automatically adds the library. A constant is defined to denote the pin to which the DHT sensor 11 is connected, and a constant is defined to denote a variant of the DHT sensor. Variables are declared to contain the value of temperature, humidity, light intensity, motion and UV range. Variables are declared to indicate the pins that the sensors interface to and are set to digital LOW by default to make them input pins. An object of type DHT is declared.

The setup function is called when the pin connected to the infrared proximity sensor is declared as an input pin using the pinMode function. The DHT sensor is initialized by calling the Begin method on the DHT object. The setup function is executed only once at the beginning of the code.

The loop iterates infinitely. In the loop function, DHT sensor data is obtained using the getTempCelcius and getHumidity methods on the DHT object. LDR sensor data is obtained by calling the analogRead method and converted to light intensity using standard formulas. The IR sensor value is read and converted into a distance measurement. A custom pirSensor function is called to detect motion from the proximity sensor. Data from different sensors is stored in variables, grouped into suitable strings, and sent to the data logging URL using the publish method on the Particle object.

Note: The complete Photon code for AR-based sensor data display can be found in the code section.

You can find the link below to download the Android app rar file.

https://github.com/ EngenheirosGaragem/Software/bruto/ master/AR_Data.rar

 ###

 //Program to


 // based on;
 
// ome useful stuff:

 // This #include statement was automatically added by the Particle IDE.

 #include "Adafruit_DHT/Adafruit_DHT.h"


 // This #include statement was automatically added by the Particle IDE.


 //DHTparameters

 #define DHTPIN A0

 #define DHTTYPE DHT11


 //Variables

 int temperature;

 int humidity;

 int light;

 int motion;

 int ultraviolet;

 int t = 100; // team

 int light_sensor_pin = A1;

 int pir = D0;

 int pirState = LOW;

 int uv = A2;


 // DHT sensor

 DHT dht(DHTPIN, DHTTYPE);


 void setup {

 // Start DHT sensor

 pinMode(D7,OUTPUT);

 pinMode(pir,INPUT);

 dht.begin;

 }


 void loop {

 digitalWrite(D7,HIGH);

 // Humidity measurement

 temperature = dht.getTempCelcius;

 // Humidity measurement

 humidity = dht.getHumidity;

 // Light level measurement

 float light_measurement = analogRead(light_sensor_pin);

 light = (int)(light_measurement/4096*100);



 int uvValue=analogRead(uv);

 ultraviolet = (uvValue*100)/1023;

 // Publish data

 Particle.publish("temperature", String(temperature));// + " °C");
 
delay

 Particle.publish("humidity", String(humidity));// + "%");

 delay

 Particle.publish("light", String(light));// + "%");

 delay

 pirSensor; // motion sensor information

 delay

 Particle.publish("ultraviolet", String(ultraviolet));

 delay

 digitalWrite(D7,LOW);

 delay

 }


 void pirSensor {

 motion = digitalRead(pir); // read input value

 Particle.publish("motion", "1"); // or 1 - We only want to print on the output change, not state

 if(motion==false)

 Particle.publish("motion", "0"); // or 0 - Serial.println("Motion ended!");


 }