Bachelor Research Project: Enhancing Augmented Reality Tracking
|
|
The Bachelor Research Project is a selective 20 week research project at the TU Eindhoven where groups of 2 or 3 students perform a research project on an advanced topic.
I did this project under the supervision of professors Robert van Liere and Andrei Jalba where the goal is to combine optical and inertial tracking for enhanced pose estimation. Indeed, reducing tracking inaccuracies is of great importance in Augmented Reality. When there is an interaction in such an environment, the position and orientation of the objects must be tracked accurately in order to not inhibit the sense of immersion.
The issue addressed is about finding a stable position and orientation of an object in an open space. Optical tracking suffers from occlusion caused by motion blur or blocking. Inertial tracking suffers seriously from drift due to double integration of noise in measured acceleration. By using a series of filters and models on the raw sensor data, we can improve the reliability of the inertial tracking and use that data to supplement optical tracking in cases of occlusion. We propose combining optical tracking with inertial tracking in order to reduce the tracking inaccuracies and use fused data output a more stable position and orientation.
My focus is on the optical tracking using OpenCV and Aruco libraries. The images presented above show the detection of an Aruco marker, which compares every marker on the screen to a dictionary of markers (presented on the right). Using this optical tracking and sensors (work of my partner) on a 3D printed cube, we try to combine these two different systems in order to improve the reliability of estimating the pose to reduce latency, with each tracking system reducing the effect of the disadvantage presented by the other tracking system.
Abstract: In order to fully experience the world of virtual and augmented reality, reducing tracking inaccuracies from latency and noise is extremely important. Current tracking systems have a very high cost. We present an approach to a prototype that combines optical and inertial tracking systems to track and predict the pose of a 3D printed cube with low cost commodity parts.
The research paper written for this project is available here:
I did this project under the supervision of professors Robert van Liere and Andrei Jalba where the goal is to combine optical and inertial tracking for enhanced pose estimation. Indeed, reducing tracking inaccuracies is of great importance in Augmented Reality. When there is an interaction in such an environment, the position and orientation of the objects must be tracked accurately in order to not inhibit the sense of immersion.
The issue addressed is about finding a stable position and orientation of an object in an open space. Optical tracking suffers from occlusion caused by motion blur or blocking. Inertial tracking suffers seriously from drift due to double integration of noise in measured acceleration. By using a series of filters and models on the raw sensor data, we can improve the reliability of the inertial tracking and use that data to supplement optical tracking in cases of occlusion. We propose combining optical tracking with inertial tracking in order to reduce the tracking inaccuracies and use fused data output a more stable position and orientation.
My focus is on the optical tracking using OpenCV and Aruco libraries. The images presented above show the detection of an Aruco marker, which compares every marker on the screen to a dictionary of markers (presented on the right). Using this optical tracking and sensors (work of my partner) on a 3D printed cube, we try to combine these two different systems in order to improve the reliability of estimating the pose to reduce latency, with each tracking system reducing the effect of the disadvantage presented by the other tracking system.
Abstract: In order to fully experience the world of virtual and augmented reality, reducing tracking inaccuracies from latency and noise is extremely important. Current tracking systems have a very high cost. We present an approach to a prototype that combines optical and inertial tracking systems to track and predict the pose of a 3D printed cube with low cost commodity parts.
The research paper written for this project is available here:
OpenGL: Power Towers
Power Towers is 2D tower defense game created entirely with Java and using the OpenGL library (OpenGraphicsLibrary).
The goal of the game is to defend the land by putting the three different types of towers available and not let the enemy towers go through the entire (sand) path. By putting a tower, the player's cash is reduced. When a tower hits an enemy, the player's cash increases. Each of the available towers have different characteristics: The red "alien" is cheap, fast but does not damage a lot, the blue "alien" is more expensive, slower but damages more while the blue "ice" tower reduces the speed of the enemy making him slower and hence more vulnerable.
It is not possible to place towers on the water nor the sand, but only on the grass.
Some extremely interesting aspects of programming in this projects were:
The entire commented source code is available on github.
The goal of the game is to defend the land by putting the three different types of towers available and not let the enemy towers go through the entire (sand) path. By putting a tower, the player's cash is reduced. When a tower hits an enemy, the player's cash increases. Each of the available towers have different characteristics: The red "alien" is cheap, fast but does not damage a lot, the blue "alien" is more expensive, slower but damages more while the blue "ice" tower reduces the speed of the enemy making him slower and hence more vulnerable.
It is not possible to place towers on the water nor the sand, but only on the grass.
Some extremely interesting aspects of programming in this projects were:
- 2D Artificial Intelligence: Calculating the angles before shooting, making sure the tower follows the enemy, making sure the project hits the enemy in the center etc...
- OpenGL: It was the very first time I used OpenGL, and it was interesting to build the graphics using this library instead of the simple JFrame in Java
- Multi-threading: I used multi-threading for the music of the game
- Mathematics: The math behind putting the menu in the right for the towers required with the towers inside required calculations.
- Drag & Drop: The code behind clicking on a tower and putting it in the map (drag & drop)
The entire commented source code is available on github.
Computer Vision & Path Planning: Cyclone
|
|
|
Cyclone is an autonomous racing drone team of 12 students (6 new members joined since March 2018) of which I am the leader. This project is part of my Honors Program, where I follow the High Tech Systems track focusing on the development of drones. The image on the left represents the drone we are building from scratch, and on the right an example of a hula-hoop detection system I am working on.
On top of my leadership position, I am also working in the Computer Vision sub-team of the drone. My current task is to develop a path planning system for the autonomous drone and a hula-hoop detection algorithm.
This is done using C++ and OpenCV libraries for the software, and an NVIDIA Jetson TX2 for the hardware part.
This project is teaching me camera coding and giving me a practical knowledge of Computer Vision, a topic that has my interest for the past 2 years.
At the end of the academic year of 2017/2018, Cyclone has won the Jury award for the best Honors Program project out of more than 25 other projects! (Picture in the right in which I am holding the "Certificate of Excellence")
On top of my leadership position, I am also working in the Computer Vision sub-team of the drone. My current task is to develop a path planning system for the autonomous drone and a hula-hoop detection algorithm.
This is done using C++ and OpenCV libraries for the software, and an NVIDIA Jetson TX2 for the hardware part.
This project is teaching me camera coding and giving me a practical knowledge of Computer Vision, a topic that has my interest for the past 2 years.
At the end of the academic year of 2017/2018, Cyclone has won the Jury award for the best Honors Program project out of more than 25 other projects! (Picture in the right in which I am holding the "Certificate of Excellence")
Hackathon: The Resurrection of Cthulu
Play THE RESURRECTION OF CTULHU: https://ahmedahres.itch.io/the-resurrection-of-cthulhu
The Resurrection of Ctulhu is 2D Puzzle game developed with Unity by myself and two people from Aalto University (Helsinki, Finland) during Junction 2016, Europe's largest Hackathon. I flew to Helsinki for a week-end in November 2016 specifically for this hackathon.
I was mainly a Gameplay programmer taking care of the camera movement, core gameplay, and assets integration.
The theme of the game jam was RESURRECTION.
The goal of this puzzle game is to resurrect Ctulhu by giving him his 8 legs back. Throughout the game, different scenarios will happen with a story that will lead to the ending. It takes an average of 20 minutes to complete it.
Using the left mouse for the left hand and the right mouse for the right one along with A/W/S/D, the player is able to interact with different objects and needs to use them in the correct way to continue with the story.
The Resurrection of Ctulhu is 2D Puzzle game developed with Unity by myself and two people from Aalto University (Helsinki, Finland) during Junction 2016, Europe's largest Hackathon. I flew to Helsinki for a week-end in November 2016 specifically for this hackathon.
I was mainly a Gameplay programmer taking care of the camera movement, core gameplay, and assets integration.
The theme of the game jam was RESURRECTION.
The goal of this puzzle game is to resurrect Ctulhu by giving him his 8 legs back. Throughout the game, different scenarios will happen with a story that will lead to the ending. It takes an average of 20 minutes to complete it.
Using the left mouse for the left hand and the right mouse for the right one along with A/W/S/D, the player is able to interact with different objects and needs to use them in the correct way to continue with the story.
Android Application: Shapion
Play SHAPION: https://ahmedahres.itch.io/shapion-v01 (web version)
Link to Android Store: https://play.google.com/store/apps/details?id=com.Ahres.Shapion
Today, Shapion has a rating of 4.9/5 in the Google Play Store and has over 1000 installs worldwide.
Shapion is a 2D game developed in Unity and C# by myself and Jelle Schukken. We both worked as game designers and programmers. It took us a total of 4 months of work from developing to publishing.
In this game, the goal is to have the highest score possible. You can move the bird up and down using W and S or UP and DOWN arrows, and make the bird bigger using A and D or RIGHT and LEFT arrows.
Getting a coin gives the player 1 point, killing an enemy gives him 2 points. However, in order to kill an enemy, your bird needs to be bigger than him.
The initial idea was about having obstacles and change its shape (circle, triangle, square etc..) depending on the obstacle. This is where the name Shapion came from.
Then after careful consideration we have decided to go for this type instead: an endless 2D game in which the player controls the bird to get the highest score possible.
After releasing the first version, we had feedback about it being too hard and not very intuitive. We then made it easier, changed the controls to more intuitive ones, added animation to be more immersive and made it much smoother.
Here are some reviews I had from worldwide players and my friends:
- "Very cool concept! I like the art style and the gameplay mechanic!"
- "Amazing game, got me addicted already after 1 game!"
- "The ability to get bigger makes the game very innovative and changes everything"
- "Just like Flappy Bird, this game releases a lot of adrenaline and dopamine in one's body"
- "I like it! It's a bit difficult but once you get the hang of it, it's really fun."
- "Wonderful! I keep playing non-stop when taking breaks while studying"
Link to Android Store: https://play.google.com/store/apps/details?id=com.Ahres.Shapion
Today, Shapion has a rating of 4.9/5 in the Google Play Store and has over 1000 installs worldwide.
Shapion is a 2D game developed in Unity and C# by myself and Jelle Schukken. We both worked as game designers and programmers. It took us a total of 4 months of work from developing to publishing.
In this game, the goal is to have the highest score possible. You can move the bird up and down using W and S or UP and DOWN arrows, and make the bird bigger using A and D or RIGHT and LEFT arrows.
Getting a coin gives the player 1 point, killing an enemy gives him 2 points. However, in order to kill an enemy, your bird needs to be bigger than him.
The initial idea was about having obstacles and change its shape (circle, triangle, square etc..) depending on the obstacle. This is where the name Shapion came from.
Then after careful consideration we have decided to go for this type instead: an endless 2D game in which the player controls the bird to get the highest score possible.
After releasing the first version, we had feedback about it being too hard and not very intuitive. We then made it easier, changed the controls to more intuitive ones, added animation to be more immersive and made it much smoother.
Here are some reviews I had from worldwide players and my friends:
- "Very cool concept! I like the art style and the gameplay mechanic!"
- "Amazing game, got me addicted already after 1 game!"
- "The ability to get bigger makes the game very innovative and changes everything"
- "Just like Flappy Bird, this game releases a lot of adrenaline and dopamine in one's body"
- "I like it! It's a bit difficult but once you get the hang of it, it's really fun."
- "Wonderful! I keep playing non-stop when taking breaks while studying"
3D Animation: Space collision
|
Start of collision
|
During collision
|
After collision
|
Using my inspiration and some tutorials, I have created a 3D animation using Blender. This animation is 8-seconds long, in which two planets collide.
CASTLENOVA
The map inspired from the real architecture of Breda in 1589.
Of course, there is a way to locate yourself in the map with a letter G, standing for Gerard, the name of the character.
Of course, there is a way to locate yourself in the map with a letter G, standing for Gerard, the name of the character.
Castlenova is a 3D Infiltration/Stealth game developed using Unity and C# by myself and 4 other people.
My inspiration for our game came from Sid Meier's Pirates!, a game released in 1987 involving pirates as well as infiltration scenes in cities. After taking some time to choose what kind of game we wanted to create, my idea was chosen and it was the beginning of 8 weeks of developing our game.
Castlenova is a Stealth/Infiltration game, which means the player needs to get from a certain point A to a certain point B by using the objects in a smart way or by killing enemies and hiding the bodies.
Indeed, the player can choose to go pure infiltration, using the endless medieval houses, grass, trees in order to complete his mission.
On the other hand, the player can also kill the enemies silently from the back using SPACE. You will then see an animation of your character standing up and using the knife to kill the guard.
However, when killing an enemy, the body cannot stay seen. Hence, the body needs to be hidden by the player using the LEFT mouse.
In case a dead body is detected by other guards, they will go into ALERT mode.
When going to ALERT mode, the player will hear bells. It then means that each guard will have his movement speed increased as well as his sight, making the player's strategy to infiltrate harder than ever.
As I said previously, the objective is to go from a point A to a point B without getting noticed. Castlenova has a long story behind it.
The point A represents the entrance of the city of Breda, while the point B represents a castle conquered by the Spanish army.
Our game is inspired by true events that happened in 1589 in Breda, Netherlands. In order to create an original way of explaining the story to the player, we have decided to implement a fictional letter from general Charles de Heraugiere (a general that really existed) to Gerard, the fictional character controlled by the player.
The letter does not only explain the context and what the mission is, but also gives a clue to where the final objective (castle) is. You can see the letter in the 3rd image above.
We tried to foster the problem-solving and environment analysis as much as possible by offering endless buildings to hide the bodies behind and the character himself.
I have had the pleasure to lead this project from the very beginning and it taught me a lot in terms of managing a project as well as the technical aspect of a complex game.
It has some amazing scripts such as dragging bodies, killing enemies, waypoints for guards, list of selected music chosen by the creator and much more available in my GitHub profile: https://github.com/AhmedAhres
These scripts were written by myself and the two other members responsible for the technical aspect.
Play Castlenova on MAC: https://www.dropbox.com/sh/6rl2o9x9jam1jdm/AABaio07O3IsD2jIpvLielgga?dl=0
My inspiration for our game came from Sid Meier's Pirates!, a game released in 1987 involving pirates as well as infiltration scenes in cities. After taking some time to choose what kind of game we wanted to create, my idea was chosen and it was the beginning of 8 weeks of developing our game.
Castlenova is a Stealth/Infiltration game, which means the player needs to get from a certain point A to a certain point B by using the objects in a smart way or by killing enemies and hiding the bodies.
Indeed, the player can choose to go pure infiltration, using the endless medieval houses, grass, trees in order to complete his mission.
On the other hand, the player can also kill the enemies silently from the back using SPACE. You will then see an animation of your character standing up and using the knife to kill the guard.
However, when killing an enemy, the body cannot stay seen. Hence, the body needs to be hidden by the player using the LEFT mouse.
In case a dead body is detected by other guards, they will go into ALERT mode.
When going to ALERT mode, the player will hear bells. It then means that each guard will have his movement speed increased as well as his sight, making the player's strategy to infiltrate harder than ever.
As I said previously, the objective is to go from a point A to a point B without getting noticed. Castlenova has a long story behind it.
The point A represents the entrance of the city of Breda, while the point B represents a castle conquered by the Spanish army.
Our game is inspired by true events that happened in 1589 in Breda, Netherlands. In order to create an original way of explaining the story to the player, we have decided to implement a fictional letter from general Charles de Heraugiere (a general that really existed) to Gerard, the fictional character controlled by the player.
The letter does not only explain the context and what the mission is, but also gives a clue to where the final objective (castle) is. You can see the letter in the 3rd image above.
We tried to foster the problem-solving and environment analysis as much as possible by offering endless buildings to hide the bodies behind and the character himself.
I have had the pleasure to lead this project from the very beginning and it taught me a lot in terms of managing a project as well as the technical aspect of a complex game.
It has some amazing scripts such as dragging bodies, killing enemies, waypoints for guards, list of selected music chosen by the creator and much more available in my GitHub profile: https://github.com/AhmedAhres
These scripts were written by myself and the two other members responsible for the technical aspect.
Play Castlenova on MAC: https://www.dropbox.com/sh/6rl2o9x9jam1jdm/AABaio07O3IsD2jIpvLielgga?dl=0
EVOPOINT
|
|
This is EVOPOINT, my first game created using Unity3D game engine with JavaScript and C#.
The purpose is quite simple:
All you need to do is control the ball using A/W/S/D keys and try to get the coins while avoid the enemies.
However, the tricky part is the following: You need to get all the coins in every level to go to the next one. Missing one and not being able to get it back results in a reset to the first level. It goes without saying that, otherwise, this would be extremely easy.
The scripts are available in my GitHub profile: https://github.com/AhmedAhres
The purpose is quite simple:
All you need to do is control the ball using A/W/S/D keys and try to get the coins while avoid the enemies.
However, the tricky part is the following: You need to get all the coins in every level to go to the next one. Missing one and not being able to get it back results in a reset to the first level. It goes without saying that, otherwise, this would be extremely easy.
The scripts are available in my GitHub profile: https://github.com/AhmedAhres
COMPETITION WINNER: A SORTING MACHINE
This constitutes a huge achievement. Indeed, among exactly 24 groups of 6 people each, my group was one of the 4 chosen for the competition after a lot of hard work. We had the opportunity to present the final machine in front of a jury made by high ranked people from the Computer Science department on the 1st of April 2016. We made it to the first place.
As you may have read in the title, this is a sorting machine. The black tube contains at most 12 disks that leave the tube to be pushed by a linear actuator. Then, a color detector detects the current processed disk's color which goes to a conveyor belt and, depending on the result, is sorted to its respective tray. There are only white and black disks.
The requirement was to sort 12 disks in 5 minutes, which means 25 seconds per disk.
Our machine sorts 12 disks in 5.9 seconds, which means more than 2 disks per second. This is one of the highest record ever achieved by students at the Eindhoven University of Technology.
The software design was made in Java, and the implementation in C using an Arduino. A screen displays the colour of each processed disk, the number of sorted disks as well as errors detected.
For instance, if a black disk goes to the white tray, the machine stops and the screen displays the error. This is only one example out of many others.
One of the great features that our machine has is that it can run on DC and on battery as well.
The project is supposed to be done using a PP2 processor. However, being one of the students that completed all the assignments in 2IC30 Computer Systems using Assembly, I had the opportunity to choose between an Arduino and a PP2 with my team. There were no documentation to help us on the Arduino from the University since it is the first time it is used in this project. Hence, we had to independently learn coding pertaining to the Arduino implementation, which was a challenge for us.
This project taught me a lot about software documentation and development.
After the result, I have been personally congratulated for my presentation skills and scored the perfect score in every criteria.
Video of the machine: https://www.youtube.com/watch?v=sdNZMEfJHxM&feature=youtu.be
As you may have read in the title, this is a sorting machine. The black tube contains at most 12 disks that leave the tube to be pushed by a linear actuator. Then, a color detector detects the current processed disk's color which goes to a conveyor belt and, depending on the result, is sorted to its respective tray. There are only white and black disks.
The requirement was to sort 12 disks in 5 minutes, which means 25 seconds per disk.
Our machine sorts 12 disks in 5.9 seconds, which means more than 2 disks per second. This is one of the highest record ever achieved by students at the Eindhoven University of Technology.
The software design was made in Java, and the implementation in C using an Arduino. A screen displays the colour of each processed disk, the number of sorted disks as well as errors detected.
For instance, if a black disk goes to the white tray, the machine stops and the screen displays the error. This is only one example out of many others.
One of the great features that our machine has is that it can run on DC and on battery as well.
The project is supposed to be done using a PP2 processor. However, being one of the students that completed all the assignments in 2IC30 Computer Systems using Assembly, I had the opportunity to choose between an Arduino and a PP2 with my team. There were no documentation to help us on the Arduino from the University since it is the first time it is used in this project. Hence, we had to independently learn coding pertaining to the Arduino implementation, which was a challenge for us.
This project taught me a lot about software documentation and development.
After the result, I have been personally congratulated for my presentation skills and scored the perfect score in every criteria.
Video of the machine: https://www.youtube.com/watch?v=sdNZMEfJHxM&feature=youtu.be
WAVE
|
|
The main menu
|
This game is called WAVE. It has been written entirely in Java.
The player is supposed to control the white player and needs to avoid the enemies (without being able to eliminate them) using the arrows in the keyboard. It is of course possible to pause the game by pressing P.
The red enemy is a BASIC enemy, he just moves randomly and reduces your health whenever there is a collision between your player and him. The cyan is a FAST enemy, which implies that he has greater speed than the red one. The yellow one is the HARD enemy, and creates more damage than the others. The best one is the green one since that is the SMART enemy. Indeed, this is one example of Artificial Intelligence created through Java. The SMART enemy does not move randomly, but keeps following you by calculating the distance between you and him using the coordinates (x,y) and tries to reduce it at any time. Hence, it is not possible to camp in this game.
Last but not least, the big red enemy is the BOSS. He appears at a quite high level of the game and starts sending tons of bullets until your player dies.
The score increases with the time in which you stay alive (until your health goes down). Each time the score increases by 300, which takes 5 seconds, the level goes up and new enemies spawn. Hence, from 0 to 300, it is level 1. From 300 to 600 it is level 2, etc...
You may have noticed that in the down-right photo, there are two difficulties. Indeed, they differ quite a bit.
For instance, SMART and HARD enemies as well as the BOSS appear at a lower level in the hard mode to make you survive less time, there are no BASIC enemies in hard mode etc..
This has been an amazing project since it taught some of the basics of Game Programming:
It is possible to play Wave. The whole code of this game is available in my GitHub profile: https://github.com/AhmedAhres
The player is supposed to control the white player and needs to avoid the enemies (without being able to eliminate them) using the arrows in the keyboard. It is of course possible to pause the game by pressing P.
The red enemy is a BASIC enemy, he just moves randomly and reduces your health whenever there is a collision between your player and him. The cyan is a FAST enemy, which implies that he has greater speed than the red one. The yellow one is the HARD enemy, and creates more damage than the others. The best one is the green one since that is the SMART enemy. Indeed, this is one example of Artificial Intelligence created through Java. The SMART enemy does not move randomly, but keeps following you by calculating the distance between you and him using the coordinates (x,y) and tries to reduce it at any time. Hence, it is not possible to camp in this game.
Last but not least, the big red enemy is the BOSS. He appears at a quite high level of the game and starts sending tons of bullets until your player dies.
The score increases with the time in which you stay alive (until your health goes down). Each time the score increases by 300, which takes 5 seconds, the level goes up and new enemies spawn. Hence, from 0 to 300, it is level 1. From 300 to 600 it is level 2, etc...
You may have noticed that in the down-right photo, there are two difficulties. Indeed, they differ quite a bit.
For instance, SMART and HARD enemies as well as the BOSS appear at a lower level in the hard mode to make you survive less time, there are no BASIC enemies in hard mode etc..
This has been an amazing project since it taught some of the basics of Game Programming:
- How to use GUI efficiently
- How to relate different classes and use inheritance as well as Enum in order to create the enemies
- How Artificial Intelligence works in 2D games
It is possible to play Wave. The whole code of this game is available in my GitHub profile: https://github.com/AhmedAhres
SPACE SHOOTER 2D
This is SpaceShooter2D, a 2D game that I built using Unity2D and C#.
It is a simple game that I challenged myself to finish in the shortest time possible in order to test my skills.
The player controls the red ship with A/W/S/D and shoots using SPACE.
It is also possible to pause using ESC.
The goal is simply to survive. The score increases with the time you are alive in. I have made it such that the enemies (blue ships) will spawn increasingly over time, which keeps the game quite challenging. A score of about 6000 is descent. Moreover, if the player shoots an enemy bullet, then both bullets will be canceled. Each time the player's ship is damaged, a small animation will appear and the ship becomes invincible for about 0.5 seconds, which will give the player a very short time period to reposition himself.
The scripts are available in my GitHub profile: https://github.com/AhmedAhres
It is a simple game that I challenged myself to finish in the shortest time possible in order to test my skills.
The player controls the red ship with A/W/S/D and shoots using SPACE.
It is also possible to pause using ESC.
The goal is simply to survive. The score increases with the time you are alive in. I have made it such that the enemies (blue ships) will spawn increasingly over time, which keeps the game quite challenging. A score of about 6000 is descent. Moreover, if the player shoots an enemy bullet, then both bullets will be canceled. Each time the player's ship is damaged, a small animation will appear and the ship becomes invincible for about 0.5 seconds, which will give the player a very short time period to reposition himself.
The scripts are available in my GitHub profile: https://github.com/AhmedAhres