![]() This is the first blog post I write in a few months, I missed it! I have recently completed a project involving satellite image classification and it has inspired me to write a post. Today, I will be talking about convolutional neural networks which have gained a lot of attention especially for computer vision and image classification. They are highly proficient in areas like identification of objects, faces, and traffic signs apart from generating vision in self-driving cars and robots. All the advanced vision technologies you see out there (robots, machines, self-driving cars to name a few) are probably using these types of neural networks to achieve their goals. Guess what? You will actually learn how it works and have a rough idea about what is going on. I will try to explain it assuming that you have just a rough idea of what machine and deep learning are. OverviewAs stated earlier, Convolutional Neural Networks (CNN) represent a type of neural networks. A neural network is simply a "mechanism" vaguely inspired by the biological neural networks that constitute animal brains. The neural network itself is not an algorithm, but rather a framework for many different machine learning algorithms to work together and process complex data inputs. Such systems "learn" to perform tasks by considering examples, generally without being programmed with any task-specific rules. In our case, the goal of CNN would be to use images to either detect specific objects, extract relevant information (such as face recognition), perform image segmentation and so on. The images below show object detection and image segmentation using CNN, where one can notice how accurate it is. Example of object detection. Source: https://arxiv.org/pdf/1506.01497v3.pdf Separating roads from the rest in satellite images. Source: my Machine Learning project Before diving into the how the CNN actually works (intuitively without detailing the mathematics), we first need to understand how an image is represented in the computer, how this neural network sees an input image. Image RepresentationWhen a computer takes an image as input, it will see an array of pixel values. For example, let's say we have an image in JPG form with size 480 x 480. The representative array will be 480 x 480 x 3 (3 refers to the RGB values). Each of these numbers is given a value from 0 to 255 which describes the pixel intensity at that point (assuming it is an 8-bit image which is the standard). These numbers, while meaningless to us when we perform image classification, are the only inputs available to the computer. Example of a 2D array given to the computer. Source: https://www.slideshare.net Now that we know how a computer sees an image, we want to extract information from it. As human beings, seeing which is a dog and which is a cat is a fairly easy task, however for a computer it really is more complicated than using only numbers as inputs. What this neural network will do is take the image, pass it through a series of convolutional, nonlinear, pooling and fully connected layers to get a certain output. The output can be a single class or a probability of classes that best describes the image (example of classes can be human, dog, cat, horse etc). In the example below, we see that the neural network predicts that this image represents a cat with a probability of 82% which is quite accurate. However, you will see that using it will start by being not accurate and will "learn" how to correctly predict an image. The final goal would be to predict it with 99-100% accuracy. Example of how a computer sees an image and trying to label it. Source: http://cs231n.github.io/classification/ Architecture Details of CNNNeural networks are made of layers which contain neurons. The particular architecture of the layers constitutes a very important concept because different types of layers yield different learning types. In this section, I will describe the basic architecture used by CNN. Of course, a number of optimization techniques can be applied, but we will stick to the basics. Extracting the features - Convolution LayerConvolution is one of the main building blocks of a CNN. The term convolution refers to the mathematical combination of two functions to produce a third function. It is highly used in the field of signal processing. In the case of a CNN, the convolution is performed on the input data with the use of a filter (also called kernel) to then produce a certain result called feature map. We execute a convolution by sliding the filter over the input. At every location, a matrix multiplication is performed and sums the result onto the feature map. In the animation below, you can see the convolution operation. You can see the filter (the yellow square) is sliding over our input (the green square) and the sum of the convolution goes into the feature map (the red square). Note that the filter shown below is of dimensions 3x3 (but you can use any dimension you want, however 3x3 is a popular choice) Convolution in images As you may notice, the pixel value in a certain location of the feature map only depends on its local neighbours. This is actually one key component of CNN. By using only the neighboring values, then it can analyze a particular object in image independently of where it is in the image. For example, if we shift our input image then the neighboring elements will also be shifted, making the analysis shift-invariant. Introducing non-linearity: ReLU Activation FunctionThe Rectified Linear Unit (ReLU) is a non-linear function used after the convolution layer. It is extremely important to remember that this function is not linear. Since most of the problems we deal with practically are not linear (otherwise life would be so easy), then by applying a non-linear function we generalise our problem to non-linear problems as well. The ReLU function is shown below: This function acts as an element wise operation (applied per pixel) and replaces all negative pixel values in the feature map by zero. Reducing computation while maintaining what is important: PoolingIf we wish to reduce the dimensionality of individual feature maps and yet keep the crucial information, we use the Pooling layer after the Convolution layer. Different types of Pooling exist: Average, Sum, Maximum, etc. For Max Pooling, we first specify a spatial neighborhood (such as a 2×2 window) and then pick out the largest element of that feature rectified map within that window. Instead of largest element, if we pick the Average one it’s called Average Pooling and when the summation of elements in the window is taken, we call it Sum Pooling. Summing up what we have seenSo far I have shown you the architecture of the CNN and the various operations used to compute the result, without explicitly explaining how the result is computed leading to the following question: How does it compute the result?As I stated earlier, neural networks are made of layers, in which we have nodes representing neurons (similar to the brain). Between those layers exist edges (some sort of connections, again similar to the brain) that have weights or filter values. Those filter values need to be updated each time to produce better results, using what we call backpropagation. Before the CNN starts, the weights or filter values placed in the connections are randomly initialized. Let's assume we have a training set that has thousands of images of dogs, and cats and each of the images has a label of what animal that picture is. The final goal is to have those weights be numbers in such a way that the label given in the training set matches the label. An overview of the steps that the neural network takes are: 1. Initialize all filters and parameters / weights with random values (as discussed above). 2. The network takes a training image as input, goes through the forward propagation step (convolution, ReLU and pooling operations along with forward propagation in the Fully Connected layer) and finds the output probabilities for each class.
4. Use Backpropagation to calculate the gradients of the error with respect to all weights in the network and use (stochastic) gradient descent to update all filter values / weights and parameter values to minimize the output error.
SummaryIn summary, CNN is a type of neural network that is based on the idea of convolution. By using a convolution layer along with other types of layers, it can learn how to perform object segmentation or detection. However, CNN does not stop at images! It is also widely used in natural language processing.
If you are interested in creating your own convolutional neural networks, I highly recommend the Keras library. It is a very intuitive Python library that uses TensorFlow and can help you learn neural networks in a practical way.
4 Comments
![]() I love reading books. In fact, one of my 2017 resolutions was to read at least one book per month, which meant a minimum of 12 books a year (which is a success until now). I am extremely passionate about self-development books, they have a unique style to inspire me and probably a lot of people in the world. I started these books at the age of 13. Since then, I have never stopped being in love with them. Today, I will be giving some of the lessons I learned in the past years. But first, I want to discuss the some benefits of reading for pleasure. When reading online about tips on how to be smarter, guess what pops instantly? Reading books. In this sense, I have been doing my research on answering why that is the case, and here are some interesting facts: Reading improves focus and concentration Unfortunately, nowadays it is extremely easy to get distracted. We have to keep an eye on so many different things, which eventually leads us to losing focus. Reading is one way to improve it. “The moment I find it hard to concentrate, I just grab the book I’m currently reading”, says Paul Ashton, a manager from Bestessaytips. “It’s a wonderful tool, whenever I lose focus.” When we need that concentration right away and it just doesn’t come that easy, reading is the ultimate solution. The way reading grabs our attention is a cure for our ability to focus and our concentration skills. Reading enhances memory When reading a book, one needs to remember an assortment of characters, their backgrounds, ambitions, histories, and nuances, as well as the various arcs and sub-plots that weave their way through every story. As a result, every new memory created forges new synapses (brain pathways) and strengthens existing ones, which assists in short-term memory recall. Reading is a brain-stimulating activity and it has been proven to stop cognitive decline (such as preventing dementia and the Alzheimer). And it influences our brain while rapidly improving our memory. Reading has an amazing positive effect on analytical skills Have you ever read an amazing mystery novel and solved the mystery yourself before finishing the book? If so, you were able to put your critical thinking skills to work by taking note of all the details provided and sorting them out to infer things by yourself. That same ability to analyze details comes in handy when it comes to critiquing the plot, determining whether it was a well-written piece, if the characters were properly developed, if the storyline ran smoothly, etc. Should you ever have an opportunity to discuss the book with others, you’ll be able to state your opinions clearly, as you’ve taken the time to really consider all the aspects involved. Even in real life, critical thinking skills are essential. Being able to solve problems given certain elements are a part of daily life – from finishing a project for work to figuring out how to navigate difficult relationships, possessing critical thinking skills are necessary in all walks of life. Reading expands your vocabulary This benefit seems trivial (and it is), but we tend to forget how important this is for the health of our brain. The more you read, the more words you gain exposure to, and they’ll inevitably make their way into your everyday vocabulary. The fact that you articulate well, speak with a good vocabulary gives more you more confidence as you speak, and that can only be beneficial. Moreover, this could be helpful when learning new languages. Indeed, if you read a book in a language that you are trying to learn, this will teach you new vocabulary, therefore making you better at speaking that language. ----------------------------------------------------------------------------------------------------------------------------------- Back to specific books, the first one I read in terms of self-development , which is still until now my favorite one, is called: How to Win Friends and Influence People by Dale Carnegie (originally published in 1936). This book is simply a master piece that teaches you everything you need to know about social skills. It taught me so much, and I use its lessons every single day of my life. (What is funny is that the first time I looked at the title of the book, I laughed and said "Come on, I have friends, I can make friends". This reaction is very common among people. A close family member then told me that this is the best book I'll ever read, and he could not be more right.) Lessons from How to Win Friends and Influence PeopleDo not condemn, and do not complain!
This is a very powerful message that the author starts the book with. Dale Carnegie tells us that, instead of condemning people, we should be understanding them. We should understand what motivated them to do a certain action. This is much better than criticising and condemning. He states: "God himself does not give a judgement to people before death. How dare we?" We have all been in a situation where we either criticised or judging someone on his acts, then realising that we were all wrong. "Any fool can criticize, condemn, and complain — and most fools do," Carnegie wrote. "But it takes character and self-control to be understanding and forgiving." Praise others in an honest and sincere way We should stop thinking about ourselves, our desires, what we want and what we need. Let's start thinking about the others; their desires, what they want and what they need in a very sincere and helpful way. "Abilities wither under criticism; they blossom under encouragement," Carnegie wrote. Be lavish with praise, but only in a genuine way, he advised. "Remember, we all crave appreciation and recognition, and will do almost anything to get it," he said. "But nobody wants insincerity. Nobody wants flattery." Be interested in others, encourage them to talk about themselves Think about those cute little dogs that come run to you when you go to guest's house. Those dogs did not read any single book, nor practised any of these principles I am describing, but somehow they make you feel welcomed and loved. If you want others to feel the same way around you, do like those dogs. Forget yourself, think about others because you will make more friends in two months being interested in others than in two years talking only about yourself. Listening closely to someone "is one of the highest compliments we can pay anyone," Carnegie wrote. Acknowledge your own mistakes When we know we are right, we should share it in a respectful and humble way. However, when we are wrong, we should acknowledge our mistakes. Not only this will show better results for the relationship in question, but it is also much more entertaining then trying to justify ourselves. "Admitting one's own mistakes — even when one hasn't corrected them — can help convince somebody to change his behavior," Carnegie wrote. Do not try to "win" an argument Even if you manage to tear apart someone else's argument, you don't actually achieve anything. Carnegie cited an old saying: "A man convinced against his will/Is of the same opinion still." If you're looking to actually persuade somebody, avoid an argument in the first place, he said. ----------------------------------------------------------------------------------------------------------------------------------- I only gave a few lessons, the book itself shows exactly 30 lessons, or "principles" and I strongly advise everyone to have a look at them at least once in a lifetime. I hope this motivates you to read the book because, as I said, it is a real master piece! ![]() Whenever you try to find some of the best life tips that have instant benefits, exercising is one of those tips that usually comes at the top. Now, people tend to think that others exercise to have a "body summer" or to just "look good". However, exercising has by far greater importance than just the esthetics. Most importantly, exercising has an amazing benefit on your mental health. In this article, I will be highlighting the most important benefits that exercising has on your brain. Exercising helps the control of emotionsThe ability to control our emotions is a very important skill of cognitive control. Whenever you reign in an outburst of anger, or continue your day despite feelings of sadness, you are exercising emotional regulation which may or may not transform your good day into a bad one. Scientists wanted to track changes in self-regulation. So they conducted a 2 month long study where participants are exposed to a program of regular physical exercise. The researchers noted a number of positive changes in behavioral patterns. Among these was a decrease in emotional stress and an increase in emotional control. If you have a tendency to blow up at people or lose your calm, exercise can help you keep centered. This can make a huge difference in the daily life. Exercising sharpens long-term memoryThere is a strong link between regular exercising and improved memory. One study found that 35 minutes of interval exercise on a bike strengthened long term memory. The timing of the exercise was important. Better memory was found for subjects who exercised four hours after learning. No benefit was seen for those exercising immediately after learning. Interesting right? Another study split participants up into three groups. Each group was told to recall as much information as possible from two paragraphs. The first group received the information after exercise, the second before exercise, and the last completed no exercise. They found that the group that was exposed to exercise before being given the information performed significantly better at recall than the other two groups. Exercising reduces stress and anxietyThis benefit is a well known psychological benefits and one of the biggest reasons why people start exercising. The science behind it is well documented, as well as it’s calming effect on a stressed mind. First of all, exercising releases neuro-chemicals into the brain. The most important ones being endorphins, dopamine, and norepinephrine. These chemicals are associated with better cognitive functioning, alertness and elevated moods. In addition to dumping feel good chemicals into your head, it also helps purge stress hormones from your body – cortisol and adrenaline . From a psychological perspective, exercising gives a way to distract yourself from focusing on daily stressors. This could be from teachers, bosses, romantic partner (hopefully not) or any number of personal problems. When the mind has nothing else to focus on, it will drift. Many people can fixate on immediate issues, specific stressful problems, or strong emotional feelings. So exercise can simply give you an immediate task to focus your energy on. So while this benefit of exercise is not a surprise, it is still one of the best, time tested reasons to get out there and get moving. exercise is an efficient stress management technique. Exercising is a very powerful weapon against depressionDepression is one of the most common mental conditions that affects people worldwide. It is also one of the biggest (direct or indirect) causes of suicide worldwide. The National Institute of Mental Health (NIMH) estimates that 16.2 million U.S. adults had at least one major depressive episode in 2016. This represents 6.7 percent of the U.S. adult population. A large meta-analysis analyzed the effect of exercise on alleviating symptoms of depression. Two things were found from the review. First, positive results from a significant and moderate relief from depression. The second result came from the comparison of exercise to other forms of psychological therapy or drugs. Exercise was found to be just as effective as the other alternatives. I am pretty sure that for those taking pills everyday, it is time to review their decisions because exercising can be a much cheaper and equally effective way to treat their problems. Exercising improves self-esteemIn psychology, the term self-esteem is used to describe a person's overall sense of self-worth or personal value. In other words, how much you appreciate and like yourself. Self esteem is important if we want to live a happy life. Low self esteem creates stress, depression, and anxiety. Also, it can negatively impact our job and academic performance. If gone unchecked it can also cause a number of other unhealthy behaviors. Exercise has been shown to affect self-esteem positively in all ages. From your development as a child up until the twilight years. A large quantitative review of 113 studies reported finding a change in self-esteem through exercise. In fact, the more physically fit one was, the higher one’s self esteem. One final result reported was that the type of exercise program could also affect the level of self-esteem. Exercising improves sleep qualitySleep is one of the major areas of efficient brain functioning. It has a direct effect on memory, self regulation, and cognition. One study looked at adults that had numerous complaints about lack of quality sleep. The study examined several factors. Gender, age, and physical function were observed over a period of 12 months. The results found that adults that were the least active benefited most from exercise when it came to quality of sleep. Exercise can do more than help those with insomnia or sleep disturbances. It’s good for normal healthy adults and adolescents as well. A meta-analysis found that total sleep time and fewer disturbances during the night were positive side effects of acute bouts of exercise. It’s important to note, the relationship between sleep and exercise has had some varying results. A lot of the variation has occurred when subjects have known sleep issues such as insomnia. Typically, exercise in the long term has beneficial results. Certain variables like the type of exercise, duration, and intensity are still being explored to find the cause of certain types of research results. Exercising increases productivityExercise impacts more than your academic or personal life. It can also have a positive effect on your professional life as well.
Over 200 white collar workers from 3 organizations were used to study exercise in the workplace. Researchers found that people reported several positive effects in the workplace from working out during their break or if they exercised before work. The benefits included better time management, better mood, and increased employee tolerance. On days were employees didn’t exercise, the benefits were not seen. Another study from Bringham Young University looked at different factors that contributed to the loss of productivity in the workplace in almost 20,000 employees. Researchers found that people who exercised only occasionally or not at all were more likely to report lower productivity than employees who exercised regularly. ---------------------------------------------------------------------------------------------------------------------------------- That's it for my article! Of course, exercising has much more benefits, but I chose to share this with you. Personally, I try to exercise at least 4 times a week, and that has amazing results. I strongly recommend everyone to do the same. This is my first blog post in 6 months, honestly I was extremely busy with graduate school applications. However, I plan to be back in writing blog posts more frequently. Today, I will be talking about how to detect a hula-hoop using a camera and estimate its relative position from the camera. This method I will be talking about is currently being developed in the Cyclone project in which I work on an autonomous racing drone using C++ and OpenCV. The detection of a hula-hoop can be very beneficial for instance when developing a drone that tries to go within the hoop or avoids it when racing. The method presented here is not optimal, but is good practice for beginners in the computer vision field. This method assumes that the camera used is properly calibrated, that the hoop is filled with red LEDs, and that the camera is facing the center of the hoop. Color thresholding First of all, the image (frame from the live video) is converted into HSV (Hue, Saturation, Value) color space, which is more useful than the BGR image that comes straight out of the camera, as in this representation the intensity information is separated from color, which makes thresholding much easier. This in turn reduces the effect of light intensity changes while tracking the target [2]. The LEDs that need to be detected have a consistently high luminosity, so by setting a relatively narrow threshold at a high intensity level, already most of the unwanted pixels can be discarded. Empirically and through research, the following values are excellent ranges for the red color in HSV: H = [45,255] S = [170,245] V = [170,255] This thresholding will allow to detect only the red points in the frame, with the following results: As you can see, the red LEDs are extracted (we discard the green color in our example). Contouring In order to be able to apply circle fitting, the edges of the detected image need to be extracted, which is done by using a Canny edge extraction. This canny edge extraction is a built-in function inside OpenCV. Ellipse fitting Finally, an ellipse is fit over all the coordinate points remaining in the array which describes the position of the hula-hoop on the image. This is done using the function fitEllipse from OpenCV. This function returns the height of the detected ellipse as well as the width and the angle ω of tilt of the major axis. From this, we can extract k, which is the length of minor axis (b) divided by the length of major axis (a). In theory, these are the values extracted from the function: The following figure shows the result of the detection on our hoop: Great, now we have detected the ellipse as needed and have some important values such as the angle and k. Now, in order to get the position in cartesian coordinates, it becomes a little tricky. Here, we assume that the origin is the center of the hoop, x is the horizontal axis (left and right from the center), y is the vertical axis (up and down from the center) and z is the axis from the camera facing the center of the hoop. We wish to determine the relative position (x,y and z) of the hoop with respect to the camera, where the origin is the center of the hoop. In order to do so, we need to determine the distance from the camera to the hoop as well as the vertical and horizontal angle of the hoop. Indeed, we need the following values: α2: Angle of rotation about the camera frame's vertical axis between the disc axis and the horizontal direction of view of the camera (0 = no rotation; 90 degrees = viewed left edge-on, disc facing right). This is the horizontal angle. β2: Angle of rotation about the camera frame's horizontal axis ('lie back') between the disc axis and the real world horizontal plane (0 = no rotation, disc is vertical; 90 degrees = disc is flat, facing up). This is the vertical angle. D: Distance from the camera to the center of the hoop. Determining the distance First, the measurements of the hoop and the focal length of the camera lens have to be performed. The focal length, in turn can be determined with the following equation: F = PD/R where F is the focal length, R is radius of the hoop in meters, D is the distance at which the reference shot is taken and P is the representation of the radius of the hoop in pixels in a captured frame. In order to determine the distance to the object, the semi-major axis of the ellipse in the captured photo frame is compared to the ellipse (circle) in the reference frame that represents the perceived hoop at a distance of 1 meter with a perpendicular view. The distance can then be determined using the focal length F calculated with the following equation: d = FR/pmajor where pmajor is the semi-major axis of the perceived ellipse. Determining the angles The following source [1] suggests that, using k and ω, we can comput the following horizontal and vertical angles: The math behind this is a bit complicated so I will avoid going into the details.
Determining the position Using the computed values, we can now compute the values of x, y and z: x = sin(β2) * cos(α2) * d y = sin(α2) * d z = cos(α2) * d And that’s it! If the assumptions stated at the beginning of this post hold, then the output results should be correct. A small trick: This method proved to be inaccurate sometimes when looking at the hoop from the center (where x and y should be nearly 0). This is due to the fact that the tilted angle returns inaccurate result because the detection does not know which axis is the major and which axis is the minor. Hence, whenever k (remember k is the minor axis divided by the major axis) is nearly 1 then we can assume that the ellipse is a circle, and thus that we are looking at it from the center. As a result, we can then assume that x and y have a value of 0. This is not the best way to fix this, but it works for most of the cases. [1] http://web.ncf.ca/aa456/scale/ellipse.html ![]() I think this is the fifth blog post I talk about VR. I just can't help it. It is just such a passionate subject. I was walking in MediaMarkt store here in Eindhoven, and saw all the new virtual reality headsets being sold, so it inspired me to write this blog post. The thing is, to quote Jay Borenstein from Stanford CS department: “If you believe that VR is coming… that it’s going to be transformative, that it’s going to be ubiquitous, and very significant for the way that society interacts with one another, then it’s exciting to be at the very early stages.” And yes, I believe in this, I am more than excited to be at the very early stages. As you may all know, Facebook, Google, Sony and a lot of big companies are working on it. What you might not know is that there are a lot of small start-ups as well, so many VR headsets that are not as famous as the Oculus Rift or the HTC Vive. I talk more about this in my post about the Game Developers Conference 2017 that happened in San Francisco, California. Today, I will not be talking about the technology behind VR, but rather how to get into that world. Why don't I apply all these strategies? I am trying my best to do so, by working on VR with a PhD student, writing these posts and making my own research on this subject, but I am also extremely busy with studies and graduate applications as well as sports. Science fiction writers and futurists dreamt up VR decades ago, and hackers have been attempting to build it ever since. Today, the technology is rapidly advancing with the promise to shift modern computing into a new paradigm, not unlike smartphones did one decade ago. Some people are skeptical about whether VR will stick around, because the technology is still very much plagued by issues like high cost, unwieldiness, and simulator sickness. But I strongly believe that this will be resolved with time, and investments. Is VR a new technology? To quote Jeremy Bailenson from the Stanford VHIL (with whom I had the opportunity to exchange some e-mails with): “VR is not a new technology. It just became accessible.” And that is true, the technology was being developed since the 1960s. Only recently have developments in VR become more visible to the public eye. CPU/GPUs have reached a point where they can provide high-fidelity, immersive experiences for reasonable prices. Smartphones have enabled mobile VR as cheaper and more accessible options that don’t require you to be tethered to high-end computer. As more people have an opportunity to try VR, it’s becoming increasingly clear how the technology might reach consumers in a big way. VR Gaming is cool, but there is more to that I mean, when you first think about VR, it is about driving cars in video games, destroying robots and infiltrating military camps. Those things are really fun, but there is a lot more than that.. and that is what makes VR so interesting. But VR will also enable immersive concerts, reinvented museums, and live, court-side sporting events. With VR, videoconferencing will improve, with better eye contact and the inclusion of nuanced, non-verbal cues. The cost of training will plummet without the need for human trainers in industries like construction or manufacturing. At the same time, the efficacy of repeatable, hands-on training will increase. Academics will conduct social psychology research with more reproducibility, diverse sample sizes, and day-to-day realism without the need for human confederates. VR will provide a scalable way to introduce true experiential learning into education. Let's lay down some of the technical fields of VR
My point here is that any experience in one of these fields will you get into the VR world. But there are also non-technical fields
It is possible to create content.. RIGHT NOW Download a game engine, like Unity or Unreal Engine, and start hacking. I have the chance to have extended Unity experience, and hope you also do! Otherwise, it is never too late to start learning! Usually, the game engines are very intuitive and it is not too hard to learn the basics, mastering them is something else. Some useful links that can help you:
Some Computer Vision and Imaging resources:
My advice..START SMALL This is very similar to Game Development, everyone wants to build the next Call of Duty as a first project. Check my project, it was a simple space shooter 2D, and it was a great learning point! Try to embody the perspective of a beginner: be willing to learn and absorb. Don’t start something because you feel like you have to. Now is a great time to learn, experiment, fail and become part of an amazing community. Another advice..TALK TO PEOPLE I honestly believe that this is my main strength, I always go talk to professionals. Don't be shy. During GDC 2017 in California, I talked to senior software engineers from Oculus, Facebook, Google, Unity and senior programmers from Treyarch, Psynox, Infinity Ward etc.. Because you don't lose anything by doing so, all you do is learn. The process of building a beautiful experience requires the work of not only engineers, but artists, designers, and storytellers as well (game developers will understand this). Ask for feedback in online forums. Join clubs. Work at a VR company. The world of VR is still very small, and that’s really special for someone looking to make a big dent in the field. Here is a Facebook VR group you can join as well :) A bright future is coming, let's make sure to contribute to it! ![]() In the week from 28th of February 2017 to 5th of March, I flew to San Francisco, California to attend the Game Developers Conference (GDC) 2017 happening in the Moscone Center and visit my brother who lives in the same city. Attending GDC has a been a huge dream for me, as I have followed it for the past years through Twitter. Without hesitation, I went there to see the latest in the entertainment industry and listen to talks from companies investing on gaming platforms and infrastructures (Google, Facebook, Amazon to name a few). It was one of the best experiences in my life. I fell even more in love with the gaming world, not just the content but the whole world being built around it, namely Virtual Reality and Augmented Reality. After spending more than 15 hours in the event talking to people from all companies and studios, here is an overview of it. VR is the next big thing Yes I know, I said this a bunch of times already! I wrote a bunch on VR explaining how it represents the future, but this is different. In the expo, nearly EVERYTHING was about VR. It was like heaven for me, literally heaven. I was walking there like a little kid, fascinated by everything around me. You put your eyes up, and all you read is "PlayStation", "XBOX", "Google", "Facebook". I attended different tech talks, 2 from Facebook and one from PlayStation. It was interesting to see how facebook is putting so much money into gaming, not game development, but rather into the platform (instant games, gameroom). Google is doing something different, they are more into analytics rather than ads and platforms. I also talked to some people from Naughty Dog (Uncharted), Treyarch (Call of Duty), Psyonix (Rocket League), Oculus, PlayStation and were all telling me how it is to work at their companies. I had the chance to try several Oculus exclusives, played a tournament of Robo Recall to try to win an Oculus Rift, but did not manage. Anyway, it was an amazing experience to meet engineers from their team. I also had the opportunity to try the PlayStation VR with the Aim Controller to play an FPS game. It was amazing and disgusting at the same time, as spiders jump on you in the game. Honestly, if I felt disgusted, it means the game was well designed, doesn't it ? :) Here are some photos about what happened there in downtown San Francisco Facebook + Oculus This is Facebook and Oculus hubs, where Oculus was making the Robo Recall tournament and showing the demos of other games. Facebook was mostly there to give tech talks, give Facebook swags (cool right ?), and recruit. Having seen the GDC, my perspectives and opinions changed quite a bit. Before this, I wanted to become involved in Game Development (Programming mainly), working in game studios. Now, even though I still want to do that, GDC expanded my horizons. I realized I also want to enable game developers to do their job and provide awesome interactive experiences to players. I saw so much potential there, and started thinking about specializing in Human-Computer Interaction, Software Engineering in VR companies such as Oculus. Of course, the entertainment industry is still where I am heading (and that will hopefully never change). GDC did expand my horizons, offering me a broader vision and more opportunities to consider for my future. Here are some other photos I got from the event It was kind of funny to see Unity and Unreal (Epic Games) battling and trying to get the maximum of players. I spent some time talking to the analytics team and met their product manager. He explained to me their plans for the future in terms of data analysis.
It was also funny to see XBOX vs PlayStation and Google vs Facebook. As a passionate game developer, I focused on trying the maximum of knowledge from every company. A photo of the PS VR + Aim Controller is part of this slide show. Last year, I spent a week living on the campus of Stanford University. This year, I had the chance to visit the campus of UC Berkeley and Facebook HQ in Menlo Park, and all of them were really, really amazing! A funny selfie at GDC :) ![]() Technology... it just never stops growing. Amazing things happen every single day. You wake up, a Google robot beats the Go game world champion. You wake up, Amazon demonstrates Amazon Go, which is a grocery store without check out using Computer Vision, Deep Learning algorithms and other complex stuff. What is happening there in the Silicon Valley (and everywhere in the world) transforms the world into a sort of a dreamland for those interested deeply in technology like me. I currently work as a VR Game Developer in the Game Lab here at the TU/e and doing research on how to create empathy through VR experiences. Sometimes I meet people that are more interested into Augmented Reality (AR) and we start arguing about who will take the other in the future. I honestly believe that it will be the next big technology battle, where different big companies will invest into one of them and start developing crazy stuff. Microsoft is already investing a lot on the HoloLens, which is "the first fully untethered, holographic computer, enabling you to interact with high‑definition holograms in your world." This means that Microsoft is in the Augmented Reality side. On the other hand, there is Facebook who acquired Oculus currently building the Oculus Rift. Moreover, there is HTC and Steam, working together on the HTC Vive. Both of these headsets are Virtual Reality headsets. This shows that tech battle already started. Both AR and VR are not yet consumer ready: Yes, they are being sold and shipped, but they are still too expensive and not perfect yet. Those who possess them are usually developers and huge VR or AR enthusiasts. Of course this will not last for long, I believe that in 2-3 years, everyone will start buying the headsets at home for them or their family and friends. Mark Zuckerberg, the man who spent $2 billion of Facebook's money on Luckey's VR company Oculus, is a huge VR enthusiast: "We're working on VR because I think it's the next major computing and communication platform after phones," he said in July 2016. "we'll have the power to share our full sensory and emotional experience with people whenever we'd like." Both Virtual Reality and Augmented Reality are similar in the goal of immersing the user, though both systems do this in different ways. With AR, users continue to be in touch with the real world while interacting with virtual objects around them. With VR, the user is isolated from the real world while immersed in a world that is completely fabricated. As it stands, VR might work better for video games and social networking in a virtual environment, such as Second Life, or even PlayStation Home. On top, you can see the Microsoft Hololens and in the bottom the Oculus Rift headset.
In terms of experience, they both differ a lot. AR is more about wearing something light such as glasses while VR is about wearing a huge headset. While AR will eventually be neatly tucked into the sides of your sports sunnies, though, VR is always going to have to enclose your eyes and ears with lenses, displays and headphones to work. AR glasses will cause etiquette problems as they 'disappear' whereas VR will go the other way with us very clearly 'plugging in' to a virtual world for a session. With pass-through cameras there could even be some kind of hybrid wearable that offers both. In general, AR specs are lighter and more comfortable than VR headsets and they are more likely to be wireless. This is important because, at least for now and the next few years, no one will go out in the street with a heavy headset, but going out with a modified glass is possible. What I am saying is that I see VR more as a technology where the person stays home to play video games while AR will be much more there outside. Now, which of the two technologies represents the future ? This is really the question of whether humans want what they experience in the future to be based in well, reality, or constructed, artificial and cut off from what we now refer to as reality. That's a big question. We're social creatures. So you might think VR has a slight disadvantage - augmented reality demos such as HoloLens' Halo 5 demo allow a group of users to stand in a circle around 'holograms' simultaneously. But steps are already being taken to make VR more social - Facebook is betting big on it and we are beginning to see real investment in platforms which allow virtual hangouts such as AltSpaceVR. And look no further than at art projects such as the telepresence installation Me and My Shadow which connected visitors from four cities in a virtual space. Neither of these technologies is as passive or as social in a physical location sense of the word as say, hanging out half-watching TV. And this could be a big factor. To play a game or experience on Oculus or HTC Vive, you have to commit yourself fully. And VR is thrilling and entertaining enough that gamers will do this, not to mention students, say, getting their heads around anatomy. Perhaps the difference will be VR as an at-home treat, as console gaming or kicking back to a Blu-ray is today and AR for more of a social, everyday experience that doesn't take you away from smartphone alerts, walking down the street or playing with your kids. The answer, then? Well, it's both isn't it? Note: I just published Shapion on Android. It represents an achievement for me as this has been a small dream. Here is a link: https://play.google.com/store/apps/details?id=com.Ahres.Shapion Onto the next project :) ![]() Serious games are getting more and more popular. They are a source of education, motivation and I have had the chance to work on one already. My project was about getting people to change their minds of nuclear energy in favor of Sustainable energy. I am pretty sure that it will get much more importance and will have a bigger impact on the world, especially with the rise of VR. But first, let's define serious games! Have you ever wondered how playing games can help us to train people, deal with societal challenges or raise awareness of contemporary social issues? This is exactly what serious gaming is about: A game designed for any purpose rather than entertainment. Serious games are considered useful to those who wish to use simulation for training and education e.g., flight simulations. They also make use of games engines, a good platform for development and play. That is where the "serious" expression comes from: education, persuasion, motivation or whatever is other than entertainment. It is quite hard to imagine them for some people, because games are usually seen as "fun". Thing is that, they can be both at the same time. However, in this case, being fun is not the most important thing for the game to have. A study made in 2012 by the Serious Games Association showed some really interesting results for the future of this industry [2]:
Now that we have covered a bit about serious games (currently developing one), one idea came to my mind not a long time ago, and I will start working on it just after publishing Shapion on mobile. Here, I will give a quick overview of it and the reason why it can be amazing to develop it, without going too much into details. What if life-lessons are not meant to be said, but to be experienced? An introduction to motivational games Today, Youtube possesses more than 300 motivational videos with up to 35 million views [1], and this number is increasing daily. Watching motivational videos is becoming a well-known advice in order to help people overcome failures, loss of motivation and social difficulties. They are a strong tool in helping humans achieve their highest potential, accomplish more than expected, and visualize success. Moreover, most of the well received speeches are the ones about past failures. For instance, Steve Job’s speech at Stanford University (2005) is considered as one of the best speeches ever made. Another example is J.K. Rowling’s speech at Harvard Commencement (2008). What if we enhanced this motivation by making people live these videos as experiences instead of just being a simple Youtube viewer? What if games can foster learning life lessons? In this section, I will be presenting motivational games as a new type of serious games. The informations highlighted in the first part of this paper will guide serious games to a great design, a great user experience allowing players to be truly satisfied. But first, let’s talk about what current entertainment games teach us about life. Needless to say, some videos games currently possess life-lessons, explicit or not, that the play can extract to use it daily. Here are some of the example: Embrace failure and never give up "Game Over" never means game over. It means press start and try again. A gamer’s video game history is plagued with failure. So many missed leaps, bad timing, and shots to the head. This doesn't mean he gives up, though. Failing in a game just sets him up for success in the future. In life, like in video games, if you take a path with no enemy, it means you took the wrong one When confronted to different paths, usually in games including adventure, a player knows midway though the taken path whether he took the correct one by looking at the number of enemies coming in his way. By making an analogy with life, one can understand how it relates to the path we take in terms of career or even relationships. One last example that is deeper, emotionally appealing to humans is Journey. In this multiple awards-winning PlayStation game [3], the player controls a creature in a beautifully crafted desert with one goal: go to the top of the mountain. The studio stated their interpretation of the meaning life with this game. It is in the title, and it is there until the end of the game. The meaning of life is to never truly know the meaning. All that matters is the Journey you make to discover it. From the beginning of the game you have no idea where you are or what your reason for even being there is. You are basically a newborn, and you are about to enter a beautifully mysterious world, much like the lives we live. Then your creature travels on, learning more about itself, and its culture, and the history behind it. Much like a child. Then at sunset you discover true beauty. You surf over waves of sand that might as well be water. The analogy could be made with your first steps into the world as a young adult. You seek adventure and find things you have never seen before. Then darkness falls, and your delve into the underbelly of Journey's world. It is a dark and scary place down there. Just like the challenges of adult hood. Then you surface and enter the cold unrelenting wind. It blows you backward, but you continue on. This is much like the hindrances of old age. Then you reach the brink of death. There is no beauty, just hopelessness, but you are given a second chance to complete your quest. It's almost like surviving a heart attack.
Then peacefully, on your own terms, you walk slowly into the light accepting your fate. As a player, you never knew for sure what your true purpose in life was, but you're at peace with that. Because you have seen such amazing beauties that you do not need a sentence to describe your true meaning. The meaning behind Journey is that there is beauty in life, in triumph, in fear, and in death. You just have to slow down and realize it. My point through these examples is that the meaning of life and life-lessons are not meant to be said only, but they have to be experienced. In the first two examples, those lessons were taught un-directly. What if we design these motivational games to teach them directly? The example of Journey can be used as a reference. In their website, the studio that developed Journey called thatgamecompany explicitly stated the following: we design and develop artistically crafted, broadly accessible video games that push the boundaries of interactive entertainment. We respect our players and want to contribute meaningful, enriching experiences that touch and inspire them. [4] Journey is a 45-min game that won multiple awards including best PS3 game and came in front of the AAA games such as FIFA, Call of Duty, Assassin's creed and so on. If a 45-min game explaining the meaning of life made this impact, what if we adapt Paulo Coelho's "Manual of Warrior of Light" or "Like a Flowing River" into 2D games with a length of 5-10min ? For those who have read these books, we all know that reading them when feeling bad (pressure, depression, failure..) creates such an amazing feeling that makes us feel alive again, and those are only words. Experiencing it would make it much much better. I am confident that this type of games has a huge potential with VR, and it is time to start building them from now. ![]() When you read some of the important skills one has to develop, you often read: "learn how to learn". For some people, this may sound weird. Yet, it is fundamental to our development. I have always had this problem to memorise information instead of learning. The key is to understand how the brain works when learning something new, and take advantage out of it. In this post, I will share some of the knowledge I learned in the online course called "Learning How to Learn: Powerful mental tools to help you master tough subjects" and from my personal experience One day, as I was reading some answers on Quora, someone asked: What is something unusual about you ? An answer showed up, the best answer among all the others for that question. I cannot remember the exact words, but these sentences stayed in my mind the whole time: "I don't study. Seriously, I don't. The key is to learn, people tend to memorise. I learn". Well, you could say that for a high-school student, it is easy to answer like this. The material covered is still simple. This person was doing a PhD in Mathematics at MIT. Focused vs Diffuse Mode When learning, our brain has two different modes that can be activated. These modes cannot be activated at the same time: FOCUSED MODE: it is just what is sounds like, a concentrated, focused form of thinking DIFFUSED MODE: it is a more relaxed thinking state, one the the brain settles into at resting. One way of imagining it is the FLASHLIGHT ANALOGY. This involves visualising your brain as a flashlight: Diffused mode of thinking could be thought of as a setting on the flashlight designed to cast a broad light not very strongly, while focused mode would cast a very strong light in smaller area. Now, why am I talking about these ? It is important to know how to joggle between both of them. Did it ever happen to you to focus on a problem, not find a solution, then you take a break and during that break the solution just comes to you ? That is because you joggled between both of them. Sometimes, I wake up at 3am, my mind solves an algorithmic question, then come to sleep. Funny right ? Learning in focused mode is usually what people think of when hearing the word “learning.” It is using our focused attention to think solely about the information we are trying to learn. During focused mode thinking, we are sitting down and deliberately practicing something or trying to solve a problem, without distracting ourselves with anything else. When you are sitting down and writing a paper, doing a math problem, or practicing a specific dance move, you are in focused mode. The focused mode can be thought of as the foundation of knowledge, laying the initial memory traces for us to form our knowledge base. The focused practice and repetition of triple axles, free throws, roundhouse kicks, math problems, or vocabulary is what allows us to build a foundation of knowledge to ultimately apply it to what we are ultimately learning to do—whether it be figure skating, playing basketball, doing karate, acing a math examination, or learning to speak a foreign language. Focused mode learning is centered in and around the prefrontal cortex, the area right behind the forehead. The prefrontal cortex is responsible for much of our executive functions that has to do with decision-making and problem-solving, in addition to controlling our attention and memory. Unlike focused mode, diffused mode doesn’t seem to have one central area in the brain that is mainly responsible—it seems to be a division of labor of multiple areas of the brain. When you are trying to grasp a new concept, you do not have a preexisting neural patterns to help guide your thoughts—there is no fuzzy underlying pathway to help guide you. This is when diffused mode becomes handy. To further explain the difference between focused and diffused, it is useful to use the flashlight analogy used in the book. When you are in focused mode, you are shining a flashlight that is tightly focused on one small area. However, when using diffused mode you are casting the flashlight in a broad area, with the light not shining brightly in any one specific area. When you are in diffused mode, you are not intently focused on so-called deliberate practice. Rather, you are just letting the limited knowledge run in the background, kind of like background programs running on your smart phone while you actively use one program. Thinking in diffused mode can be done by just playing a game of basketball if you are learning how to be a better basketball player, by playing random chords on the guitar if you are learning how to play the guitar, or just mentally thinking about math problems while taking a walk. Let me give you an example: Salvador Dali was an extremely famous surrealist painter. Here is his face and most famous painting: This surrealist painting is called the "Persistence of memory" and was painted in 1931. I will leave all the art analysis, as it is not my goal here. However, any person interested in art should check its analysis. It is a masterpiece.
Now, let's come back to our initial point: Salvador Dali was said to relax and drift off to sleep with a key dangling and upon his dozing off to sleep he would drop the key, it would jangle and startle him awake. This is an example of Dali bouncing between thought modes. He would clear his head and relax his mind until sleep came, entering the diffuse mode, then he would wake up and drag all those wonderful diffuse mode ideas back the focused mode and the results were brilliant works like the painting above. I hope this example gives more insight to the joggle of both modes. Now, I will give some learning techniques that anyone should use in order to learn faster, work smarter and be more productive. These techniques are from my personal opinion and some resources on the internet:
I honestly believe it is quite a shame that this is not taught in school. We only learn about new materials, but the first thing we should be taught as children is how to learn. ![]() VR is awesome! Right? To quote Mark Zuckerberg: "VR has the potential to be the most social platform". It opens a whole new world to visualisation: Gaming, Immersion, Movies, Experiences. A few months ago, I have had the opportunity to try the HTC VIVE on a stealth game. It was literally one of the best experiences of my life. It really made an impact on me. Since then, I have started learning deeply how to develop applications on it and have done a lot of research on how VR works exactly. Today, I will be explaining how Virtual Reality works, and will try to go deep into it from an engineering perspective. Virtual Reality tricks your brain into believing you are in a 3D world. The first way VR does this is with the stereoscopic display. This works by displaying two slightly different angles of the scene to each eye, stimulating depth. the image above is an example of the stereoscopic display. In order to create a life like experience, other techniques are developed. For instance, parallax defines the farther objects to the player seem to move slower, making it more realistic. As you can see in the picture, the angle of the weapon is slightly different on each side, as is the crosshair. Once you put the headset, everything lines up perfectly, as if you actually a real weapon in your hand. For VR to work convincingly, the person needs to feel completely immersed in an environment, but also have the ability to interact with it. It's the interaction - known as telepresence - that sets VR apart from other virtual world systems and 3D cinema. Moreover, there is the notion of feedback to be taken into account. When interacting with objects in real life, we can touch, pick up and feel the objects. It is referred as force feedback. In order to create this in the VR world, things such as vibrating controllers have been implemented. They are called haptic systems. A very important part of VR is about tracking the movement and maintaining the illusion. Virtual Reality just wouldn't be Virtual Reality without the ability to look around. I mean, otherwise, the whole thing would not be immersive. Tracking a user's motion is nothing new in the world of computer technology. The mouse that you are currently using is already a device for tracking hand motion, a very simple one compared to an HUD like the Oculus Rift or the HTC Vive. A mouse could be described in engineering terms as a "two degrees of freedom" tracking device, since you can move it left-to-right and front-to-back across a desk. Mathematically speaking, it represents the x and y axis. Most VR headsets have at least three degrees of freedom: you can "pitch" your head by tilting up and down , "roll " your head from shoulder to shoulder, or "yaw" your head from side to side (for example to say "no"). A few even let you move around a physical room in three dimension (such as the HTC Vive), for a total of six degrees of freedom. The image above represents the tracking system of the HTC Vive. The trackers take some time to setup, but do the work efficiently. As you can imagine, tracking someone or something moving around in a combination of six different ways is more complicated than a mouse. One device for recording the kind of information needed is called an Inertial Measurement Unit (IMU).
For VR to work, the illusion must be extraordinarily slick. Humans are extremely sensitive to visual inconsistencies; even small snags can cause "VR sickness" (also called Cyber-sickness), an affliction like motion-sickness. So images must update very quickly. That requires beefy computing hardware capable of generating 90 or more frames per second, also called FPS (standard TV, and most video games, target only 30 updates per second). And the sensors that track the user's head must be able to talk to the computer at least that fast: any delay can cause an unpleasant dragging sensation. Despite the difficulties, engineers are convinced that such problems have, at last, been banished. Different VR platforms also have different specifications on the headsets themselves. The HTC Vive and Oculus Rift both have 90Hz displays, while the Playstation VR has a 60Hz display. It is a rule of thumb that you want your frames per second to match your monitor’s refresh rate, so it is recommended that the Vive and Rift both maintain 90 FPS while the PS VR maintains 60 FPS. Mobile is a different story, as different phones have different resolutions, but maintaining at least 60 FPS is the goal. Expanding more on how FPS and refresh rate work, FPS and the refresh rate of a monitor are two separate things independent from each other. Frames per second is how fast your GPU can display images, per second. 60 FPS means that the GPU is outputting 60 images every second. The refresh rate of a monitor is how fast the monitor can display images per second, measured in hertz (Hz). This means that if you are playing a game and the FPS is 120 but your monitor refresh rate is 60 Hz, you will only be able to display 60 FPS. You are essentially losing half of your frames, which is not a good thing as “tearing” can occur. Tearing is the phenomenon of objects in a game breaking up into a few pieces and being displayed in two different locations along the X axis giving a tearing effect. The battle of VR headsets 1. HTC Vive + Most complete and best overall VR experience + Software partnership with Valve - Requires a high-end GPU - Expensive As of right now, the HTC Vive is the most complete VR experience on the market. As well as a headset and two base stations (which are used for tracking the headset's movement) the Vive also includes two motion controllers in the box. This is important, since it allows the Vive to offer a much more immersive experience than using a traditional controller. 2. Oculus Rift + VR pioneers + Facebook backing - No room-scale The current VR arms race is all thanks to one man: Oculus founder Palmer Luckey. As a teenager, Luckey collected VR tech and was fascinated with making his own headset in his garage. Numerous prototypes and a $2Billion Facebook buyout later, Oculus is still the biggest name in VR. Now the consumer version of the Oculus Rift is finally out and we can get our hands on the headset that started it all. 3. Samsung Gear VR + Good build quality + Works with many of the most popular phones - A bit bulky - Expensive compared to other mobile VR offerings Powered by technology from Oculus, the Samsung Gear VR was effectively the first VR headset on the market. To use it, you simply grab a Samsung phone, download apps and games from the Oculus store, and clip it into the headset. The original model supported the Note 4, but subsequent models have expanded compatibility to a number of phones including the Note 5, Galaxy S6 and Galaxy S7. Below, you will find the specs needed for a computer to support VR. Actually, the majority of people who own desktops are unable to use virtual reality, as their computers are not powerful enough. Steam recommends an Intel i5 Haswell or newer and either an Nvidia GTX 970 or AMD Radeon R9 290 for a smooth experience.
I hope this gave you some insight on Virtual Reality and how it works. |
AuthorAhmed Ahres, 24. |