Live Perception and Real Time Motion Prediction with Deep Neural Networks and Machine Learning
Citation
Zielinski, Edward. 2021. Live Perception and Real Time Motion Prediction with Deep Neural Networks and Machine Learning. Master's thesis, Harvard University Division of Continuing Education.Abstract
The research in this project explores the intersection of human computer interaction (HCI) and deep neural networks. Advances in real-time output has reduced latency making webcam skeletal model output useful for fine motor skill motion research. The newer Live Perception model no longer relies on distant servers resulting in reduction of both latency and privacy issues. Here we take advantage of the advances and develop an interface with low latency and increased privacy to make predictions and inferences entirely with local processing. The interface customizes JavaScript on the client browser to use MediaPipe Pose, TensorFlow.js and Python’s Keras. We call the new interface the Foul Shot Training Mirror. The live perception application provides a blueprint to create motion predictions from deep computer vision models by customizing the real time output.Using this interface, researchers can create time series analysis with the real-time data. This advances HCI research by analyzing how a tight feedback loop can improve the fine motor skills involved in shooting a basketball. Our methods train sequenced motion data from real-time vision models to optimize Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). Our research shows this novel approach to training the motion data is successful in training a Gated Recurrent Unit (GRU). In this new approach, we successfully implement a prototype and use computer vision data from the skeletal data points and angular velocities to predict motion from the deep learning simulation data.
The interface will scale to other applications by using real-time results in a more private and efficient manner. The predictions provide immediate feedback, allowing for immediate forward and backward chain learning. This style of learning aids in improving fine motor skills, which can be used in other research, such as to improve motor skills of people with injuries or disabilities, or to monitor and maintain proper motor skills as people age.
Terms of Use
This article is made available under the terms and conditions applicable to Other Posted Material, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#LAACitable link to this page
https://nrs.harvard.edu/URN-3:HUL.INSTREPOS:37370061
Collections
- DCE Theses and Dissertations [1329]
Contact administrator regarding this item (to report mistakes or request changes)