package 'rsl' -- Robotic Sign Language (rsl) recognition system using Leap Motion Controller This project is the software part of the CIVEMSA 2018 paper "Teaching a Robot Sign Language using Vision-Based Hand Gesture Recognition", D. Zhi, T. E.A. Oliveira, V. P. Fonseca, E. Petriu
If you have any questions please contact [email protected]
This project was developed under ROS framework and Gesture Recognition Toolkits (GRT) library on Linux Ubuntu 14.04 LTE.
Libraries: -ROS -- indigo -GRT -Leap Motion SDK -- version 2.3.1
For more details of the installation of ROS, leapSDK, and GRT library, please refer to: -http://wiki.ros.org/turtlebot/Tutorials -http://www.nickgillian.com/wiki/pmwiki.php/GRT/GestureRecognitionToolkit -https://developer.leapmotion.com/documentation/v2/cpp/index.html
Source code files in ../src/rsl/src folder:
-binary_motor_handNode.cpp: the Ada robotic hand part ROS node
-wrist_node.cpp: the wrist motor part ROS node
-leapmotion_tf_broadcaster.cpp: the tf broadcaster ROS node of the leap motion controller
-leapmotion_tf_listener.cpp: the tf listener ROS node of the leap motion controller
-LeapSDK_tfbroadcaster.cpp: the new tf braodcaster ROS node of the leap motion controller
using the LeapSDK integrated into the script.
-SVMExample.cpp: the classification module ROS node using the SVM classifier
-DTWExample.cpp: the classification module ROS node using the DTW classifier
-KNNExample.cpp: the classification module ROS node using the KNN classifier
Training dataset files in ../src/rsl/data folder:
-DTWTrainingData_10_gestures.grt: the 10 dynamic hand gestures training dataset
-DTWTrainingData_new.grt: The modified 10 dynamic hand gestures training dataset
(more clean and can be used as test dataset)
-TrainingData_static_10numbers.grt: the training dataset of the 10 static digits based on ASL
-testData_static_10numbers.grt: the test dataset of the 10 static digits
Connect Leap Motion through USB
$sudo leapd
$LeapControlPanel (Optional: Sometimes the leap motion connection will lose even the terminal
says "is connected", open the leap control panel to make sure the device
connection is ture.)
With the Leap Motion enabled:
right click its icon > Resume tracking > Diagnostic Visualizer
If it's not calibrated yet, go to 'Recalibrate Device' > Recalibrate and Apply
Build
$ cd [workspace]
$ catkin_make
Connect the hand and wrist to the PC through USB
Use
$ roscore
$ rosrun rsl binary_motor_handNode (Enable the hand ROS node)
$ rosrun rsl wrist_node (Enable the wrist ROS node)
$ rosrun rosserial_server serial_node _port:=/ttyACM0 (Allocate a serial port for hand node)
$ rosrun rosserial_server serial_node _port:=/ttyACM1 __name:=wrist_node(Allocate another serial port for wrist node)
$ rosrun rsl SVMExample (start the static hand postures recognition)
(optional) $ rosrun rsl DTWExample (start the dynamic hand gestures recognition)
To reset calibration:
$rosrun leapmotion reset_leapmotion_calibration_client