Abstract

Multimodal Skills Capture and Transfer is a challenging topic of research that is aimed to analyze and transfer the skilled component of human activities with the use of multimodal technologies. This field requires the contribution of researches from neuroscience, psychophysics, capturing technologies and rendering technologies. It ranges from the multimodal capture of a human task, the analysis of the task in terms of skills, to the rendering of the skill using human modalities, e.g. through haptic interfaces and advanced visualization techniques. At the same time the data acquired in real time is processed by machine learning algorithms for the identification of performance, comparing the user with an existing database of skilled user, producing as an outcome the required stimuli for improving the user task.

Intended Audience

Researchers who want to get up to speed with the major tools for indexing and processing large motion-capture databases. Also practitioners who want a concise, intuitive overview of the state of the art.

Organizers

Speakers

Schedule

1st August 2008
Time Title Speaker PDF
14:00 14:20 Introduction to SKILLS Project Carlo Alberto Avizzano (PERCRO-SSSA)
Modelling and Analysis of Skills
14:20 14:50 Metastability of human perceptuo-motor skills Benoit Bardy (UM1) PDF
14:50 15:20 Motor influences on visual action perception Antonino Casile (Tubingen University) PDF
Capturing and Representing Skills
15:20 15:50 Mathematical modelling of skills Emanuele Ruffaldi (PERCRO-SSSA)
15:50 16:20 Surface EMG for hand control Patrick van der Smagt (DLR)
16:20 16:40 Break
Rendering of Skills
16:40 17:10 Supporting skill acquisition using vibrotactile feedback Chris Jansen (TNO) PDF
17:10 17:40 Skills Rendering and Augmented Reality Didier Stricker (Kaiserslautern University-FhG) PDF
Applications of the SKILLS Project
17:40 18:05 Overview of the Demonstration Activities Carsten Preusche (DLR)
18:05 18:15 Closing Comments and Discussion Carlo Alberto Avizzano (PERCRO-SSSA)

Profiles

Carlo Alberto Avizzano is Assistant Professor of Mechatronics at University of Pisa and Researcher on Control and Automation at SSSA since 2003. Carlo Alberto Avizzano is currently the scientific responsible of research at PERCRO (PERCeptual RObotics) Laboratory of Scuola Superiore Sant’Anna, Pisa – Italy, directed by prof. Massimo Bergamasco. At present, he has authored. co-authored more than 70 scientific papers in the field of haptics, control, robotics and virtual reality. Carlo Alberto Avizzano is the technical responsible of the SKILLS IP for PERCRO, in which is coordinating the research on the digital encoding of Skills and the demonstration activities. Website.

Benoit Bardy is Professor at Montpellier-1 university, junior member (2001-2006) of the Institut Universitaire de France (www.cpu.fr/Iuf/), responsible for UM1 of the Enactive Interfaces network of excellence (www.enactivenetwork.org, visit the web site of the 2006 congress in Montpellier www.enactive2006.org), and of the european SKILLS integrated project (www.skills-ip.eu). I currently coordinate the EuroMov project (www.euromov.eu), a new european center for research, business incubation and technological development in movement sciences in the Languedoc-Roussillon region. Website.

Antonino Casile is independent researcher in the Department of Cognitive Neurology of the Hertie Institute for Clinical Brain Research. His research is focused on investigating the possible influences of motor behaviors on both low- and high-level perception. For his research he uses psychophysical, theoretical and neurophysiological techniques. Website

Chris Jansen is a researcher/project leader at TNO Human Factors (The Netherlands). His research concentrates on designing man-machine interfaces for optimizing performance. Areas of interest are aircraft cockpit design, man-robot interactions, tactile displays.

Carsten Preusche is research scientist at the Institute of Robotics and Mechatronics at the German Aerospace Center (DLR) since 1999. He is head of the Telerobotics and Telepresence Group and participates in several national and international research projects. Up to now he authored or co-authored more than 40 scientific papers in the field of telerobotics, haptic human system interfaces, haptic rendering, bilateral control and space robotics. Carsten Preusche is scientific and administrative responsible of the SKILLS IP on behalf of DLR. He currently represents the Demonstration Activities, of wich he is coordinating the Industry and Maintenance WP, within the steering committee of the IP. Website.

Emanuele Ruffaldi is Assistant Professor at Scuola Superiore Sant'Anna,Pisa in Applied Mechanics in which he obtained his PhD in 2006 discussing a thesis on perceptually inspired Haptic Algorithms. He is member of the research division VRTS "Virtual Reality & Telerobotic Systems" at PERCRO (PERCeptual RObotics) Laboratory of Scuola Superiore Sant’Anna, directed by prof. Massimo Bergamasco. Emanuele has been the author of more than 15 papers published on International Journals and proceedings of scientific workshops. The main project in which he is currently involved is the SKILLS IP, aimed to capture and transfer human skills. His research interests are in the field of haptics, human gesture analysis and system integration for virtual reality applications. He is currently involved in the design of the Multimodal Database System for the management of Skills. Website

Patrick van der Smagt is head of the bionics group at the Institute of Robotics and Mechatronics, German Aerospace Center (DLR Oberpfaffenhofen). His award-winning group, for which he acquired over 16M€ funding in the last 4 years, is inspired by biological systems and uses that to build the most advanced robotic and sensor concepts, as well as the control thereof. As Auden had Tom Rakewell say: “to follow nature as my teacher”. Patrick is Editor and Book Editor for Neural Networks, conference chair and board member, leader of several EC projects, has published numerous papers and holds several patents and awards. Website.

Didier Stricker is a member of the Management Board of the German Research Center for Artificial Intelligence (DFKI GmbH) in Kaiserslautern where he is leading the newly founded research department "Augmented Vision". At the same time, he was appointed Professor at the Computer Science Department of the University of Kaiserslautern, where he hold a chair on visual computing. From 2002 to 2008 Prof. Stricker lead the department "Virtual and Augmented Reality" at the Fraunhofer Institute for Computer Graphics (Fraunhofer IGD) in Darmstadt, Germany. In this function he was initiator and coordinator of the European project MATRIS (Markerless real-time Tracking for Augmented Reality Image Synthesis) and ULTRA (Ultra Portable Augmented Reality System for Augmented Reality Maintenance Applications). He was Technical Coordinator of the project ARVIKA (Augmented Reality for Development, Production and Services) and ARTESAS (Augmented Reality for Industrial Applications). He also served as general chair of the "IEEE&ACM International Symposium on Mixed and Augmented Reality" (ISMAR) 2002 and as reviewers of different journals and conferences. His research interests are virtual and augmented reality, computer vision, edutainment, and multimedia information services.". Website

Abstracts

Metastability of human perceptuo-motor skills (PDF)

Benoit Bardy
University of Montpellier UM1,France

Human perceptuo-motor skills simultaneously exhibit stability and flexibility.
These three fundamental properties:
(i) can be observed at various levels (e.g., muscular synergies vs. perceptual adaptations)
(ii) operate at various spatio-temporal scales (fast, slow), and
(iii) often interact together to produce metastability, i.e., the tendendy for a
complex system to exhibit a flexible form of self-organization where the
system's components express their individual behavior while working together.
In this presentation, recent data obtain in human skills will
be presented demonstrating the metastable nature of human coordinative behaviors.
The consequences for the development of adaptive mutimodal or robotic interfaces
will be adressed.

Motor influences on visual action perception (PDF)

Antonino Casile
Department of Cognitive Neurology
Hertie Institute for Clinical Brain Research
University of Tuebingen, Germany

The perception of movements of people around us is possibly the most
behaviorally important task that we have to carry out in our everyday
life. It is thus not surprising that humans exhibit a high degree of
sensitivity in perceiving the actions of con-specifics. It has been
suggested that this high sensitivity might be due, at least partially,
to a functional link between the motor and the visual systems.

In this talk I will present a brief overview of recent results from
psychophysical and neurophysiological experiments that explore the
possible characteristics of this putative link. In particular, in the
first part, I will present psychophysical results suggesting that
motor learning, even in the complete absence of visual feedback, does
influence the visual perception of similar movements. Furthermore,
visual recognition performance after motor training correlate strongly
with the accuracy of the execution of the learned motor pattern. These
results suggest that motor learning has a direct and highly selective
influence on visual action recognition that is not mediated by visual
learning. In the second part of my talk I will present
neurophysiological results in monkey concerning the response
characteristics of mirror neurons, a possible neuronal substrate for
the interaction between motor and visual representations of
actions. In particular, our data show that (1) given the appropriate
paradigm mirror neurons do respond also to filmed actions and (2) the
activity of mirror neurons is modulated by the point of view under
which actions are observed.

Supporting skill acquisitimon using vibrotactile feedback (PDF)

Chris Jansen
TNO Human Factors, Netherlands


Tactile displays are being used to present information to an individual
in an intuitive way. Especially when cognitive and visual task load is high
or performance in a fast perception-action loop should be optimized, tactile
displays have proven to be successfull. Example applications are navigation in
complex task settings or in suboptimal visual conditions, and land vehicle and
aircraft control.
Another promising applications is perceptual-motor skill acquisition for athletes.
Earlier we learned that simple movements can be best communicated to a person using
an extrinsic (where to move the body part to with respect to the environment) rather
than an intrinsic reference frame (e.g., which muscle to contract).
In a follow up study, we investigated the application of tactile feedback in
the acquisition of correct coordination patterns by beginning competition rowers
having a couple of months experience. The data suggested that tactile feedback
enhanced movement efficiency, indicated by the lower heart rates in the post-training
test. We will present the method and set up of this experiment and present
the results in the context of theoretical approach to coordination patterns inspired
by dynamical systems theory.

Overview of the Demonstration Activities

Carsten Preusche
DLR, Germany

The SKILLS projects aim to develop methods and multimodal interfaces to
capture and transfer human skills from one person (expert) to another
(trainee). These concept and the developped technologie have a common
philosophy such that they can be used in a wide area of applications. To show
this demonstrators in three different areas are built within the project:

Mathematical Modeling of Skills

Emanuele Ruffaldi
PERCRO, Scuola Superiore S.Anna, Italy

Design of Virtual Reality training systems has focused mostly on technological
aspects and specific domains. The application of multimodal training to new areas
stimulates the research on more general approaches of the skill transfer process
and of the design of such trainers. Talk presents a information-action model
that describes how the interaction between agent, environment and trainer can take place,
in particular with the possibility of taking into account different solutions of virtual reality.
The application of this model will be presented woth specific examples of skills
in the area of sport and entertainment.

Skills Rendering and Augmented Reality (PDF)

Didier Stricker
University of Kaiserslautern, Germany

The central idea of augmented reality is to add virtual objects into a real
scene, either by displaying them in a see-through head-mounted display, or
by superimposing them on an image of the scene captured by a camera.
Depending of the application and the skill to be transfered many set-ups and
ways to present the virtual instructions can be designed. In this
presentation we review different augmented reality systems developed within
industrial projects, mainly for supporting users during the achievement of
an industrial maintenance task. Then, several technical issues and
challenges are addressed and first new results are presented. Finally, their
appropriateness to skill rendering and skill transfer is discussed.

Surface EMG for hand control

Patrick van der Smagt
DLR,Germany


Recent advances in forearm surface EMG show remarkable results, being
applicable to the control of dexterous robotic hands. We will
demonstrate some preliminary results and a roadmap towards complete
flexibility of EMG-based control.

Funding

We thank the EU-funded SKILLS (FP6-IST-2005-035005-IP) for support.