Deep Learning Models for Visual Sensory-Perceptual-Cognitive Dynamical Systems from Eye Movement Data and Categories of Natural Images

Abstract

Purpose : To model dynamical systems arising in human visual processes in health and disease using BIG Data Analytics, Deep Learning in collaboration with clinical researchers.
Specific Aims -
(A) Data driven computational modeling for dynamics of sensory-perceptual-cognitive processes for categorization of visual stimuli in natural scenes, using eye movement data, video and other imaging modalities (when appropriate & available).
(B) Apply the model in (A) and Deep Learning for early prediction of Autism Spectrum Disorder in infants from response to facial/non-facial videos.

Methods : Visual categorization model by Poggio et al (hybrid bottom-up/top-down) and hierarchical deep recurrent networks are adapted to learn from natural data (faces, interior spaces in architecture, paintings ...), and to construct an empirical visual space (EVS). Eye movement data is used to reconstruct dynamical systems (DS) in EVS at three sensory, perceptual, cognitive levels. EVS & DS conform with experimental results in vision psychophysics and neuronal processing pathways. The reconstructed DS are suitable for Deep Learning and enhance further modeling paradigms.

Results : 1) We have applied the model in (A) for modeling visual memorability, attention and recall for image categories in architecture. These results are partially included in a forthcoming UW Madison PhD thesis.
2) Work in progress includes ASD hierarchical classification of responses by infants/children 6 months-3 years of age from eye movement response in facial/non-facial visual processing.
3) Work in progress includes visual categorization and abstraction in Baroque paintings versus pre-Baroque and post-Baroque paintings.

Conclusions : The proposed experimental data driven model of “empirical visual space” is suitable for reconstruction of sensory-perceptual-cognitive dynamical systems using eye movement data. These dynamical systems are useful for deep learning from experimental data to model visual search, visual memorability, visual attention and visual recall using realistic image/video categories. The computationally predicted results for visual perception of interior spaces agree with experimental results from human subjects.

This is an abstract that was submitted for the 2018 ARVO Annual Meeting, held in Honolulu, Hawaii, April 29 - May 3, 2018.