Analyzing human activity is a basic component of any system, be it biological or artificial, that aims to predict future behavior. Tracking and recognizing voluntary and involuntary action traits are basic endeavors for artificial vision systems that aim to predict cognitive fatigue, a major cause for road and workplace accidents. In this work, we developed a vision system to detect the early onset of fatigue.
In collaboration with the Magnetoencephalography Lab at MIT, we collected synchronous brain (MEG) and behavioral (high-speed camera) data from 14 subjects in a 3-hour task that was designed to induce cognitive exhaustion. We derived a set of 8 eye-movement and 6 head-movement features and trained classifiers for two classes (fatigue, non-fatigue) and three classes (fatigue, transition stage, non-fatigue). We trained Random Forest, K- Nearest Neighbor, and Support Vector Machines classifiers, first and achieved average test accuracies of 98%, 97%, 92% (two classes) and 92%, 90%, 87% (three classes) respectively. To further validate our models, we used the alpha band power in the MEG data as the neural indicator of fatigue. A regression analysis between the camera-based features and the alpha band power revealed an average ρ^2 = 0.59.
Here, we also propose a new method to detect the early stages of fatigue by using the classification error as our behavioral marker. Specifically, we found that the accuracy of the classifiers was higher when the distance between the time intervals of labels for “non-fatigue” and “fatigue” was larger; We estimated the total number of the mis-classified “fatigue” and “non-fatigue” data points in a sliding window: the “fatigue” (“non-fatigue”) number was high (low) in the beginning -non-fatigue stage- and became lower (higher) with time, signifying clear periods of fatigue and non-fatigue. We also observed a sharp change in the labels from non-fatigue to fatigue after 40-50 minutes, which can be attributable to detecting early stages of fatigue. Our results are promising in terms of designing a fully automated system that can predict one’s effective “operation range”, based on behavioral and neurophysiological cues.