CS Events

Qualifying Exam

Generative Scene Graph Networks


Download as iCal file

Thursday, December 23, 2021, 09:00am


Speaker: Fei Deng

Location : Via Zoom


Prof. Sungjin Ahn (Advisor)

Prof. Karl Stratos

Prof. Hao Wang

Prof. Dong Deng

Event Type: Qualifying Exam

Abstract: Human perception excels at building compositional hierarchies of parts and objects from unlabeled scenes that help systematic generalization. Yet most work on generative scene modeling either ignores the part-whole relationship or assumes access to predefined part labels. In this work, we propose Generative Scene Graph Networks (GSGNs), the first deep generative model that learns to discover the primitive parts and infer the part-whole relationship jointly from multi-object scenes without supervision and in an end-to-end trainable way. We formulate GSGN as a variational autoencoder in which the latent representation is a tree-structured probabilistic scene graph. The leaf nodes in the latent tree correspond to primitive parts, and the edges represent the symbolic pose variables required for recursively composing the parts into whole objects and then the full scene. This allows novel objects and scenes to be generated both by sampling from the prior and by manual configuration of the pose variables, as we do with graphics engines. We evaluate GSGN on datasets of scenes containing multiple compositional objects, including a challenging Compositional CLEVR dataset that we have developed. We show that GSGN is able to infer the latent scene graph, generalize out of the training regime, and improve data efficiency in downstream tasks.


Rutgers University School of Arts and Sciences

Contact  Prof. Sungjin Ahn

Zoom link: https://rutgers.zoom.us/j/91477502287?pwd=eHh0Q0FncmtkVkF1WFQyUEhxYXBmUT09