CS Events Monthly View

Abstract:

Computational food analysis (CFA) has drawn substantial attention due to its importance in health and general wellbeing. For instance, being able to extract food information including ingredients and calories from a meal image could help us monitor our daily nutrient intake and manage our diet. However, the unstructured nature and diversity of meal images limit utilities of deep learning based CFA.

We investigate the possibility of generating structured meal image from a set of ingredients using Generative Adversarial Networks (GANs). To tackle the difficulties mentioned above, we propose a two-phase framework: 1. In order to extract ingredients feature, we train an attention based cross-modal association model to match ingredient sets and their corresponding images in a joint latent space. 2. The generative adversarial network uses the ingredients feature in the latent space as input and generates the corresponding meal image. Furthermore, a cycle-consistent constraint is added to further improve image quality and control appearance. Extensive experiments show our model is able to generate meal image corresponding to the ingredients, which could be used to augment existing dataset for solving other CFA problems.