For autonomous agents to be able to effectively follow instructions and hold conversational dialogue, it is necessary to resolve the various semantic, pragmatic, and common sense ambiguities that exist in everyday language. My research addresses the two fundamental requirements for robust language understanding: leveraging external world knowledge that is missing in standard NLP training datasets, and formulating machine learning methods that extract the precise disambiguation cues lying latent in this noisy, multimodal data. We develop structured and neural models that learn such world knowledge semantics via unlabeled Web-scale features, weakly supervised language embeddings, and cross-modal cues from vision and speech. We achieve state-of-the-art performance on various core NLP tasks and multimodal applications in vision and robotics.