People typically learn through exposure to visual concepts associated with linguistic descriptions. For instance, teaching visual object categories to children is often accompanied by descriptions in text or speech. In a machine learning context, these observations motivates us to ask whether this learning process could be computationally modeled to improve visual object recognition task. Specifically, the main question we address in this work is how to use purely textual description of categories with no training images to learn visual classifiers for these categories. We propose an approach for zero-shot learning of object categories where the description of unseen categories comes in the form of typical text such as an encyclopedia entry, without the need to explicitly defined attributes. We propose and investigate two baseline formulations, based on regression and domain adaptation. Then, we propose a new constrained optimization formulation that combines a regression function and a knowledge transfer function with additional constraints to predict the classifier parameters for new classes. We further developed the framework such that it can predict kernelized classifiers for classes with no training data (unseen classes), where the knowledge about the classes is generalized to belong to an arbitrary domain ( e.g. Semantic concepts from ontologies, attribute representation, or textual description). The predicted classifier is assumed to have the form of the generalized representer theorem. There are several features of the proposed framework that distinguish it from other zero-shot learning approaches; most importantly explicitly predicting classifiers in a kernelized format; and requiring weak annotation, where the secondary information is only available at the category level. Our approach was applied for the challenging task of zero-shot learning on fine-grained categories. We applied the proposed approach on two challenging fine-grained categorization datasets, and the results indicate successful classifier prediction.