Building facade extraction is the primary step in the recognition process in outdoor scenes. It is also a challenging task since each building can be viewed from different angles or under different lighting conditions. In outdoor imagery, regions, such as sky, trees, pavement cause interference for a successful building facade recognition. In this paper we propose a knowledge based approach to automatically segment out the whole facade or major parts of the facade from outdoor scene. The found building regions are then subjected to recognition process. The system is composed of two modules: segmentation of building facades region module and facade recognition module. In the facade segmentation module, color processing and objects position coordinates are used. In the facade recognition module, Chamfer metrics are applied. In real time recognition scenario, the image with a building is first analyzed in order to extract the facade region, which is then compared to a database with feature descriptors in order to find a match. The results show that the recognition rate is dependent on a precision of building extraction part, which in turn, depends on a homogeneity of colors of facades.