3D building models, being the main part of a digital city scene, are essential to all applications related to human activities in urban environments. The development of range sensors and Multi-View Stereo (MVS) technology facilitates our ability to automatically reconstruct level of details 2 (LoD2) models of buildings. However, because of the high complexity of building structures, no fully automatic system is currently available for producing building models. In order to simplify the problem, a lot of research focuses only on particular buildings shapes, and relatively simple ones. In this paper, we analyze the property of topology graphs of object surfaces, and find that roof topology graphs have three basic elements: loose nodes, loose edges, and minimum cycles. These elements have interesting physical meanings: a loose node is a building with one roof face; a loose edge is a ridge line between two roof faces whose end points are not defined by a third roof face; and a minimum cycle represents a roof corner of a building. Building primitives, which introduce building shape knowledge, are defined according to these three basic elements. Then all buildings can be represented by combining such building primitives. The building parts are searched according to the predefined building primitives, reconstructed independently, and grouped into a complete building model in a CSG-style. The shape knowledge is inferred via the building primitives and used as constraints to improve the building models, in which all roof parameters are simultaneously adjusted. Experiments show the flexibility of building primitives in both lidar point cloud and stereo point cloud.