Forest fires cause major damage to human habitats and forest ecosystems. Early detection may prevent serious consequences of fast fire spread. Although there are many smoke detection algorithms employed by various optical remote sensing systems, there is still a major misdetection of images containing fog. Fog exhibits similar visual characteristics to smoke. Furthermore, when monitoring dense forests many smoke detection algorithms would fail in robust recognition due to fog covering the trees at dawn. There have been more or less successful attempts to separate smoke from a fog in optical imagery however, these algorithms are strongly connected to a specific application area or use a semi-automatic approach. This work aims to propose a novel smoke and fog separation algorithm based on color space model calculation followed by rule-based shape analysis. In addition, the internal properties of the smoke candidate areas are examined for linear attenuation towards higher energy wavelength. Those areas are then investigated for internal shape properties such as convex hull and eccentricity. Several tests conducted on various high-resolution aerial images suggest that the system is effective in differentiating smoke and fog and thus considered to be robust in early fire detection in forest areas.