This paper discusses the results of the engineering geological investigation and geotechnical studies carried out at the proposed Khersan 3 dam site. The dam will be built on Khersan River, is located about 50 km of west of Lordegan city in west of Iran. The dam is mainly founded on sedimentary rocks of the Tercier age and on Quaternary deposits. Geotechnical information obtained from both at the field and the laboratory studies. Field studies include engineering geological mapping, surface discontinuity mapping, drilled borehole and sampling for laboratory testing. Samples obtained from drilling were tested in the laboratory, included of uniaxial, triaxial and tensile strength tests and deformation parameters, unit weight and porosity. Rock mass classifications of the dam site rock units have been performed according to RMR (Bieniawski 1989), Q(Barton et al., 1980) and GSI systems (Hoek and Brown 1997). A detailed geological and geotechnical study was carried out in the project area to determine the engineering geological characteristics of the rock masses. In the study, rock mass properties was classified using Geological Strength System (GSI), and the Hoek-Brown equation was used to obtain rock mass strength parameters and modulus of elasticity. Finally, regarding to instability analysis in tunnel axis and probably instability wedges, suitable software based on finite element method has been used.