This paper proposes a novel hybrid technique called enhanced grey wolf optimization‐sine cosine algorithm‐cuckoo search (EGWO‐SCA‐CS) algorithm to improve the electrical power system stability. The proposed method comprises of a popular grey wolf optimization (GWO) in an enhanced and hybrid form. It embraces the well‐balanced exploration and exploitation using the cuckoo search (CS) algorithm and enhanced search capability through the sine cosine algorithm (SCA) to elude the stuck to the local optima. The proposed technique is validated with the 23 benchmark functions and compared with state‐of‐the‐art methods. The benchmark functions consist of unimodal, multimodal function from which the best suitability of the proposed technique can be identified. The robustness analysis also presented with the proposed method through boxplot, and a detailed statistical analysis is performed for a set of 30 individual runs. From the inferences gathered from the benchmark functions, the proposed technique is applied to the stability problem of a power system, which is heavily stressed with the nonlinear variation of the load and thereby operating conditions. The dynamics of power system components have been considered for the mathematical model of a multimachine system, and multiobjective function has been framed in tuning the optimal controller parameters. The effectiveness of the proposed algorithm has been assessed by considering two case studies, namely, (i) the optimal controller parameter tuning, and (ii) the coordination of oscillation damping devices in the power system stability enhancement. In the first case study, the power system stabilizer (PSS) is considered as a controller, and a self‐clearing three‐phase fault is considered as the system uncertainty. In contrast, static synchronous compensator (STATCOM) and PSS are considered as controllers to be coordinated, and perturbation in the system states as uncertainty in the second case study. [ABSTRACT FROM AUTHOR]