Abstract The solar system boundary exploration mission has the characteristics of long flight time, high fuel consumption, complex gravity-assist sequence and strict constraints. Therefore, the number of decision variables and the search space of the transfer trajectory are very large, resulting in poor convergence and efficiency of the global search of the metaheuristic algorithm. Moreover, the existing gravity assist space pruning algorithm is no longer applicable for solar system boundary exploration. To effectively reduce the search space and improve the effect of trajectory optimization, an improved gravity assist space pruning algorithm is proposed. In this algorithm, a unique pruning procedure is used to effectively prune the search space, a shape of solution space box bounds combining rectangle and rhombus is adopted, and a method to automatically determine the solution space box bounds is presented. To verify the effectiveness of the improved gravity assist space pruning algorithm, the sensitivity of pruning effect to parameters is analyzed and the optimization effects of three typical metaheuristics are compared. The optimization results of 50 repeated runs of the differential evolution algorithm in the entire search space and the solution space box bounds are compared. Simulation results show that the performance of differential evolution algorithm is better than bat algorithm and firefly algorithm. And the improved pruning algorithm can increase the efficiency of subsequent optimization by more than eleven times and the convergence probability of the objective function by fifty of times. The applicability and efficiency of the proposed method for the solar system boundary exploration are demonstrated.