Recently, there has been an increased focus on developing an autonomous shipping technology that is safe, trustworthy, and efficient. Some enablers of this technology are strategies to advance sustainability and reducing CO2 emissions. This is achieved by making ships more efficient, increasing safety and reducing the number of accidents caused by human errors on water, and reducing operational costs. However, there are still many challenges to face before autonomous technology on the water becomes a part of our daily life. A safe and reliable path planning and collision avoidance method is an important component in autonomous shipping and plays a key role in incorporating this technology into our daily lives. Although numerous path planning algorithms for autonomous vessels have been developed, each with its own benefits and limitation, there is no one ultimate path planning and collision avoidance algorithm that is suitable for every vessel, in all water regions and in all scenarios. There is also no unified way of evaluating and comparing these algorithms to find the most suitable one for the chosen use case. In this context, the main purpose of this research is to propose a strategy for a unified evaluation and comparison of path planning and collision avoidance algorithms. To achieve this goal, it is essential to first gain an understanding of path planning and collision avoidance as a part of the autonomous surface vehicle’s guidance, navigation, and control system. There are two main application cases. First, it could be used as an offline benchmarking tool to evaluate and compare the algorithms. Second, it could be used online on an actual vessel to select the safest and most efficient path in the planning phase based on the current situation. This thesis presents an evaluation simulator platform (ESP) for evaluating path planning and collision avoidance algorithm performance for autonomous surface vehicles. In particular, the work focuses on an extended collision risk assessment (ECRA) method for evaluating the generated paths from a safety perspective. The testing results indicate that the proposed approach could be used for autonomous path planning and collision avoidance algorithm evaluation and comparison with some improvements. In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of NTNU’s products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink.