Recently, there has been a dramatic increase in the amount of data transmission within short range local area networks (LAN). Multimode fibre (MMF) is widely used in local area networks because of its coupling and alignment along with the low cost of related components. Graded index MMF has become common due to the reduction in pulse spreading; however, as demands for high bandwidth increase towards a future gigabit rate network, the typical MMF using conventional transmission methods will not be suitable. Meanwhile, this increasing demand for high speed data transmission will soon reach the Shannon capacity limit of single mode fibres. After multiple input and multiple output (MIMO) technology was successfully used in wireless communication, the researcher realised that the same idea could also be applied to an optical fibre network. Optical MIMO techniques are gaining interest in order to create parallel channels over orthogonal modes in a MMF or a few mode fibre (FMF). This approach could lead to a significant increase in the bandwidth distance product and be employed in the next 40Gb/s or even 100Gb/s optical fibre transmission systems. Generally speaking, optical MIMO appears to be the best solution to the bandwidth limitation problem in either short distance MMF or long distance FMF systems. This thesis focuses on designing a simple, cost-effective, and energy efficient optical MIMO system based on MMFs. This proposed system can be realised by combining radial offset launching and annular multi-segment detectors. First, in the initial work, we performed a theoretical and numerical study of the key impairments of MMFs, and the mode propagation in an MMF was analysed mathematically. The variation in electrical field intensity for linearly polarised (LP) modes in the core region of an MMF and the analytical solutions for power coupling coefficients in either radial offset launching or centre launching were presented. In addition, the modal time delays, impulse response, and transfer function were all introduced. Subsequently, the near field intensity pattern (NFP) was simulated at the output facet of the MMF, which indicated that the overall NFP suffered from blurring when it contained mode mixing, and that the intensity pattern was particularly sensitive to the random phase. According to the spatial distribution of the NFP, the annular detector can be exploited more efficiently. All of the results were calculated and plotted using the MATLAB program. Secondly, the optical MIMO model in the multimode fibre was briefly summarised, including the MIMO channel matrix H expression, a mathematical expression of optical MIMO capacity, MIMO channel estimation and an equalization method. Two metrics can be used to characterise the MIMO channel performance: condition number and crosstalk at each receiver. The numerical results demonstrated that the new type of annular multi-segment detector exhibits superior performance compared to the conventional multiple single mode fibre (SMF) detectors, making them attractive for future optical MIMO systems. Finally, the core work of this thesis can be divided into two parts: the modelling of a 10Gb/s intensity modulation direct detection (IM-DD) optical MIMO MMF system; and the modelling of an advanced 10Gb/s coherent differential phase shift keying (DPSK) MIMO FMF system. In both simulation systems, the important transmission parameters of intra-group mode mixing, modal dispersion, chromatic dispersion, and mode attenuation were considered and discussed in detail. In the IM-DD optical MIMO system, the optimization of the transceiver can be based upon the laser spot size and the power flux distribution emitted by the transmitter. Results from the simulation showed that the intra-group mode mixing had a limited impact on system performance, and due to its inability to compensate for linear impairments, the IM-DD optical MIMO was not favourable for long distance transmission systems. Nevertheless, the new type of optical fibre FMF seems to be the most promising candidate for use in long haul transmission systems. Therefore, the well-known DPSK modulation format in conjugation with the coherent detection deployed in FMF was studied. Both heterodyne and intradyne detection schemes were analysed followed by mathematical derivation and numerical simulation; the results illustrated that similar system performances can be achieved in both schemes. Meanwhile, the coherent DPSK simulation results also demonstrated that the linear impairments were almost compensated by the frequency domain MIMO equalization process, which resulted in system performance being independent to transmission distance for up to 10km. This advantage proved that the coherent optical DPSK MIMO system can be employed in long haul networks. As with an IM-DD optical MIMO system, optimization of a coherent MIMO system was also possible. However, in contrast to the optimization of an IM-DD MIMO system, a trade-off had to be made between sufficient spatial diversity at the transceiver and differential modal delay caused by modal dispersion; consequently, the numerical results showed that the proposed coherent optical DPSK MIMO gained reasonable good results without using any active device, such as a spatial light modulator and a mode converter. In conclusion, this proposed optical MIMO system provided easy implementation and integration and is feasible for use in future optical communication systems.