Search

Training in Feed Forward Deep Neural Networks is a memory-intensive operation which is usually performed on GPUs with limited memory capacities. This may force data scientists to limit the depth of the models or the resolution of the input data if data does not fit in the GPU memory. The re-materialization technique, whose idea comes from the checkpointing strategies developed in the Automatic Differentiation literature, allows data scientists to limit the memory requirements related to the storage of intermediate data (activations), at the cost of an increase in the computational cost. This paper introduces a new strategy of re-materialization of activations that significantly reduces memory usage. It consists in selecting which activations are saved and which activations are deleted during the forward phase and then recomputing the deleted activations when they are needed during the backward phase. We propose an original computation model that combines two types of activation savings: either only storing the layer inputs or recording the complete history of operations that produced the outputs. This paper focuses on the fully heterogeneous case, where the computation time and the memory requirement of each layer is different. We prove that finding the optimal solution is NP-hard and that classical techniques from Automatic Differentiation literature do not apply. Moreover, the classical assumption of memory persistence of materialized activations, used to simplify the search of optimal solutions, does not hold anymore. Thus, we propose a weak memory persistence property and provide a dynamic program to compute the optimal sequence of computations. This algorithm is made available through the Rotor software, a PyTorch plug-in dealing with any network consisting of a sequence of layers, each of them having an arbitrarily complex structure. Through extensive experiments, we show that our implementation consistently outperforms existing re-materialization approaches for a large class of networks, image sizes, and batch sizes. [ABSTRACT FROM AUTHOR]

The carbon capture, utilisation and storage (CCUS) process chain is subject to increasing interest, and its overwhelming implementation on the industrial scale appears to be one of the main ways to reduce CO 2 emissions. In this context, the optimisation of a CO 2 purification process for oxy-combustion cement plant flue gases is proposed. This optimisation is based on a multidimensional study on the energy, exergy, economy, and environmental impacts of the process. The results of the optimisations carried out show that it is more favourable to increase the CO 2 recovery above 90%, from an energy, exergy and economic point of view. The analysis of the evolution of the capture cost as a function of the CO 2 recovery shows that for a given carbon tax, there is a minimum for the total cost which includes the sum of the carbon tax contributions for the uncaptured CO 2 and the capture cost. As the unit uses only electrical energy, the cost and the electricity generation will directly impact the capture cost as well as the overall balance in terms of CO 2 avoided. As the electricity price increases from 50 to 250 €/MWh, the CO 2 capture cost increases by almost 250%. An analysis of the parameter uncertainties allows to observe their impacts on the results, and to define a standard deviation from the optimised points and show the robustness of these. Considering the technical parametric uncertainties, the standard deviation on the electrical consumption (3.65 kWh/t CO2), CO 2 recovery (0.09%) and exergy efficiency (0.92%) is limited. • Study of hybrid CO 2 capture system combining cryogenic and membrane technologies. • 4E analyses considering recovery, energy consumption, exergy yield and capture cost. • Maximum exergy efficiency for the cases where the CO 2 recovery is above 90%. • High sensitivity of the electricity price on the CO 2 capture cost. • Uncertainty quantification of modelling assumptions and impact on 4E analyses. [ABSTRACT FROM AUTHOR]