At the end of the course, each student should be able to:
LO1: Explain the fundamental principles of classical numerical optimization methods in the context of AI.
LO2: Implement and adapt metaheuristics to solve complex problems in AI.
LO3: Apply gradient descent methods in the training of deep neural networks, adjusting hyperparameters and analyzing convergence graphs.
LO4: Select and justify the choice of cost functions suitable for different AI tasks, evaluating their impact on model performance.
LO5: Critically evaluate and compare the performance of the optimization methods used, justifying the decisions made and proposing improvements.
LO6: Integrate and adapt the chosen optimization strategies, considering practical constraints (time, resources, scalability) and robustness criteria, to obtain solutions suitable to the needs of the problem.
S1: Classical numerical methods, derivatives and gradients, unconstrained and constrained optimization, introduction to convergence analysis.
S2: Simulated Annealing Metaheuristics, Tabu Search, Evolutionary Algorithms; applications to complex problems (model selection, hyperparameter tuning, global optimization search).
S3: Gradient descent (SGD, Momentum, Adagrad, Adam); implementation in deep neural networks; analysis of convergence and stability graphs of training and validation processes.
S4: Loss/cost functions in Deep Learning: impact on convergence, generalization capability, and model robustness.
S5: Comparison of methods, stopping criteria, scalability, result analysis, improvement proposals, and discussion of practical cases.
As this curricular unit is of a very practical and application-oriented nature, it follows the semester-long assessment model (provided for in the RGACC), and therefore does not have a final exam.
The assessment of this UC has the following composition:
- 2 Quizzes (10% each, total 20%):
Individual short online quizzes, distributed throughout the semester, to assess the theoretical understanding of optimization concepts, metaheuristics, gradient descent methods, and cost functions. There is no minimum score on the quizzes, but completing them helps consolidate knowledge and provides immediate feedback to the student.
- Final Project (80%):
The final project, carried out in groups, is developed in two phases (tutorials), reflecting the PBL (Project-Based Learning) approach:
-- 1st Tutorial (20%):
Delivery of a work plan, identifying the optimization problem in AI, the techniques to be used, as well as the evaluation criteria and convergence metrics.
-- 2nd Tutorial (60%):
Online submission and defense of the complete project, including code, experimental results, critical performance analysis, method comparison, and improvement proposals.
A minimum score of 9.5 is required for the final project as a whole.
The final grade results from the weighted sum of the quizzes (20%) and the final project (80%).
For approval, it is necessary to obtain at least 10 values.
If the student does not pass, they may retake the UC in the following academic year, according to the regulations.
The improvement of the grade can be obtained in the following academic year by repeating the assessment components, in accordance with the current regulations.
The 1st Period and 2nd Period may be used for conducting assessment moments.
Presence during synchronous moments is not mandatory, but recommended, given the opportunity for interaction and immediate feedback.
In addition to the RGACC, the student must consult the Regulation for Students with Special Status (REEE) and the Academic Code of Conduct (CCA).
Mandatory
"1. Sra, Suvrit, Sebastian Nowozin, and Stephen J. Wright, eds., Optimization for machine learning, Mit Press, 2012. 2. Dréo, Johann, et al., Metaheuristics for hard optimization: methods and case studies, Springer Science \& Business Media, 2006. 3. Postek, Krzysztof and Zocca, Alessandro and Gromicho, Joaquim and Kantor, Jeffrey, Hands-On Mathematical Optimization with AMPL in Python, 2024. https://ampl.com/mo-book 4. Dive into DeepLearning, Chapter 11, Optimization algorithms, https://d2l.ai/chapter_optimization/"
Optional
"J. A. E. Andersson, J. Gillis, G. Horn, J. B. Rawlings and Moritz Diehl, CasADi -- A software framework for nonlinear optimization and optimal control, Mathematical Programming Computation, vol. 11, no. 1, pp. 1--36, 2019. https://doi.org/10.1007/s12532-018-0139-4, https://web.casadi.org/"