The LION Manifesto:
Learning and Intelligent Optimization for self-improving Artificial Intelligence
Optimization for faster and more effective Machine Learning
Machine Learning for online and offline customization of Optimization
Artificial Intelligence is booming, and the wild transformative power of the current technoscience lies in the strong coupling between Optimization and Machine Learning.
Optimization drives Machine Learning, and Machine Learning improves Optimization by exploiting data produced while searching for better and better solutions, a spiral of continuously improving AI. Currently, this happens mostly by manually improving one with the help of the other, with a long-term vision of more and more self-improvement through automated (data-driven) creativity and innovation with human goals and control.
The recognition of this powerful symbiosis motivated the LION conference (Learning and Intelligent Optimization) two decades ago. We want to continue this union and encourage research in the following directions.
ML4Opt: Machine Learning for Optimization
- Use the “learning from data” (Machine Learning) paradigm to build digital twins (models/simulators/surrogates) of reality to be used by Optimization schemes. The interconnection of Optimization and software models allows the current exponential acceleration in problem-solving and Decision Science.
- Raise the level of automation in problem-solving by:
- Selecting the proper algorithm to apply to a given problem. This issue includes choosing the ideal combination of algorithmic building blocks, exact, heuristic, and synergistic combinations of them
- Self-tuning algorithm parameters (meta-parameters / hyper-parameters) to a given problem
- Adapting hyper-parameters of an Optimization scheme while it runs, in a kind of “algorithmic learning on the job” (Reactive Search Optimization).
- Automate the creation of new algorithms through Reinforcement Learning, often through Deep Learning of internal representations.
- Increase the robustness and avoid the fragility of Optimization to deal with noise, stochasticity, and partial knowledge of most real-world problems, hence considering both aleatoric and epistemic uncertainties.
- Solve problems with human metaphors. Human problem-solving leans on memory, learning, and rapid adaptation: Optimization metaphors should follow more human learning-on-the-job abilities, rather than ad hoc biological metaphors (e.g., by referring to insects, viruses, etc.). The original LION logo hints at this human orientation.
- Learn to optimize through human feedback. If the measurable goal is not fully specified, human feedback from a set of initial solutions can fine-tune the function(s) to be optimized (e.g., in Multi-Objective Optimization dealing with the search for trade-off solutions).
Opt4ML: Optimization for Machine Learning
- Optimization algorithms are the motor under the hood of Artificial Intelligence. Simple variations of gradient descent fuel the current successes of AI in dealing with language models (automated translation, ChatGPT…), images, etc.
- Investigate more efficient Optimization algorithms tuned for Machine Learning to decrease the “brute-force” attitude of many recent applications, requiring enormous amounts of hardware resources and energy, with environmental implications.
- Reach more automation in defining complex chains leading from raw data to features, Machine Learning architectures, training, evaluation, deployment, and continuous adaptation (AutoML is a notable example)
- The phenomenon of emergence in Deep Learning (e.g., for Large Language Models) promises big steps towards automating ML and developing intelligent systems (almost) without programming.
- Parallel population-based Optimization methods based on a group of mathematically well-informed agents, both cooperating and competing, is a paradigm behind many successes in ML (see also memetic algorithms, ensemble learning).
- Optimization (e.g., Mixed-Integer Linear Programming) can be used for analyzing and verifying ML systems and ensuring that relevant properties are satisfied (like resistance to attacks)
- Unconventional computing and Optimization methods (not limited to quantum computing) should be considered. Theoretical and “first principles” research in this area may lead to proposals for novel types of energy-efficient devices.
LION for Science
- Science is based on reproducibility and falsifiability. In many cases, the success of an Optimization (or Machine Learning) algorithm relies on a human in the loop (“human, too human”). Human creativity seeds ideas, but if the availability of the original scientist is essential to reproduce results (because of poorly documented choices, fine-tuning, and strategic choice of partial results to present) the scientific method suffers. The process of selecting algorithms and fine-tuning hyperparameters should become an integrated part accompanying the presentation of a new method.
- The representation problem is at the base of the design of both Optimization and Machine Learning methods. Scientists should concentrate on publishing novel ideas in the search for new representations more than in competition races on subsets of instances of specific problems.
- Towards a code of ethics. A rising call exists about developing a code of ethics for Computer and Data Science, with important implications for Machine Learning (AI) and Optimization.
- Maximise the potential of international cooperation: Optimization has largely enjoyed international co-development. The infrastructure problems addressed by governments are similar in nature (timetabling, rostering, railway scheduling, power-grid optimization, etc.). Our community largely supported each other irrespective of racial, social, geographical, and political circumstances. A cloud of darkness seems to be rising on the horizon due to the world-changing nature of AI applications. We need governments to align on the cooperative support of basic science in these areas.
LION for business and society
- The convergence of Optimization and learning from data considered in LION creates a qualitative landscape change in technoscience. Businesses, society, and politicians must become aware of the underlying technology, its strengths and limitations, to take back our destiny, freedom, and human centrality in this accelerated evolution.