Zellij Documentation

Zellij is an open source Python framework for HyperParameter Optimization (HPO) which was orginally dedicated to Decomposition based algorithms 1 2 . It includes tools to define mixed search space, manage objective functions, and a few algorithms. To implements metaheuristics and other optimization methods, Zellij uses DEAP 3 for the Evolutionary Algorithms part and BoTorch 4 for Bayesian Optimization.

In Zellij we consider a minimization problem with a loss function:

$f:\mathcal{X} \subset \mathbb{R}^n \rightarrow \mathbb{R}: \hat{x} = \mathrm{argmin}_{x \in \mathcal{X}}f(x)$ .

With, $\hat{x}$ the global optima, $f$ the objective function, and $\mathcal{X}$ a compact set made of inequalities (e.g. upper and lower bounds of decision variables).

Currently Zellij supports mono-objective problems and mixed, non-constrained and non-dynamic search spaces.

Hyperparameter optimization problem

A hyperparameter in machine learning is a parameter set before the learning phase, those features will impact the learning and the model design. Whereas, parameters, such as synaptic weights in a neural network, are learned. Hyperparameters can be of multiple types, real, discrete, ordinal, categorical, binary… For example, the learning rate of a Stochastic Gradient Descent, the filter size for a convolution, the activation function for a group of neurons…

When doing HPO of neural networks, one have to face several challenges:

Black box loss functions: the only knowledge are the inputs and ouputs of the function. One cannot computes the derivatives or the Jacobian matrix for example.
Expensive loss functions: the evaluation takes minutes up to days.
Noisy loss function: the value of $f(x)$ with $x$ fixed is not necessary the same over multiple runs.
High dimensional and mixed search space.

Popular algorithms for such problems are Grid Search, Random Search, Genetic Algorithms* or Bayesian Optimization.

Defining the search space is a critical phase. Selecting too much hyperparameters can result to a combinatorial explosion. Moreover selecting non significant hyperparameters will have a low impact on the training or model. Therefore, before running any optimization algorithms, one should be carefull on the search space design. Optimizing in a poorly defined search space will lead to worthless optimization.

Install Zellij

Basic version

To install the base version of Zellij, you can use:

pip install zellij

Distributed Zellij

This version requires a MPI library, such as MPICH or Open MPI. It is based on mpi4py.

pip install zellij[mpi]

User will then be able to use the MPI=True option of zellij.core.Loss().

Then the python script must be executed using mpiexec:

mpiexec -machinefile <path/to/hostfile> -n <number of processes> python3 <path/to/python/script>

First steps

from zellij.core import FloatVar, ArrayVar, ContinuousSearchspace, Loss
from zellij.strategies import Bayesian_optimization
from zellij.utils.benchmarks import himmelblau
values = ArrayVar(FloatVar("float_1", 0,1),FloatVar("float_2", 0,1))
lf = Loss(save=False)(himmelblau)
sp = ContinuousSearchspace(values,lf)
bo = Bayesian_optimization(sp, 500)

best, score = bo.run()
print(f"Best solution found:\nf({best}) = {score}")

Dependencies

Python >=3.6
numpy>=1.21.4
DEAP>=1.3.1
botorch>=0.6.3.1
gpytorch>=1.6.0
pandas>=1.3.4
enlighten>=1.10.2
[mpi]: mpi4py>=3.1.2

Contributors

Thomas Firmin: thomas.firmin@univ-lille.fr
El-Ghazali Talbi: el-ghazali.talbi@univ-lille.fr

References

1

Nakib, A., Ouchraa, S., Shvai, N., Souquet, L. & Talbi, E.-G. Deterministic metaheuristic based on fractal decomposition for large-scale optimization. Applied Soft Computing 61, 468–485 (2017).

2

Demirhan, M., Özdamar, L., Helvacıoğlu, L. & Birbil, Ş. I. FRACTOP: A Geometric Partitioning Metaheuristic for Global Optimization. Journal of Global Optimization 14, 415–436 (1999).

3

Félix-Antoine Fortin, François-Michel De Rainville, Marc-André Gardner, Marc Parizeau and Christian Gagné, “DEAP: Evolutionary Algorithms Made Easy”, Journal of Machine Learning Research, vol. 13, pp. 2171-2175, jul 2012.

4

Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. Advances in Neural Information Processing Systems 33, 2020

Zellij Documentation

Contents