Quickstart

In this tutorial, for computation time issues, we will use the Himmelblau benchmark function. It is a continuous optimization problem. As a reminder in Zellij we consider by default a minimization problem. See Objective. (e.g. training a neural network is time consumming). More applications here: Decomposition Based Algorithm.

Defining the Loss Function

A Loss Function is made of a Python callable defined by the user. In Zellij, we consider this function as black-box, it can be whatever you want, provided it follows a certain pattern. Indeed, this Loss Function must be of the form $f(x)=y$ . With $x$ a set of hyperparameters. Zellij uses a wrapping function called zellij.core.Loss() to add features to the user defined function.

Loss function inputs

In your defined Loss Function, inputs can be of two types, it can be a list of hyperparameters. But, if kwarg_mode=True in zellij.core.Loss(), then each hyperparameters will be passed as a kwarg to the Loss Function. Each key will be the label of the corresponding Variables.

Loss function outputs

Zellij supports alternative output pattern: $f(x)=y,model$ for example. Where:

$y$ can be a list or a dictionary.
$model$ is optionnal, it is an object with a save() method. (e.g. a neural network from Tensorflow or PyTorch)

import numpy as np
from zellij.core import Loss

@Loss(save=False, verbose=True)
def himmelblau(x):
  x_ar = np.array(x)
  return np.sum(x_ar**4 -16*x_ar**2 + 5*x_ar) * (1/len(x_ar))

print(himmelblau)

Defining the search space

Here we will work in 2 dimensions. But Zellij supports high dimensional problems. To define a searchspace one need to define Variables and a Loss Function. Available Variables are:

Floats: zellij.core.FloatVar allows to model with upper and lower bounds a float decision variable. You can even change the sampler.
Integers: zellij.core.IntVar allows to model with upper and lower bounds a integer decision variable. You can even change the sampler.
Categorical: zellij.core.CatVar allows to model a categorical variable with a list of features.
Arrays: zellij.core.ArrayVar allows to model an array of Variables.

from zellij.core import FloatVar, ArrayVar, ContinuousSearchspace

values = ArrayVar(FloatVar("float_1", 0,1),FloatVar("float_2", 0,1))
sp = ContinuousSearchspace(values,himmelblau)

p1,p2 = sp.random_point(), sp.random_point()
print(p1)
print(p2)

Once your search space is defined, you can use some of its functionnalities. You can draw random points, random attributes…

rand_att = sp.random_attribute(5)
rand_pts = sp.random_point(10)

print(f"Random Attributes: {rand_att}")
print(f"Random Points: {rand_pts}")

See Search space for more information.

Now we can use the loss function and the search space:

scores = himmelblau(rand_pts)
print(f"Best solution found:\nf({himmelblau.best_point}) = {himmelblau.best_score}")
print(f"Number of evaluations:{himmelblau.calls}")
print(f"All evaluated solutions:{himmelblau.all_solutions}")
print(f"All loss values:{himmelblau.all_scores}")

# Reset the loss function for other usage
himmelblau.reset()

Implementing an optimization strategy

Here we will implement a Bayesian Optimization, which uses BoTorch. In Zellij all optimization algorithms are based on the abstract class Metaheuristic. An optimization algorithm will be defined by a Search space, a Loss Function, a budget (number of calls to Loss Function).

Here we use an additive exponential cooling schedule.

from zellij.strategies import Bayesian_optimization

bo = Bayesian_optimization(sp, 500)

best, score = bo.run()
print(f"Best solution found:\nf({best}) = {score}")

Quickstart

Contents