EvoTorch

EvoTorch 是一个直接构建在 PyTorch 之上的高级进化算法库，由 NNAISENSE 创建。EvoTorch 旨在加速进化算法的研究和应用，并专门支持 NeuroEvolution。

pip install evotorch

使用 EvoTorch，可以解决各种优化问题，而不必担心手头的这些问题是否可微。EvoTorch 可解决的问题类型包括：

• Black-box 优化问题（连续或离散）
• 强化学习任务
• 监督学习任务
• 等等

EvoTorch 中提供了各种进化计算算法：

• 基于分布的搜索算法：
  • PGPE：基于参数探索的策略梯度。
  • XNES：指数自然进化策略。
  • SNES：可分离的自然进化策略。
  • CEM：交叉熵方法。
• 基于群体的搜索算法：
  • SteadyStateGA：一个完全精英化的遗传算法实现。还支持多个目标，在这种情况下，其行为类似于 NSGA-II。
  • CoSyNE：协同突触神经进化。

上面提到的所有这些算法都是在 PyTorch 中实现的，因此可以从 PyTorch 的矢量化和 GPU 功能中受益。此外，在 Ray 库的帮助下，EvoTorch 可以通过将工作负载分散到以下方面来进一步扩展这些算法：

• 多个 CPU
• 多个 GPU
• Ray 集群上的多台计算机

black-box 优化示例

from evotorch import Problem
from evotorch.algorithms import SNES
from evotorch.logging import StdOutLogger, PandasLogger
import math
import matplotlib.pyplot as plt
import torch

# Declare the objective function
def rastrigin(x: torch.Tensor) -> torch.Tensor:
    A = 10
    (_, n) = x.shape
    return A * n + torch.sum((x ** 2) - A * torch.cos(2 * math.pi * x), 1)

# Declare the problem
problem = Problem(
    "min",
    rastrigin,
    initial_bounds=(-5.12, 5.12),
    solution_length=100,
    vectorized=True,
    # device="cuda:0"  # enable this line if you wish to use GPU
)

# Initialize the SNES algorithm to solve the problem
searcher = SNES(problem, popsize=1000, stdev_init=10.0)

# Initialize a standard output logger, and a pandas logger
_ = StdOutLogger(searcher, interval=10)
pandas_logger = PandasLogger(searcher)

# Run SNES for the specified amount of generations
searcher.run(2000)

# Get the progress of the evolution into a DataFrame with the
# help of the PandasLogger, and then plot the progress.
pandas_frame = pandas_logger.to_dataframe()
pandas_frame["best_eval"].plot()
plt.show()

强化学习示例

from evotorch.algorithms import PGPE
from evotorch.logging import StdOutLogger
from evotorch.neuroevolution import GymNE

# Declare the problem to solve
problem = GymNE(
    env_name="Humanoid-v4",  # Solve the Humanoid-v4 task
    network="Linear(obs_length, act_length)",  # Linear policy
    observation_normalization=True,  # Normalize the policy inputs
    decrease_rewards_by=5.0,  # Decrease each reward by 5.0
    num_actors="max",  # Use all available CPUs
    # num_actors=4,    # Explicit setting. Use 4 actors.
)

# Instantiate a PGPE algorithm to solve the problem
searcher = PGPE(
    problem,

    # Base population size
    popsize=200,

    # For each generation, sample more solutions until the
    # number of simulator interactions reaches this threshold
    num_interactions=int(200 * 1000 * 0.75),

    # Stop re-sampling solutions if the current population size
    # reaches or exceeds this number.
    popsize_max=3200,

    # Learning rates
    center_learning_rate=0.0075,
    stdev_learning_rate=0.1,

    # Radius of the initial search distribution
    radius_init=0.27,

    # Use the ClipUp optimizer with the specified maximum speed
    optimizer="clipup",
    optimizer_config={"max_speed": 0.15},
)

# Instantiate a standard output logger
_ = StdOutLogger(searcher)

# Run the algorithm for the specified amount of generations
searcher.run(500)

# Get the center point of the search distribution,
# obtain a policy out of that point, and visualize the
# agent using that policy.
center_solution = searcher.status["center"]
trained_policy = problem.make_net(center_solution)
problem.visualize(trained_policy)

更多示例可以在这里找到。