README
¶
Natural Evolution Strategies
An implementation of Natural Evolution Strategies for black box optimization.
How it works
Natural Evolution Strategies optimizes a black box function parameterized with a set of weights by mutating a baseline set of weights with noise for each member of the population, then biasing the noise provided to the rewards achieved and summing that accross the entire population. The baseline set of weights is then updated at each generation based on the Natural Gradient of the entire population.
Examples
See the experiments folder for example implementations.
Roadmap
- More enviornments
- K8s
- Support multiple weights.
References
- Paper - http://www.jmlr.org/papers/volume15/wierstra14a/wierstra14a.pdf
- Wiki - https://en.wikipedia.org/wiki/Natural_evolution_strategy
- Karpathy sample code - https://gist.github.com/karpathy/77fbb6a8dac5395f1b73e7a89300318d
- A Visual Guide to Evolution Strategies - http://blog.otoro.net/2017/10/29/visual-evolution-strategies
- OpenAI Release - https://openai.com/blog/evolution-strategies
- Lilian Weng Overview - https://lilianweng.github.io/lil-log/2019/09/05/evolution-strategies.html
Documentation
¶
Overview ¶
Package nes is an agent implementation of the Natural Evolution Strategies algorithm.
Index ¶
- Variables
- func MakePolicy(config *PolicyConfig, base *agentv1.Base, env *envv1.Env) (modelv1.Model, error)
- type Agent
- type AgentConfig
- type BlackBox
- type BlackBoxResult
- type Evolver
- type EvolverConfig
- type EvolverHyperparameters
- type LayerBuilder
- type PolicyConfig
- type SolvedChecker
- type SphereBlackBox
- type SphereBlackBoxConfig
Constants ¶
This section is empty.
Variables ¶
var DefaultAgentConfig = &AgentConfig{ PolicyConfig: DefaultPolicyConfig, }
DefaultAgentConfig is the default config for a dqn agent.
var DefaultEvolverHyperparameters = &EvolverHyperparameters{
NPop: 50,
NGen: 300,
Sigma: 0.1,
Alpha: 0.001,
}
DefaultEvolverHyperparameters are the default hyperparams for the evolver.
var DefaultFCLayerBuilder = func(x, y *modelv1.Input) []layer.Config { return []layer.Config{ layer.FC{Input: x.Squeeze()[0], Output: y.Squeeze()[0], Activation: layer.Linear, NoBias: true}, } }
DefaultFCLayerBuilder is a default fully connected layer builder.
var DefaultPolicyConfig = &PolicyConfig{ LayerBuilder: DefaultFCLayerBuilder, Track: false, }
DefaultPolicyConfig are the default hyperparameters for a policy.
var DefaultSphereBlackBoxConfig = &SphereBlackBoxConfig{ NumEpisodes: 100, EnvName: "CartPole-v0", AgentConfig: DefaultAgentConfig, Logger: log.DefaultLogger, SolvedChecker: func(reward float32) bool { if reward >= 195 { return true } return false }, }
DefaultSphereBlackBoxConfig is the default config for a sphere black box.
Functions ¶
func MakePolicy ¶
MakePolicy makes a model.
Types ¶
type Agent ¶
type Agent struct {
// Base for the agent.
*agentv1.Base
// Policy for the agent.
Policy model.Model
// contains filtered or unexported fields
}
Agent is a dqn agent.
type AgentConfig ¶
type AgentConfig struct {
// PolicyConfig for the agent.
PolicyConfig *PolicyConfig
}
AgentConfig is the config for a dqn agent.
type BlackBox ¶
type BlackBox interface {
// Run the black box.
Run(weights *t.Dense) (reward float32, err error)
// RunAsync the black box.
RunAsync(populationID int, weights *tensor.Dense, results chan BlackBoxResult, wg *sync.WaitGroup)
// Initialize the weights.
InitWeights() *t.Dense
}
BlackBox function we wish to optimize.
type BlackBoxResult ¶
type BlackBoxResult struct {
// Reward from black box run.
Reward float32
// Error from run.
Err error
// PopulationID is the ID of the population.
PopulationID int
// Solved tells if the agent solved the problem.
Solved bool
// Weights if solved.
Weights *tensor.Dense
}
BlackBoxResult is the result of a black box run.
type Evolver ¶
type Evolver struct {
*EvolverHyperparameters
*agent.Base
// contains filtered or unexported fields
}
Evolver of agents.
type EvolverConfig ¶
type EvolverConfig struct {
// Hyperparameters for the evolver.
*EvolverHyperparameters
// BlackBox function to be optimized.
BlackBox BlackBox
// Base agent.
Base *agent.Base
}
EvolverConfig is the config for the evolver.
type EvolverHyperparameters ¶
type EvolverHyperparameters struct {
// NPop is the population size.
NPop int
// NGen is the number of generations.
NGen int
// Sigma is the noise standard deviation.
Sigma float32
// Alpha is the learning rate.
Alpha float32
}
EvolverHyperparameters are the hyperparameters for the evolver.
type LayerBuilder ¶
LayerBuilder builds layers.
type PolicyConfig ¶
type PolicyConfig struct {
// LayerBuilder is a builder of layer.
LayerBuilder LayerBuilder
// Track is whether to track the model.
Track bool
}
PolicyConfig are the hyperparameters for a policy.
type SolvedChecker ¶
SolvedChecker checks if the environment is solved.
type SphereBlackBox ¶
type SphereBlackBox struct {
// contains filtered or unexported fields
}
SphereBlackBox is a sphere environment runner.
func NewSphereBlackBox ¶
func NewSphereBlackBox(config *SphereBlackBoxConfig, server *envv1.Server) (*SphereBlackBox, error)
NewSphereBlackBox returns a new sphere black box.
func (*SphereBlackBox) InitWeights ¶
func (s *SphereBlackBox) InitWeights() *tensor.Dense
InitWeights for the test.
func (*SphereBlackBox) Run ¶
func (s *SphereBlackBox) Run(weights *tensor.Dense) (reward float32, err error)
Run env.
func (*SphereBlackBox) RunAsync ¶
func (s *SphereBlackBox) RunAsync(populationID int, weights *tensor.Dense, results chan BlackBoxResult, wg *sync.WaitGroup)
RunAsync runs the black box async
type SphereBlackBoxConfig ¶
type SphereBlackBoxConfig struct {
// NumEpisodes is the number of episodes.
NumEpisodes int
// EnvName is the environment name.
EnvName string
// AgentConfig is the agent config.
AgentConfig *AgentConfig
// Logger for the box.
Logger *log.Logger
// SolvedChecker checks if the environment is solved.
SolvedChecker SolvedChecker
}
SphereBlackBoxConfig is the sphere black box config.
Source Files
Directories