Smelt Internals

Commands and Targets

Smelt has two abstractions to represent "work," the Target interface and the Command interface. Semantically, the lifetime of "work" moves through smelt in the following way:

        ┌───────────────────┐                    ┌───────────┐                   ┌─────────────────┐
        │                   │     LOWERED TO     │           │    EXECUTED ON    │                 │
        │  Target (Python)  │───────────────────►│  Command  │──────────────────►│  Smelt runtime  │
        │                   │                    │           │                   │                 │
        └───────────────────┘                    └───────────┘                   └─────────────────┘

This relationship generalizes to testlists -- a test list, after all, is just a list of targets. Each target is lowered to a set of Commands, and those Commands are executed on the smelt runtime.

To be precise, the target to command boundary is a hard boundary -- the smelt runtime is completely agnostic as to targets, and only interfaces with commands.

Targets

Concretely, a Target is any Python class that inherits from the target Python dataclass in smelt. End users interact with targets in two ways:

1. Defining targets: end users can define their own targets. Semantically, the definition of a class that inherits from the Target interface is directly analogous to a rule in build systems like bazel or buck2.
2. Instantiating targets: end users create targets in testlists, either yaml based or procedural.

The Target interface is defined at in the pysmelt.interfaces.target module.

Each target can create up to three commands:

1. The original command Required -- this is the command that should be executed "normally".
2. The @rerun command Optional -- this is the command is executed if the original command fails -- this is how smelt implements automatic re-running of tests. the @rerun command will depend on the @rebuild variants of the dependencies of the original command. the @rerun is command is optional. Command re-running is described in more detail below.
3. the @rebuild command Optional -- this is the command that is executed if a command depending on this command fails. This is required because sometimes, when recreating a failure, you will want to rebuild part of the system under test with debug flags. If no @rebuild command is generated by the target, then the @rebuild target will be aliased to the original command.

Commands

A Command is the lowest level of work, and is generally derived from a Target. It contains:

• A name describing the command
• A bash command that is to be executed
• Runtime requirements -- how much CPU, memory, and the timeout for this command
• A "CommandType" -- describes the type of work the command does, either test, build or stimulus creation
• Dependencies -- these are names of other commands that must execute before this command can be executed
• An on_failure field -- the name of a command that should be executed if this command fails

While targets are implemented in Python, commands are designed to be language agnostic and completely decoupled from the Target interface.

The smelt runtime is only aware of commands -- it will receive a list of commands and then directions to execute all commands, or particular sets of commands.

Each command will create a bash script that can be interactively re-run by end-users, outside of the smelt runtime

Smelt Runtime

The smelt runtime is implemented in Rust, structured in client-server architecture.

Any time smelt executes one or more commands, the smelt runtime will create a channel that will give a stream of protobuf events, describing the execution of each command.

Executors

The smelt runtime has a concept of executors that can execute each command -- for instance, you can use the docker executor and execute each command in a transient container. Executors are configured globally -- every command must use the same executor, but that may change in the future.

Events and subscribers

As the smelt runtime executes commands, it broadcasts events describing the progress of the commands.

These events include information such as:

• When a command starts and stops executing
• Profiling information for each command -- records how much memory and CPU load is being taken by each command
• The artifacts created by each command

Command re-running

Smelt features automatic re-running of failed commands. This allows you to automatically re-run failing tests with debug flags enabled.

Let's consider an example to contextualize this feature:

- name: build_simulator
  rule: raw_bash_build
  rule_args:
    cmds:
      - build_release_simulator.sh
    rebuild_cmds:
      - build_debug_simulator.sh

- name: simple_test
  rule: raw_bash
  rule_args:
    cmds:
      - bin/release_simulator --seed 1000
    rerun_cmds:
      - bin/debug_simulator --seed 1000
    deps:
      - build_simulator

In the happy path if we run the simple_test test, what will happen is:

1. The release simulator will be built
2. The simple_test will execute successfully
3. (Optional) know peace -- the system under test is functionally correct

But simple_test will fail occasionally -- what happens then?

1. The release simulator will be built
2. simple_test fails
3. The smelt runtime tries to execute the simple_test@rerun command: a. simple_test@rerun has an explicit dependency on build_simulator@rebuild -- so first build_simulator@rebuild is called, rebuilding the simulator in debug mode b. simple_test@rerun has all of its dependencies met -- now it will execute

If we visualize the graph of commands that is generated from the test list above, it will look like the following:

  ┌────────────────┐
  │build_simulator │                         ┌────────────────────────┐
  │                │                         │build_simulator@rebuild │
  └────────────────┘                         │                        │
         ▲                                   └────────────────────────┘
         │depends on                               ▲
         │                                         │
 ┌──────────────────┐                              │depends on
 │simple_test       │                              │
 │                  │                              │
 └──────────────────┘                              │
               ▲                                   │
               │  depends on the failure of  ┌──────────────────┐
               └─────────────────────────────│simple_test@rerun │
                                             │                  │
                                             └──────────────────┘