Testing Frameworks
You can add as many projects and repositories as you want, but testing frameworks are limited because Lode needs to understand what it should run and how. The following sections will guide you through what's required to make your testing workflow work in Lode.
The following frameworks are supported by Lode:
- PHPUnit
- Jest
If you'd like to see another framework supported contact us.
Adding frameworks to Lode
In order to add testing frameworks to your repositories in Lode, you will have to trigger a scan.
A scan can be triggered upon adding a new repository:
Alternatively, you can right-click over an existing repository in the sidebar, then select "Scan for Frameworks…"
Scanning strategies
Lode will try to parse your repository to understand what testing frameworks are available for it. It has different scanning strategies for each framework, so if Lode doesn't pick up on a supported framework, make sure to read that framework's "Scanning strategies" section and see if there is something that can be done to correct it.
You can also suggest improvements in scanning strategies by contacting us.
Adding frameworks manually
It is currently not possible to add frameworks manually to your repositories in Lode. We hope to change this very soon.
Managing frameworks
Upon finding frameworks in a repository scan, or at any point when you choose to manage existing frameworks in a repository, you will see the "Manage test frameworks" screen. It will show a list of found/pending frameworks (i.e. after a scan), of existing frameworks, or even of frameworks that were added at a given point and can no longer be found by Lode (they will be flagged as "removed").
After you edit a framework's settings you must click on "Save changes", otherwise the changes will not take effect.
Understanding framework settings
The following settings can be set for each of your frameworks.
Setting | Required | What it does |
---|---|---|
Name | Yes | Allows you to customize your framework's name inside Lode |
Framework | Yes | Which testing framework this is. |
Command | Yes | What binary command should Lode run in order to execute the testing framework. |
Tests path | No | If all of your tests are inside a given path, specifying it here will make Lode hide that path when displaying the test file paths in the list |
Runs in remote machine | Yes | Whether the framework is run in a remote machine (e.g. Vagrant or Docker). You must specify this, even if you don't set any of the remote machine configurations, otherwise Lode will not be able to properly transform your framework's results. |
Remote repository path | No | Much like "Tests path" above, specifying this will make Lode understand which portions of a file path are irrelevant and should be hidden. |
SSH settings | No | If Lode needs to use SSH to connect to the remote machine, specify which settings it should use. |
NOTE
Commands are not executed by a shell, which means it doesn't support variables nor will it use any aliases defined in your shell environment. If you need more flexibility, you can write your own shell script.
Running locally
If a test framework in your repository runs on your local machine, setup is much easier. In most cases, the default command will do. Transformation of the results occurs directly inside of Lode and no changes in your repository should be necessary.
Running in remote machines
Lode can also run tests inside remote machines, as long as you instruct it how it should connect. This can be done in Lode's framework settings, or -- since Lode will merely execute a given script -- using a custom executable file with all required settings (for maximum flexibility).
WARNING
In order to process test results from remote machines, Lode will temporarily inject its reporters into the parent repository and remove them once testing is concluded. It is nevertheless highly recommended that you add the .lode
folder to your .gitignore
file or equivalent.
Vagrant / Homestead
In order to run testing frameworks inside of a Vagrant machine, make sure to configure the SSH settings in your "Framework Settings" menu with all the same parameters that the vagrant ssh
command would use. For example, here are some common settings:
Setting | Value |
---|---|
SSH Host | 127.0.0.1 |
SSH User | vagrant |
SSH Port | 2222 |
Identity file | (Leave blank) |
TIP
If you are unsure what your settings are, running vagrant up && vagrant ssh-config
should print them on the command-line.
Since Laravel Homestead uses Vagrant, the above should work for your Laravel's PHPUnit suite, too.
Docker
Since Docker is extremely flexible, your mileage may vary. But assuming you have a test
command in your tests container, something like this could do the trick: docker exec mymachine test
. Just remember to omit flags like --interactive
, -i
, --tty
and -t
.
Running in custom environments
For custom environments that are not like what's described above, a solution might be to create a custom shell script and use that as the Lode command. Whatever that command does, though, it is very important that it supports the passing of parameters. Testing framework executables make use of parameters to customise runs, and Lode's features like running tests selectively depend on being able to pass them. In practice this means that your custom mycommand.sh
could look something like this (note the use of "$@"
).
bash
#!/usr/bin/env bash
./vendor/bin/phpunit "$@"
NOTE
Lode does not run interactive commands.
PHPUnit
Lode supports various versions of PHPUnit and provides a number of advantages over the native command-line interface. You will be able to see results in real-time, with current running files marked as such, and any errors or exceptions will be visible as soon as they happen with diffs and traces, if available.
There are also a number of refinements built specifically for Laravel assertions, with many more coming soon. (We love Laravel.)
Supported PHPUnit versions
The following table represents the assessed support in our latest testing rounds.
PHPUnit Version | Support | Framework |
---|---|---|
11.0 or later | Full | PHPUnit |
10.0 or later | Full | PHPUnit |
9.0 or later | Full | PHPUnit (Legacy) |
8.0 or later | Full | PHPUnit (Legacy) |
7.0 or later | Full | PHPUnit (Legacy) |
6.0 or later | Full | PHPUnit (Legacy) |
5.7 or later | Full | PHPUnit (Legacy) |
5.0 or later | Partial | PHPUnit (Legacy) |
4.8 or later | Partial | PHPUnit (Legacy) |
Please ensure you have the latest stable version of a major release before trying out Lode. Note that support is tentative, as we cannot guarantee that all testing conditions will work with Lode out-of-the-box. If you think something should be working and isn't, contact us.
PHPUnit scanning strategies
In order for Lode to discover PHPUnit in your repository, it must meet one of these two conditions:
- Have a
phpunit.xml
orphpunit.xml.dist
file in the repository's root folder - Have the
phpunit/phpunit
Composer dependency installed, withvendor/bin/phpunit
executable available
A note on PHPUnit framework command
Using the Laravel command php artisan test
to run PHPUnit works on PHPUnit 10 or later, but the normal output from that command cannot be suppressed. This won't be visible in Lode's interface in normal circumstances, but if the process is interrupted (due to an error or a dd
in a test), Lode will attempt to show unexpected output, including Laravel's own reporting. This is because Laravel (via Collision) will create a new PHP process to run the tests with, and Lode ceases to have full control over that process. For that same reason, Lode cannot work when using the --quiet
flag on an Artisan command, as that will suppress all output from the child process which ultimately runs Lode's reporting.
Jest
The following table represents the assessed support in our latest testing rounds.
Jest Version | Support |
---|---|
24.0 or later | Full |
23.0 or later | Full |
22.0 or later | Full |
Please ensure you have the latest stable version of a major release before trying out Lode. Note that support is tentative, as we cannot guarantee that all testing conditions will work with Lode out-of-the-box. If you think something should be working and isn't, contact us.
Jest scanning strategies
In order for Lode to discover Jest in your repository, it must meet one of these two conditions:
- Have a script declaration in your
package.json
that executes jest, like the following example:
json
{
"scripts": {
"test": "jest"
}
}
- As a fallback, have a
jest
property with Jest settings in yourpackage.json
(although this will likely require you to configure Lode's Jest settings manually, as the default ones will likely fail)
Known limitations for Jest
- When listing tests, Jest only returns the parent suites. Therefore, individual tests are only known after first run, but will subsequently be remembered by Lode.
- Lode cannot run individual Jest tests selectively because the Jest CLI does not support it. Instead, running Jest test selectively is done in the test code itself by using the
describe.only
orit.only
(ortest.only
) methods, like in the following example:
js
describe('Describe test', () => {
it.only('Runs selectively', () => {
// ...
})
it('Is skipped', () => {
// ...
})
})