* don't run test262 on push anymore
* wrap in an expression
* remove writing to the gh-pages branch
* checkout data repo for comparison
* check dirs
* add back in PR checks
This Pull Request improves our test results display per edition and cleanups our edition detector logic.
It changes the following:
- Adds a new `edition` flag to limit the maximum edition that will be tested.
- Adds a new `versioned` flag to display all tested editions in a table.
- Adds utility methods to `SpecEdition` to detect the edition of a test and get all the available editions.
- Cleanups logic.
Output with this PR ~(We only collect ES5, ES6 and ES13 stats, so all other editions are a WIP)~:
~Marking as a draft since I need to determine the version of the remaining features, but feel free to review everything else.~ Finished!
This is calculated based the tests `es5id` or `es6id`.
Judging by [this](https://github.com/tc39/test262/issues/1557) it's probably not the best method of doing it, but the alternative would require a lot of rework on the boa_tester so that it can pull an older version of the test262 spec which has it's own problems since there's not really an "ES5" only version.
I would think that if we're 100% passing on es5id's then it's safe to assume boa supports es5 (assuming test262's es5id is covering all the es5 cases)
This Pull Request fixes/closes #2629.
It changes the following:
- Store `spec_version` in TestResult based on the tests `es[6/5]id`
- Count all es5, es6 and their test outcome during `TestSuite::run()`
- Print the conformance.
I'm serializing the `spec_version` outcomes so that it can be displayed in test262 github page. I'd like to work on that too if possible. Let me know if there's anything more I should cover in this PR, I'll gladly do it :)
Co-authored-by: jedel1043 <jedel0124@gmail.com>
Just some quality changes to improve the maintainability of the tester:
- Replaces `anyhow` with `color_eyre` to have a better output on errors/panics.
- Changes the ignore file to a TOML file and replaces all parsing logic with the `toml` crate.
- Adds a `ignored` field on all `Test`s to simplify run logic.
- Replaces the global `IGNORED` with an `ignored` argument on the CLI.
This Pull Request replaces `contains`, `contains_arguments`, `has_direct_super` and `function_contains_super` with visitors. (~1000 removed lines!)
Also, the new visitor implementation caught a bug where we weren't setting the home object of async functions, generators and async generators for methods of classes, which caused a stack overflow on `super` calls, and I think that's pretty cool!
Next is `var_declared_names`, `lexically_declared_names` and friends, which will be on another PR.
We were always getting a "VM Implementation" headline in Test262 results for PRs. This was a leftover from the time when we had 2 distinct implementations for the execution. This is no longer the case, so we can remove it to reduce clutter.
<!---
Thank you for contributing to Boa! Please fill out the template below, and remove or add any
information as you feel neccesary.
--->
This Pull Request removes two dependencies that were not really needed, and fixes#2244 by no longer having the package in the dependency tree.
It changes the following:
- The `structopt` dependency in `boa_tester` has been replaced by `clap` v3, the same way as we did in `boa_cli`. This means that we have one less dependency (at least), and that `clap` v2 is only used as a dev-dependency by `criterion` (which will probably be removed in 0.4, as per https://github.com/bheisler/criterion.rs/issues/596).
- The no-longer-updated `num-format` dependency has been removed from `boa_tester`. We were only using it to add comma thousands separator on results, so I added a simple function to do it (not very performant, but it will only be used a few times when showing results).
Looking at this, I noticed a couple of things:
- The `csv` dependency, used by `criterion` has not been updated in more than a year, and it's using a very old `itoa` dependency. They updated the dependency in the repository in March, but unfortunately, the release is taking some more time than expected, and a tracking issue can be found here: https://github.com/BurntSushi/rust-csv/issues/271
- `cargo update` fails, because the latest update to `tinystr` in the ICU4x breaks ICU4x 0.6. I have reported this here: https://github.com/unicode-org/icu4x/issues/2428 and their recommendation is for us to use a beta version of the library, but I don't think we should go for that, since this is a semver breakage.
This removes the only use of the `git2` and `hex` dependencies by reading the test262 submodule commit id directly from the `.git` directory.
Because `git2` depends on a lot of other crates, this removes a bunch of dependencies.
This Pull Request fixes/closes #1645.
It changes the following:
- Add `features` field to `SuiteResult` structure
- Fetch features from `TestSuite` and propagate them via `SuiteResult`
- Add `FeaturesInfo` structure and serialize it to `features.json`
This PR adds some Clippy lints. Mainly, it adds the list of pedantic lints excluding some lints that were causing too many warnings. I also denied some useful restriction and pedantic lints, to make sure we use `Self` all the possible times (for better maintainability), and that we pass elements by reference where possible, for example, or that the documentation is properly written.
This might even have some small performance gains.
I also added a perfect hash function for the CLI keywords, which should be more efficient than a `HashSet`. This is something we could use elsewhere too.
This PR adds conformance results for the VM branch both for PRs and for the conformance results in GitHub pages (even if these are not currently being shown). I'm not 100% sure how this will really work, as I'm not very used to the syntax to concatenate strings.
Co-authored-by: João Borges <rageknify@gmail.com>
Co-authored-by: RageKnify <RageKnify@gmail.com>
* Don't consider panic fixes as "new failures"
* Improved coding style
* Fixed Test262 tester for the testing update
* Ignore JSON module tests
* Test262 uses _FIXTURE at any point of the file name
* Added some result changes for Test262 result comparisons
* Added a bug and a panic to test the changes in the comparator
* Remove explicit todo!()
* Fixing output for test links
* Force comment
* Reducing failures so that commenting works
* New test
* Fixed a bit of formatting
* Reverted purposedly introduced bugs
* Added a "details" section for the test262 test diff
* Fix bold
When the test262 conformance comments are added to a PR, it will now
show the percentage symbol for the conformance, to show up it's
actually a conformance percentage.
* Improved CI workflows
This improves several things in the CI workflows:
- More conformant Test262 result generation
- Benchmarks should now show comments for all users
- Added Test262 result comparison comments to Pull Requests
* Fixed typo
* Checking the comment generation
* Fixing conditions to test comments
* Fix a couple of bugs on the comparator
* Fixed format
* Trying to fix comment updating
* Removing commit autor when searching
* Replace the comment instead of appending
* Switched back to the `pull_request_target` event
Jason Williams
José Julián Espina
Haled Odat
raskad
Zack Mitkin
Iban Eguia Moraza
NorbertGarfield
RageKnify
Halid Odat
0x7D2B
