* stable-5.1:
Run JMH benchmarks using bazel
Benchmark for creating files and FileSnapshots
Implement benchmark for looking up FileStore of a given Path
JMH benchmark for SimpleLruCache
Remove unused API problem filters
Silence API errors for new API added since 5.1.0
Change-Id: If91c55a192d3b2c441d9c8d414f2e24a7261b1b6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The benchmarks can be built and run using bazel by running:
$ bazel run //org.eclipse.jgit.benchmarks:benchmarks
Change-Id: I6679750eaa0f2be30ed9d45036e013b0ea4fcc86
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Results on Mac OS 10.14.6 using APFS:
Result "org.eclipse.jgit.benchmarks.LookupFileStoreBenchmark.testLookupFileStore":
44583.277 ±(99.9%) 2666.096 ns/op [Average]
(min, avg, max) = (43707.073, 44583.277, 45320.685), stdev = 692.377
CI (99.9%): [41917.181, 47249.373] (assumes normal distribution)
Secondary result
"org.eclipse.jgit.benchmarks.LookupFileStoreBenchmark.testLookupFileStore:·stack":
Stack profiler:
....[Thread state
distributions]........................................................
100.0% RUNNABLE
....[Thread state:
RUNNABLE].............................................................
42.5% 42.5% sun.nio.fs.UnixNativeDispatcher.realpath0
37.9% 37.9% sun.nio.fs.UnixNativeDispatcher.stat0
11.0% 11.0% sun.nio.fs.BsdNativeDispatcher.getfsstat
7.6% 7.6% sun.nio.fs.BsdNativeDispatcher.fsstatEntry
0.7% 0.7% sun.nio.fs.BsdNativeDispatcher.endfsstat
0.1% 0.1% sun.nio.fs.UnixPath.initOffsets
0.0% 0.0% sun.nio.fs.UnixFileAttributes.get
0.0% 0.0% java.util.zip.Inflater.inflateBytes
0.0% 0.0% sun.misc.Unsafe.compareAndSwapInt
0.0% 0.0% sun.nio.fs.BsdFileStore.findMountEntry
This shows that FS.attrCacheByPath should be useful to reduce overhead
of looking up the FileStore of a given file.
Change-Id: I7213086b42e0453f0ee149660d507dac2a4644cc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
See [1] for JMH documentation and [2] how to use JMH in Eclipse.
The benchmarks pom currently cannot use the JGit parent pom due to an
ecj bug [3] regarding annotation processing. Hence for now do not
inherit from the JGit parent pom and copy the compiler plugin
configuration for javac from the parent pom.
After running the Maven build the benchmark can be run using Maven:
$ java -jar org.eclipse.jgit.benchmarks/target/benchmarks.jar
or in Eclipse by running the main method of the SimpleLruCacheBenchmark
class.
[1] https://openjdk.java.net/projects/code-tools/jmh/
[2] http://alblue.bandlem.com/Page/3/index.html
[3] https://eclip.se/532029
CQ: 20517
CQ: 20518
Change-Id: Idca8a9e0980f0b8a9c741c4c9e97d03c62f07c8d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
- filter errors for new APIs added in service release
- remove unused filters
Change-Id: Ifbf532b8a3c46d4ed78a38f6c75073a072b7f669
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
# By Matthias Sohn (2) and Saša Živkov (1)
* stable-5.1:
Prepare 5.1.13-SNAPSHOT builds
JGit v5.1.12.201910011832-r
Do not rely on ArrayIndexOutOfBoundsException to detect end of input
Change-Id: Iaae4b171eaa0081f9142489de8df94ab455d65f7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
In the Config#StringReader we relied on ArrayIndexOutOfBoundsException
to detect the end of the input. Creation of exception with (deep) stack
trace can significantly degrade performance in case when we read
thousands of config files, like in the case when Gerrit reads all
external ids from the NoteDb.
Use the buf.length to detect the end of the input.
Change-Id: I12266f25751373a870ce3fa623cf2a95d882d521
Older JGit stored only milliseconds timestamps in the index. Newer
JGit may get finer timestamps from the file system. This leads to
slow index diffs when a new JGit runs against an index produced
by older JGit because many timestamps will differ and JGit will
then do many content checks. See [1].
Handle this migration case by only comparing milliseconds if the
index entry has only millisecond precision.
The inverse may also occur; also compare only milliseconds if the
file timestamp has only millisecond precision.
Do the same also for microsecond resolution. On Windows, NTFS may
provide 100ns resolution and may be used by external programs writing
the index, but Java's WindowsFileAttributes may provide only
microseconds.
File timestamp precision in Java depends not only on the Java APIs
used by different JGit versions but may also change when running the
same Java code on different VMs. And of course the resolution may
vary among operating and file systems. Moreover, timestamp precision
in the index depends on the program that wrote the index. Canonical
git may use a different resolution, maybe even different between git
versions.
[1] https://www.eclipse.org/forums/index.php/t/1100344/
Change-Id: Idfd08606c883cb98787b2138f9baf0cc89a57b56
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If CheckStat is MINIMAL or timestamps have no nanosecond part
WorkingTreeIterator.compareMetaData only checks the second part of
timestamps and ignores nanoseconds which may have ended up in the index
by using native git.
If
fileLastModified.getEpochSecond() == cacheLastModified.getEpochSecond()
we currently proceed comparing fileLastModified and cacheLastModified
with full precision which is wrong since we determined that we detected
reduced timestamp resolution.
Fix this and also handle smudged index entries for CheckStat.MINIMAL.
Change-Id: I6149885903ac63d79b42d234cc02aa4e19578f3c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.1:
Format BUILD files with buildifier
Change-Id: I5a8413470fd0fbb6db3ca553f455b3e3dc10ab33
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Using buildifier from master branch and the command:
$ buildifier -r -lint fix -warnings all .
Change-Id: I19c8ff183081093cb73bed7221a78a91b6cba4dc
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
We don't need to update time atomically since it's only used to order
cache entries in LRU order.
Change-Id: I756fa6d90b180c519bf52925f134763744f2c1f1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.1:
Bazel: Fix warning about deprecated lib.bzl
Format lib/BUILD with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Bump minimum supported version to 0.29.0
Bazel: Bump skylib library version to 0.8.0
Use bazelisk to switch between used bazel version
Bazel: Require minimum bazel version 0.17.1
Fix wrong placeholder index in error message packInaccessible
JGitText: Remove unused externalized strings
RepoText: Remove unused externalized string
CLI: Remove unused externalized strings
Change-Id: Iea37a8e39e9d4872dc607c9222dcf191ce4e4757
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Bazelisk is utility to switch to the right Bazel version, that we
used to have with Buck build tool: [1].
Bazelisk will download the right Bazel version only once and will use
it in subsequent calls:
$ bazelisk build :release
2019/06/06 16:22:15 Downloading \
https://releases.bazel.build/0.26.1/release/bazel-0.26.1-linux-x86_64...
Bazelisk is storing the binaries in user's cache directory: [2], e.g.
on Linux OS:
$ ls -1 ~/.cache/bazelisk/bin
bazel-0.26.1-linux-x86_64
* [1] https://github.com/bazelbuild/bazelisk
* [2] https://golang.org/pkg/os/#UserCacheDir
Change-Id: Ia9180fb75f8cc17a0a0232622cf33a13bfad6b60
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Check the bazel version using the checker from bazel_skylib, and
require at least version 0.17.1 which is the minimum version that
does not suffer from the Java API mismatch issue [1].
The implementation is borrowed from the Gerrit project.
[1] https://github.com/bazelbuild/bazel/issues/6027
Change-Id: I224250088a1f5072fcaa3ec81228f4d2cb8cb389
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-5.1:
Prepare 5.1.12-SNAPSHOT builds
JGit v5.1.11.201909031202-r
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Iac5dc9683cea97db04d20f27c10f2e103d3ae7b5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.0:
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: I172136a031ff0730e575327cafb3527c9650a71d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.11:
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Ifb6a4dbea2f48fd2ffa66eb737d61920aefedfbd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.10:
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: If672b4f0c350f4e8ff7e1e706485cffd8137236d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.9:
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Id5bf44645655fca40ad22bb1f1ad20a7c2e8f6db
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The latest revision includes various fixes to allow the build
to work with recent versions of Bazel.
Change-Id: I72c100b99762010946d9b2784286af560bbdf185
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
FileTreeIteratorWithTimeControl was deleted in a024759, but was
not removed from the BUILD file, thus causing the bazel build to
fail.
Change-Id: I892c0ffcac947298d0d6009374ee2c5d9afefb66
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit e54fde8616)
PackedBatchRefUpdate was creating a new packed-refs list that was
potentially unsorted. This would be papered over when the list was
read back from disk in parsePackedRef, which detects unsorted ref
lists on reading, and sorts them. However, the BatchRefUpdate also
installed the new (unsorted) list in-memory in
RefDirectory#packedRefs.
With the timestamp granularity code committed to stable-5.1, we can
more often accurately decide that the packed-refs file is clean, and
will return the erroneous unsorted data more often. Unluckily timed
delays also cause the file to be clean, hence this problem was
exacerbated under load.
The symptom is that refs added by a BatchRefUpdate would stop being
visible directly after they were added. In particular, the Gerrit
integration tests uses BatchRefUpdate in its setup for creating the
Admin group, and then tries to read it out directly afterward.
The tests recreates one failure case. A better approach would be to
revise RefList.Builder, so it detects out-of-order lists and
automatically sorts them.
Fixes https://bugs.eclipse.org/bugs/show_bug.cgi?id=548716 and
https://bugs.chromium.org/p/gerrit/issues/detail?id=11373.
Bug: 548716
Change-Id: I613c8059964513ce2370543620725b540b3cb6d1
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The only usage of this test iterator was removed in df637928d. Hence
delete this iterator and associated test.
Change-Id: I47710133ec3edc675c21db210960c024982668c6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit a024759cf5)
This test case assumed file system timestamp resolution of 1 second. On
filesystems with a finer resolution this test fails since the index
entry is only smudged if the file index entry's lastModified and the
lastModified of the git index itself are within the same filesystem
timer tick. Fix this by ensuring that these timestamps are identical
which should work for any filesystem timer resolution.
Bug: 548188
Change-Id: Id84d59e1cfeb48fa008f8f27f2f892c4f73985de
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
(cherry picked from commit 1159f9dd7c)
Matthias Sohn
David Pursehouse
Saša Živkov
Thomas Wolf
David Ostrovsky
Han-Wen Nienhuys