Every caller would need to check if bitmaps are available in the repo to
instantiate a reachability checker.
Offer a method to create the reachability checker in the walk: the
caller has already a walk over the repo, and the walk has all the
information required.
This allows us to make the implementation classes package-private.
Change-Id: I355e47486fcd9d55baa7cb5700ec08dcc230eea5
Signed-off-by: Ivan Frade <ifrade@google.com>
* master:
ReachabilityCheckers: @since 5.4
Update Orbit to S20190521195709 for 2019-06 M3
Tune max heap size for tests
GPG: check that the key found is a signing key
GPG: use key fingerprint suffix to compare id for signing key
Recognize ReachabilityCheckerTestCase as helper
UploadPack: restore inadvertently deleted line
UploadPack: Use reachability checker to validate non-advertised wants
Bazel: Simplify names of bouncy castle libraries
Convert to lambda or member reference
BitmappedReachabilityChecker: Reachability check using bitmaps
BitmapCalculator: Get the reachability bitmap of a commit
ReachabilityChecker: Default implementation with a RevWalk
Change-Id: I323c460853230abcffa94c4f34f15dbaff8c330c
The new ReachabilityChecker interface and its implementations are marked
as @since 5.5, but they will make it to the 5.4 release.
Change-Id: I88c31b3300ccf35d18c35faddb2517f0a57bdcfd
Signed-off-by: Ivan Frade <ifrade@google.com>
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].
Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test
Decrease it to 768m for
- org.eclipse.jgit.test
[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console
Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).
Some filesystems expose unique file identifiers, e.g. inodes in Posix
filesystems which are named filekeys in Java's BasicFileAttributes. Use
them as another means to detect file modifications based on stat
information.
Running git gc on a repository yields a new packfile with the same id as
a packfile which existed before the gc if these packfiles contain the
same set of objects. The content of the old and the new packfile might
differ if a different PackConfig was used when writing the packfile.
Considering filekeys in FileSnapshot may help to detect such packfile
modifications.
Bug: 546891
Change-Id: I711a80328c55e1a31171d540880b8e80ec1fe095
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
FileSnapshot.notRacyClean() assumed a worst case filesystem timestamp
resolution of 2.5 sec (FAT has a resolution of 2 sec). Instead measure
timestamp resolution to avoid unnecessary IO caused by false positives
in detecting the racy git problem caused by finite filesystem timestamp
resolution [1].
Cache the measured resolution per FileStore since timestamp resolution
depends on the respective filesystem type. If timestamp resolution
cannot be measured or fails due to an exception fallback to the worst
case FAT timestamp resolution and avoid caching this value.
Add a 10% safety margin in FileSnapshot.notRacyClean(), though running
FsTest.testFsTimestampResolution() 1000 times which is not using a
safety margin didn't fail on Mac using APFS and Java 8, 11, 12.
Measured Java file timestamp resolution: [2]
[1] https://github.com/git/git/blob/master/Documentation/technical/racy-git.txt
[2] https://docs.google.com/spreadsheets/d/1imy0y6WmRqBf0kjCxzxj2X7M50eIVfa7oaUIzEOHmjo
Bug: 546891
Change-Id: I493f3b57b6b306285ffa7d392339d253e5966ab8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* fix equals() and hashCode() methods to consider size
* fix toString() to show size
Change-Id: I5485db55eda5110121efd65d86c7166b3b2e93d0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Check whether the value of the git config user.signingKey is a suffix
of the full fingerprint of the key. This was already used for finding
keys in secring.gpg, but not in pubring.kbx. This mechanism allows a
user to use any unique suffix to identify keys; to avoid needless
collisions it's recommended to use at least the last 16 characters of
the hex representation of the fingerprint, which is the key id.[1]
[1] https://tools.ietf.org/html/rfc4880#section-12.2
Bug: 545673
Change-Id: If6fb4879502b6ee4b8c26c21b2714aeac4e4670c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
4e196faa1b ("ReachabilityChecker: Default implementation with a
RevWalk", 2019-05-15) added ReachabilityCheckerTestCase but did not add
it as a test helper in the corresponding BUILD file, making tests no
longer runnable with Bazel. Resolve this issue.
Change-Id: Iccc00b0d169dbaa137e130ce2ddd1b7669960b52
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
In 7b96bd812e ("UploadPack: Use reachability checker to validate
non-advertised wants", 2019-05-16), a "walk.setRetainBody(false);"
statement was inadvertently deleted. (An earlier version of this commit
had this line in another part of the code and a review comment suggested
to move it back here; the line was then deleted from the other part of
the code but not readded.) Restore this line.
Change-Id: I96ff6106ba9e4eef429388c83e898b3363295f69
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
* changes:
UploadPack: Use reachability checker to validate non-advertised wants
BitmappedReachabilityChecker: Reachability check using bitmaps
BitmapCalculator: Get the reachability bitmap of a commit
ReachabilityChecker: Default implementation with a RevWalk
In "Reachable commit" request validators, we need to check that a "want"
in the request, that hasn't been advertised, is reachable from the refs
visible to the user.
Current code has intermixed the translation of ObjectIds to RevCommits
(and its error handling) with the actual walk, with the delegation to
bitmaps in restricted circunstances.
Refactor the code to make it "flatter" and more readable. Move ObjectIds
to RevCommits translation to its own functions. Use the reachability
checker instead of a newly defined walk.
Before the non-advertised wants were validated with bitmaps only if any
"want" refered to an non-commit. Now they will be validated with bitmaps
also if the "wants" refer all to commits.
Change-Id: Ib925a48cde89672b07a88bba4e24d0457546baff
Signed-off-by: Ivan Frade <ifrade@google.com>
Name bouncy castle libraries consistently. This also allows to use
local jgit tree as dependency in gerrit.
Change-Id: Ie0a995a70ecee907d25b3cdda9034a25c0290efe
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
The "basic" reachability check walks the graph starting from the tips
marking things as "uninteresting". If the target commit is marked as
"uninteresting" it was reached; it is reachable from those tips.
This requires a lot of walking and can be solved directly with bitmaps.
Most of the time the bitmaps are already calculated or a short walk
away.
This should improve the performance of reachability checks, for example
in Gitiles.
Change-Id: I83d33271f58d95d2dc9ed151967b3eda513c99f7
Signed-off-by: Ivan Frade <ifrade@google.com>
To make reachability checks with bitmaps, we need to get the
reachability bitmap of a commit, which is not always precalculated.
There is already a class returning such bitmap (BitmapWalker) but it
does too much unnecessary work: it calculates ALL reachable objects from
a commit (i.e. including trees and blobs), when for reachability the
commits are just enough.
Introduce BitmapCalculator to get the bitmap of a commit: either because
it is precalculated or generating it with a walk only over commits.
Change-Id: Ibb6c78affe9eeaf1fa362a06daf4fd2d91c1caea
Signed-off-by: Ivan Frade <ifrade@google.com>
It is common to check if a certain commit is reachable from some
starting points. For example gitiles does it to check if a commit
is visible to a user based on its permissions.
Offer this functionality in JGit.
Split the interface as the next commit will introduce an implementation
using bitmap indices.
Change-Id: I0933b305c8d734f7a64502910ff4d9ef4fc92ae1
Signed-off-by: Ivan Frade <ifrade@google.com>
Add encrypted ed25519 keys in the tests; sshd 2.2.0 can finally
decrypt encrypted new-style OpenSSH key files. (Needs the "unlimited
strength" JCE, which is the default since Java 8u161. On older JREs,
users should install the policy files available from Oracle.)
The "expensive" key added has been generated with OpenSSH's
ssh-keygen -t ed25519 -a 256, i.e., with 256 bcrypt KDF rounds
instead of the default 16. On my machine it takes about 2sec to
decrypt.
Bug: 541703
Change-Id: Id3872ca2fd75d8f009cbc932eeb6357d3d1f267c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Update target platforms, maven and bazel builds to use sshd 2.2.0.
Adapt internal classes to changed sshd interfaces and remove previous
work-arounds for asking repeatedly for key passwords and for loading
keys lazily; both are now done by sshd.
CQ: 19034
CQ: 19035
Bug: 541425
Change-Id: I85e1df6ebb8a94953a912d9b2b8a7b5bdfbd608a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
- ObjectWalk.getTreeDepth() returns int hence there is no need to use
long depths in the lowestDepthVisited map.
- Also fix boxing warnings introduced in 0a15cb3a.
Change-Id: I6d73b6f41d5d20975d02f376c8588e411eaff0ec
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Replace:
in the "core section"
in the "diff section"
in the "merge section"
with:
in the "core" section
in the "diff" section
in the "merge" section
Change-Id: Ided7bf73e9c8aae4fc4e43d5d5b9f6d7e3066f0a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-5.3:
Prepare 5.3.2-SNAPSHOT builds
JGit v5.3.1.201904271842-r
Prepare 5.2.3-SNAPSHOT builds
JGit v5.2.2.201904231744-r
Revert 4678f4b and provide another solution for bug 467631
Apache MINA sshd: make sendKexInit() work also for re-keying
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I8a8671f7767444a77b809bd66a27d776c8332736
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.2:
Prepare 5.2.3-SNAPSHOT builds
JGit v5.2.2.201904231744-r
Revert 4678f4b and provide another solution for bug 467631
Apache MINA sshd: make sendKexInit() work also for re-keying
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: Ie7e572ac7e346f21fe0c387d7448be168a9c127a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Same version as used in the latest Eclipse platform. Updating the
JGit target platform results in the new JSch version being included
in the JGit feature in the p2 repo, and thus ultimately in the EGit
update site.
CQ: 19588
Bug: 546130
Change-Id: Id7e4318096f6f29dc4573e15542abdf54582af34
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Making gitlinks and folders match in a tree walk was the wrong
approach to fix bug 467631. The problem is that in such a conflict
the tree walk may then not descend into the folder.
Revert the changes to Paths.java and PathsTest.java from commit
4678f4b. Instead test for the problem case from bug 467631 explicitly
in IndexDiff. Add Daniel's test case from bug 545162, and add yet
another test case for DiffEntry.scan() that covers the problem
originally reported in bug 545162.
Bug: 545162
Change-Id: Ie2214c5d5ee32ac6596b621f0f1c7b86d38fa9b7
Also-by: Daniel Veihelmann <daniel.veihelmann@gmail.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The message delaying for the initial SSH messages (client
identification followed by the initial key exchange request)
was broken. sendKexInit() is _also_ called when a new key exchange
is requested. We inadvertently also re-sent the client identification
at that point, which is wrong and makes the server terminate the
connection.
Re-keying occurs from time to time during an SSH connection depending
on time, the number of messages (packets/blocks) exchanged, or the
amount of data exchanged. The net result was that for large
repositories data-intensive operations failed on the first re-keying.
Change the initial message delay such that the two messages for the
client identification and the initial key exchange can be buffered
individually while the proxy protocol is still in progress. The
AbstractClientProxyConnector can now buffer several commands, which
should also resolve bug 544715.
Bug: 545920
Change-Id: If09ee963a439b39098a0f52a1510237b428df8dd
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The callback interface is invoked after object upload was
completed.
Change-Id: I705d8becaf4f35188caf098aa75cff8963d64a60
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Ivan Frade
Matthias Sohn
Thomas Wolf
Jonathan Nieder
Jonathan Tan
David Ostrovsky
Carsten Hammer
David Pursehouse
Michael Keppler