When fetching from a shallow clone, the client sends "have" lines
to tell the server about objects it already has and "shallow" lines
to tell where its local history terminates. In some circumstances,
the server fails to honor the shallow lines and fails to return
objects that the client needs.
UploadPack passes the "have" lines to PackWriter so PackWriter can
omit them from the generated pack. UploadPack processes "shallow"
lines by calling RevWalk.assumeShallow() with the set of shallow
commits. RevWalk creates and caches RevCommits for these shallow
commits, clearing out their parents. That way, walks correctly
terminate at the shallow commits instead of assuming the client has
history going back behind them. UploadPack converts its RevWalk to an
ObjectWalk, maintaining the cached RevCommits, and passes it to
PackWriter.
Unfortunately, to support shallow fetches the PackWriter does the
following:
if (shallowPack && !(walk instanceof DepthWalk.ObjectWalk))
walk = new DepthWalk.ObjectWalk(reader, depth);
That is, when the client sends a "deepen" line (fetch --depth=<n>)
and the caller has not passed in a DepthWalk.ObjectWalk, PackWriter
throws away the RevWalk that was passed in and makes a new one. The
cleared parent lists prepared by RevWalk.assumeShallow() are lost.
Fortunately UploadPack intends to pass in a DepthWalk.ObjectWalk.
It tries to create it by calling toObjectWalkWithSameObjects() on
a DepthWalk.RevWalk. But it doesn't work: because DepthWalk.RevWalk
does not override the standard RevWalk#toObjectWalkWithSameObjects
implementation, the result is a plain ObjectWalk instead of an
instance of DepthWalk.ObjectWalk.
The result is that the "shallow" information is thrown away and
objects reachable from the shallow commits can be omitted from the
pack sent when fetching with --depth from a shallow clone.
Multiple factors collude to limit the circumstances under which this
bug can be observed:
1. Commits with depth != 0 don't enter DepthGenerator's pending queue.
That means a "have" cannot have any effect on DepthGenerator unless
it is also a "want".
2. DepthGenerator#next() doesn't call carryFlagsImpl(), so the
uninteresting flag is not propagated to ancestors there even if a
"have" is also a "want".
3. JGit treats a depth of 1 as "1 past the wants".
Because of (2), the only place the UNINTERESTING flag can leak to a
shallow commit's parents is in the carryFlags() call from
markUninteresting(). carryFlags() only traverses commits that have
already been parsed: commits yet to be parsed are supposed to inherit
correct flags from their parent in PendingGenerator#next (which
doesn't happen here --- that is (2)). So the list of commits that have
already been parsed becomes relevant.
When we hit the markUninteresting() call, all "want"s, "have"s, and
commits to be unshallowed have been parsed. carryFlags() only
affects the parsed commits. If the "want" is a direct parent of a
"have", then it carryFlags() marks it as uninteresting. If the "have"
was also a "shallow", then its parent pointer should have been null
and the "want" shouldn't have been marked, so we see the bug. If the
"want" is a more distant ancestor then (2) keeps the uninteresting
state from propagating to the "want" and we don't see the bug. If the
"shallow" is not also a "have" then the shallow commit isn't parsed
so (2) keeps the uninteresting state from propagating to the "want
so we don't see the bug.
Here is a reproduction case (time flowing left to right, arrows
pointing to parents). "C" must be a commit that the client
reports as a "have" during negotiation. That can only happen if the
server reports it as an existing branch or tag in the first round of
negotiation:
A <-- B <-- C <-- D
First do
git clone --depth 1 <repo>
which yields D as a "have" and C as a "shallow" commit. Then try
git fetch --depth 1 <repo> B:refs/heads/B
Negotiation sets up: have D, shallow C, have C, want B.
But due to this bug B is marked as uninteresting and is not sent.
Change-Id: I6e14b57b2f85e52d28cdcf356df647870f475440
Signed-off-by: Terry Parker <tparker@google.com>
Gerrit's superproject subscription feature uses RefSpecs to formalize
the ACLs of when the superproject subscription feature is allowed.
As this is a slightly different use case than describing a local/remote
pair of refs, we need to be more permissive. Specifically we want to allow:
refs/heads/*
refs/heads/*:refs/heads/master
refs/heads/master:refs/heads/*
Introduce a new constructor, that allows constructing these RefSpecs.
Change-Id: I46c0bea9d876e61eb2c8d50f404b905792bc72b3
Signed-off-by: Stefan Beller <sbeller@google.com>
Example usage:
$ ./jgit push \
--push-option "Reviewer=j.doe@example.org" \
--push-option "<arbitrary string>" \
origin HEAD:refs/for/master
Stefan Beller has also made an equivalent change to CGit:
http://thread.gmane.org/gmane.comp.version-control.git/299872
Change-Id: I6797e50681054dce3bd179e80b731aef5e200d77
Signed-off-by: Dan Wang <dwwang@google.com>
We observe in Gerrit 2.12 that useCnt can become negative in rare cases.
Log this to help finding the bug.
Change-Id: Ie91c7f9d190a5d7cf4733d4bf84124d119ca20f7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
1f86350 added initial support for include.path. Relative path and path
with tilde are not yet supported but config load was failing if one of
those 2 unsupported options was encountered. Another problem was that
config load was failing if the include.path file did not exist.
Change the behavior to be consistent with native git. Ignore unsupported
or nonexistent include.path.
Bug: 495505
Bug: 496732
Change-Id: I7285d0e7abb6389ba6983e9c46021bea4344af68
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
File.listFiles() returns null if the File is not a directory, improve
validation of directory and gitDir to fix this.
Change-Id: I763d08835faf96a0beb8e706992df0908526bd2c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
As per [1], but limited to absolute paths indeed. No support yet for
tilde or $HOME expansion. Support for the --[no-]includes options
([1]) is not part of this commit scope either, but those options'
defaults are in effect as described in [1].
[1] https://git-scm.com/docs/git-config
Included path can be a config file that includes other path-s in turn.
An exception is thrown if too many recursions (circular includes)
happen because of ill-specified config files.
Change-Id: I700bd7b7e1625eb7de0180f220c707d8e7b0930b
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Repurpose RefDatabase#performsAtomicTransactions() slightly, to
indicate that the backend _supports_ atomic transactions, rather than
the current definition, which is that the backend always _uses_ atomic
transactions regardless of whether or not the caller actually wants
them. Allow BatchRefUpdate callers to turn off atomic transactions by
calling setAtomic(false). Defaulting to true means this is backwards
compatible.
Change-Id: I6df78d7df65ab147b4cce7764bd3101db985491c
Experimental flag to turn on the KetchLeader within this daemon JVM.
This is a manually elected leader process, set from the command line.
Remote followers for each repository are configured per-repository
using remote sections with ketch-type = FULL. For example:
Manually elected leader's $GIT_DIR/config:
[ketch]
name = A
[remote "A"]
ketch-type = FULL
[remote "B"]
url = git://127.0.0.1:9421/sample.git
ketch-type = FULL
[remote "C"]
url = git://127.0.0.1:9422/sample.git
ketch-type = FULL
Replica B and C daemons:
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9421 \
--base-path=$HOME/ketch_test/follower_one \
$HOME/ketch_test/follower_one &
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9422 \
--base-path=$HOME/ketch_test/follower_two \
$HOME/ketch_test/follower_two &
Change-Id: I165f85970a77e16b5263115290d685d8a00566f5
Git Ketch is a multi-master Git repository management system. Writes
are successful only if a majority of participant servers agree. Acked
writes are durable against server failures as a majority of the
participants store all required objects.
Git Ketch is modeled on the Raft Consensus Algorithm[1]. A ketch
sailing vessel is faster and more nimble than a raft. It can also
carry more source codes.
Git Ketch front-loads replication costs, which vaguely resembles a
ketch sailing vessel's distinguishing feature of the main mast on the
front of the ship.
[1] https://raft.github.io/
Change-Id: Ib378dab068961fc7de624cd96030266660b64fb4
Accept some of the same section keys that fsck does in git-core,
allowing repositories to skip over specific kinds of acceptable
broken objects, e.g.:
[fsck]
duplicateEntries = ignore
zeroPaddedFilemode = ignore
The zeroPaddedFilemode = ignore is a synonym for the JGit specific
allowLeadingZeroFileMode = true. Only accept the JGit key if git-core
key was not specified.
Change-Id: Idaed9310e2a5ce5511670ead1aaea2b30aac903c
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.
Change-Id: I24c8a600ec962d61d5f40abf73eac4203e115240
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
This should mirror the behavior of `git push --atomic` where the
client asks the server to apply all-or-nothing. Some JGit servers
already support this based on a custom DFS backend. InMemoryRepository
is extended to support atomic push for unit testing purposes.
Local disk server side support inside of JGit is a more complex animal
due to the excessive amount of file locking required to protect every
reference as a loose reference.
Change-Id: I15083fbe48447678e034afeffb4639572a32f50c
When filters are defined for certain paths in gitattributes make
sure that clean filters are processed when adding new content to the
object database.
Change-Id: Iffd72914cec5b434ba4d0de232e285b7492db868
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
org.eclipse.jgit.lib.Repository class is an example of the API which
should be written with Java 8 java.util.Optional<T> type. Unfortunately
this API is already released and widely used. The good clients are
currently doing their best with checking return values for null and bad
clients do not know how bad their code is.
I've tried not to change any logic and to be as less intrusive as
possible. Most of the JGit code was well prepared to this, only few
classes needed some smaller fixes.
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.
Because annotating getDirectory() and getFS() as Nullable would cause
lot of additional changes in JGit and EGit they are postponed.
Change-Id: Ie8369d2c9c5fac5ce83b3b1b9bc217d7b55502a3
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
File, FileInputStream and friends may throw FileNotFoundException even
if the file is existing e.g. when file permissions don't allow to access
the file content. In most cases this is a severe error we should not
suppress hence rethrow the FileNotFoundException in this case.
This may also fix bug 451508.
Bug: 451508
Change-Id: If4a94217fb5b7cfd4c04d881902f3e86193c7008
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If the index file exists but can't be read for example because of wrong
filesystem permissions we should throw a specific exception. This allows
EGit to handle this error situation.
Bug: 482607
Change-Id: I50bfcb719c45caac3cb5550a8b16307c2ea9def4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
See previous attempt: https://git.eclipse.org/r/#/c/16674/
Here we preserve as much of JetS3t mode as possible
while allowing to use new Java 8+ PBE algorithms
such as PBEWithHmacSHA512AndAES_256
Summary of changes:
* change pom.xml to control long tests
* add WalkEncryptionTest.launch to run long tests
* add AmazonS3.Keys to to normalize use of constants
* change WalkEncryption to support AES in JetS3t mode
* add WalkEncryptionTest to test remote encryption pipeline
* add support for CI configuration for live Amazon S3 testing
* add log4j based logging for tests in both Eclipse and Maven build
To test locally, check out the review branch, then:
* create amazon test configuration file
* located your home dir: ${user.home}
* named jgit-s3-config.properties
* file format follows AmazonS3 connection settings file:
accesskey = your-amazon-access-key
secretkey = your-amazon-secret-key
test.bucket = your-bucket-for-testing
* finally:
* run in Eclipse: WalkEncryptionTest.launch
* or
* run in Shell: mvn test --define test=WalkEncryptionTest
Change-Id: I6f455fd9fb4eac261ca73d0bec6a4e7dae9f2e91
Signed-off-by: Andrei Pozolotin <andrei.pozolotin@gmail.com>
Since 4.0 we require Java 7 so there is no longer a need to override the
following methods in FS_POSIX, FS_Win32, FS_Win32_Cygwin
- lastModified()
- setLastModified()
- length()
- isSymlink()
- exists()
- isDirectory()
- isFile()
- isHidden()
Hence implement these methods in FS and remove overrides in subclasses.
Change-Id: I5dbde6ec806c66c86ac542978918361461021294
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If a client mistakenly tries to send a tag object as a shallow line
JGit blindly assumes this is a commit and tries to parse the tag
buffer using the commit parser. This can cause an obtuse error like:
InvalidObjectIdException: Invalid id: t c0ff331234...
The "t" comes from the "object c0ff331234..." line of the tag tring
to be parsed as though it where the "tree" line of a commit.
Run any client supplied shallow lines through the RevWalk to lookup
the object types. Fail fast with a protocol exception if any of them
are non-commit.
Skip objects not known to this repository. This matches behavior
with git-core's upload-pack, which sliently skips over any shallow
line object named by the client but not known by the server.
Change-Id: Ic6c57a90a42813164ce65c2244705fc42e84d700
On a local filesystem the packed-refs file will be orphaned if it is
replaced by another client while the current client is reading the old
one. However, since NFS servers do not keep track of open files, instead
of orphaning the old packed-refs file, such a replacement will cause the
old file to be garbage collected instead. A stale file handle exception
will be raised on NFS servers if the file is garbage collected (deleted)
on the server while it is being read. Since we no longer have access to
the old file in these cases, the previous code would just fail. However,
in these cases, reopening the file and rereading it will succeed (since
it will reopen the new replacement file). So retrying the read is a
viable strategy to deal with stale file handles on the packed-refs file,
implement such a strategy.
Since it is possible that the packed-refs file could be replaced again
while rereading it (multiple consecutive updates can easily occur with
ref deletions), loop on stale file handle exceptions, up to 5 extra
times, trying to read the packed-refs file again, until we either read
the new file, or find that the file no longer exists. The limit of 5 is
arbitrary, and provides a safe upper bounds to prevent infinite loops
consuming resources in a potential unforeseen persistent error
condition.
Change-Id: I085c472bafa6e2f32f610a33ddc8368bb4ab1814
Signed-off-by: Martin Fick<mfick@codeaurora.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Catch unexpected PatternSyntaxException and convert it to
InvalidPatternException. Log such errors, do not silently ignore them.
Bug: 463581
Change-Id: Id0936d9816769ec0cfae1898beda0f7a3c146e67
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Inspired by a proposal from gitolite[1], where we store a file in
a tree for each ref name, and the contents of the file is the latest
push cert to affect that ref.
The main modification from that proposal (other than lacking the
out-of-git batching) is to append "@{cert}" to filenames, which allows
storing certificates for both refs/foo and refs/foo/bar. Those
refnames cannot coexist at the same time in a repository, but we do
not want to discard the push certificate responsible for deleting the
ref, which we would have to do if refs/foo in the push cert tree
changed from a tree to a blob.
The "@{cert}" syntax is at least somewhat consistent with
gitrevisions(7) wherein @{...} describe operators on ref names.
As we cannot (currently) atomically update the push cert ref with the
refs that were updated, this operation is inherently racy. Kick the can
down the road by pushing this burden on callers.
[1] cf062b8bb6/contrib/hooks/repo-specific/save-push-signatures
Change-Id: Id3eb32416f969fba4b5e4d9c4b47053c564b0ccd
In most texts we use "cannot" hence instead of escaping the apostroph in
"can't" use "cannot".
Bug: 471796
Change-Id: Icda5b4db38076789d06498428909306aef3cb68b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The invalidId message in JGitText and the asAscii bad id both contain a
colon, so the resulting message would say
Invalid id: : a78987c98798ufa
Fix it by keeping the colon in the translated message and not adding
another colon programmatically.
Noticed by code inspection.
Change-Id: I13972eebde27a4128828e6c64517666f0ba6288b
Signed-off-by: Jonathan Nieder <jrn@google.com>
- Consistently return structured data, such as actual ReceiveCommands,
which is more useful for callers that are doing things other than
verifying the signature, e.g. recording the set of commands.
- Store the certificate version field, as this is required to be part
of the signed payload.
- Add a toText() method to recreate the actual payload for signature
verification. This requires keeping track of the un-chomped command
strings from the original protocol stream.
- Separate the parser from the certificate itself, so the actual
PushCertificate object can be immutable. Make a fair attempt at deep
immutability, but this is not possible with the current mutable
ReceiveCommand structure.
- Use more detailed error messages that don't involve NON-NLS strings.
- Document null return values more thoroughly. Instead of having the
undocumented behavior of throwing NPE from certain methods if they
are not first guarded by enabled(), eliminate enabled() and return
null from those methods.
- Add tests for parsing a push cert from a section of pkt-line stream
using a real live stream captured with Wireshark (which, it should
be noted, uncovered several simply incorrect statements in C git's
Documentation/technical/pack-protocol.txt).
This is a slightly breaking API change to classes that were
technically public and technically released in 4.0. However, it is
highly unlikely that people were actually depending on public
behavior, since there were no public methods to create
PushCertificates with anything other than null field values, or a
PushCertificateParser that did anything other than infinite loop or
throw exceptions when reading.
Change-Id: I5382193347a8eb1811032d9b32af9651871372d0
This exception's detail message states
Service not permitted
and according to the Javadoc it indicates that the current user does not
have access to the service. In practice, though, callers handle this
exception by presenting a '401 Unauthorized' response to the client,
meaning that the user is unauthenticated and should authenticate.
Clarify the documentation and detail message to match the practice.
The exception message is not used anywhere except logs. No
client-visible effect intended.
Change-Id: I2c6be9cb74af932f0dcb121a381a64f2ad876766
Signed-off-by: Jonathan Nieder <jrn@google.com>
- throw an API exception instead of an internal exception to allow
applications to handle this problem
- improve error message to give hints about possible root causes
Bug: 464660
Change-Id: Ib7d18bb2eeeac0fc218daea375b290ea5034bda1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The primary goal is to improve exception readability. Since this is a
standalone thread, just logging the stack trace of the caught
exception is not very useful:
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
at java.io.BufferedInputStream.read(BufferedInputStream.java:258)
at org.eclipse.jgit.util.FS$2.run(FS.java:451)
Providing a named class eliminates the "FS$2", and including the
command name provides a little more context in the error message.
A future improvement might include the stack trace that created the
GobblerThread as well.
Change-Id: Ibf16d15b47a85b6f41844a177e398c2fc94f27b0
This error happens on nfs file system when you try to read a file that
was deleted or replaced.
When the error happens because the file was deleted, removing it from
the list is the proper way to handle the error, same use case as
FileNotFoundException. When the error happens because the file was
replaced, removing the file from the list will cause the file to be
re-read so it will get the latest version of the file.
Bug: 462868
Change-Id: I368af61a6cf73706601a3e4df4ef24f0aa0465c5
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
This reverts commit 6bc48cdc62.
Until git v1.7.10.2~29^2~1 (builtin/merge.c: reduce parents early,
2012-04-17), C git merge would make merge commits with duplicate parents
when asked to with a series of commands like the following:
git checkout origin/master
git merge --no-ff origin/master
Nowadays "git merge" removes redundant parents more aggressively
(whenever one parent is an ancestor of another and not just when
duplicates exist) but merges with duplicate parents are still permitted
and can be created with git fast-import or git commit-tree and history
viewers need to be able to cope with them.
CommitBuilder is an interface analagous to commit-tree, so it should
allow duplicate parents. (That said, an option to automatically remove
redundant parents would be useful.)
Reported-by: Dave Borowitz <dborowitz@google.com>
Change-Id: Ia682238397eb1de8541802210fa875fdd50f62f0
Signed-off-by: Jonathan Nieder <jrn@google.com>
When setting the parents of a commit with setParentIds() or
addParentId() it should be checked that we don't have duplicate parents.
An IllegalArgumentException should be thrown in this case.
Change-Id: I9fa9f31149b7732071b304bca232f037146de454
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Hooks are now obtained via a convenient API like git commands, and
callers don't have to check for their existence.
The pre-commit hook has been updated accordingly.
Change-Id: I3383ffb10e2f3b588d7367b9139b606ec7f62758
Signed-off-by: Laurent Delaigue <laurent.delaigue@obeo.fr>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This allows for testing arbitrary sets of push/fetch hooks (e.g.
PreReceiveHook) without depending on either an external protocol (e.g.
HTTP) or the local filesystem.
Change-Id: I4ba2fff9c8a484f990dea05e14b0772deddb7411
Push certificates ("git push --signed") have been part of
git-core since version 2.2.0 (released Nov 26 2014). We also
want to support that feature.
This is not complete and is lacking the actual functionality
to validate the signature for now.
Change-Id: I249869cadb2d55aef016371b9311b8583591b9cf
Signed-off-by: Stefan Beller <sbeller@google.com>
Mention packfile path in exceptions thrown when we detect that a
packfile is invalid and make excplicit that corrupt packs are removed
from the pack list.
Change-Id: I454ada5f8e69307d3f34d1c1b8f3cb87607ddf35
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Introduce support for the pre-commit hook into JGit, along with the
--no-verify commit command option to bypass it when rebasing /
cherry-picking.
Change-Id: If86df98577fa56c5c03d783579c895a38bee9d18
Signed-off-by: Laurent Goubet <laurent.goubet@obeo.fr>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This introduces the background plumbing necessary to run git hooks from
JGit. This implementation will be OS-dependent as it aims to be
compatible with existing hooks, mostly written in Shell. It is
compatible with unix systems and windows as long as an Unix emulator
such as Cygwin is in its PATH.
Change-Id: I1f82a5205138fd8032614dd5b52aef14e02238ed
Signed-off-by: Laurent Goubet <laurent.goubet@obeo.fr>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If accessing a pack throws FileNotFoundException the pack was deleted
and we need to remove it from the pack list. This can be caused e.g. by
git gc.
Change-Id: I5d10f87f364dadbbdbfb61b6b2cbdee9c7457f3d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If we hit a corrupt object or invalid pack remove the pack from the pack
list. Other IOException could be transient hence we should not remove
the pack from the list to avoid the problem reported on the Gerrit list
[1]. It looks like in the reported case the pack was removed from the
pack list causing MissingObjectExceptions which disappear when the
server is restarted.
[1] https://groups.google.com/forum/#!topic/repo-discuss/Qdmbl-YZ4NU
Change-Id: I331626110d54b190e46cddc2c40f29ddeb9613cd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Terry Parker
Dave Borowitz
Stefan Beller
Dan Wang
Matthias Sohn
Hugo Arès
Michael Keppler
Marco Miller
Shawn Pearce
Andrey Loskutov
Christian Halstrick
Andrei Pozolotin
Martin Fick
Jonathan Nieder
Laurent Delaigue
Laurent Goubet