* stable-4.8:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: I6ab2b951dd94a9fc1c4f5283847a3e2ec37d0895
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.7:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: Iadc8275fbaa3d6f7d08a96ab66d49f392f6aab78
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Remove completely the empty directories under refs/<namespace>
including the first level partition of the changes, when they are
completely empty.
Bug: 536777
Change-Id: I88304d34cc42435919c2d1480258684d993dfdca
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Get the full IOException of the reason why a directory
cannot be removed during GC.
Change-Id: Ia555bce009fa48087a73d677f1ce3b9c0b685b57
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When processing a fetch using protocol v2, UploadPack#fetchV2 sends an
extraneous flush pkt when also sending a packfile (#sendPack sending its
own flush pkt). Update that method to only send the flush pkt if the
packfile is not being sent.
Change-Id: I7117a264bccd2d7f3a048645fcb8425a9d78d526
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Any additional statements after the statement that is expected to
throw will never be executed in a passing test. This can lead to
inappropriately passing tests where later incorrect assertions are
skipped by the thrown exception. See [1] for examples.
There are no cases of this in the code base, but by enabling the
checker with severity ERROR we can ensure that we don't overlook
any future misuse of ExpectedException.
[1] http://errorprone.info/bugpattern/ExpectedExceptionChecker
Change-Id: I266d4f73f6075bfc1e39fa3d8aee7dee96db61b9
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
In a0c9016abd ("upload-pack: send refs' objects despite "filter"",
2018-07-09), Git updated the "filter" option in the fetch-pack
upload-pack protocol to not filter objects explicitly specified in
"want" lines, even if they match the criterion of the filter. Update
JGit to match that behavior.
Change-Id: Ia4d74326edb89e61062e397e05483298c50f9232
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Propagating more information from the manifest to the superproject
tends to work out well. These particular settings mean:
Recording the remote branch in .gitmodules allows
- "git submodule update --remote" to fetch the right branch
- Gerrit's submodule subscription feature to subscribe from the
right branch, if requested
Recording groups in .gitattributes allows commands like "git clone
--recurse-submodules=:(attr:platform-linux) $superproject" to clone
the correct set of projects.
Recording the shallow bit in .gitmodules means that "git clone
--recurse-submodules" will use shallow clone in submodules where that
was requested in the manifest.
Gerrit's supermanifest plugin records the remote branch and shallow
bit by default already, and most users of that plugin configure it to
record groups in .gitattributes as well, so for most users this will
be a no-op.
Change-Id: Id2ed47cbca5ce822bde517494673c86ab8c58da9
Signed-off-by: Jonathan Nieder <jrn@google.com>
Don't try to delete folders if the merger chooses THEIRS, but all of
BASE, OURS, and THEIRS contain the folder.
Add a test for rebase with auto-stash and subdirectories that
verifies this case. The needless directory deletion and reporting
such directories in getModifiedFiles() was the root cause of bug
536880.
Note even with this fix, bug 536880 will not be fixed in all cases
yet. There may still be cases where the set of modified files ends
up containing directories. This will be dealt with in EGit where
this set is used. (See https://git.eclipse.org/r/#/c/126242/ .)
Bug: 536880
Change-Id: I62b4571a1c1d4415934a6cb4270e0c8036deb2e9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
RacyGitTests depend on filesystem timer resolution. We wait for a file
system timer tick, remember that time as t1, modify a file and assume
that this file has a lastmodified of t1.
If this assumption is not fulfilled then ignore the test result.
Bug: 526111
Change-Id: Ia38b7d2f99171ef54b8f9fe5be343cf9fcfd3971
04b9f4436 fixed places where compressed bitmaps were holding on to their
full buffers, but missed this StoredBitmap.getBitmap() case where a
bitmap is resonstituted from an xor chain.
Change-Id: I7cf75d9e49c18a1a8a880a4df7e821502edc68a4
Signed-off-by: Terry Parker <tparker@google.com>
Currently SubmoduleAddCommand always uses the path as submodule name.
This patch lets the caller specify a submodule name.
SubmoduleUpdateCommand still does not make use of the submodule name
(see bug 535027) but Git does. To avoid triggering CVE-2018-11235,
do some validation on the name to avoid '..' path components.
[jn: fleshed out commit message, mostly to work around flaky CI]
Change-Id: I6879c043c6d7973556e2080387f23c246e3d76a5
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Make the method names more consistent and their semantics simpler:
hasRef and seekRef to look up a single exact reference by name and
hasRefsByPrefix and seekRefsByPrefix to look up multiple references by
name prefix.
In particular, splitting hasRef into two separate methods for its
different uses makes DfsReftableDatabase.isNameConflicting easier to
follow.
[jn: fleshed out commit message]
Change-Id: I71106068ff3ec4f7e14dd9eb6ee6b5fab8d14d0b
Signed-off-by: Minh Thai <mthai@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Reftable implementation of RefDatabase.getRefsByPrefix() should be
more performant, as references are filtered directly by prefix;
instead of fetching the whole subtree then filter by prefix.
Change-Id: If4f5f8c08285ea1eaec9efb83c3d864cea7a1321
Signed-off-by: Minh Thai <mthai@google.com>
The "Building bitmaps" GC phase fails for large repositories (repos with
10M objects use 1.25MB per uncompressed bitmap, and those with long
histories may build >25k bitmaps). Since these bitmaps xor well against
each other, the actual space needed for each compressed bitmap is
usually no more than a few KB. Calling trim() will ensure we aren't
holding on to excess memory.
Change-Id: I40bf78c730b9f6051da6025f9777ce27220a5b0a
Signed-off-by: Terry Parker <tparker@google.com>
This may be convenient for downstream implementers who require a dummy
StoredConfig implementation, rather than making them reimplement the two
abstract StoredConfig methods.
Change-Id: I2b7bc6250d722c2b95d9f99e4eff1e5bf97cb567
After packaging references, the folders containing these references are
not deleted. In a busy repository, this causes operations to slow down
as traversing the references tree becomes longer.
Delete empty reference folders after the loose references have been
packed.
To avoid deleting a folder that was just created by another concurrent
operation, only delete folders that were not modified in the last 30
seconds.
Signed-off-by: Hector Oswaldo Caballero <hector.caballero@ericsson.com>
Change-Id: Ie79447d6121271cf5e25171be377ea396c7028e0
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Log as warning when an attempt to remove a directory
fails. This helps troubleshooting some bugs like the GC leaving
behind empty directories.
Change-Id: Idb94ce17f8be9668a970c7ecae31436bf434073c
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
* stable-5.0:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: I8e6d0c92144db3bf57ea05f93852ac015458260b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.11:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: I40b7b4b3fd6d53bbad32f8f550e885c0e698faf1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.10:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: I2f02298d0ff7caafeca4020cde4fdfa29a46e585
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.9:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: Ibd8f2a041b0de6e008a1ea84b92823f8cbc6e3d2
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.8:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: Id6a85804695d5dcb32f26ed1d861b7c93577c5e4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.7:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: If17328fbd101d596a8a16d9c4a190e9b6e120902
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Try to give as much information as possible. The connection's
response message might contain additional hints as to why the
connection could not be established.
Bug: 536541
Change-Id: I7230e4e0be9417be8cedeb8aaab35186fcbf00a5
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
It is convenient for TestRepository to return fully parsed
objects from its commit()/tree()/blob() methods, so that test
code doesn't have to remember to parse them before making
assertions about them.
Update TestRepostiory to return fully parsed objects.
Adjust the tests that are affected by this change in behavior.
Change-Id: I09d03d0c80ad22cb7092f4a2eaed99d40a10af63
Signed-off-by: Terry Parker <tparker@google.com>
In a new RevWalk, if the first object parsed is one of the
shallow commits, the following happens:
1) RevCommit.parseCanonical() is called on a new "r1" RevCommit.
2) RevCommit.parseCanonical() immediately calls
RevWalk.initializeShallowCommits().
3) RevWalk.initializeShallowCommits() calls lookupCommit(id),
creating and adding a new "r2" version of this same object and
marking its parents empty.
4) RevCommit.parseCanonical() initializes the "r1" RevCommit's
fields, including the parents.
5) RevCommit.parseCanonical()'s caller uses the "r1" commit that
has parents, losing the fact that it is a shallow commit.
This change passes the current RevCommit as an argument to
RevWalk.initializeShallowCommits() so that method can set its
parents empty rather than creating the duplicate "r2" commit.
Change-Id: I67b79aa2927dd71ac7b0d8f8917f423dcaf08c8a
Signed-off-by: Terry Parker <tparker@google.com>
The previous algorithm selected commits by creating bitmaps at
each branch tip, doing a revwalk to populate each bitmap, and
looping in this way:
1) Select the remaining branch with the most commits (the branch
whose bitmap has the highest cardinality)
2) Select well-spaced bitmaps in that branch
3) Remove commits in the selected branch from the remaining
branch-tip bitmaps
4) Repeat at #1
This algorithm gave good commit selection on all branches but
a more uniform selection on "important" branches, where branch
length is the proxy for "important". However the algorithm
required N bitmaps of size M solely for the purpose of commit
selection, where N is the number of branch tips in the primary
GC pack, and M is the number of objects in the pack.
This new algorithm uses branch modification date as the proxy for
"important" branches, replacing the N*M memory allocation with a
single M-sized bitmap and N revwalks from new branch tips to
shared history (which will be short when there is a lot of shared
history).
GcCommitSelectionTest.testDistributionOnMultipleBranches verifies
that this algorithm still yields good coverage on all branches.
Change-Id: Ib6019b102b67eabb379e6b85623e4b5549590e6e
Signed-off-by: Terry Parker <tparker@google.com>
Test that the commit selection algorithm provides appropriate
coverage across all branches.
Change-Id: I82c71b52068f01e8cef2398aecfec8e144d9a68b
Signed-off-by: Terry Parker <tparker@google.com>
Since no files are actually deleted it makes no sense to fire such an
event.
Change-Id: I66e87afc1791f27fddaa873bafe8bb8b61662535
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-4.11:
Empty merge because of previous merge from stable-4.9 into
stable-5.0 and commits subsequently merged into stable-4.10
and stable-4.11.
Change-Id: I69b8d3e96173ef58b77186dfe462148075954ce4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
This change fixes the issue [1]. Before this fix, a merge involving
the caching of consecutive yet similar filenames with Norwegian
characters [2] used to throw an IllegalStateException: Duplicate
stages not allowed. This was caused by inaccurate decoding of the
filenames, using string values assuming default encoding. In the
toString method of DirCacheEntry, used before through getPathString,
UTF-8 encoding is used, but the end result becomes default encoding,
through Object's default toString usage. The special characters in
those two consecutive (particular) filenames [2] were becoming the
very same decoded /single character, lending consecutive -but then
identical- filenames. Thus the perceived duplicate 0-staging of the
file(s).
Replace getPathString usage with getRawPath for this specific case,
or use byte array representations of cached entries instead of string.
Adding a test for this change is not possible, as there is no known
way to change the default encoding for filenames such as [2] (e.g.).
JGitTestUtil does write file contents through UTF-8, but encoding like
so does not apply to the actual file name. Hence there is no way to
create files with names properly made of special characters such as
[2]'s. And the test that is necessary for this case assumes such
Norwegian (or similar characters) filenames. Changing the default
locale programmatically in a test has no effect either. And changing
the LANG value passed to the JVM is only possible upon starting it.
[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=9153
[2] <=>
(...)
"a/b/SíÒr-Norge.map",
"a/b/Sør-Norge.map",
(...)
Change-Id: Ib9f2f5297932337c9817064cc09d9f774dd168f4
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Matthias Sohn
David Pursehouse
Luca Milanesio
Jonathan Nieder
Jonathan Tan
Gabriel Couto
Thomas Wolf
Christian Halstrick
Terry Parker
Masaya Suzuki
Minh Thai
Dave Borowitz
Hector Caballero
Marco Miller