Reftable is a binary, block-based storage format for the ref-database.
It provides several advantages over the traditional packed + loose
storage format:
* O(1) write performance, even for deletions and transactions.
* atomic updates to the ref database.
* O(log N) lookup and prefix scans
* free from restrictions imposed by the file system: it is
case-sensitive even on case-insensitive file systems, and has
no inherent limitations for directory/file conflicts
* prefix compression reduces space usage for repetitive ref names,
such as gerrit's refs/changes/xx/xxxxx format.
FileReftableDatabase is based on FileReftableStack, which does
compactions inline. This is simple, and has good median performance,
but every so often it will rewrite the entire ref database.
For testing, a FileReftableTest (mirroring RefUpdateTest) is added to
check for Reftable specific behavior. This must be done separately, as
reflogs have different semantics.
Add a reftable flavor of BatchRefUpdateTest.
Add a FileReftableStackTest to exercise compaction.
Add FileRepository#convertToReftable so existing testdata can be
reused.
CQ: 21007
Change-Id: I1837f268e91c6b446cb0155061727dbaccb714b8
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Several methods were protected when they were defined in BaseReceivePack
which has since been squashed into ReceivePack. Those methods no longer
need to be protected, and can now be private instead.
Change-Id: Ic6bb5e66f353fcbdc7919847423234af398c88b4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Prevent adding a null parent to a commit's parent array. Doing so
can cause NPEs later on.
Bug: 552160
Change-Id: Ib24b7b9b7b08e0b6f246006b4a4cade7eeb830b9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
core.quotePath = false means that "bytes higher than 0x80 are not
considered "unusal" anymore"[1], i.e., they are not escaped. In
essence this preserves non-ASCII characters in path names in output.
Note that control characters and other special characters in the
ASCII range will still be escaped.
Add a new QuotedString.GIT_PATH_MINIMAL singleton implementing this.
Change the normal GIT_PATH algorithm to use bytes instead of characters
so it can be re-used. Provide a setter in DiffFormatter for the quoting
style so that an application can override the default, which is the
setting from the git config (and by default "true"). Use the new
QuotedString.GIT_PATH_MINIMAL when core.quotePath == false.
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-corequotePath
Bug: 552467
Change-Id: Ifcb233e7d10676333bf42011e32d01a4e1138059
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Some applications using JGit use their own repository caching. In
such applications, it may be needlessly inefficient to create new
submodule repositories from a SubmoduleWalk or in an IndexDiff. It
can be much more efficient to use an already cached repository
instance.
Provide a way to configure a SubmoduleWalk with a factory to create
BaseRepositoryBuilders to use to create repositories, and use it in
IndexDiff. Provide new IndexDiff.diff() operations that take such an
additional factory as parameter.
An application that caches Repository instances (for instance EGit)
can use a factory that provides builders that don't create a new
Repository instance but that return the already cached instance, if
one is available. Note that in such a case, the application may need
to be prepared to deal with IndexDiff.diff() also _closing_ the
obtained repository; if the application expects its cached Repository
instances to remain open while being cached, it'll have to use
Repository.incrementOpen() to prevent that the repository instance
gets closed.
Bug: 550878
Change-Id: Icc1b34dfc4cebd8ed4739dd09d37744d41adf711
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
IndexDiff would apply ignore mode ALL from .gitmodules to all remaining
submodules, and would ignore other settings from .gitignore and always
apply the setting defined on the IndexDiff instead. Correct that.
In canonical git the ignore setting from .gitmodules can also be
overridden by .git/config.[1] Implement that override in SubmoduleWalk.
[1] https://git-scm.com/docs/gitmodules#Documentation/gitmodules.txt-submoduleltnamegtignore
Bug: 521613
Change-Id: I9199fd447e41c7838924856dce40678370b66395
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
I1ce92869435d5eebb7d671be44561e69c6233134 merged BaseReceivePack into
ReceivePack which breaks API but is only affecting the few jgit based
servers out in the wild.
Change-Id: Iad856a2afaf3cad95d01ad81a0116cebcd9de2d9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Move the BaseReceivePack implementation back into ReceivePack. This is a
backward-incompatible change. For example, BaseReceivePack.FirstLine no
longer exists and cannot be referenced. However, most of the code
should just work by replacing BaseReceivePack with ReceivePack.
Although this is an API change, it only affects callers using JGit as a
server, and there are very few of those in the wild.
Change-Id: I1ce92869435d5eebb7d671be44561e69c6233134
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
This will provide exponential backoff with jitter to other JGit
components too.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Idd44e3bbaef6d71134ce2e3f7d405f35e7397cbd
The getPackSize method will throw IllegalStateException if it is
called when the pack size is not set. This is the case for example
when the received commands are all DELETE and there is no pack.
Add a new method hasReceivedPack that can be called prior to calling
getPackSize, to avoid causing the IllegalStateException.
See [1] for context.
[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=11918
Change-Id: I56397256a05e92c8398e65c07a859cee59b46317
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
SystemReader.updateAll() must _not_ test whether the file exists. In
tests at least there are FileBasedConfigs with a null file. Test
configs should (and do) override isOutdated() to deal with this case.
Change-Id: I56303fe0d56afeb9f2203ee807a92c5dcf3809e9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
This ensures the measured filesystem timestamp resolution will be only
used on the machine where it was measured and avoid errors in case the
~/.jgitconfig file is copied to another machine.
Bug: 551850
Change-Id: Iff2a11be62ca94c3bbe4a955182988dc50852f9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This avoids polluting hand-crafted user level config with
auto-configured options which might disturb in environments where
the user level config is replicated between different machines.
Add a jgit config as parent of the system level config. Persist
measured timestamp resolutions always in this jgit config and read it
via the user global config. This has the effect that auto-configured
timestamp resolution will be used by default and can be overridden in
either the system level or user level config.
Store the jgit config under the XDG_CONFIG_HOME directory following the
XDG base directory specification [1] in order to ensure that we have
write permissions to persist the file. This has the effect that each OS
user will use its jgit config since they typically use different
XDG_CONFIG_HOME directories.
If the environment variable XDG_CONFIG_HOME is defined the jgit config
file is located at $XDG_CONFIG_HOME/jgit/config otherwise the default is
~/.config/jgit/config.
If you want to avoid redundant measurement for different OS users
manually copy the values measured and auto-configured for one OS user to
the system level git config.
[1] https://wiki.archlinux.org/index.php/XDG_Base_Directory
Bug: 551850
Change-Id: I0022bd40ae62f82e5b964c2ea25822eb55d94687
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Change-Id: Ia77f442e47c5670c2d6d279ba862044016aabd86
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
- filter errors for new APIs added in service release
- remove unused filters
Change-Id: Ifbf532b8a3c46d4ed78a38f6c75073a072b7f669
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Pass the CheckoutMetadata to checkoutEntry(), otherwise cr-lf settings
and smudge filters would be ignored.
Change-Id: Ifb1c4cb098a0b47c9752d0231d530db6a92b25a4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
A merge may write files to the working tree. After a successful
merge one must fire a WorkingTreeModifiedEvent explicitly if
getModifiedFiles() is not empty.
Also, any touched files must be reported by the
WorkingTreeModifiedEvent fired by DirCacheCheckout.checkout().
Bug: 552636
Change-Id: I5fab8279ed8be8a4ae34cddfa726836b9277aea6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
MergedReftableTest#scanDuplicates tests whether we can write duplicate
keys in a merged reftable. Apparently, the first key appearing should
get precedence, and this works because the sort() algorithm on ordered
collections is stable.
This is potentially confusing behavior, because you can write data
into the table that cannot be retrieved (Merged table can only have
one entry per key), and the APIs such as exactRef() only return a
single value.
Make this consistent with behavior introduced in I04f55c481 "reftable:
enforce ordering for ref and log writes" by considering a duplicate key
in sortAndWriteRefs as a fatal runtime error.
Change-Id: I1eedd18f028180069f78c5c467169dcfe1521157
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Some URLs cannot be converted via URL.toURI(). So don't convert
the full URL but only the bits that are needed to find a proxy
via java.net.ProxySelector.
Bug: 549690
Change-Id: I55b5ecee70c6b52f72f9bdba9ce552fde7f33976
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The RepositoryState's description is shown in EGit's interactive rebase
view. Use the grammatically correct form.
Bug: 552359
Change-Id: I860ef730a67f2b34d7fabbfbda5ea65ead9270d2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Instead log the problem. There is no reason to panic when we fail to
delete a single temporary file in java.io.tmpdir.
Change-Id: Idb867b3f07b090c7453ccd3688e94097df3b2672
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This fixes OperatorPrecedence error raised by errorprone on FS#findHook.
Change-Id: Ia15f61902c7deff7328c1afa066fc53152949bbf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Instead of just looking for a substring match of user.signingKey
in a key's user ID implement the GPG matching formats[1] for:
'=' Full exact match
'<' Full exact match of the e-mail address
'@' Substring match within the e-mail address only
'*' General case-insensitive substring match (default)
When user.signingKey is not set, the committer's e-mail address is
used by default. In that case, use '<', i.e., require an exact match
on the OpenPGP e-mail address.
Also handle the optional "0x" prefix for (partial) key fingerprints.
[1] https://www.gnupg.org/documentation/manuals/gnupg/Specify-a-User-ID.html
Bug: 550335
Change-Id: I6ce482a099ff1a0dc9de45435cd4d3ec5b504f12
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The effect of assert is defined by compiler flags, so this code
introduced a potential vector for corruption.
Change-Id: I12197432e4351a5bd4aa24d352a19937721845c3
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Currently when a GPG key is looked up using a user identity the first
key from the keyring that has this user identity is returned.
The code was changed to instead return the first signing [S] key in this
keyring and only return the master key if no such signing key was found.
If the master key also does not have the signing flag set null is
returned instead.
Bug: 552288
Change-Id: I194862991d13c2c7ff34a60a54a227167f88f53b
Signed-off-by: Roan Hofland <roan.hofland@hotmail.com>
Currently when a subkey is configured for signing via the git
user.signingkey configuration option the first key from the keyring for
this subkey would be returned for use (master key). The code has been
changed to return the requested key from the keyring instead.
Bug: 552288
Change-Id: I1c1cdf64c1667316a274ff9d829fc2b563797f2a
Signed-off-by: Roan Hofland <roan.hofland@hotmail.com>
Support the core.hooksPath git config. This can be an absolute or
relative path of a directory where to find git hooks; a relative
path is resolved relative to the directory the hook will run in.
Bug: 500266
Change-Id: I671999a6386a837e897c31718583c91d8035f3ba
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Since [1] the gerrit project includes jgit as a submodule, and has this
warning enabled, resulting in 100s of warnings in the console.
Also enable the warning here, and fix them.
At the same time, add missing braces around adjacent and nearby one-line
blocks.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/227897
Change-Id: I81df3fc7ed6eedf6874ce1a3bedfa727a1897e4c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
This introduces ReftableBatchRefUpdate and ReftableDatabase, as
generic classes, with some code moved to DfsReftableBatchRefUpdate and
DfsReftableDatabase.
Clarify thread-safety requirements by asserting locked status in
accessors, and acquiring locks in callers. This does not fix threading
problems, because ReftableBatchRefUpdate already wraps the whole
transaction in a lock.
This also fixes a number of bugs in ReftableBatchRefUpdate:
* non-atomic updates should not bail on first failure
* isNameConflicting should also check for conflicts between names that
are added and removed in the BatchRefUpdate.
Change-Id: I5ec91173ea9a0aa19da444c8c0b2e0f4e8f88798
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
On changing a ref, the old SHA1 is not updated in the object => ref
mapping. This means search by object ID may still turn up a ref from
deeper within the stack. To fix this, check all refs produced by the
merged iterator against the merged reftables.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I41e9cd395b0608eedeeaead0a9fd997238d747c9
The flag enabling sideband-all is used in two places: in UploadPack
for advertisement and in the protocol parser to read it from the
request.
This leds to problems in distributed deployments where the two requests of
a fetch can go to different servers with different configurations.
Use the existing allowsidebandall to accept the sideband-all request
(and respond to it) and introduce a new "advertisesidebandall" to toggle
the advertising of the feature.
Change-Id: I892d541bc3f321606c89bad1d333b079dce6b5fa
Signed-off-by: Ivan Frade <ifrade@google.com>
If the SubModuleWalk isn't closed its TreeWalk's ObjectReader won't
be closed. Re-loading the DirCache during an IndexDiff is not only
inefficient but could also give strange results if an external
process had modified the index in the meantime: file diffs would
be based on a "before" state, but submodule diffs on an "after"
state.
Change-Id: Iab948c08ac342138b37263c9028d80b84101f6d6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Add to statistics the amount and size of packfiles offloaded to HTTP
download.
Change-Id: I895a7219ecac2794368bfc4fdfae74c1238deed9
Signed-off-by: Ivan Frade <ifrade@google.com>
The object identifying packfiles to send them via packfile-uri contains
only the uri and the hash. This is the information that goes through the
wire. It would be useful to know also the size of those packfile, for
example to track how many bytes have been offloaded to HTTP.
Add size field the CachedPackUriProvider.PackInfo object.
Change-Id: If6b921b48a4764d936141c777879b148cc80bbd3
Signed-off-by: Ivan Frade <ifrade@google.com>
Exception handling can be isolated from UploadPack. This makes it
possible to make the exception handler pluggable.
Change-Id: Ieebbd6711963c7f2e47a98783b4ad815793721c7
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Michael Keppler
Han-Wen Nienhuys
David Pursehouse
Matthias Sohn
Thomas Wolf
Masaya Suzuki
Roan Hofland
Ivan Frade