This enables higher file timestamp resolution on filesystems like ext4,
Mac APFS (1ns) or NTFS (100ns) providing high timestamp resolution on
filesystem level.
Note:
- on some OSes Java 8,9 truncate milliseconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes truncates timestamp resolution to microseconds when
converting the internal representation to FileTime exposed in the API,
see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution
Change-Id: I25ffff31a3c6f725fc345d4ddc2f26da3b88f6f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use options
- StandardOpenOption.CREATE to create touched file if not existing
- StandardOpenOption.SYNC to enforce synch of data and meta data changes
- StandardOpenOption.WRITE
Also set mtime explicitly in FileUtils#touch to the current system time.
This should fix that the previous implementation didn't work on
- locally cached Windows network share (CSC-CACHE filesystem) mapped as
a drive
- nfsv4 mounts on Linux
and that it didn't create unborn file like Linux command "touch".
Apache common's and Guava's touch() use the same approach.
Immediately after creating the probe file used to measure timestamp
resolution touch it. This ensures we always use the local system clock
when measuring filesystem timestamp resolution. This should prevent that
clock skew could influence the measured timestamp resolution in case of
a mounted network filesystem.
Bug: 548598
Change-Id: Iaeaf5967963f582395a195aa637b8188bfadac60
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
To enable persisting filesystem timestamp resolution per FileStore add a
new config section to the user global git configuration:
- Config section is "filesystem"
- Config subsection is concatenation of
- Java vendor (system property "java.vm.vendor")
- runtime version (system property "java.vm.version")
- FileStore's name
- separated by '|'
e.g.
"AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
The prefix is needed since some Java versions do not expose the full
timestamp resolution of the underlying filesystem. This may also
depend on the underlying operating system hence concrete key values
may not be portable.
- Config key for timestamp resolution is "timestampResolution" as a time
value, supported time units are those supported by
DefaultTypedConfigGetter#getTimeUnit
If timestamp resolution is already configured for a given FileStore
the configured value is used instead of measuring the resolution.
When timestamp resolution was measured it is persisted in the user
global git configuration.
Example:
[filesystem "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"]
timestampResolution = 1 seconds
If locking the git config file fails retry saving the resolution up to 5
times in order to workaround races with another thread.
In order to avoid stack overflow use the fallback filesystem timestamp
resolution when loading FileBasedConfig which creates itself a
FileSnapshot to help checking if the config changed.
Note:
- on some OSes Java 8,9 truncate to milliseconds or seconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes up to Java 12 truncates timestamp resolution to
microseconds when converting the internal representation to FileTime
exposed in the API, see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution up to
Java 12
Hence do not attempt to manually configure a higher timestamp resolution
than supported by the Java version being used at runtime.
Bug: 546891
Bug: 548188
Change-Id: Iff91b8f9e6e5e2295e1463f87c8e95edf4abbcf8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
In order to avoid blocking on the main thread during measurement
interactive applications like EGit may want to measure the filesystem
timestamp resolution asynchronously.
In order to enable measurement in the background call
FileStoreAttributeCache.setAsyncfileStoreAttrCache(true)
before the first access to cached FileStore attributes.
Bug: 548188
Change-Id: I8c9a2dbfc3f1d33441edea18b90e36b1dc0156c7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If
- pack.waitPreventRacyPack = true (default is false)
- packfile size > pack.minSizePreventRacyPack (default is 100 MB)
wait after a new packfile was written and before it is opened until it
cannot be racy anymore.
If a new packfile is accessed while it's still racy at least the pack's
index will be reread by ObjectDirectory.scanPacksImpl(). Hence it may
save resources to wait one tick of the file system timer to avoid this
reloading. On filesystems with a coarse timestamp resolution it may be
beneficial to skip this wait for small packfiles.
Bug: 546891
Change-Id: I0e8bf3d7677a025edd2e397dd2c9134ba59b1a18
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
FileSnapshot.notRacyClean() assumed a worst case filesystem timestamp
resolution of 2.5 sec (FAT has a resolution of 2 sec). Instead measure
timestamp resolution to avoid unnecessary IO caused by false positives
in detecting the racy git problem caused by finite filesystem timestamp
resolution [1].
Cache the measured resolution per FileStore since timestamp resolution
depends on the respective filesystem type. If timestamp resolution
cannot be measured or fails due to an exception fallback to the worst
case FAT timestamp resolution and avoid caching this value.
Add a 10% safety margin in FileSnapshot.notRacyClean(), though running
FsTest.testFsTimestampResolution() 1000 times which is not using a
safety margin didn't fail on Mac using APFS and Java 8, 11, 12.
Measured Java file timestamp resolution: [2]
[1] https://github.com/git/git/blob/master/Documentation/technical/racy-git.txt
[2] https://docs.google.com/spreadsheets/d/1imy0y6WmRqBf0kjCxzxj2X7M50eIVfa7oaUIzEOHmjo
Bug: 546891
Change-Id: I493f3b57b6b306285ffa7d392339d253e5966ab8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Keep track of the original cause for a packfile invalidation.
It is needed for the sysadmin to understand if there is a real
underlying filesystem problem and repository corruption or if it is
simply a consequence of a concurrency of Git operations (e.g. repack
or GC).
Change-Id: I06ddda9ec847844ec31616ab6d17f153a5a34e33
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).
Read and consider file size also, so that differing file size can help
to more accurately detect file changes without reading the file content.
Use bulk read to avoid multiple stat calls to retrieve file attributes.
Change-Id: I974288fff78ac78c52245d9218b5639603f67a46
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
In order to validate .gitmodules files, we first need to find them
in the incoming pack.
Do it in the ObjectChecker stage. Check in the tree objects if they
point to a .gitmodules file and report the tree id and the .gitmodules
blob id.
This can be used later to check if the file is in the root of the
project and if the contents are good.
While we're here, make isMacHFSGit more accurate by detecting variants
of filenames that vary in case.
[jn: tweaked NTFS and HFS+ checking; added more tests]
Change-Id: I70802e7d2c1374116149de4f89836b9498f39582
Signed-off-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
In C git versions before 2.19.1, the submodule is fetched by running
"git clone <uri> <path>". A URI starting with "-" would be interpreted
as an option, causing security problems. See CVE-2018-17456.
Refuse to add submodules with URIs, names or paths starting with "-",
that could be confused with command line arguments.
[jn: backported to JGit 4.7.y, bringing portions of Masaya Suzuki's
dotdot check code in v5.1.0.201808281540-m3~57 (Add API to specify
the submodule name, 2018-07-12) along for the ride]
Change-Id: I2607c3acc480b75ab2b13386fe2cac435839f017
Signed-off-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The following commits introduced in stable-4.5 and stable-4.9
introduced some minor API additions in service releases.
f7ceeaa2 FileRepository: Add pack-based inserter implementation
085d1f95 Make PackInserter public
10e65cb4 Fix LockFile semantics when running on NFS
Change-Id: I4afed7e0395cf93d828e671080e3ec9ddf20987d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This avoids that we have to suppress API errors whenever we add a new
constant in a minor release. This change affects implementors only which
is ok to do in a minor release following OSGi semantic versioning rules.
Change-Id: Iece841886fbe00f1ba567c5ff68093c542ba265e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
These methods were introduced for 4.11.1 so we have to silence the API
error adding API in a service release raises.
Change-Id: Ic847cebbed439912d3979ec2ec1809f77a28f61e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Respect merge=lfs and diff=lfs attributes where required to replace (in
memory) the content of LFS pointers with the actual blob content from
the LFS storage (and vice versa when staging/merging).
Does not implement general support for merge/diff attributes for any
other use case apart from LFS.
Change-Id: Ibad8875de1e0bee8fe3a1dffb1add93111534cae
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Merges are performed using the raw text as stored in the git
repository. When we write the merge result, we must apply the
correct CRLF settings. Otherwise the line endings in the result
will be wrong.
Bug: 499615
Change-Id: I37a9b987e9404c97645d2720cd1c7c04c076a96b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This was silenced before but suppression was unintentionally lost in
merge commit 6858339c1e.
This method was removed in 4.9.0 and reintroduced in 4.9.1 to avoid
breaking EMF compare versions which were built against older versions.
See: abf420302b
Change-Id: I152d58ac885e044bcab682b9423f6cc83b667989
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
- this is a new warning option in Eclipse 4.7 and higher
- we always change version of all bundles in a release to keep release
engineering simple
Change-Id: Ic7523d77b67b2802f1bab3bc70af250d712a034f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When running on NFS there was a chance that JGits LockFile
semantic is broken because File#createNewFile() may allow
multiple clients to create the same file in parallel. This
change provides a fix which is only used when the new config
option core.supportsAtomicCreateNewFile is set to false. The
default for this option is true. This option can only be set in the
global or the system config file. The repository config file is not
taken into account in this case.
If the config option core.supportsAtomicCreateNewFile is true
then File#createNewFile() is trusted and the behaviour doesn't
change.
But if core.supportsAtomicCreateNewFile is set to false then after
successful creation of the lock file a hardlink to that lock file is
created and the attribute nlink of the lock file is checked to be 2. If
multiple clients manage to create the same lock file nlink would be
greater than 2 showing the error.
This expensive workaround is described in
https://www.time-travellers.org/shane/papers/NFS_considered_harmful.html
section III.d) "Exclusive File Creation"
Change-Id: I3d2cc48d8eb280d5f7039eb94da37804f903be6a
This method was removed in 4.9 and reintroduced in
I48ba4308dee73925fa32d6c2fd6b5fd89632c571 as deprecated in 4.9.1 in
order to help EMF Compare to avoid breakage.
Change-Id: Ia638517178313da42ae13ebcf88ad535d9a02723
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
So far we follow OSGi semantic versioning [1] which says the following:
"A change in the second (minor) part of the version signals that the
change is backward compatible with consumers of the API package but not
with the providers of that API. That is, when the API package goes from
version 1.5 to 1.6 it is no longer compatible with a provider of that
API but consumers of that API are backward compatible with that API
package."
The change Ib5fbf17bdaf727bc5d0e106ce88f2620d9f87a6f broke EMF Compare
which subclasses ResolveMerger since we added a new parameter to the
protected ResolveMerger.processEntry() method. According to the above
cited OSGi semantic versioning this is ok, implementers should expect
that they break on minor version changes of the API they implement.
This change reintroduces the old processEntry() method in order to help
avoid breakage for existing EMF Compare versions which expect breakage
also for the implementer case only for major version change (in this
case from JGit 4.x to 5.x).
[1] http://www.osgi.org/wp-content/uploads/SemanticVersioning1.pdf
See: https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03431.html
Change-Id: I48ba4308dee73925fa32d6c2fd6b5fd89632c571
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Do not automatically organize imports using a save action since this
seems to be buggy and removed some annotations org.eclipse.jgit.pgm
needs to use args4j.
Change-Id: I5a91292c3b9241ce2dde3e4ecce14ad460097129
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Revert the following save actions which were introduced in c0ad77d8:
- always use braces around blocks
- remove unused imports
Other than I expected save actions are run globally on edited files -
and not only on edited code lines only.
Hence revert the save action "Convert control statement bodies to
blocks" which would affect a large number of code lines not affected by
the change editing some small part of a class. This would generate a
large number of changes which may lead to many unnecessary conflicts.
Total number of affected lines across jgit would be around 10k lines.
Also revert "Remove unused imports" since it erroneously removes imports
of some annotations needed by pgm classes using args4j.
Change-Id: I879a47f68e664129e6124cf25c1ae1f6a2d7a5aa
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add the following Eclipse save actions executed when saving modified
lines. This should help to reduce manual work needed to maintain a clean
and consistent code style:
- organize imports
- always use braces around blocks
- add missing annotations
- @Override including implementation of interface methods
- @Deprecated
- remove
- unused imports
- unnecessary $NON-NLS$ tags
- redundant type arguments
Also add default values for new settings that were introduced in recent
Eclipse versions up to Neon since we updated save rules the last time.
Change-Id: Idc90b249df044d0552f04edf01a5f607c4846f50
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
git-core follows HTTP redirects so JGit should also provide this.
Implement config setting http.followRedirects with possible values
"false" (= never), "true" (= always), and "initial" (only on GET, but
not on POST).[1]
We must do our own redirect handling and cannot rely on the support
that the underlying real connection may offer. At least the JDK's
HttpURLConnection has two features that get in the way:
* it does not allow cross-protocol redirects and thus fails on
http->https redirects (for instance, on Github).
* it translates a redirect after a POST to a GET unless the system
property "http.strictPostRedirect" is set to true. We don't want
to manipulate that system setting nor require it.
Additionally, git has its own rules about what redirects it accepts;[2]
for instance, it does not allow a redirect that adds query arguments.
We handle response codes 301, 302, 303, and 307 as per RFC 2616.[3]
On POST we do not handle 303, and we follow redirects only if
http.followRedirects == true.
Redirects are followed only a certain number of times. There are two
ways to control that limit:
* by default, the limit is given by the http.maxRedirects system
property that is also used by the JDK. If the system property is
not set, the default is 5. (This is much lower than the JDK default
of 20, but I don't see the value of following so many redirects.)
* this can be overwritten by a http.maxRedirects git config setting.
The JGit http.* git config settings are currently all global; JGit has
no support yet for URI-specific settings "http.<pattern>.name". Adding
support for that is well beyond the scope of this change.
Like git-core, we log every redirect attempt (LOG.info) so that users
may know about the redirection having occurred.
Extends the test framework to configure an AppServer with HTTPS support
so that we can test cloning via HTTPS and redirections involving HTTPS.
[1] https://git-scm.com/docs/git-config
[2] 6628eb41db
[3] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
CQ: 13987
Bug: 465167
Change-Id: I86518cb76842f7d326b51f8715e3bbf8ada89859
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Matthias Sohn
Luca Milanesio
Ivan Frade
Thomas Wolf
David Pursehouse
Markus Duft
Christian Halstrick