Error Prone reports:
[ReferenceEquality] Comparison using reference equality instead of
value equality
The END_EDIT instance is used as a marker, and thus it's OK to use
a reference equality comparison. Factor the comparison to a method
and add a suppression.
Change-Id: I7d9dc1fa21f46c984787056b0b5d163e313026a6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Error Prone reports:
[NarrowingCompoundAssignment] Compound assignments from long to int
hide lossy casts
and
[NarrowingCompoundAssignment] Compound assignments from int to byte
hide lossy casts
See https://errorprone.info/bugpattern/NarrowingCompoundAssignment
Fix the warnings by adding explicit casts or changing types as
necessary.
Now that all occurrences of the warning are fixed, increase its
severity to ERROR.
Change-Id: Idb3670e6047b146ae37daee07212ff9455512623
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Android unconditionally throws a SecurityException;[1] getFileStore()
is not supported. Catch the exception and don't attempt the hard-
linking atomic file mechanism.
[1] https://android.googlesource.com/platform/libcore/+/21e6175e25
Bug: 548947
Change-Id: Idfba2d9dbcbc80ea15ab2ae7889e5142444c1581
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Increase the safety factor to 2.5x for extra safety if max of measured
timestamp resolution and measured minimal racy threshold is < 100ms, use
1.25 otherwise since for large filesystem resolution values the
influence of finite resolution of the system clock should be negligible.
Before, not yet using the newly introduced minRacyThreshold measurement,
the threshold was 1.1x FS resolution, and we could issue the
following sequence of events,
start
create-file
read-file (currentTime)
end
which had the following timestamps:
create-file 1564589081998
start 1564589082002
read 1564589082003
end 1564589082004
In this case, the difference between create-file and read is 5ms,
which exceeded the 4ms FS resolution, even though the events together
took just 2ms of runtime.
Reproduce with:
bazel test --runs_per_test=100 \
//org.eclipse.jgit.test:org_eclipse_jgit_internal_storage_file_FileSnapshotTest
The file system timestamp resolution is 4ms in this case.
This code assumes that the kernel and the JVM use the same clock that
is synchronized with the file system clock. This seems plausible,
given the resolution of System.currentTimeMillis() and the latency for
a gettimeofday system call (typically ~1us), but it would be good to
justify this with specifications.
Also cover a source of flakiness: if the test runs under extreme load,
then we could have
start
create-file
<long delay>
read
end
which would register as an unmodified file. Avoid this by skipping the
test if end-start is too big.
[msohn]:
- downported from master to stable-5.1
- skip test if resolution is below 10ms
- adjust safety factor to 1.25 for resolutions above 100ms
Change-Id: I87d2cf035e01c44b7ba8364c410a860aa8e312ef
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Since we now measure file time resolution we can use it to replace the
hard coded wait time of 25ms. FileSnapshot#equals will return true until
the mtime of the old (o) and the new FileSnapshot (n) differ by at least
one file time resolution.
Change-Id: Icb713a80ce9eb929242ed083406bfb6650c72223
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Cache FileStoreAttributeCache entries since looking up FileStore for a
file may be expensive on some platforms.
Implement a simple LRU cache based on ConcurrentHashMap using a simple
long counter to order access to cache entries.
Change-Id: I4881fa938ad2f17712c05da857838073a2fc4ddb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
Use the fallback timestamp resolution as already described in the
javadoc of these methods. Using zero file timestamp resolution doesn't
make sense.
Change-Id: Iaad2a0f99c3be3678e94980a0a368181b6aed38c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
To enable persisting the minimal racy threshold per FileStore add a
new config option to the user global git configuration:
- Config section is "filesystem"
- Config subsection is concatenation of
- Java vendor (system property "java.vendor")
- Java version (system property "java.version")
- FileStore's name, on Windows we use the attribute volume:vsn instead
since the name is not necessarily unique.
- separated by '|'
e.g.
"AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
The same prefix is used as for filesystem timestamp resolution, so
both values are stored in the same config section
- The config key for minmal racy threshold is "minRacyThreshold" as a
time value, supported time units are those supported by
DefaultTypedConfigGetter#getTimeUnit
- measure for 3 seconds to limit runtime which depends on hardware, OS
and Java version being used
If the minimal racy threshold is configured for a given FileStore the
configured value is used instead of measuring it.
When the minimal racy threshold was measured it is persisted in the user
global git configuration.
Rename FileStoreAttributeCache to FileStoreAttributes since this class
is now declared public in order to enable exposing all attributes in one
object.
Example:
[filesystem "AdoptOpenJDK|11.0.3|/dev/disk1s1"]
timestampResolution = 7000 nanoseconds
minRacyThreshold = 3440 microseconds
Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
By running FileSnapshotTest#detectFileModified we found that the sum of
measured filesystem timestamp resolution and measured clock resolution
may yield a too small interval after a file has been modified which we
need to consider racily clean. In our tests we didn't find this behavior
on all systems we tested on, e.g. on MacOS using APFS and Java 8 and 11
this effect was not observed.
On Linux (SLES 15, kernel 4.12.14-150.22-default) we collected the
following test results using Java 8 and 11:
In 23-98% of 10000 test runs (depending on filesystem type and Java
version) the test failed, which means the effective interval which needs
to be considered racily clean after a file was modified is larger than
the measured file timestamp resolution.
"delta" is the observed interval after a file has been modified but
FileSnapshot did not yet detect the modification:
"resolution" is the measured sum of file timestamp resolution and clock
resolution seen in Java.
Java version filesystem failures resolution min delta max delta
1.8.0_212-b04 btrfs 98.6% 1 ms 3.6 ms 6.6 ms
1.8.0_212-b04 ext4 82.6% 3 ms 1.1 ms 4.1 ms
1.8.0_212-b04 xfs 23.8% 4 ms 3.7 ms 3.9 ms
1.8.0_212-b04 zfs 23.1% 3 ms 4.8 ms 5.0 ms
11.0.3+7 btrfs 98.1% 3 us 0.7 ms 4.7 ms
11.0.3+7 ext4 98.1% 6 us 0.7 ms 4.7 ms
11.0.3+7 xfs 98.5% 7 us 0.1 ms 8.0 ms
11.0.3+7 zfs 98.4% 7 us 0.7 ms 5.2 ms
Mac OS
1.8.0_212 APFS 0% 1 s
11.0.3+7 APFS 0% 6 us
The observed delta is not distributed according to a normal gaussian
distribution but rather random in the observed range between "min delta"
and "max delta".
Run this test after measuring file timestamp resolution in
FS.FileAttributeCache to auto-configure JGit since it's unclear what
mechanism is causing this effect.
In FileSnapshot#isRacyClean use the maximum of the measured timestamp
resolution and the measured "delta" as explained above to decide if a
given FileSnapshot is to be considered racily clean. Add a 30% safety
margin to ensure we are on the safe side.
Change-Id: I1c8bb59f6486f174b7bbdc63072777ddbe06694d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
We should not list the complete cache but only show the cache entry at
hand.
Change-Id: I22be2a4dcbf0145155e23f2389bfcf5662cf23a6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Repeat the test 10000 times to get statistics if measured
fsTimestampResolution is working in practice to detect racy git
situations.
Add a class to compute statistics for this test. Log delta between
lastModified and time when FileSnapshot failed to detect modification.
This happens if the racy git limit determined by measuring filesystem
timestamp resolution and clock resolution is too small. If it would be
correct FileSnapshot would always detect modification or mark it
modified if time since modification is smaller than the racy git limit.
Change-Id: Iabe7af1a7211ca58480f8902d4fa4e366932fc77
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Measure granularity of timestamps stored in the filesystem by setting
and then getting lastModified timestamp until the read value changed.
Increase increment exponentially to limit number of iterations starting
with 1 microsecond since Java's FileTime (up to Java 12) truncates
timestamps to 1 microsecond resolution. The chosen algorithm yields 2000
steps between 1 ms and 2.5 s.
Also measure clock resolution and add that for the total timestamp
resolution. This avoids systematic measurement errors introduced by
doing IO to touch a file.
Change-Id: I9b37138619422452373e298d9d8c7cb2c384db3f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
We should not use configuration when creating FileSnapshot when
accessing FileBasedConfig.
Change-Id: Ic521632870f18bb004751642b9d30648dd94049a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This enables higher file timestamp resolution on filesystems like ext4,
Mac APFS (1ns) or NTFS (100ns) providing high timestamp resolution on
filesystem level.
Note:
- on some OSes Java 8,9 truncate milliseconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes truncates timestamp resolution to microseconds when
converting the internal representation to FileTime exposed in the API,
see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution
Change-Id: I25ffff31a3c6f725fc345d4ddc2f26da3b88f6f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
On Android FS#getFsTimestampResolution always throws a
SecurityException, handle this by falling back to the fallback timestamp
resolution.
Bug: 548947
Change-Id: I0ee6cb3c20e189bdc8d488434a930427ad6f2df2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
It seems on cygwin creating a file under a writable directory can fail
with AccessDeniedException. Log a warning in this case and fallback to
worst case timestamp resolution of 2 seconds.
Bug: 548648
Change-Id: Ic50c31ce9dc9ccadd4db5247df929418ac62d45c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Checking lastModified is time critical hence debug trace is the only way
to analyze issues since debugging is impractical.
Also add configuration for buffering of log4j output to reduce runtime
impact when debug trace is on. Limit buffer to 1MiB and comment this
configuration out since we may not always want to use buffering.
Change-Id: Ib1a0537b67c8dc3fac994a77b42badd974ce6c97
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use options
- StandardOpenOption.CREATE to create touched file if not existing
- StandardOpenOption.SYNC to enforce synch of data and meta data changes
- StandardOpenOption.WRITE
Also set mtime explicitly in FileUtils#touch to the current system time.
This should fix that the previous implementation didn't work on
- locally cached Windows network share (CSC-CACHE filesystem) mapped as
a drive
- nfsv4 mounts on Linux
and that it didn't create unborn file like Linux command "touch".
Apache common's and Guava's touch() use the same approach.
Immediately after creating the probe file used to measure timestamp
resolution touch it. This ensures we always use the local system clock
when measuring filesystem timestamp resolution. This should prevent that
clock skew could influence the measured timestamp resolution in case of
a mounted network filesystem.
Bug: 548598
Change-Id: Iaeaf5967963f582395a195aa637b8188bfadac60
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
To enable persisting filesystem timestamp resolution per FileStore add a
new config section to the user global git configuration:
- Config section is "filesystem"
- Config subsection is concatenation of
- Java vendor (system property "java.vm.vendor")
- runtime version (system property "java.vm.version")
- FileStore's name
- separated by '|'
e.g.
"AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
The prefix is needed since some Java versions do not expose the full
timestamp resolution of the underlying filesystem. This may also
depend on the underlying operating system hence concrete key values
may not be portable.
- Config key for timestamp resolution is "timestampResolution" as a time
value, supported time units are those supported by
DefaultTypedConfigGetter#getTimeUnit
If timestamp resolution is already configured for a given FileStore
the configured value is used instead of measuring the resolution.
When timestamp resolution was measured it is persisted in the user
global git configuration.
Example:
[filesystem "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"]
timestampResolution = 1 seconds
If locking the git config file fails retry saving the resolution up to 5
times in order to workaround races with another thread.
In order to avoid stack overflow use the fallback filesystem timestamp
resolution when loading FileBasedConfig which creates itself a
FileSnapshot to help checking if the config changed.
Note:
- on some OSes Java 8,9 truncate to milliseconds or seconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes up to Java 12 truncates timestamp resolution to
microseconds when converting the internal representation to FileTime
exposed in the API, see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution up to
Java 12
Hence do not attempt to manually configure a higher timestamp resolution
than supported by the Java version being used at runtime.
Bug: 546891
Bug: 548188
Change-Id: Iff91b8f9e6e5e2295e1463f87c8e95edf4abbcf8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
In order to avoid blocking on the main thread during measurement
interactive applications like EGit may want to measure the filesystem
timestamp resolution asynchronously.
In order to enable measurement in the background call
FileStoreAttributeCache.setAsyncfileStoreAttrCache(true)
before the first access to cached FileStore attributes.
Bug: 548188
Change-Id: I8c9a2dbfc3f1d33441edea18b90e36b1dc0156c7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add a unittest.
In commit I5485db55 ("Fix FileSnapshot's consideration of file size"),
the special casing of UNKNOWN_SIZE was forgotten.
This change, together with I493f3b57b ("Measure file timestamp
resolution used in FileSnapshot") introduced a regression that would
occasionally surface in Gerrit integration tests marked UseLocalDisk,
with the symptom that creating the Admin user in NoteDb failed with a
LOCK_FAILURE.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I7ffd972581f815c144f810481103c7985af5feb0
It was reported that measuring file timestamp resolution may hang
indefinitely on nfs. Hence timeout this measurement at the known worst
filesystem timestamp resolution (FAT) of 2 seconds.
Bug: 548188
Change-Id: I17004b0aa49d5b0e76360a008af3adb911b289c0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Error Prone reports:
[ClassNewInstance] Class.newInstance() bypasses exception checking;
prefer getDeclaredConstructor().newInstance()
See https://errorprone.info/bugpattern/ClassNewInstance
This was the only occurrence of the warning in the code base; now it's
fixed, increase the severity to ERROR to prevent future occurrences.
Change-Id: Ic04d1c5d2bd458bbb4bb399d6ce9d147bd48d0b1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Archives created by the ArchiveCommand didn't produce deterministic
archive hashes. For RevCommits RevWalk.parseTree returns the root tree
instead of the RevCommit hence retrieving the commit's timestamp didn't
work. Instead use RevWalk.parseAny and extract the tree manually.
Archive entries store timestamps with 1 second resolution hence we need
to wait longer when creating the same archive twice and compare archive
hashes. Otherwise hash comparison in tests wouldn't fail without this
patch.
Bug: 548312
Change-Id: I437d515de51cf68265584d28a8446cebe6341b79
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
update Maven plugins
- jacoco-maven-plugin to 0.8.4
- japicmp-maven-plugin to 0.14.1
- maven-compiler-plugin to 3.8.1
- maven-deploy-plugin to 3.0.0-M1
- maven-enforcer-plugin to 3.0.0-M2
- maven-install-plugin to 3.0.0-M1
- maven-jar-plugin to 3.1.2
- maven-javadoc-plugin to 3.1.0
- maven-jxr-plugin to 3.0.0
- maven-pmd-plugin to 3.12.0
- maven-resources-plugin to 3.1.0
- maven-shade-plugin to 3.2.1
- maven-source-plugin to 3.1.0
- maven-surefire-plugin to 3.0.0-M3
- spotbugs-maven-plugin to 3.1.12
- tycho to 1.3.0
- tycho-pack200a-plugin to 1.3.0
- tycho-pack200b-plugin to 1.3.0
Cleanup Maven warnings
- pin version of all used Maven plugins
- remove deprecated way to declare minimum Maven version
Change-Id: If23e2e2bb03e5e1e7b1eb9d4924a8faa0aa3704e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
As reported by Error Prone:
An inner class should be static unless it references members of its
enclosing class. An inner class that is made non-static unnecessarily
uses more memory and does not make the intent of the class clear.
See https://errorprone.info/bugpattern/ClassCanBeStatic
Change-Id: Ib99d120532630dba63cf400cc1c61c318286fc41
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit ee40efcea4)
Error Prone reports the warning on several classes:
[NonOverridingEquals] equals method doesn't override Object.equals;
if this is a type-specific helper for a method that does override
Object.equals, either inline it into the callers or rename it to
avoid ambiguity.
See https://errorprone.info/bugpattern/NonOverridingEquals
Most of these are in the public API, so we can't rename or inline them
without breaking the API. FileSnapshot is not part of the public API,
but clients may be using it anyway, so we also shouldn't change that.
Suppress all the warnings instead. Having the check at severity ERROR
will at least make sure we don't introduce any new occurrences.
Change-Id: I92345c11256f06b4fa03ccc13337f72af5a43591
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
ParseableSimpleDateFormat is an enum, and enums must be immutable,
hence the member should be final. At the same time, make it private
since it does not need to be publicly visible.
Change-Id: I7e181f591038d556f1123b6e37adf8441059e99a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Reference comparison is intentional. The END and DELIM string
constants are used as sentinels and will always be the same
instances.
Suppress both ReferenceEquality and StringEquality warnings.
Change-Id: I4ce0495702c56b3911f42f26c2f81d28073cbe19
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Reference comparison with EMPTY and MISSING_FILE is intended; these
are static instances used as markers, and will always be the same
instances.
Change-Id: Ic27f5b797bdb9370cf8f6b3b7bb3f1523d4a454c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Issues reported by downstream analyzers.
The "hash" method can be static.
It is a good practice to group overloaded methods. Move the write(URL)
method with the other writes.
Change-Id: Ia42c0d7081333edcb77e58d5e627929d29672490
Signed-off-by: Ivan Frade <ifrade@google.com>
Downstream analyzers reported empty fields in the javadoc. I corrected
few more details:
* Fill empty javadoc fields.
* Use <p> to separate description paragraphs.
* End description paragraphs with a period.
* Remove period at the end of field descriptions.
Change-Id: I749e4b821fc855999caddc442ac788fa514386ea
Signed-off-by: Ivan Frade <ifrade@google.com>
Reported by downstream analyzers. Suppress the warning on reference
equality for isMissing and fill an empty javadoc field.
Change-Id: I3494423daf2a53ca10e0a9c66553f00204c35396
Signed-off-by: Ivan Frade <ifrade@google.com>
Using the | and & operators in boolean conditions results in a warning
from Error Prone:
[ShortCircuitBoolean]
Prefer the short-circuiting boolean operators && and || to & and |.
see https://errorprone.info/bugpattern/ShortCircuitBoolean
Change-Id: I182f986263b8b9ac189907f4bd1662b4092a52d8
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Using the | and & operators in boolean conditions results in a warning
from Error Prone:
[ShortCircuitBoolean]
Prefer the short-circuiting boolean operators && and || to & and |.
see https://errorprone.info/bugpattern/ShortCircuitBoolean
Change-Id: I6cccca3fdd28bf93b302a9b8a66e68ac912cb60d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Using the | and & operators in boolean conditions results in a warning
from Error Prone:
[ShortCircuitBoolean]
Prefer the short-circuiting boolean operators && and || to & and |.
see https://errorprone.info/bugpattern/ShortCircuitBoolean
Change-Id: I4275c60306e43c74030c4465ba02cb853ad444e1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Fix all remaining instances of the OperatorPrededence warning, by adding
parentheses to make the precedence explicit.
Change-Id: Ib296dfed09f9be042d0ff0f7fad8214e4dd766b4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Matthias Sohn
David Pursehouse
Thomas Wolf
Han-Wen Nienhuys
Ivan Frade