When reading from a packfile, make sure that is valid
and has a non-null file-descriptor.
Because of concurrency between a thread invalidating a packfile
and another trying to read it, the read() may result into a NPE
that won't be able to be automatically recovered.
Throwing a PackInvalidException would instead cause the packlist
to be refreshed and the read to eventually succeed.
Bug: 544199
Change-Id: I27788b3db759d93ec3212de35c0094ecaafc2434
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
When a packfile is invalid, throw an exception explicitly
outside any catch scope, so that is not accidentally caught
by the generic catch-all cause, which would set the packfile
as valid again.
Flagging an invalid packfile as valid again would have
dangerous consequences such as the corruption of the in-memory
packlist.
Bug: 544199
Change-Id: If7a3188a68d7985776b509d636d5ddf432bec798
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Do not redundantly call File.lastModified() for extracting the
timestamp of the PackFile but rather use consistently the FileSnapshot
which reads all file attributes in a single bulk call.
Change-Id: I932675ae4fe56dcd3833dac249816f097303bb09
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).
Read and consider file size also, so that differing file size can help
to more accurately detect file changes without reading the file content.
Use bulk read to avoid multiple stat calls to retrieve file attributes.
Change-Id: I974288fff78ac78c52245d9218b5639603f67a46
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The pack reload mechanism from the filesystem works only by name
and does not check the actual last modified date of the packfile.
This lead to concurrency issues where multiple threads were loading
and removing from each other list of packfiles when one of those
was failing the checksum.
Rely on FileSnapshot rather than directly checking lastModified
timestamp so that more checks can be performed.
Bug: 544199
Change-Id: I173328f29d9914007fd5eae3b4c07296ab292390
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
In case of concurrent pack file access, threads may wait on the idx()
function even for already open files. This happens especially with a
slow file system.
Performance numbers are listed in the bug report.
Bug: 543739
Change-Id: Iff328d347fa65ae07ecce3267d44184161248978
Signed-off-by: Juergen Denner <j.denner@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.0:
Fix error log message in ObjectDirectory.handlePackError()
Properly format pack checksums in PackFile.idx()
Cancel gc if thread was interrupted
PackFile: report correct message for checksum mismatch
ObjectDirectory: Clean up logging
Bazel: Stop using native.git_repository
ObjectDirectory: extra logging on packfile exceptions
Change-Id: I056f41f619a1bb22ca6be75ec21c32c1f6d4fd2f
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.11:
Fix error log message in ObjectDirectory.handlePackError()
Properly format pack checksums in PackFile.idx()
Cancel gc if thread was interrupted
PackFile: report correct message for checksum mismatch
ObjectDirectory: Clean up logging
Bazel: Stop using native.git_repository
ObjectDirectory: extra logging on packfile exceptions
Change-Id: If75b149e693005dd3fe06b523e6e6784bedf44c1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.10:
Fix error log message in ObjectDirectory.handlePackError()
Properly format pack checksums in PackFile.idx()
Cancel gc if thread was interrupted
PackFile: report correct message for checksum mismatch
ObjectDirectory: Clean up logging
Bazel: Stop using native.git_repository
ObjectDirectory: extra logging on packfile exceptions
Change-Id: I9052e318b5d920770f7c7121d36e3c58df9d5f5a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.9:
Fix error log message in ObjectDirectory.handlePackError()
Properly format pack checksums in PackFile.idx()
Cancel gc if thread was interrupted
PackFile: report correct message for checksum mismatch
ObjectDirectory: Clean up logging
Bazel: Stop using native.git_repository
ObjectDirectory: extra logging on packfile exceptions
Change-Id: I0847251eb010616a705e0b91df4bdebc225fa95d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
When the packfile checksum does not match the expected one
report the correct checksum error instead of reporting that
the number of objects is incorrect.
Change-Id: I040f36dacc4152ae05453e7acbf8dfccceb46e0d
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit 436c99ce59)
Externalize the message and log the pack file with absolute path.
Change-Id: I019052dfae8fd96ab67da08b3287d699287004cb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit 9665d86ba1)
The native.git_repository method doesn't work in the latest version
of bazel, and causes the build to fail with:
type 'struct' has no method git_repository()
Change-Id: Id6a57369b681c0afe811e9e3740b141fb7fb4653
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit ec5fc57b79)
Display extra logging, including the exception with the associated
stacktrace, whenever a packFile can't be read and thus removed
from the packlist.
Change-Id: I97a4e31dc427bfcc0baae438dcbe2dcd4704b824
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
(cherry picked from commit 962babc4b2)
When the packfile checksum does not match the expected one
report the correct checksum error instead of reporting that
the number of objects is incorrect.
Change-Id: I040f36dacc4152ae05453e7acbf8dfccceb46e0d
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Externalize the message and log the pack file with absolute path.
Change-Id: I019052dfae8fd96ab67da08b3287d699287004cb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The native.git_repository method doesn't work in the latest version
of bazel, and causes the build to fail with:
type 'struct' has no method git_repository()
Change-Id: Id6a57369b681c0afe811e9e3740b141fb7fb4653
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Display extra logging, including the exception with the associated
stacktrace, whenever a packFile can't be read and thus removed
from the packlist.
Change-Id: I97a4e31dc427bfcc0baae438dcbe2dcd4704b824
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
The AdvertiseRefsHook can be called twice if the following conditions
hold:
1. This AdvertiseRefsHook doesn't set this.refs.
2. getAdvertisedOrDefaultRefs is called after getFilteredRefs.
For example, this can happen when fetchV2 is called after lsRefsV2
when using a stateful bidirectional transport.
The second call does not accomplish anything useful. Guard it with
'if (!advertiseRefsHookCalled)' to avoid wasted work.
Reported-by: Jonathan Tan <jonathantanmy@google.com>
Change-Id: Ib746582e4ef645b767a5b3fb969596df99ac2ab5
Signed-off-by: Jonathan Nieder <jrn@google.com>
In the longer term, we can add support for this to the
RequestValidator interface. In the short term, this is a minimal
band-aid to ensure any refs the client requests are visible to the
client.
Change-Id: I0683c7a00e707cf97eef6c6bb782671d0a550ffe
Reported-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
ProtocolV2Parser explains:
// TODO(ifrade): This validation should be done after the
// protocol parsing. It is not a protocol problem asking for an
// unexisting ref and we wouldn't need the ref database here.
Do so. This way all ref database accesses are in one place, in the
UploadPack class.
No user-visible change intended --- this is just to make the code
easier to manipulate.
Change-Id: I68e87dff7b9a63ccc169bd0836e8e8baaf5d1048
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
AdvertiseRefsHook is used to limit the visibility of the refs in Gerrit.
If this hook is not called, then all refs are treated as visible.
In protocol v2, the hook is not called, causing the server to advertise
all refs. This bug was introduced in v5.0.0.201805221745-rc1~1^2~9
(Execute AdvertiseRefsHook only for protocol v0 and v1, 2018-05-14).
Even before then, the hook was not called in requests after the
capability advertisement, so in transports like HTTP that do not retain
state between round-trips, the server would advertise all refs in
response to an ls-refs (ls-remote) request.
Fix both cases by using getAdvertisedOrDefaultRefs to retrieve the
advertised refs in lsRefs, ensuring the hook is called in all cases that
use its result.
[jn: backported to stable-5.0; split out from a larger patch that also
fixes protocol v0; avoided filtering this.refs by ref prefix]
Change-Id: I64bce0e72d15b90baccc235c067e57b6af21b55f
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
AdvertiseRefsHook is used to limit the visibility of the refs in Gerrit.
If this hook is not called, then all refs are treated as visible,
causing the server to serve commits reachable from branches the client
should not be able to access, if asked to via a request naming a guessed
object id.
Until 3a529361a76e8267467071e0b13ebb36b97d8fb2 (Call AdvertiseRefsHook
before validating wants, 2018-12-18), UploadPack would invoke this hook
at ref advertisement time but not during negotiation and when serving a
pack file. Add a test to avoid regressing. Stateful bidirectional
transports were not affected, so the test uses HTTP.
[jn: split out when backporting the fix to stable-4.5. The test passes
as long as v4.9.0.201710071750-r~169 (fetch: Accept any SHA-1 on lhs of
refspec, 2017-06-04) is cherry picked along with it.]
Change-Id: I8c017107336adc7cb4c826985779676bf043e648
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn
Luca Milanesio
Juergen Denner
David Pursehouse
Jonathan Nieder
Masaya Suzuki