We can detect index changes using FileSnapshot. This is more efficient
and removes usage of a deprecated class.
Change-Id: I4a679102c9a1bd8e82b9ca93eb9dbbde445e9be4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Clients cache the set of advertised references at the start of a
negotiation, and keep replaying the same "want SHA1" list to the
server on each negotiation step. If another client pushes into
a branch and moves it by fast-forward, any request to obtain that
branch's prior SHA-1 is still valid, the commit is reachable from
the new position of the reference. Unfortunately the fast-forward
causes smart HTTP negotations to fail, as the server no longer is
advertising that prior SHA-1.
Instead of causing clients to fail out with a "want invalid" error
and forcing the end-user retry, possibly getting into a never ending
try-fail-retry race while other clients are pushing into the same
busy repository, allow the slightly stale want request so long as
it is still reachable.
C Git implemented this same change recently to fix races on the
smart HTTP protocol when the C Git git-http-backend is used.
The new RequestPolicy feature also allows server authors to make
an even more lenient configuration that exports any SHA-1 to the
client. This might be useful in certain settings where a server
has authenticated the client as the "repository owner" and wants
to allow them to grab any content from the server as a complete
unbroken history chain.
The new setAdvertisedRefs() method allows server authors to manually
fix the references that are advertised, possibly bypassing the
getAllRefs() call on the Repository object.
Change-Id: I7cdb563bf9c55c83653f217f6e53c3add55a0541
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Export the shallow pack information, and also a handy function to
sum up the total times. Include the time writing out the index file,
if it was created.
Change-Id: I7f60ae6848455a357b25feedb23743bbf6c153cf
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If the no-done capability was enabled on the connection, don't
queue up the state vector again once the ACK %s ready message
is observed from the remote. The pack will be following in this
response stream, so the state vector is no longer required.
Change-Id: I7bd1e76957cb58c7ff1cdaeef227f1b02a7e5d24
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The client's use of UnionInputStream was broken when combined with a
8192 byte buffer used by PackParser. A smart HTTP client connection
always pushes in the execute stateless RPC input stream after the
data stream has ended from the remote peer. At the end of the pack,
PackParser asked to fill a 8192 byte buffer, but if only e.g. 1000
bytes remained UnionInputStream went to the next stream and asked
it for input, which triggered a new RPC, and failed because there
was nothing pending in the request buffer.
Change UnionInputStream to only return what it consumed from a
single InputStream without invoking the next InputStream, just in
case that second InputStream happens to be one of these magical
ones that generates an RPC invocation.
Change-Id: I0e51a8e6fea1647e4d2e08ac9cfc69c2945ce4cb
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Calls to unlock the DirCache before throwing an exception
were not needed since checkout calls doCheckout wrapped
in a try block that calls DirCache.unlock in a finally
block.
Change-Id: I2b249a784f9e363430e288aad67fcefb7fac0a6e
Signed-off-by: Kevin Sawicki <kevin@github.com>
We do not yet check or validate submodules, but can accept that
someone staged a change in a submodule with other tools.
Change-Id: I642ede382314bfbd1892dd509a2222885cc5350a
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
The purpose of this commit is to prevent destruction of
submodules on checkout from a tree with a submodule to
another. For consistency we handle the reverse case too,
when we checkout a branch that has a submodule and the
submodule directory exists. And finally we ignore the
case where the submodule changes.
We do not update the submodules, we just try to ignore
them harder.
Bug: 356664
Change-Id: I202c695a57af99b13d0d7220803fd08def3d9b5e
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Since we already have assigned i.getDirCacheEntry() to dce,
use dce instead.
Change-Id: I107713ad0b356516d75c29203f945b056bad3ac7
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
IndexOutOfBoundException is thrown from Repository.resolveSimple() when
'-g' string is located less then 4 characters from the end of this
string.
Change-Id: I1128c2cdfec9db3023d4d0f1f40d863e84b75950
Signed-off-by: Dariusz Luksza <dariusz@luksza.org>
We should use a template for Mylyn commit messages that matches with our
guidelines for commit messages.
http://wiki.eclipse.org/EGit/Contributor_Guide#Commit_message_guidelines
Bug: 337401
Change-Id: I05812abf0eb0651d22c439142640f173fc2f2ba0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
We were diverging from the reference implementation. Always use the
ref we checkout to as the to-branch the reflog and avoid the
refs/heads both in the from-name and to-name.
Change-Id: Id973d9102593872e4df41d0788f0eb7c7fd130c4
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Change-Id: I91c7e08c4afd2562df2226887a933d93c78a0371
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
JGit currently identifies loose objects as 'corrupt' if they've been
deflated using a window size less than 32Kb, because the
isStandardFormat() function doesn't recognise the header
byte as a zlib header. This patch makes the method tolerant of
all valid window sizes (15-bit to 8-bit) - but doesn't sacrifice
it's accuracy in distingushing the standard loose-object format
from the experimental (now abandoned) format. It's based on a patch
which has been merged into C-Git master branch:
https://github.com/git/git/commit/7f684a2aff636f44a506
On memory constrained systems zlib may use a much smaller window
size - working on Agit, I found that Android uses a 4KB window;
giving a header byte of 0x48, not 0x78. Consequently all loose
objects generated by the Android platform appear 'corrupt' :(
It might appear that this patch changes isStandardFormat() to the
point where it could incorrectly identify the experimental format as
the standard one, but the two criteria (bitmask & checksum) can only
give a false result for an experimental object where both of the
following are true:
1) object size is exactly 8 bytes when uncompressed (bitmask)
2) [single-byte in-pack git type&size header] * 256
+ [1st byte of the following zlib header] % 31 = 0 (checksum)
As it happens, for all possible combinations of valid object type
(1-4) and window bits (0-7), the only time when the checksum will be
divisible by 31 is for 0x1838 - ie object type *1*, a Commit - which,
due the fields all Commit objects must contain, could never be as
small as 8 bytes in size.
Given this, the combination of the two criteria (bitmask & checksum)
always correctly determines the buffer format, and is more tolerant
than the previous version.
References:
Android uses a 4KB window for deflation:
http://android.git.kernel.org/?p=platform/libcore.git;a=blob;f=luni/src/main/native/java_util_zip_Deflater.cpp;h=c0b2feff196e63a7b85d97cf9ae5bb2583409c28;hb=refs/heads/gingerbread#l53
Code snippet searching for false positives with the zlib checksum:
https://gist.github.com/1118177
Change-Id: Ifd84cd2bd6b46f087c9984fb4cbd8309f483dec0
Create separate loops based on whether the ref specs
collection is empty or not.
Change-Id: I2861b45fe083675e12ff47f591e104f8cf6dd216
Signed-off-by: Kevin Sawicki <kevin@github.com>
If the ResetCommand should reset to a invalid ref (e.g. HEAD in a repo
whithout a single commit) it was throwing an NPE. This is fixed now by
throwing a JGitInternalExcpeption. It would be nicer if we could throw
a InvalidRefException, but this would modify our API.
Bug: 339610
Change-Id: Iffcb4f2cca9f702176471d93c3a71e5cb3e700b1
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
This implements the server side of shallow clones only (i.e.
git-upload-pack), not the client side.
CQ: 5517
Bug: 301627
Change-Id: Ied5f501f9c8d1fe90ab2ba44fac5fa67ed0035a4
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
This can be useful when implementing garbage collection and there
are packs that should not be copied, such as huge packs that have
a sibling ".keep" file alongside of them.
Callers driving PackWriter need to initialize the list of packs not
to include objects from by passing each index to excludeObjects().
Change-Id: Id7f34df69df97be406bcae184308e92b0e8690fd
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Test was added which reproduce the ClassCastException when ours or
theirs merge strategy is set to MergeCommand. Merger and MergeCommand
were updated in order to avoid exception.
Change-Id: I4c1284b4e80d82638d0677a05e5d38182526d196
Signed-off-by: Denys Digtiar <duemir@gmail.com>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Adds method into DiffEntry class that allows to specify whether changed
trees are included in scanning result list. By default changed trees
aren't added, but in some cases having changed tree would be useful.
Also adds check for tree count in TreeWalk and when it is different from
two it will thrown an IllegalArgumentException.
This change is required by egit
I7ddb21e7ff54333dd6d7ace3209bbcf83da2b219
Change-Id: I5a680a73e1cffa18ade3402cc86008f46c1da1f1
Signed-off-by: Dariusz Luksza <dariusz@luksza.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
During parsing these are used with contains(). If they are a List
type, the contains operation is not efficient. Some callers such
as UploadPack often pass a List here, so convert to Set when the
type isn't efficient for contains().
Change-Id: If948ae3bf1f46e756bd2d5db14795e12ba7a6207
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This reverts commit 67b064fc9f.
The "tiny optimization" introduced by 67b0 turns out to have a big
savings on wall-clock time when the object store is very slow (e.g.
the DHT support in JGit), but comes with a much bigger penalty in
space used by the output stream. CGit packed with 67b0 enabled is
7 MiB larger than it should be (36 MiB rather than 28/29 MiB). The
much bigger Linux kernel repository gained over 200 MiB, though some
of this may have been caused by a smaller window setting.
Revert this patch as PackWriter should be optimizing for space used
rather than time spent, since its primary use is network transfer, and
that isn't free.
Change-Id: I7413a9ef89762208159b4a1adc5a22a4c9245611
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The "Counting objects" phase of packing is the most time consuming
part for any server providing access to Git repositories. Scanning
through the entire project history, including every revision of
every tree that has ever existed is expensive and takes an incredible
amount of CPU time.
Inline the tree parsing logic, unroll a number of loops, and setup
to better handle the common case of seeing another occurrence of
an object that was already marked SEEN.
This change boosts the "Counting objects" phase when JGit is acting
as a server and is packing the linux-2.6 repository for its client.
Compared to CGit on the same hardware, a JGit daemon server is now
21883 objects/sec faster:
CGit:
Counted 2058062 objects in 38981 ms at 52796.54 objects/sec
Counted 2058062 objects in 38920 ms at 52879.29 objects/sec
Counted 2058062 objects in 39059 ms at 52691.11 objects/sec
JGit (before):
Counted 2058062 objects in 31529 ms at 65275.21 objects/sec
Counted 2058062 objects in 30359 ms at 67790.84 objects/sec
Counted 2058062 objects in 30033 ms at 68526.69 objects/sec
JGit (this commit):
Counted 2058062 objects in 28726 ms at 71644.57 objects/sec
Counted 2058062 objects in 27652 ms at 74427.24 objects/sec
Counted 2058062 objects in 27528 ms at 74762.50 objects/sec
Above the first run was a "cold server". For JGit the JVM had just
started up with `jgit daemon`, and for CGit we hadn't touched the
repository "recently" (but it was certainly in kernel buffer cache).
The second and third runs were against the running JGit JVM, allowing
timing tests to better reflect the benefits of JGit's pack and index
caching, as well as any optimizations the JIT may have performed.
The timings are fair. CGit is opening, checking and mmap'ing both
the pack and index during the timer. JGit is opening, checking
and malloc+read'ing the pack and index data into its Java heap
during the timer. Both processes are walking the same graph space,
and are computing the "path hash" necessary to sort objects in the
object table for delta compression. Since this commit only impacts
the "Counting objects" phase, delta compression was obviously not
included in the timings and JGit may still be performing delta
compression slower than CGit, resulting in an overall slower server
experience for clients.
Change-Id: Ieb184bfaed8475d6960a494b1f3c870e0382164a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This is useful when the result needs to be displayed and it's only of
interest if the operation was successful or not (in egit, it could be
used in MultiPullResultDialog).
Change-Id: Icfc9a9c76763f8a777087a1262c8d6ad251a9068
Signed-off-by: Robin Stocker <robin@nibor.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The offset32 format is used for objects <= 2^31-1, while the offset64
format is used for all other objects. This condition was missing
the = needed to ensure an object placed exactly at 2^31 would have
its 64 bit offset in the index.
Change-Id: I293fac0e829c9baa12cb59411dffde666051d6c5
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
A non-thin pack does not need to worry about preferred bases, the pack
will be self-contained and all required delta base objects will appear
within the pack itself. Obtaining the path buffer and length from the
ObjectWalk to build the preferred base table is "expensive", so avoid
the cost unless a thin pack is being constructed.
Change-Id: I16e30cd864f4189d4304e7957a7cd5bdb9e84528
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The previous comment stated that the value set was used
to keep track of the branch in the remote repository
which was incorrect.
Updated the method comment to match the format used
for the PushCommand.setRemote and FetchCommand.setRemote
methods.
Change-Id: I11b81eb3125958af29247b485da56fd88c3bfdf5
Signed-off-by: Kevin Sawicki <kevin@github.com>
If the "Finding sources" phase will complete in <1 second with no
delta compression enabled, don't bother showing the progress meter for
this phase. Small repositories on the local filesystem tend to rip
through this phase always subsecond and the ProgressMonitor display
can actually slow the operation down.
If delta compression is enabled, there are two phases that may run
very quickly. Set the timer to 500 milliseconds instead, reducing the
risk that the user has to wait longer than 1 second before any sort of
output from the packer occurs.
Change-Id: I58110f17e2a5ffa0134f9768b94804d16bbb8399
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If delta resolution completes in < 1000 milliseconds, don't bother
showing the progress meter. This is actually very common for a Gerrit
Code Review server, where the client is probably sending 1 commit and
only a few trees/blobs modified... and the base objects are hot in the
process buffer cache.
The 1000 millisecond delay is just a guess at a reasonable time to wait.
Change-Id: I440baa64ab0dfa21be61deae8dcd3ca061bed8ce
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This progress meter never reached 100% as it did not update while
resolving the external bases in thin packs.
Instead of updating in batches at the top level, update once per delta
that is resolved. The batching progress meter type should smooth out
the frequent updates to an update rate that is more reasonable to send
to the UI, while also ensuring a successful pack parse always reaches
100% deltas resolved.
Change-Id: Ic77dcac542cfa97213a6b0194708f9d3c256d223
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Repository inspection tools may find building a reverse index on a
pack useful, as they can then locate an object by offset. As both
C Git and JGit sometimes produce error messages with the offset
rather than the SHA-1, it may be useful to expose this type.
Change-Id: I487bf32e85a8985cf8ab382d4c82fcbe1fc7da6c
There isn't a good reason to hide all of these as package-private.
Make them public so applications can inspect pack files.
Change-Id: Ia418daf65d63e9e015b8dafdf3d06a1ed91d190b
Matthias Sohn
Shawn O. Pearce
Kevin Sawicki
Robin Rosenberg
Dariusz Luksza
Roberto Tyley
Tomasz Zarna
Christian Halstrick
Matt Fischer
Denys Digtiar
Ketan Padegaonkar
Robin Stocker