The scoring method was not taking into account the similarity of
the file paths and file names. I changed the metric so that it is 99%
based on content (which used to be 100% of the old metric), and 1%
based on path similarity. Of that 1%, half (.5% of the total final
score) is based on the actual file names (e.g. "foo.java"), and half
on the directory (e.g. "src/com/foo/bar/").
Change-Id: I94f0c23bf6413c491b10d5625f6ad7d2ecfb4def
Added support for converting DiffEntrys to FileHeaders. FileHeaders
are DiffEntrys with a buffer containing the diff output as well as
a list of HunkHeaders. The HunkHeaders contain EditLists. The
createFileHeader(DiffEntry) method in DiffFormatter performs a Myers
Diff on the files refered to by the DiffEntry, then puts the returned
EditList into a single HunkHeader, which is then put into the
FileHeader to be returned. It also generates the appropriate diff
header an puts it into the FileHeader's buffer. The rest of the diff
output, which would normally be parsed to generate the HunkHeaders,
is not generated. In fact, the purpose of this method is to avoid
the costly diff output generation and parsing normally required to
create a FileHeader.
Change-Id: I7d8b18c0f6c85e3d02ad58995d3d231e69af5887
Passing around the OutputStream and the Repository is crazy. Instead
put the stream in the constructor, since this formatter exists only to
output to the stream, and put the repository as a member variable that
can be optionally set.
Change-Id: I2bad012fee7f40dc1346700ebd19f1e048982878
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Content similarity based rename detection is performed only after
a linear time detection is performed using exact content match on
the ObjectIds. Any names which were paired up during that exact
match phase are excluded from the inexact similarity based rename,
which reduces the space that must be considered.
During rename detection two entries cannot be marked as a rename
if they are different types of files. This prevents a symlink from
being renamed to a regular file, even if their blob content appears
to be similar, or is identical.
Efficiently comparing two files is performed by building up two
hash indexes and hashing lines or short blocks from each file,
counting the number of bytes that each line or block represents.
Instead of using a standard java.util.HashMap, we use a custom
open hashing scheme similiar to what we use in ObjecIdSubclassMap.
This permits us to have a very light-weight hash, with very little
memory overhead per cell stored.
As we only need two ints per record in the map (line/block key and
number of bytes), we collapse them into a single long inside of
a long array, making very efficient use of available memory when
we create the index table. We only need object headers for the
index structure itself, and the index table, but not per-cell.
This offers a massive space savings over using java.util.HashMap.
The score calculation is done by approximating how many bytes are
the same between the two inputs (which for a delta would be how much
is copied from the base into the result). The score is derived by
dividing the approximate number of bytes in common into the length
of the larger of the two input files.
Right now the SimilarityIndex table should average about 1/2 full,
which means we waste about 50% of our memory on empty entries
after we are done indexing a file and sort the table's contents.
If memory becomes an issue we could discard the table and copy all
records over to a new array that is properly sized.
Building the index requires O(M + N log N) time, where M is the
size of the input file in bytes, and N is the number of unique
lines/blocks in the file. The N log N time constraint comes
from the sort of the index table that is necessary to perform
linear time matching against another SimilarityIndex created for
a different file.
To actually perform the rename detection, a SxD matrix is created,
placing the sources (aka deletions) along one dimension and the
destinations (aka additions) along the other. A simple O(S x D)
loop examines every cell in this matrix.
A SimilarityIndex is built along the row and reused for each
column compare along that row, avoiding the costly index rebuild
at the row level. A future improvement would be to load a smaller
square matrix into SimilarityIndexes and process everything in that
sub-matrix before discarding the column dimension and moving down
to the next sub-matrix block along that same grid of rows.
An optional ProgressMonitor is permitted to be passed in, allowing
applications to see the progress of the detector as it works through
the matrix cells. This provides some indication of current status
for very long running renames.
The default line/block hash function used by the SimilarityIndex
may not be optimal, and may produce too many collisions. It is
borrowed from RawText's hash, which is used to quickly skip out of
a longer equality test if two lines have different hash functions.
We may need to refine this hash in the future, in order to minimize
the number of collisions we get on common source files.
Based on a handful of test commits in JGit (especially my own
recent rename repository refactoring series), this rename detector
produces output that is very close to C Git. The content similarity
scores are sometimes off by 1%, which is most probably caused by
our SimilarityIndex type using a different hash function than C
Git uses when it computes the delta size between any two objects
in the rename matrix.
Bug: 318504
Change-Id: I11dff969e8a2e4cf252636d857d2113053bdd9dc
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
JGit does not currently do rename detection during diffs. I added
a class that, given a TreeWalk to iterate over, can output a list
of DiffEntry's for that TreeWalk, taking into account renames. This
class only detects renames by SHA1's. More complex rename detection,
along the lines of what C Git does will be added later.
Change-Id: I93606ce15da70df6660651ec322ea50718dd7c04
Added code to support ignoring leading, trailing, and changed
whitespace when performing a diff operation. I also added command
line options to Diff to enable the various whitespace ignoring
methods. These match the flags for git diff.
Change-Id: Ie56301aafad59ee3f0fe5de62719f5023cd702c8
JGit did not have support for skipping whitespace when comparing
lines in RawText objects. I added a subclass of RawText that skips
whitespace in its equals and hashCode methods. I used a subclass
rather than adding functionality into RawText so that performance
would not be impacted by extra logic.
This class only supports ignoring all whitespace. Others will follow
that allow other forms of whitespace ignoring.
Change-Id: Ic2f79e85215e48d3fd53ec1b4ad13373dd183a4a
This stream was used only to determine how many bytes had been
written thus far. Except we're always dumping it into a simple
ByteArrayOutputStream, which also knows that. Drop the dependency
on the pack stream and use ByteArrayOutputStream directly.
This lets us later move this test into the new storage.file
package without dragging along the pack stream that is an internal
implementation detail of PackWriter, which is more general than
just the file storage layer.
Change-Id: I291689c0b1ed799270c213ee73b710b2637fb238
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Setting this value is pointless, because its automatically set
by the refs.newUpdate call that created the update operation.
The API is protected by default, because application level code,
including this test, should not be calling it.
Change-Id: I8867a4e8007892e2bd44a05d7dec619081081943
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The mapCommit API is being deprecated because it doesn't run very
fast. Leaving tests around to test how fast it is relative to C Git
isn't instructive. Remove them, which should help aid the transition
away from the mapCommit API.
Change-Id: I27e1c844610d7da5b2c44b33a00602706973c9cc
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some sources had dos line endings. Also configure all projects to use
unix line endings and UTF-8 text encoding.
Change-Id: I8fc9a1dbb219ffa91d1b3011b3b11b7e48e74ca7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If a repository is "bare", it currently still returns a working directory.
This conflicts with the specification of "bare"-ness.
Bug: 311902
Change-Id: Ib54b31ddc80b9032e6e7bf013948bb83e12cfd88
Signed-off-by: Mathias Kinzler <mathias.kinzler@sap.com>
Currently, there is no way to read the content
of the Git Configuration in a way that would
allow to list all configured values generically.
This change extends the Config class in such a
way as to being able to get a list of sections and
to get a list of names for any given section or
subsection.
This is required in able to implement proper
configuration handling in EGit (show all the
content of a given configuration similar to
"git config -l").
Change-Id: Idd4bc47be18ed0e36b11be8c23c9c707159dc830
Signed-off-by: Mathias Kinzler <mathias.kinzler@sap.com>
Created wrong tags for 0.8.3 hence creating another version.
Change-Id: I4e00bbcffe1cf872e2d7e3f3d88d068701fb5330
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
On Windows, FS_Win32_Cygwin has been used if a Cygwin Git installation
is present in the PATH. Assuming that the user works with the Cygwin
Git installation may result in unnecessary overhead if he actually
does not.
Applications built on top of jgit may have more knowledge on the
actually used Git client (Cygwin or not) and hence should be able to
configure which FS to use accordingly.
Change-Id: Ifc4278078b298781d55cf5421e9647a21fa5db24
A Change-Id helps tools like Gerrit Code Review to keeps different
versions of a patch together. The Change-Id is computed as a SHA-1
hash of some of the same basic information as a commit id on the first
commit intended to solve a particular problem and then reused for
updated solutions.
Change-Id: I04334f84e76e83a4185283cb72ea0308b1cb4182
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
ReadTreeTest contains a lot of useful tests for "checkout"
implementations. But ReadTreeTest was hardcoded to test only
WorkDirCheckout. This change doesn't add/modify any tests semantically
but refactors ReadTreeTest so that a different implementations of
checkout can be tested. This was done to allow DirCacheCheckout to be
tested without rewriting all these tests.
Change-Id: I36e34264482b855ed22c9dde98824f573cf8ae22
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Merges the current head with one other commit.
In this first iteration the merge command supports
only fast forward and already up-to-date.
Change-Id: I0db480f061e01b343570cf7da02cac13a0cbdf8f
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
The CommitCommand should take care to create a merge commit if the file
$GIT_DIR/MERGE_HEAD exists. It should then read the parents for the merge
commit out of this file. It should also take care that when commiting
a merge and no commit message was specified to read the message from
$GIT_DIR/MERGE_MSG.
Finally the CommitCommand should remove these files if the commit
succeeded.
Change-Id: I4e292115085099d5b86546d2021680cb1454266c
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
The strings are externalized into the root resource bundles.
The resource bundles are stored under the new "resources" source
folder to get proper maven build.
Strings from tests are, in general, not externalized. Only in
cases where it was necessary to make the test pass the strings
were externalized. This was typically necessary in cases where
e.getMessage() was used in assert and the exception message was
slightly changed due to reuse of the externalized strings.
Change-Id: Ic0f29c80b9a54fcec8320d8539a3e112852a1f7b
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Rather than keep track of both the position of the object, and the
position of its data, just keep track of the number of bytes used
by the object's header in the pack. This shaves 4 bytes out of the
size of the PackedObjectLoader instances.
We also can defer the addition instruction to the materialize()
operation, avoiding it entirely if the caller never actually uses
the loader. This may be relevant for PackWriter invocations,
where only 1 loader gets chosen for a given object, even though
the object may appear on disk in more than one pack file.
Error reporting is now simplified, as we can rely on the object
offset rather than its data offset. This is the value displayed
by pack debugging tools like `git verify-pack -v`, so its better
to use that in our own errors.
Because nobody needs getDataOffset() now, we can drop that from
the public API.
Change-Id: Ic639c0d5a722315f4f5c8ffda6e26643d90e5f42
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Added a new package org.eclipse.jgit.api and a builder-style API for
jgit. Added also the first implementation for two git commands: Commit
and Log.
This API is intended to be used by external components when
functionalities of the standard git commands are required. It will also
help to ease writing JGit tests.
For internal usages this API may often not be optimal because the git
commands are doing much more than required or they expect parameters of
an unappropriate type.
Change-Id: I71ac4839ab9d2f848307eba9252090c586b4146b
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
The repository state tells in which state the repo is and also which actions
are currently allowed. The state MERGING is telling that a commit is not
possible. But this is only true in the case of unmerged paths in the index.
When we are merging but have resolved all conflicts then we are in a special
state: We are still merging (means the next commit should have multiple
parents) but a commit is now allowed.
Since the MERGING state "canCommit()" cannot be enhanced to return true/false
based on the index state (MERGING is an enum value which does not have a
reference to the repository its state it is representing) I had to introduce a new
state MERGING_RESOLVED. This new state will report that a commit is possible.
CAUTION: there might be the chance that users of jgit previously blindly did a
plain commit (with only one parent) when the RepositoryState allowed them to
do so. With this change these users will now be confronted with a RepositoryState
which says a commit is possible but before they can commit they'll have to
check the MERGE_MESSAGE and MERGE_HEAD files and use the info from these
files.
Change-Id: I0a885e2fe8c85049fb23722351ab89cf2c81a431
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If two keys are the same length, but don't share the same sequence
of characters, we were incorrectly claiming they still matched due
to a bug in the for loop condition. I used the wrong variable and
the loop never executed, resulting in equality anytime the two keys
being compared were the same length.
Use the proper local variable to loop through the arrays, and add
a JUnit test to verify equality works as expected.
Change-Id: I4a02400e65a9b2e0da925b05a2cc4b579e1dd33a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The pack files were left open after the test ended, which meant
we could not delete them automatically when the test was over.
Make sure we close the repositories (and thus their underlying packs)
before the tear down finishes.
Bug: 310367
Change-Id: I4d2703efa4b2e0c347ea4f4475777899cf71073e
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This option was mis-named from day 1. Its not checking that the
objects provided by the client are reachable, its actually doing
a scan to prove that objects referenced by the client are already
reachable through another reference on the server, or were sent
as part of the pack from the client.
Rename it checkReferencedObjectsAreReachable, since we really are
trying to validate that objects referenced by the client's actions
are reachable to the client.
We also need to ensure we run checkConnectivity() anytime this is
enabled, even if the caller didn't turn on fsck for object formats.
Otherwise the check would be completely bypassed.
Change-Id: Ic352ddb0ca8464d407c6da5c83573093e018af19
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If ensureProvidedObjectsVisible is enabled we expected any trees or
blobs directly reachable from an advertised reference to be marked
with UNINTERESTING. Unfortunately ObjectWalk doesn't bother setting
this until the traversal is complete. Even then it won't necessarily
set it on every tree if the corresponding commit wasn't popped.
When we are going to check the base objects for the received pack,
ensure the UNINTERESTING flag gets carried into every immediately
reachable tree or blob, because these are the ones that the client
might try to use as delta bases in a thin pack.
Change-Id: I5d5fdcf07e25ac9fc360e79a25dff491925e4101
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If we need to append less than 20 bytes in order to fix a thin pack
and make it complete, we need to set the length of our file back to
the actual number of bytes used because the original SHA-1 footer was
not completely overwritten. That extra data will confuse the header
and footer fixup logic when it tries to read to the end of the file.
This isn't a very common case to occur, which is why we've never
seen it before. Getting a delta that requires a whole object which
uses less than 20 bytes in pack representation is really hard.
Generally a delta generator won't make these, because the delta
would be bigger than simply deflating the whole object. I only
managed to do this with a hand-crafted pack file where a 1 byte
delta was pointed to a 1 byte whole object.
Normally we try really hard to avoid truncating, because its
typically not safe across network filesystems. But the odds of
this occurring are very low. This truncation is done on a file
we have open for writing, will append more content onto, and is
a temporary file that we won't move into position for others to
see until we've validated its SHA-1 is sane. I don't think the
truncate on NFS issue is something we need to worry about here.
Change-Id: I102b9637dfd048dc833c050890d142f43c1e75ae
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The JSch bundle in Eclipse 3.4 does not export its packages with
version numbers. Use Require-Bundle on version 0.1.37 that comes
with Eclipse 3.4
There is no 0.1.37 in the maven repositories so the pom still refers
to 0.1.41 so the build can get the compile time dependencies right.
Bug: 308031
CQ: 3904 jsch Version: 0.1.37 (using Orbit CQ2014)
Change-Id: I12eba86bfbe584560c213882ebba58bf1f9fa0c1
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Since the API is changing relative to 0.7.0, we'll call our next
release 0.8.1. But until that gets released, builds from master
will be 0.8.0.qualifier.
Change-Id: I921e984f51ce498610c09e0db21be72a533fee88
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some transports actually provide stream buffering on their own,
without needing to be wrapped up inside of a BufferedInputStream in
order to smooth out system calls to read or write. A great example
of this is the JSch SSH client, or the Apache MINA SSHD server.
Both use custom buffering to packetize the streams into the encrypted
SSH channel, and wrapping them up inside of a BufferedInputStream
or BufferedOutputStream is relatively pointless.
Our SideBandOutputStream implementation also provides some fairly
large buffering, equal to one complete side-band packet on the main
data channel. Wrapping that inside of a BufferedOutputStream just to
smooth out small writes from PackWriter causes extra data copies, and
provides no advantage. We can save some memory and some CPU cycles
by letting PackWriter dump directly into the SideBandOutputStream's
internal buffer array.
Instead we push the buffering streams down to be as close to the
network socket (or operating system pipe) as possible. This allows
us to smooth out the smaller reads/writes from pkt-line messages
during advertisement and negotation, but avoid copying altogether
when the stream switches to larger writes over a side band channel.
Change-Id: I2f6f16caee64783c77d3dd1b2a41b3cc0c64c159
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Instead of relying on our callers to wrap us up inside of a
BufferedOutputStream and using the proper block sizing, do the
buffering directly inside of SideBandOutputStream. This ensures
we don't get large write-throughs from BufferedOutputStream that
might overflow the configured packet size.
The constructor of SideBandOutputStream is also beefed up to check
its arguments and ensure they are within acceptable ranges for the
current side-band protocol.
Change-Id: Ic14567327d03c9e972f9734b8228178bc448867d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The tests were using a Locale.ROOT constant which was introduced
in Java 6. However, we need to retain Java 5 support.
Change-Id: I75c5648fcfc728a9aea2e839d2ad0320f5cf742f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Sasa Zivkov <sasa.zivkov@sap.com>
The support for NLS relies on java.util API to load a standard
ResourceBundle and then uses java reflection API to inject localized
strings into public String fields of the corresponding instance
of TranslationBundle.
Locale setting is supported per thread to enable concurrent threads
to use different locales. This is useful when JGit runs in a server
context where (error) messages might need to differ per-request to
suit the user's preference.
Change-Id: Ie0e63a0d7bb74eaad495dbe8248595d8a3a76883
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
If the build server is really busy, we might wait longer than 250 ms
before being interrupted, simply because one of our threads couldn't
be scheduled onto a CPU. Don't make that cause a test failure.
Instead tolerate longer than expected waits, but not shorter waits.
Change-Id: I64511eec24b49e33928451e4c8b8c124eddaf0c2
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
A 0 file mode in a DirCacheEntry is not a valid mode. To C git
such a value indicates the record should not be present. We already
were catching this bad state and exceptioning out when writing tree
objects to disk, but we did not fail when writing the dircache back
to disk. This allowed JGit applications to create a dircache file
which C git would not like to read.
Instead of checking the mode during writes, we now check during
mutation. This allows application bugs to be detected sooner and
closer to the cause site. It also allows us to avoid checking most
of the records which we read in from disk, as we can assume these
are formatted correctly.
Some of our unit tests were not setting the FileMode on their test
entry, so they had to be updated to use REGULAR_FILE.
Change-Id: Ie412053c390b737c0ece57b8e063e4355ee32437
Originally: http://thread.gmane.org/gmane.comp.version-control.git/128214/focus=128213
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Adam W. Hawks <awhawks@writeme.com>
Actually set the range of versions we are willing to accept for
each package we import, lest we import something in the future
that isn't compatible with our needs.
Change-Id: I25dbbb9eaabe852631b677e0c608792b3ed97532
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The supplied test case comes out of the example tree identified by
Robert de Wilde and Ilari on #git:
$ git ls-tree -rt a54f1a85ebf6a7f53aa60a45a1be33f8b078fb7e
040000 tree bfe058ad536cdb12e127cde63b01472c960ea105 A
040000 tree 4b825dc642 A/A
040000 tree 4b825dc642 A/B
100644 blob abbbfafe3129f85747aba7bfac992af77134c607 B
In this tree, "B" was being skipped because "A/A" as an empty tree
was immediately followed by "A/B", also an empty tree, but the
ObjectWalk broke out too early and never visited "B".
Bug: 286653
Change-Id: I25bcb0bc99d0cbbbdd9c2bd625ad6a691a6d0335
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We didn't skip the correct number of bytes when we skipped over an
unrecognized but optional dircache extension. We missed skipping
the 8 byte header that makes up the extension's name and length.
We also didn't include the skipped extension's payload as part of
our index checksum, resuting in a checksum failure when the index
was done reading. So ensure we always scan through a skipped
section and include it in the checksum computation.
Add a test case for a currently unsupported index extension, 'ZZZZ',
to verify we can still read the DirCache object even though we
don't know what 'ZZZZ' is supposed to mean.
Bug: 301287
Change-Id: I4bdde94576fffe826d0782483fd98cab1ea628fa
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Jeff Schumacher
Shawn O. Pearce
Matthias Sohn
Mathias Kinzler
Marc Strapetz
Robin Rosenberg
Christian Halstrick
Stefan Lay
Sasa Zivkov