Implement LfsProtocolServlet handling the "Git LFS v1 Batch API"
protocol [1]. Add a simple file system based LFS content store and the
debug-lfs-store command to simplify testing.
Introduce a LargeFileRepository interface to enable additional storage
implementation while reusing the same protocol implementation.
At the client side we have to configure the lfs.url, specify that
we use the batch API and we don't use authentication:
[lfs]
url = http://host:port/lfs
batch = true
[lfs "http://host:port/lfs"]
access = none
the git-lfs client appends the "objects/batch" to the lfs.url.
Hard code an Authorization header in the FileLfsRepository.getAction
because then git-lfs client will skip asking for credentials. It will
just forward the Authorization header from the response to the
download/upload request.
The FileLfsServlet supports file content storage for "Large File
Storage" (LFS) server as defined by the Github LFS API [2].
- upload and download of large files is probably network bound hence use
an asynchronous servlet for good scalability
- simple object storage in file system with 2 level fan-out
- use LockFile to protect writing large objects against multiple
concurrent uploads of the same object
- to prevent corrupt uploads the uploaded file is rejected if its hash
doesn't match id given in URL
The debug-lfs-store command is used to run the LfsProtocolServlet and,
optionally, the FileLfsServlet which makes it easier to setup a
local test server.
[1]
https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md
[2] https://github.com/github/git-lfs/tree/master/docs/api
Bug: 472961
Change-Id: I7378da5575159d2195138d799704880c5c82d5f3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
The Branch class has a member named 'branch', which was being
hidden by the local variable of the same name used in a for-loop.
Change-Id: I334092010a9c80686fb79713852d4bfa166ce12f
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
Experimental flag to turn on the KetchLeader within this daemon JVM.
This is a manually elected leader process, set from the command line.
Remote followers for each repository are configured per-repository
using remote sections with ketch-type = FULL. For example:
Manually elected leader's $GIT_DIR/config:
[ketch]
name = A
[remote "A"]
ketch-type = FULL
[remote "B"]
url = git://127.0.0.1:9421/sample.git
ketch-type = FULL
[remote "C"]
url = git://127.0.0.1:9422/sample.git
ketch-type = FULL
Replica B and C daemons:
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9421 \
--base-path=$HOME/ketch_test/follower_one \
$HOME/ketch_test/follower_one &
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9422 \
--base-path=$HOME/ketch_test/follower_two \
$HOME/ketch_test/follower_two &
Change-Id: I165f85970a77e16b5263115290d685d8a00566f5
HEAD is not part of getRefs(ALL) and must be copied explicitly.
This allows `jgit debug-rebuild-ref-tree --enable` to convert an
existing repository to use RefTree with a local working tree:
mkdir testRepo
cd testRepo
jgit init
jgit debug-rebuild-ref-tree --enable
touch a
jgit add a
jgit commit -m initial
Change-Id: I46cbc2611b9ae683ef7319dc46af277925dfaee5
This experimental code can be enabled in $GIT_DIR/config:
[core]
repositoryformatversion = 1
[extensions]
refsBackendType = RefTree
When these are set the repository will read references from the
RefTree rooted by the $GIT_DIR/refs/txn/committed reference.
Update debug-rebuild-ref-tree to rebuild refs/txn/committed only from
the bootstrap layer. This avoids misuse by rebuilding using packed-refs
and $GIT_DIR/refs tree.
Change-Id: Icf600e4a36b2f7867822a7ab1f1617d73c710a4b
This tool scans all references in the repository and writes out a new
reference pointing to a single commit whose root tree is a RefTree
containing the current refs of this repository.
It alway skips storing the reference it will write to, avoiding the
obvious cycle.
Change-Id: I20b1eeb81c55dc49dd600eac3bf8f90297394113
Currently execution of tests in pgm uses CLIGitCommand which
re-implements few things from Main. Unfortunately this can results in a
different test behavior compared to the real CLI runtime.
The change let CLIGitCommand extend Main and only slightly modifies the
runtime (stream redirection and undesired exit() termination).
Change-Id: I87b7b61d1c84a89e5917610d84409f01be90b70b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
git branch -d -h reports an error (because of missing -d option value)
but does not print the help as expected.
To fix this, CmdLineParser must catch, print but do not propagate
exceptions if help is requested.
Bug: 484951
Change-Id: I51265ebe295f22da540792c6a1980b8bdb295a02
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
In different places (Main, TextBuiltin, CLIGitCommand) we report fatal
errors and at same time want to check for fatal errors in the tests.
Using common API simplifies the error testing and helps to navigate to
the actual error check implementation.
Change-Id: Iecde79beb33ea595171f168f46b0b10ab2f339bb
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Run git describe during the build to determine the lineage of this
working directory and stamp this information into the binary.
Change-Id: I0ad24125c31e4280ccf900bac4065924087b05aa
Compile each test in its own java_test() target so they can run in
parallel, reducing total time spent testing on large machines.
$ buck test --all
[-] PROCESSING BUCK FILES...FINISHED 0.3s [100%]
[-] BUILDING...FINISHED 2.9s [100%] (351/383 JOBS, 351 UPDATED, 0.0% CACHE MISS)
[-] TESTING...FINISHED 98.1s (3360 PASS/15 SKIP/0 FAIL)
Change-Id: I8d6541268315089299f933ed23d785b1b3431133
Construct the java_application JAR wrapped with the shell script
header. This is enough to clone a repository over HTTPs:
$ buck build :jgit_bin
$ buck-out/gen/jgit_bin/jgit_bin clone https://...
Change-Id: I4aceb4e77b2ec9be76a32ec93d94f2dafe9acce6
The "--" path option (and all other similar options consuming all
remaining arguments) should be placed at the end of the command line
help.
Currently jgit reset -h shows this:
jgit reset [commit-ish] [path ... ...] [-- path ... ...] [--hard]
[--help (-h)] [--mixed] [--soft]
After the patch the help shows this:
jgit reset [commit-ish] [path ... ...] [--hard] [--help (-h)] [--mixed]
[--soft] [-- path ... ...]
Bug: 484951
Change-Id: I3db332bf293ca8d6bfaab0d546cd35af689bd46e
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
This commit changes the jgit "reset" command line options help from
this:
jgit reset name [VAL ...] [-- path ... ...] [--hard] [--help (-h)]
[--mixed] [--soft]
name : Reset current HEAD to the specified state
[...]
to this:
jgit reset [commit-ish] [path ... ...] [-- path ... ...] [--hard]
[--help (-h)] [--mixed] [--soft]
commit-ish : Reset to given reference name
[...]
Bug: 484951
Change-Id: I614e71101b4f9f46ef8f02379d1a9d135f3292d2
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
See bug 484951 comment 4: "jgit status -- a b" doesn't work and
complains that "b" is not an allowed argument
Bug: 484951
Change-Id: I86b81e7f2bab6e928bb8e973bd50c8f4b9c6fecf
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Fixes point 4 in bug 484951, where "jgit repo" or "jgit repo -h" dumps a
stack trace.
Bug: 484951
Change-Id: Ic8b362e07a40ad923dc9acde0c0983a1e7932a02
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signal early command termination due '-h' or '--help' option via
TerminatedByHelpException. This allows tests using
CLIGitCommand differentiate between unexpected command parsing errors
and expected command cancellation "on help" (which also allows
validation of expected/unexpected help messages).
Additional side-effect: jgit supports now git style of handling help
option: any unexpected command line options before help are reported as
errors, but after help ignored.
Bug: 484951
Change-Id: If45c41c0d32895ab6822a7ff9d851877dcef5771
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
JGit CLI should allow to do this: checkout -- <path>
Currently, even if "a" is a valid path in the git repo, jgit CLI can't
checkout it:
$jgit checkout -- a
error: pathspec 'a' did not match any file(s) known to git.
The fix also fixes at same time "unnamed" zombie "[VAL ...]" argument
shown on the command line.
Before fix:
$jgit -h
jgit checkout name [VAL ...] [-- path ... ...] [--force (-f)] [--help
(-h)] [--orphan] [-b]
After fix:
$jgit -h
jgit checkout [name] [-- path ... ...] [--force (-f)] [--help (-h)]
[--orphan] [-b]
Bug: 475765
Change-Id: I2b0e77959a72e4aac68452dc3846adaa745b0831
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Pushing with JGit commandline to e.g. Github failed with "unauthorized"
since HttpUrlConnection calls the configured authenticator implicitly.
The problem is that during a push two requests are sent to the server,
first a GET and then a POST (containing the pack data). The first GET
request sent anonymously is rejected with 401 (unauthorized). When an
Authenticator is installed the java.net classes will use the
Authenticator to ask the user for credentials and retry the request.
But this happens under the hood and JGit level code doesn't see that
this happens.
The next request is the POST but since JGit thinks the first GET request
went through anonymously it doesn't add authentication headers to the
POST request. This POST of course also fails with 401 but since this
request contains a lot of body-data streamed from JGit (the pack file!)
the java.net classes can't simply retry the request with authorization
headers. The whole process fails.
Fix this by using Apache httpclient which doesn't use Authenticator to
retrieve credentials. Instead initialize TransportCommand to use the
default credential provider if no other credentials provider was set
explicitly. org.eclipse.jgit.pgm.Main sets this default for the JGit
command line client.
Change-Id: Ic4e0f8b60d4bd6e69d91eae0c7e1b44cdf851b00
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.
Change-Id: I24c8a600ec962d61d5f40abf73eac4203e115240
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
This should mirror the behavior of `git push --atomic` where the
client asks the server to apply all-or-nothing. Some JGit servers
already support this based on a custom DFS backend. InMemoryRepository
is extended to support atomic push for unit testing purposes.
Local disk server side support inside of JGit is a more complex animal
due to the excessive amount of file locking required to protect every
reference as a loose reference.
Change-Id: I15083fbe48447678e034afeffb4639572a32f50c
org.eclipse.jgit.lib.Repository class is an example of the API which
should be written with Java 8 java.util.Optional<T> type. Unfortunately
this API is already released and widely used. The good clients are
currently doing their best with checking return values for null and bad
clients do not know how bad their code is.
I've tried not to change any logic and to be as less intrusive as
possible. Most of the JGit code was well prepared to this, only few
classes needed some smaller fixes.
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.
Because annotating getDirectory() and getFS() as Nullable would cause
lot of additional changes in JGit and EGit they are postponed.
Change-Id: Ie8369d2c9c5fac5ce83b3b1b9bc217d7b55502a3
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Add the --verify option to be more compatible with git
Change-Id: I225a36ecc4711fd2eb9af67ca8fb79681d94c587
Signed-off-by: Thomas Meyer <thomas.mey@web.de>
Update the project-specific Eclipse settings to replace the use of the
org.eclipse.jdt.annotation.Nullable class the new JGit-specific
@Nullable annotation. I verified that Eclipse reports errors when the
return value of a method annotated with
@org.eclipse.jgit.annotations.Nullable is dereferenced without a null
check.
Also remove the Maven and MANIFEST.MF dependencies on
org.eclipse.jdt.annotation.
Eclipse null analysis uses three annotations: @Nullable, @NonNull and
@NonNullByDefault. All three are updated in this patch because it is
invalid to set the Eclipse preferences to empty values. So far only
@Nullable has been introduced in org.eclipse.jgit.annotations.
My personal preference is to follow the advice in Effective Java and
avoid the null-return idiom, and to avoid passing null values in
general. This sets the expectation is that arguments and return types
are assumed non-null unless otherwise documented. If that is the
expectation, then consistent application of @NonNull is redundant and
hurts readability by cluttering the code, obscuring the occasional
@Nullable annotation that really requires attention.
If the JGit community decides there is value in using the @NonNull and
@NonNullByDefault annotations we can add them--this change configures
Eclipse to use them.
Change-Id: I9af1b786d1b44b9b0d9c609480dc842df79bf698
Signed-off-by: Terry Parker <tparker@google.com>
On a server also running Gerrit that is using RepoCommand to
convert from an XML manifest to a git submodule superproject
periodically, it would be handy to be able to use Gerrit's
submodule subscription feature[1] to update the superproject
automatically between RepoCommand runs as changes are merged
in each subprojects.
This requires setting the 'branch' field for each submodule
so that Gerrit knows what branch to watch. Add an option to
do that.
Setting the branch field also is useful for plain Git users,
since it allows them to use "git submodule update --remote" to
manually update all submodules between RepoCommand runs.
[1] https://gerrit-review.googlesource.com/Documentation/user-submodules.html
Change-Id: I1a10861bcd0df3b3673fc2d481c8129b2bdac5f9
Signed-off-by: Stefan Beller <sbeller@google.com>
Matthias Sohn
David Pursehouse
Shawn Pearce
Andrey Loskutov
Shawn Pearce
Rüdiger Herrmann
Kaloyan Raev
Thomas Meyer
Terry Parker
Stefan Beller