jgit.storage.dht is a storage provider implementation for JGit that
permits storing the Git repository in a distributed hashtable, NoSQL
system, or other database. The actual underlying storage system is
undefined, and can be plugged in by implementing 7 small interfaces:
* Database
* RepositoryIndexTable
* RepositoryTable
* RefTable
* ChunkTable
* ObjectIndexTable
* WriteBuffer
The storage provider interface tries to assume very little about the
underlying storage system, and requires only three key features:
* key -> value lookup (a hashtable is suitable)
* atomic updates on single rows
* asynchronous operations (Java's ExecutorService is easy to use)
Most NoSQL database products offer all 3 of these features in their
clients, and so does any decent network based cache system like the
open source memcache product. Relying only on key equality for data
retrevial makes it simple for the storage engine to distribute across
multiple machines. Traditional SQL systems could also be used with a
JDBC based spi implementation.
Before submitting this change I have implemented six storage systems
for the spi layer:
* Apache HBase[1]
* Apache Cassandra[2]
* Google Bigtable[3]
* an in-memory implementation for unit testing
* a JDBC implementation for SQL
* a generic cache provider that can ride on top of memcache
All six systems came in with an spi layer around 1000 lines of code to
implement the above 7 interfaces. This is a huge reduction in size
compared to prior attempts to implement a new JGit storage layer. As
this package shows, a complete JGit storage implementation is more
than 17,000 lines of fairly complex code.
A simple cache is provided in storage.dht.spi.cache. Implementers can
use CacheDatabase to wrap any other type of Database and perform fast
reads against a network based cache service, such as the open source
memcached[4]. An implementation of CacheService must be provided to
glue this spi onto the network cache.
[1] https://github.com/spearce/jgit_hbase
[2] https://github.com/spearce/jgit_cassandra
[3] http://labs.google.com/papers/bigtable.html
[4] http://memcached.org/
Change-Id: I0aa4072781f5ccc019ca421c036adff2c40c4295
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Introduce a http test bundle to make this functionality available for
EGit tests. A simple http server class is provided. The jetty version
was updated to a version that is also available via p2 (needed in EGit
UI tests).
Change-Id: I13bfc4c6c47e27d8f97d3e9752347d6d23e553d4
Signed-off-by: Jens Baumgart <jens.baumgart@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Created wrong tags for 0.8.3 hence creating another version.
Change-Id: I4e00bbcffe1cf872e2d7e3f3d88d068701fb5330
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Since the API is changing relative to 0.7.0, we'll call our next
release 0.8.1. But until that gets released, builds from master
will be 0.8.0.qualifier.
Change-Id: I921e984f51ce498610c09e0db21be72a533fee88
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
No Eclipse support for this project is provided, because the
Jetty project does not publish a complete P2 repository.
Change-Id: Ic5fe2e79bb216e36920fd4a70ec15dd6ccfd1468
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Tycho isn't production ready for projects like JGit to be using as
their primary build driver. Some problems we ran into with Tycho
0.6.0 that are preventing us from using it are:
* Tycho can't run offline
The P2 artifact resolver cannot perform its work offline. If the
build system has no network connection, it cannot compile a
project through Tycho. This is insane for a distributed version
control system where developers are used to being offline during
development and local testing.
* Magic state in ~/.m2/repository/.meta/p2-metadata.properties
Earlier iterations of this patch tried to use a hybrid build,
where Tycho was only used for the Eclipse specific feature and P2
update site, and maven-bundle-plugin was used for the other code.
This build seemed to work, but only due to magic Tycho specific
state held in my local home directory. This means builds are not
consistently repeatable across systems, and lead me to believe
I had a valid build, when in fact I did not.
* Manifest-first build produces incomplete POMs
The POM created by the manifest-first build format does not
contain the dependency chain, leading a downstream consumer to
not import the runtime dependencies necessary to execute the
bundle it has imported. In JGit's case, this means JSch isn't
included in our dependency chain.
* Manifest-first build produces POMs unreadable by Maven 2.x
JGit has existing application consumers who are relying on
Maven 2.x builds. Forcing them to step up to an alpha release
of Maven 3 is simply unacceptable.
* OSGi bundle export data management is tedious
Editing each of our pom.xml files to mark a new release is
difficult enough as it is. Editing every MANIFEST.MF file to
list our exported packages and their current version number is
something a machine should do, not a human. Yet the Tycho OSGi
way unfortunately demands that a human do this work.
* OSGi bundle import data management is tedious
There isn't a way in the MANIFEST.MF file format to reuse the
same version tags across all of our imports, but we want to have
a consistent view of our dependencies when we compile JGit.
After wasting more than 2 full days trying to get Tycho to work,
I've decided its a lost cause right now. We need to be chasing down
bugs and critical features, not trying to bridge the gap between
the stable Maven repository format and the undocumented P2 format
used only by Eclipse.
So, switch the build to use Apache Felix's maven-bundle-plugin.
This is the same plugin Jetty uses to produce their OSGi bundle
manifests, and is the same plugin used by the Apache Felix project,
which is an open-source OSGi runtime. It has a reasonable number
of folks using it for production builds, and is running on top of
the stable Maven 2.x code base.
With this switch we get automatically generated MANIFEST.MF files
based on reasonably sane default rules, which reduces the amount
of things we have to maintain by hand. When necessary, we can add
a few lines of XML to our POMs to tweak the output.
Our build artifacts are still fully compatible with Maven 2.x, so
any downstream consumers are still able to use our build products,
without stepping up to Maven 3.x. Our artifacts are also valid as
OSGi bundles, provided they are organized on disk into a repository
that the runtime can read.
With maven-bundle-plugin the build runs offline, as much as Maven
2.x is able to run offline anyway, so we're able to return to a
distributed development environment again.
By generating MANIFEST.MF at the top level of each project (and
therefore outside of the target directory), we're still compatible
with Eclipse's PDE tooling. Our projects can be imported as standard
Maven projects using the m2eclipse plugin, but the PDE will think
they are vaild plugins and make them available for plugin builds,
or while debugging another workbench.
This change also completely removes Tycho from the build.
Unfortunately, Tycho 0.6.0's pom-first dependency resolver is broken
when resolving a pom-first plugin bundle through a manifest-first
feature package, so bundle org.eclipse.jgit can't be resolved,
even though it might actually exist in the local Maven repository.
Rather than fight with Tycho any further, I'm just declaring it
plugina-non-grata and ripping it out of the build.
Since there are very few tools to build a P2 format repository, and
no documentation on how to create one without running the Eclipse
UI manually by poking buttons, I'm declaring that we are not going
to produce a P2 update site from our automated builds.
Change-Id: If7938a86fb0cc8e25099028d832dbd38110b9124
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Matthias Sohn
Chris Aniszczyk
Shawn O. Pearce
Jens Baumgart