纯java实现的git操作库。
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

90 lines
4.2 KiB

Store Git on any DHT jgit.storage.dht is a storage provider implementation for JGit that permits storing the Git repository in a distributed hashtable, NoSQL system, or other database. The actual underlying storage system is undefined, and can be plugged in by implementing 7 small interfaces: * Database * RepositoryIndexTable * RepositoryTable * RefTable * ChunkTable * ObjectIndexTable * WriteBuffer The storage provider interface tries to assume very little about the underlying storage system, and requires only three key features: * key -> value lookup (a hashtable is suitable) * atomic updates on single rows * asynchronous operations (Java's ExecutorService is easy to use) Most NoSQL database products offer all 3 of these features in their clients, and so does any decent network based cache system like the open source memcache product. Relying only on key equality for data retrevial makes it simple for the storage engine to distribute across multiple machines. Traditional SQL systems could also be used with a JDBC based spi implementation. Before submitting this change I have implemented six storage systems for the spi layer: * Apache HBase[1] * Apache Cassandra[2] * Google Bigtable[3] * an in-memory implementation for unit testing * a JDBC implementation for SQL * a generic cache provider that can ride on top of memcache All six systems came in with an spi layer around 1000 lines of code to implement the above 7 interfaces. This is a huge reduction in size compared to prior attempts to implement a new JGit storage layer. As this package shows, a complete JGit storage implementation is more than 17,000 lines of fairly complex code. A simple cache is provided in storage.dht.spi.cache. Implementers can use CacheDatabase to wrap any other type of Database and perform fast reads against a network based cache service, such as the open source memcached[4]. An implementation of CacheService must be provided to glue this spi onto the network cache. [1] https://github.com/spearce/jgit_hbase [2] https://github.com/spearce/jgit_cassandra [3] http://labs.google.com/papers/bigtable.html [4] http://memcached.org/ Change-Id: I0aa4072781f5ccc019ca421c036adff2c40c4295 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
14 years ago
JGit Storage on DHT
-------------------
This implementation still has some pending issues:
* DhtInserter must skip existing objects
DirCache writes all trees to the ObjectInserter, letting the
inserter figure out which trees we already have, and which are new.
DhtInserter should buffer trees into a chunk, then before writing
the chunk to the DHT do a batch lookup to find the existing
ObjectInfo (if any). If any exist, the chunk should be compacted to
eliminate these objects, and if there is room in the chunk for more
objects, it should go back to the DhtInserter to be filled further
before flushing.
This implies the DhtInserter needs to work on multiple chunks at
once, and may need to combine chunks together when there is more
than one partial chunk.
* DhtPackParser must check for collisions
Because ChunkCache blindly assumes any copy of an object is an OK
copy of an object, DhtPackParser needs to validate all new objects
at the end of its importing phase, before it links the objects into
the ObjectIndexTable. Most objects won't already exist, but some
may, and those that do must either be removed from their chunk, or
have their content byte-for-byte validated.
Removal from a chunk just means deleting it from the chunk's local
index, and not writing it to the global ObjectIndexTable. This
creates a hole in the chunk which is wasted space, and that isn't
very useful. Fortunately objects that fit fully within one chunk
may be easy to inflate and double check, as they are small. Objects
that are big span multiple chunks, and the new chunks can simply be
deleted from the ChunkTable, leaving the original chunks.
Deltas can be checked quickly by inflating the delta and checking
only the insertion point text, comparing that to the existing data
in the repository. Unfortunately the repository is likely to use a
different delta representation, which means at least one of them
will need to be fully inflated to check the delta against.
* DhtPackParser should handle small-huge-small-huge
Multiple chunks need to be open at once, in case we get a bad
pattern of small-object, huge-object, small-object, huge-object. In
this case the small-objects should be put together into the same
chunk, to prevent having too many tiny chunks. This is tricky to do
with OFS_DELTA. A long OFS_DELTA requires all prior chunks to be
closed out so we know their lengths.
* RepresentationSelector performance bad on Cassandra
The 1.8 million batch lookups done for linux-2.6 kills Cassandra, it
cannot handle this read load.
* READ_REPAIR isn't fully accurate
There are a lot of places where the generic DHT code should be
helping to validate the local replica is consistent, and where it is
not, help the underlying storage system to heal the local replica by
reading from a remote replica and putting it back to the local one.
Most of this should be handled in the DHT SPI layer, but the generic
DHT code should be giving better hints during get() method calls.
* LOCAL / WORLD writes
Many writes should be done locally first, before they replicate to
the other replicas, as they might be backed out on an abort.
Likewise some writes must take place across sufficient replicas to
ensure the write is not lost... and this may include ensuring that
earlier local-only writes have actually been committed to all
replicas. This committing to replicas might be happening in the
background automatically after the local write (e.g. Cassandra will
start to send writes made by one node to other nodes, but doesn't
promise they finish). But parts of the code may need to force this
replication to complete before the higher level git operation ends.
* Forks/alternates
Forking is common, but we should avoid duplicating content into the
fork if the base repository has it. This requires some sort of
change to the key structure so that chunks are owned by an object
pool, and the object pool owns the repositories that use it. GC
proceeds at the object pool level, rather than the repository level,
but might want to take some of the reference namespace into account
to avoid placing forked less-common content near primary content.