I implemented this (HS5). Etags match the AWS S3 ones as well.

Awesome, I will take a look! We need something for our end-to-end tests at work that we can use and trivially spin up buckets for millions of tests in parallel.

Ceph is pretty much the standard, I think. Very complex to manage, though, but that might just be necessary complexity for a distributed storage system.

My https://github.com/uroni/hs5 is designed for this use case.

One notable thing is that compared to MinIO (and others) it does not store the objects as individual files. I also have DuckDB directly integrated.

The readme has a comparison to Garage, seaweedfs, RustFS and Ceph.

Very cool, I like the conceptual simplicity of it. Even I can understand it, at least at a surface level :-)

Is there a safe way to make backups of the index.lmdb and data0 files without stopping the server?

Not without snapshot currently. I guess I could collect all writes in the (optional) WAL file while a backup is running, then the files would be consistent.

That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.

Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there. I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.

Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.

By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.

FWIW I had this issue even with the MDB_NOSYNC flag so it shouldnt be force flushing to disk unless I'm out of ram or whatever

LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.

Dropping the explicit P_DIRTY flag in 1.0 is a neat change. What tracks which pages still need to be written back at commit now that the flag is gone?

The txnID was added to the page header to enable support for incremental backup. As a consequence, it's sufficient to compare a page's txnID to the current txnID to know if it's dirty or not, and spilled pages don't need a cleanup pass to clear their dirty bit on commit so commits of large txns are faster now.

In my similar project (s3 compatible single-node storage) https://github.com/uroni/hs5 I do use proper fsync for data and metadata durability. But it can be turned of via switch. It is a pet peeve of mine that the defaults should always be to fsync. I do have a section on this in my README of the project.

I also do have an optional WAL. Maybe I should add an additional mode that disables fsync only for the WAL. I don't think it would be a good idea. My WAL does use checksums and sequence numbers etc. to prevent committing wrong data.

I build https://github.com/uroni/hs5 as replacement for single node use with a focus on high performance. I list other alternatives in the README there. Some short version:

Ceph: Robust, widely used for multi-node deployments. Would recommend for serious use.

RustFS: As an in-place replacement using the same storage format. Though, to me it is a bit suspect, e.g. if it uses fsync for durability.

seaweedfs: Multi-node alternative, that keeps the object mapping in memory (so more RAM usage and startup cost compared to alternatives).

Garage: Multi-node alternative, web-interface available separately. To me seems unsuitable for single-node use at this point.

VersityGW: Single-node alternative, which uses the same object=file in file system tree storage as MinIO (with the same disadvantages/advantages). It uses extended attributes for S3 metadata, so less filesystem overhead than MinIO/RustFS at least. Cannot find any Sync() or fsync calls in the code, though.

At least on ZFS, VersityGW absolutely does use sync IO, found it out the hard way on an HDD zpool with no SLOG...

I'd worry about file create, write, then fsync performance with btrfs, but not about reliability or data-loss.

But a quick grep across versitygw tells me they don't use Sync()/fsync, so not a problem... Any data loss occurring from that is obviously not btrfs fault.

Yup, https://github.com/awslabs/git-remote-s3 (disclaimer: never used it)

Run.

I've used it in a product for a couple of thousand repos. The big problem is architectural. Each branch is serialized into a single bundle file. There is no structural sharing of git objects. So each and every branch will download the full history from scratch. So changing branches is as expensive as a fresh clone. If you combine this with a real user's desire for images/diagrams of any kind, then boom, massive slowness.

There are also two concurrency bugs which the maintainer refuses to acknowledge.

For me it went into the multi-node direction, where I'd use Ceph anyway (or build on-top of an existing solid distributed database) if I needed it.

Also think there is an abstraction mismatch with the object stores that store the objects with a 1:1 mapping into the file system. Obvious issues are that you only get good listing performance with '/' as delimiter and things like "keys with length up to 1024 bytes" break.

I made https://github.com/uroni/hs5 -- focus is on single node and high performance. So plenty of alternatives available.