01Native S3 datastore
The bundled S3 client exposes exactly two provider quirks: skip the if-none-match header, and delete objects one at a time. Those are precisely Backblaze B2's two gaps against AWS S3, so I could make B2 a first-class datastore instead of a sync target. Two more things I only found by hitting them. B2 requires path-style addressing, and the bucket's default keep-all-versions lifecycle is a cost trap, because garbage collection on content-addressed chunks writes delete markers that stay billable forever and retention never reclaims a byte.
02Cache placement
The S3 datastore needs a local cache. I put it on the root filesystem, capped at 32 GB, not on the thin pool where the virtual machines live. That pool is over-committed by design: if every guest filled its virtual disk it would already sit at 93 percent, and a thin pool that fills takes every machine on it read-only at the same moment.
03A 29-hour freeze
Five days in, the new system caused the incident I had built it to prevent. A stalled upload left a backup block job running for 29 hours, freezing every write inside the CRM guest, and the CRM was then repointed to a stale twin, so the team opened it and saw data from April. Nothing was lost. Cancelling the stuck block job brought the machine back with data up to the freeze, and I enabled backup fleecing permanently so a cloud stall can never freeze a guest again.
04Copying the CRM
A 64 GB, 1,776-table Bitrix CRM had to move off a cloud VPS with only 35 GB free, so it could not hold an archive of itself and I had to stream the copy. My first attempt ran as an SSH-anchored pipe and died at 1,312 of 1,776 tables after nearly nine hours when the connection blipped; I rebuilt it as a detached systemd unit that pulled the dump to local disk and imported from disk with no network involved, gated on a check that the file held exactly 1,776 CREATE TABLE statements before the target database was dropped. That check then earned itself, aborting a later run whose dump came back with 1,113 tables, and the dump only became reliable once I suspended the CRM's own agent cron for the window, since it performed schema changes every minute and gave a long dump roughly that many chances to invalidate its snapshot.
05A login loop
A Bitrix control panel accepted the password and bounced straight back to the form, while its own audit log recorded every authorisation succeeding. TLS terminates on the reverse proxy and Apache only ever saw plain HTTP, so the HTTPS server variable was unset, and a request header named HTTPS reaches PHP as HTTP_HTTPS instead. With secure password cookies switched on, Bitrix authenticated the user and then declined to issue the cookie that would carry that login into the next request. Fixing it exposed a third defect, rotating CDN edge addresses tripping the session-to-IP binding, and when the engagement paused I had found and proven three real defects while the symptom itself was still not fully closed.