# In-House Server *Eight machines on paper, three never written to, the CRM somewhere else, and no backups of any kind.* **Category:** INFRASTRUCTURE / DEVOPS **Status:** live **Period:** August – September 2026 **Built by:** HM Ammar, Digital Architect & Head of IT, Kuala Lumpur **Stack:** Proxmox VE, Proxmox Backup Server, Backblaze B2 (S3), LVM-thin, nginx, Let's Encrypt, Bitrix24 / Percona MySQL, systemd ## The challenge A contractor built an in-house server for a travel company and handed over two PDFs describing eight virtual machines. I surveyed it and found a system that did not match the documents. Three of the eight machines had never had a byte written to them, the live CRM sat somewhere other than the paperwork claimed, and the mirrored enterprise disks in the plan had never been bought. Everything ran on one consumer NVMe, and there were no backups of any kind. ## What I built I installed Proxmox Backup Server on the hypervisor and put its datastore in Backblaze B2 rather than on local disk, encrypted client-side so the provider holds only ciphertext. Nightly at 02:00, 7/4/6 retention, weekly garbage collection, monthly verify: five machines at handover, six once the agent host had real data on it. I then audited the estate against the handover documents and wrote the corrections down. I migrated two sites off shared hosting, one of them replacing a WordPress install that was returning 500s, cut two more that already ran on the box over to their production domains, and rebuilt a dormant, never-installed machine in place from an Ubuntu cloud image as a Python and Node host for agent deployments. ## The detail ### Native S3 datastore The bundled S3 client exposes exactly two provider quirks: skip the if-none-match header, and delete objects one at a time. Those are precisely Backblaze B2's two gaps against AWS S3, so I could make B2 a first-class datastore instead of a sync target. Two more things I only found by hitting them. B2 requires path-style addressing, and the bucket's default keep-all-versions lifecycle is a cost trap, because garbage collection on content-addressed chunks writes delete markers that stay billable forever and retention never reclaims a byte. ### Cache placement The S3 datastore needs a local cache. I put it on the root filesystem, capped at 32 GB, not on the thin pool where the virtual machines live. That pool is over-committed by design: if every guest filled its virtual disk it would already sit at 93 percent, and a thin pool that fills takes every machine on it read-only at the same moment. ### A 29-hour freeze Five days in, the new system caused the incident I had built it to prevent. A stalled upload left a backup block job running for 29 hours, freezing every write inside the CRM guest, and the CRM was then repointed to a stale twin, so the team opened it and saw data from April. Nothing was lost. Cancelling the stuck block job brought the machine back with data up to the freeze, and I enabled backup fleecing permanently so a cloud stall can never freeze a guest again. ### Copying the CRM A 64 GB, 1,776-table Bitrix CRM had to move off a cloud VPS with only 35 GB free, so it could not hold an archive of itself and I had to stream the copy. My first attempt ran as an SSH-anchored pipe and died at 1,312 of 1,776 tables after nearly nine hours when the connection blipped; I rebuilt it as a detached systemd unit that pulled the dump to local disk and imported from disk with no network involved, gated on a check that the file held exactly 1,776 CREATE TABLE statements before the target database was dropped. That check then earned itself, aborting a later run whose dump came back with 1,113 tables, and the dump only became reliable once I suspended the CRM's own agent cron for the window, since it performed schema changes every minute and gave a long dump roughly that many chances to invalidate its snapshot. ### A login loop A Bitrix control panel accepted the password and bounced straight back to the form, while its own audit log recorded every authorisation succeeding. TLS terminates on the reverse proxy and Apache only ever saw plain HTTP, so the HTTPS server variable was unset, and a request header named HTTPS reaches PHP as HTTP_HTTPS instead. With secure password cookies switched on, Bitrix authenticated the user and then declined to issue the cookie that would carry that login into the next request. Fixing it exposed a third defect, rotating CDN edge addresses tripping the session-to-IP binding, and when the engagement paused I had found and proven three real defects while the symptom itself was still not fully closed. ## The outcome The server went from no backups of any kind to nightly encrypted offsite backups of every machine holding real data, which I proved by pulling a single file back out of the cloud and decrypting it. My first fortnight was not clean: a run that failed every machine on an ownership error, four nights lost to an exceeded storage cap, a snapshot that wrote a configuration blob with no disk image, and the 29-hour freeze. Each of those is why the configuration now looks the way it does. The backups are crash-consistent by my explicit choice, with the guest agent and database dumps deferred. The CRM copy landed with record counts matching the source exactly rather than approximately, 1,776 of 1,776 tables, and four production sites now run on the box. I wrote a team handover guide and left an offline paper copy of the encryption key, so the system does not depend on me to be understood. --- [All work](https://hmammar.world/) · [HTML version](https://hmammar.world/work/in-house-server.html)