bashclub-zfs-push-pull | zfs snapshot send/receive supporting pull | Incremental Backup library

• What protocol does your NAS use?
• Do you see errors when you run the Backup-SvnRepository PowerShell cmdlet?
• What VisualSVN Server version and the version of SVN command-line tools are you using? I.e., what svnadmin --version says?

Note that you can consider the built-in Backup and Restore feature. It supports backup scheduling, encryption, incremental backups and backups to remote shares and Azure Files cloud. See KB106: Getting Started with Backup and Restore and KB137: Choosing backup destination.

I want to set up an automatic incremental backup of my SVN repositories. Doing that to a local folder of the same PC seems to work, but if I try to write the files directly to a shared folder (I tried 2 different QNAP nas boxes) I get various errors, always a couple of hundred lines.

From what I see, an unexpected network error indeed occurs when you hotcopy the repository onto your NAS. Please, double-check that you are using up-to-date Subversion command-line tools (what svnadmin --version says?).

I've shut down the svn service, so there's no danger of someone working on svn in the meantime. Still same problem.

You don't need to stop the server's services when you run svnadmin hotcopy:

[[[

This subcommand makes a “hot” backup of your repository, including all hooks, configuration files, and, of course, database files. You can run this command at any time and make a safe copy of the repository, regardless of whether other processes are using the repository.

]]]

CreateHardLink create hard link to the symbolic link (reparse point) themselves, not to the target. so documentation simply is wrong. the lpExistingFileName opened with option FILE_OPEN_REPARSE_POINT so you can simply use CreateHardLink as is and nothing more need todo. visa versa - if you want create hard link to target, you need custom implementation of CreateHardLink and not use FILE_OPEN_REPARSE_POINT (if you will use NtOpenFile) or FILE_FLAG_OPEN_REPARSE_POINT if you use CreatFileW )

I did notice Windows' internal command MKLINK appears to be able to create hardlinks to symlinks.

if you debug cmd.exe with mklink command you easy can notice that also simply CreateHardLinkW api called (set breakpoint to it)

after you create hardlink to symlink file you can view in explorer that type of file is .symlink . for test we can remove link from target file ( by use FSCTL_DELETE_REPARSE_POINT) if hardlink point to target - any operation with symlink not affect hardlink. but if we created hardlink to symlink intself - after break symlink - hard link will also be breaked:

I didn't read the entire code because the main problem didn't seem to lay there.
Verify the disk usage of your /Backups directory with du -sh /Backups and then compare it with the sum of du -sh /Backups/Full and du -sh /Backups/Inc.

I'll show you why with a little test:

Create a directory containing a file of 1 MiB:

I solved it! I created a bash script that syncs all snapshots with a date in the name to the remote server. The date is subtracted from btrfs subvolume show.

So daily.0 can become 2021-09-20-08-44-46 on the remote.

I sync backwards. daily.30 first. daily.0 last. This way I can pass the proper parent to btrfs send. E.g.: btrfs send -p daily.30 daily.29.

If the date named snapshot exists on the remote, I check with btrfs subvolume show whether it was properly synced. If not, I delete the remote subvolume/snapshot and re-sync. If it was already synced, I will skip the sync. A proper synced subvolume/snapshot has a Received UUID and readonly flag.

After syncing I compare all snapshot names of the remote to what was just synced. The difference will deleted (thus old snapshots).

I might share the code in the future when it's all been stable for a long run. For now I hope the above information will help others!

Probably you can create a job on Azure Data Factory (aka ADF, https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-cosmos-db), use the ADF job to copy data from these containers, and save the data as files (one file for one container) in somewhere (like Azure Blob Storage).

As @MikeM said, this was happening because I was executing the code from a non UI thread.

I solved it by using a Handler, this one takes care of running the code from the UI thread:

Backups are not copies of the WAL files, they're copies of the cluster's whole data directory. As it says in the docs, an incremental backup contains:

those database cluster files that have changed since the last backup (which can be another incremental backup, a differential backup, or a full backup)

WALs alone aren't enough to restore a database; they only record changes to the cluster files, so they require a backup as a starting point.

The need for periodic backups (incremental or otherwise) is primarily to do with recovery time. Technically, you could just hold on to your original full backup plus years worth of WAL files, but replaying them all in the event of a failure could take hours or days, and you likely can't tolerate that kind of downtime.

A new backup also means that you can safely discard any older WALs (assuming you don't still need them for point-in-time recovery), meaning less data to store, and less data whose integrity you're relying on in order to recover.

If you want to know more about what pgBackRest is actually doing under the hood, it's all covered pretty thoroughly in the Postgres docs.

Yes, you can backup to S3.

You will need to configure the S3 credentials, so that MarkLogic is able to use S3 and read/write objects to your S3 bucket.

MarkLogic can't use S3 for journal archive paths, because S3 does not support file append operations. So if you want to enable journal archives, you will need to specify a custom path for that when creating your backups.

The directory you specified can be an operating system mounted directory path, it can be an HDFS path, or it can be an S3 path. For details on using HDFS and S3 storage in MarkLogic, see Disk Storage Considerations in the Query Performance and Tuning Guide.

S3 requires authentication with the following S3 credentials:

• AWS Access Key
• AWS Secret Key

The S3 credentials for a MarkLogic cluster are stored in the security database for the cluster. You can only have one set of S3 credentials per cluster. You can set up security access in S3, you can access any paths that are allowed access by those credentials. Because of the flexibility of how you can set up access in S3, you can set up any S3 account to allow access to any other account, so if you want to allow the credentials you have set up in MarkLogic to access S3 paths owned by other S3 users, those users need to grant access to those paths to the AWS Access Key set up in your MarkLogic Cluster.

To set up the AW credentials for a cluster, enter the keys in the Admin Interface under Security > Credentials. You can also set up the keys programmatically using the following Security API functions:

• sec:credentials-get-aws
• sec:credentials-set-aws

The credentials are stored in the Security database. Therefore, you cannot use S3 as the forest storage for a security database.

if you want to have Journaling enabled, you will need to have them written to a different location. Journal archiving is not supported on S3.

The default location for Journals are in the backup, but when creating programmatically you can specify a different $journal-archive-path .

Storage on S3 has an 'eventual consistency' property, meaning that write operations might not be available immediately for reading, but they will be available at some point. Because of this, S3 data directories in MarkLogic have a restriction that MarkLogic does not create Journals on S3. Therefore, MarkLogic recommends that you use S3 only for backups and for read-only forests, otherwise you risk the possibility of data loss. If your forests are read-only, then there is no need to have journals.

Unfortunately I do not have a true solution to the issue, instead a re-work was done to the entire code and then implemented within BladeLogic to make it work as needed. As such an answer to this is no longer needed. Thank you to those who attempted to assist.