msrsync | Multi-stream rsync wrapper | Incremental Backup library

 by   jbd Python Version: Current License: GPL-3.0

kandi X-RAY | msrsync Summary

kandi X-RAY | msrsync Summary

msrsync is a Python library typically used in Backup Recovery, Incremental Backup applications. msrsync has no bugs, it has no vulnerabilities, it has a Strong Copyleft License and it has low support. However msrsync build file is not available. You can download it from GitHub.

msrsync (multi-stream rsync) is a python wrapper around rsync. It only depends on python >= 2.6 and rsync. It will split the transfer in multiple buckets while the source is scanned and will hopefully help maximizing the usage of the available bandwidth by running a configurable number of rsync processes in parallel. The main limitation is it does not handle remote source or target directory, they must be locally accessible (local disk, nfs/cifs/other mountpoint). I hope to address this in a near future.
Support
    Quality
      Security
        License
          Reuse

            kandi-support Support

              msrsync has a low active ecosystem.
              It has 416 star(s) with 67 fork(s). There are 23 watchers for this library.
              OutlinedDot
              It had no major release in the last 6 months.
              There are 3 open issues and 28 have been closed. On average issues are closed in 74 days. There are 2 open pull requests and 0 closed requests.
              It has a neutral sentiment in the developer community.
              The latest version of msrsync is current.

            kandi-Quality Quality

              msrsync has 0 bugs and 0 code smells.

            kandi-Security Security

              msrsync has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
              msrsync code analysis shows 0 unresolved vulnerabilities.
              There are 0 security hotspots that need review.

            kandi-License License

              msrsync is licensed under the GPL-3.0 License. This license is Strong Copyleft.
              Strong Copyleft licenses enforce sharing, and you can use them when creating open source projects.

            kandi-Reuse Reuse

              msrsync releases are not available. You will need to build from source code and install.
              msrsync has no build file. You will be need to create the build yourself to build the component from source.
              Installation instructions, examples and code snippets are available.

            Top functions reviewed by kandi - BETA

            kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
            Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of msrsync
            Get all kandi verified functions for this library.

            msrsync Key Features

            No Key Features are available at this moment for msrsync.

            msrsync Examples and Code Snippets

            No Code Snippets are available at this moment for msrsync.

            Community Discussions

            QUESTION

            Incremental backups (mariabackup) and dropped databases
            Asked 2022-Mar-28 at 05:39

            I tried to search for this case everywhere but, couldn't find anything that answers this - probably weird - question: What happens to the incremental backups taken from a mariadb server using mariabackup if one of the databases is dropped?

            Suppose you dropped one of the databases in a mariadb server, then you created an incremental backup afterwards, where the base full backup certainly includes the dropped database, does applying the incremental backup when preparing to restore include that removal? or that dropped database will still be present in the fully prepared backup?

            PS: I realize that mariabackup uses InnoDB LSN to backup the changes / diffs only but, do those diffs include the removal of a table or a database?

            My guess is that when preparing the incremental backup over the base, it would remove the tables and / or databases which are missing from the latest delta backups but, I might be wrong so, that's why I'm asking.

            ...

            ANSWER

            Answered 2022-Mar-28 at 05:39

            Well, after trying out the scenario I've found out that the dropped databases do exist in the full integral backup but, their tables are removed.

            So, I think that a database structure changes are also included in the incremental backup e.g. modifications in table columns, foreign keys, indices, table creation and dropping etc.. are all tracked but, dropping the database itself is NOT tracked however, a dropped database will have all its tables missing from the result backup of integrating all incremental backups to the base one.

            Source https://stackoverflow.com/questions/71496869

            QUESTION

            Backup of svn repository to shared folder on NAS fails
            Asked 2022-Feb-07 at 14:08

            I want to set up an automatic incremental backup of my SVN repositories. Doing that to a local folder of the same PC seems to work, but if I try to write the files directly to a shared folder (I tried 2 different QNAP nas boxes) I get various errors, always a couple of hundred lines.

            I tried

            ...

            ANSWER

            Answered 2022-Feb-07 at 12:59
            • What protocol does your NAS use?
            • Do you see errors when you run the Backup-SvnRepository PowerShell cmdlet?
            • What VisualSVN Server version and the version of SVN command-line tools are you using? I.e., what svnadmin --version says?

            Note that you can consider the built-in Backup and Restore feature. It supports backup scheduling, encryption, incremental backups and backups to remote shares and Azure Files cloud. See KB106: Getting Started with Backup and Restore and KB137: Choosing backup destination.

            I want to set up an automatic incremental backup of my SVN repositories. Doing that to a local folder of the same PC seems to work, but if I try to write the files directly to a shared folder (I tried 2 different QNAP nas boxes) I get various errors, always a couple of hundred lines.

            From what I see, an unexpected network error indeed occurs when you hotcopy the repository onto your NAS. Please, double-check that you are using up-to-date Subversion command-line tools (what svnadmin --version says?).

            I've shut down the svn service, so there's no danger of someone working on svn in the meantime. Still same problem.

            You don't need to stop the server's services when you run svnadmin hotcopy:

            [[[

            This subcommand makes a “hot” backup of your repository, including all hooks, configuration files, and, of course, database files. You can run this command at any time and make a safe copy of the repository, regardless of whether other processes are using the repository.

            ]]]

            Source https://stackoverflow.com/questions/71018537

            QUESTION

            Hard link to a symbolic link with Win32 API?
            Asked 2021-Nov-26 at 08:59

            Quick background to this question as I'm sure it'll raise a few eyebrows: I'm developing a command line tool in C for making backups, and I am implementing incremental backups using NTFS hard links. Thus, if symbolic links exists in a prior backup, I must be able to point to the symbolic links themselves, not the target.

            Unfortunately, the page for CreateHardLink clearly states:

            Symbolic link behavior—If the path points to a symbolic link, the function creates a hard link to the target.

            Now I'm stuck wondering, what's the solution to this? How can I create a hardlink that points to a symbolic link itself as opposed to the target? I did notice Windows' internal command MKLINK appears to be able to create hardlinks to symlinks. So theoretically, I guess I could just use the system function in C, but to be honest, it feels lazy and I tend to avoid it. Is there possibly a solution using only the Win32 API?

            I also came across some code snippets from a Google developer ([1] [2]), with some details on the implementation of CreateHardLink and whatnot, but it seemed a little too low level for me to make any real sense out of it. Also, (and I could be wrong about this) the functions provided in the GitHub repo seem to only be compatible with Windows 10 and later, but I'd hope to at least support Windows 7 as well.

            ...

            ANSWER

            Answered 2021-Nov-26 at 08:59

            CreateHardLink create hard link to the symbolic link (reparse point) themselves, not to the target. so documentation simply is wrong. the lpExistingFileName opened with option FILE_OPEN_REPARSE_POINT so you can simply use CreateHardLink as is and nothing more need todo. visa versa - if you want create hard link to target, you need custom implementation of CreateHardLink and not use FILE_OPEN_REPARSE_POINT (if you will use NtOpenFile) or FILE_FLAG_OPEN_REPARSE_POINT if you use CreatFileW )

            I did notice Windows' internal command MKLINK appears to be able to create hardlinks to symlinks.

            if you debug cmd.exe with mklink command you easy can notice that also simply CreateHardLinkW api called (set breakpoint to it)

            after you create hardlink to symlink file you can view in explorer that type of file is .symlink . for test we can remove link from target file ( by use FSCTL_DELETE_REPARSE_POINT) if hardlink point to target - any operation with symlink not affect hardlink. but if we created hardlink to symlink intself - after break symlink - hard link will also be breaked:

            Source https://stackoverflow.com/questions/70119403

            QUESTION

            Rsync Incremental Backup still copies all the files
            Asked 2021-Nov-21 at 13:50

            I am currently writing a bash script for rsync. I am pretty sure I am doing something wrong. But I can't tell what it is. I will try to elaborate everything in detail so hopefully someone can help me.

            The goal of script is to do full backups and incremental ones using rsync. Everything seems to work perfectly well, besides one crucial thing. It seems like even though using the --link-dest parameter, it still copies all the files. I have checked the file sizes with du -chs.

            First here is my script:

            ...

            ANSWER

            Answered 2021-Nov-21 at 13:50

            I didn't read the entire code because the main problem didn't seem to lay there.
            Verify the disk usage of your /Backups directory with du -sh /Backups and then compare it with the sum of du -sh /Backups/Full and du -sh /Backups/Inc.

            I'll show you why with a little test:

            Create a directory containing a file of 1 MiB:

            Source https://stackoverflow.com/questions/70049407

            QUESTION

            btrfs send / receive on incremental folders that rotate
            Asked 2021-Sep-28 at 06:36

            I am using incremental backups using rsnapshot combined with a custom cmd_cp and cmd_rm to make use of btrfs snapshots, this procudes multiple daily btrfs subvolumes:

            ...

            ANSWER

            Answered 2021-Sep-28 at 06:36

            I solved it! I created a bash script that syncs all snapshots with a date in the name to the remote server. The date is subtracted from btrfs subvolume show.

            So daily.0 can become 2021-09-20-08-44-46 on the remote.

            I sync backwards. daily.30 first. daily.0 last. This way I can pass the proper parent to btrfs send. E.g.: btrfs send -p daily.30 daily.29.

            If the date named snapshot exists on the remote, I check with btrfs subvolume show whether it was properly synced. If not, I delete the remote subvolume/snapshot and re-sync. If it was already synced, I will skip the sync. A proper synced subvolume/snapshot has a Received UUID and readonly flag.

            After syncing I compare all snapshot names of the remote to what was just synced. The difference will deleted (thus old snapshots).

            I might share the code in the future when it's all been stable for a long run. For now I hope the above information will help others!

            Source https://stackoverflow.com/questions/69164668

            QUESTION

            How to copy(backup) Azure CosmosDB container to Azure blob storage?
            Asked 2021-Aug-24 at 15:47

            There are many Containers in my CosmosDB Database, And I need to backup some but not all containers everyday. Some container are backuped for 7days, some are 15 days.

            1. I don't want to use incremental backup, because we just backup once everyday.
            2. Maybe we store backup dataset into Azure blob storage

            The thing I don't know => container == collection. Azure document is so confusing!

            ...

            ANSWER

            Answered 2021-Aug-23 at 11:54

            Probably you can create a job on Azure Data Factory (aka ADF, https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-cosmos-db), use the ADF job to copy data from these containers, and save the data as files (one file for one container) in somewhere (like Azure Blob Storage).

            Source https://stackoverflow.com/questions/68891671

            QUESTION

            Why am I getting "Agent error" when trying to perform a backup?
            Asked 2021-Jul-31 at 22:03

            I am implementing a Backup system for my Android app. I'm using a custom BackupAgentHelper to back up the shared preferences and a database file:

            ...

            ANSWER

            Answered 2021-Jul-31 at 22:03

            As @MikeM said, this was happening because I was executing the code from a non UI thread.

            I solved it by using a Handler, this one takes care of running the code from the UI thread:

            Source https://stackoverflow.com/questions/68582842

            QUESTION

            Difference between incremental backup and WAL archiving with PgBackRest
            Asked 2021-Jul-09 at 10:34

            As far as I understood

            • WAL archiving is pushing the WAL logs to a storage place as the WAL files are generated
            • Incremental backup is pushing all the WAL files created since the last backup

            So, assuming my WAL archiving is setup correctly

            1. Why would I need incremental backups?
            2. Shouldn't the cost of incremental backups be almost zero?

            Most of the documentation I found is focusing on a high level implementation (e.g. how to setup WAL archiving or incremental backups) vs the internal ( what happens when I trigger an incremental backup)

            My question can probably be solved with a link to some documentation, but my google-fu has failed me so far

            ...

            ANSWER

            Answered 2021-Jul-09 at 10:34

            Backups are not copies of the WAL files, they're copies of the cluster's whole data directory. As it says in the docs, an incremental backup contains:

            those database cluster files that have changed since the last backup (which can be another incremental backup, a differential backup, or a full backup)

            WALs alone aren't enough to restore a database; they only record changes to the cluster files, so they require a backup as a starting point.

            The need for periodic backups (incremental or otherwise) is primarily to do with recovery time. Technically, you could just hold on to your original full backup plus years worth of WAL files, but replaying them all in the event of a failure could take hours or days, and you likely can't tolerate that kind of downtime.

            A new backup also means that you can safely discard any older WALs (assuming you don't still need them for point-in-time recovery), meaning less data to store, and less data whose integrity you're relying on in order to recover.

            If you want to know more about what pgBackRest is actually doing under the hood, it's all covered pretty thoroughly in the Postgres docs.

            Source https://stackoverflow.com/questions/68300406

            QUESTION

            Can we configure Marklogic database backup on S3 bucket
            Asked 2021-Apr-01 at 12:33

            I need to configure Marklogic Full/Incremental backup in the S3 bucket is it possible? Can anyone share the documents/steps to configure?

            Thanks!

            ...

            ANSWER

            Answered 2021-Apr-01 at 12:33

            Yes, you can backup to S3.

            You will need to configure the S3 credentials, so that MarkLogic is able to use S3 and read/write objects to your S3 bucket.

            MarkLogic can't use S3 for journal archive paths, because S3 does not support file append operations. So if you want to enable journal archives, you will need to specify a custom path for that when creating your backups.

            Backing Up a Database

            The directory you specified can be an operating system mounted directory path, it can be an HDFS path, or it can be an S3 path. For details on using HDFS and S3 storage in MarkLogic, see Disk Storage Considerations in the Query Performance and Tuning Guide.

            S3 Storage

            S3 requires authentication with the following S3 credentials:

            • AWS Access Key
            • AWS Secret Key

            The S3 credentials for a MarkLogic cluster are stored in the security database for the cluster. You can only have one set of S3 credentials per cluster. You can set up security access in S3, you can access any paths that are allowed access by those credentials. Because of the flexibility of how you can set up access in S3, you can set up any S3 account to allow access to any other account, so if you want to allow the credentials you have set up in MarkLogic to access S3 paths owned by other S3 users, those users need to grant access to those paths to the AWS Access Key set up in your MarkLogic Cluster.

            To set up the AW credentials for a cluster, enter the keys in the Admin Interface under Security > Credentials. You can also set up the keys programmatically using the following Security API functions:

            • sec:credentials-get-aws
            • sec:credentials-set-aws

            The credentials are stored in the Security database. Therefore, you cannot use S3 as the forest storage for a security database.

            if you want to have Journaling enabled, you will need to have them written to a different location. Journal archiving is not supported on S3.

            The default location for Journals are in the backup, but when creating programmatically you can specify a different $journal-archive-path .

            S3 and MarkLogic

            Storage on S3 has an 'eventual consistency' property, meaning that write operations might not be available immediately for reading, but they will be available at some point. Because of this, S3 data directories in MarkLogic have a restriction that MarkLogic does not create Journals on S3. Therefore, MarkLogic recommends that you use S3 only for backups and for read-only forests, otherwise you risk the possibility of data loss. If your forests are read-only, then there is no need to have journals.

            Source https://stackoverflow.com/questions/66895146

            QUESTION

            Alternatives to Select-String
            Asked 2021-Mar-21 at 14:49

            I'm looking for an alternative to Select-String to use within my function. At present, everything else in the function returns the data I need except this command:

            ...

            ANSWER

            Answered 2021-Mar-21 at 14:49

            Unfortunately I do not have a true solution to the issue, instead a re-work was done to the entire code and then implemented within BladeLogic to make it work as needed. As such an answer to this is no longer needed. Thank you to those who attempted to assist.

            Source https://stackoverflow.com/questions/66001003

            Community Discussions, Code Snippets contain sources that include Stack Exchange Network

            Vulnerabilities

            No vulnerabilities reported

            Install msrsync

            msrsync is a single python file, you just have to download it. Or if you prefer, you can clone the repository and use the provided Makefile:.

            Support

            For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries
            CLONE
          • HTTPS

            https://github.com/jbd/msrsync.git

          • CLI

            gh repo clone jbd/msrsync

          • sshUrl

            git@github.com:jbd/msrsync.git

          • Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link

            Explore Related Topics

            Consider Popular Incremental Backup Libraries

            rsnapshot

            by rsnapshot

            bitpocket

            by sickill

            RsyncOSX

            by rsyncOSX

            sshfs

            by osxfuse

            rsync

            by WayneD

            Try Top Libraries by jbd

            treewatcher

            by jbdPython

            packo

            by jbdPython

            httpfs2

            by jbdC