lfs | build linux from scratch based systems | Continuous Deployment library

The main issue was due to the files actually being file pointers themselves. When I installed git-lfs and cloned the repository via that, it all worked.

The base64 value you provided in your question is malformed, my recommendation is to not use Json Serialization for this Api response effort.

Try using Convert.ToBase64String

• Ensure the method returns string
• and the jQuery request accepts/expects text for dataType response.

It looks like you made some commits that have the big file, then you added some more commits in which you've replaced the big file with the LFS-indirection file. What this means is that you need to remove the commits that have the big files.

Remember that when you use Git-LFS, you're using two external web sites to store your data. First, you send the big files to some LFS storage site. These big files are not (yet) committed. Then, with the real files safely stored elsewhere, you create and push commits that store only the information needed to retrieve the big files from the LFS storage site. So the commits, in the form of a Git repository, exist somewhere (e.g., on GitHub) but don't have any big files in any of them. Instead, they have small files that have instructions that say don't use this file, use a big file you get with the LFS big-file-swapper-replacer-trick, here's how to get the big file. The LFS wrappers around Git intercept Git's attempt to put the small replacement file into your working tree, and put the big file there instead.

(Note that this means that anyone who clones your repository with regular Git gets, and sees, these weird small indirection files. They must set up Git-LFS themselves so that they can have the same wrapper program intercept the attempt to put the small files out for viewing or editing.)

But if you do this in the wrong order, you first commit the big files. Then you send the big files to a second site, then you remove the big files and put in the small "here's how to get the real file" files, and make another commit that contains these small files. You still have the commit with the big files in it!

Git is designed to add new commits, and not to take old commits away. This makes it very hard to remove any commit that has a big file in it. To do that automatically requires specialized tools. Fortunately, GitHub already have instructions and tools for you, to tell you how to do this. See them here. The short version is to use git lfs migrate, if you have a modern Git-LFS. See also this other GitHub page.

The link you've posted contains a token in the URL. That token exists to make it possible to view the raw URL even though the URL is on a different domain, and it contains credentials to permit someone to view that file.

Normally, the access to Git LFS files is restricted to the same access as the rest of your repository. Only parties who can read the files in your repository can get a valid raw data link like you've gotten, so as long as you don't distribute links to files in your private repository, you should be fine.

The token in the URL is specific to a file and a user, and usually it is sufficient to change your password to expire all the links.

Ok, my bad on this one. My unity project is simply too big. I've settled for simply including the scripts for now in my GitHub repo as they house the majority of my work that I would be interested in sharing. I have misunderstood the usage of GitHub here, housing entire projects is something more for a site like itch.io as opposed to GitHub, which is more for source code

History, in any Git repository, is nothing more or less than the set of commits in that repository (as found by starting from some branch names, tag names, and/or other names and working backwards, the way Git does in general). Commits themselves hold full snapshots of every file, plus metadata; the metadata hold the hash IDs of previous commits; and the resulting chains of commits form Merkle trees which guarantee the validity of the commits.

The problem is, they pushed a node_modules.zip (in a subfolder) on one of the earliest commits into the GitLab upstream and this file is over 200MB in size.

Which of course exceeds the GitHub maximum size. So this commit cannot be sent to GitHub.

Since I don't need the master branch at all, I was hoping to rewrite the history on my new branch, but I have problems to figure out how this can be done.

The rewrite is not difficult (use a tool such as filter-branch, filter-repo, or the BFG). But once the rewrite is done, this is a completely new chain of commits: a different Merkle tree. It is a different history. It can be used with itself, but if it is combined with the original history, what you now have is a tree that requires both histories to be valid and complete.

The very important part is that I am able to pull new commits from GitLab, merge them to my own branch and push my own branch to GitHub.

The short version of all of this is "you can't".

The longer version is that you can, but only by maintaining a parallel history and never combining your history with their history. This is messy, ugly, painful, and difficult or impossible to automate. There are as far as I know no tools for doing this. You use one repository (e.g., on your own machine) that has both histories in it, and another (on GitHub) that has only your rewritten history. When they have new commits, you cherry-pick them to your own history. How you keep track of which commits are new-to-your-history is up to you: this is where a tool would come in handy. Such tools might exist, or you could write one. A barely-adequate one is probably not hard to write, but a good one would be hard.

In general, the http.sslBackend option is only usable on Windows. Most Linux distros don't offer it as an option, since they don't compile with multiple TLS libraries.

The proper solution is to remove all of the http.sslBackend options:

A good real-world analogy is tricky here, but suppose you have a cement delivery service. You delivered, to some customer, 1000 cubic yards of cement, which have now set. Your customer calls you up and says they did not want that much cement: they only wanted ten cubic yards. So you have your trucks deliver them another 10 yards of cement. Did you improve their situation, or just make it worse?

Now, the reason for the above analogy is simple enough: Git commits only ever add new stuff to the repository. If you put a giant file into some commit, that giant file is in that commit forever. If someone didn't want that giant file, and you make a new commit that lacks the giant file, well, now you have two commits: one with the giant file, and one without. The second commit depends on, and therefore requires, the first commit.

No matter what else you do, if you keep adding on new commits, you're just making everything worse. What you need is not more cement (er, commits). You need some kind of jackhammer and cement-removal service (something to remove commits from your repository).

That's not all, though! While we already see that listing a file in .gitignore has no effect on any existing commit—no existing commit can ever be changed, at all; you just have to get them hauled away and stop using them entirely to get rid of them—it also has no good effect on new commits, at least not yet. If a file is tracked, it has no effect on that file in future commits either.

So:

• Your first job is to remove some commit(s). We generally do this with git reset. The git reset command is destructive! Be very careful here: you probably should work on a clone of your repository, rather than the original.

• Then, having removed the bad commit(s), you may need to explicitly remove the large files, so that they are no longer tracked. A file is tracked, in Git, if the file is in Git's index. How do you know if a file is in Git's index? Well, you know because it's tracked. This is kind of a circular problem, as you can see. :-) But if you definitely want it not-tracked, you can run:

You are still getting the quota error most likely because you still have commits in one or more branches containing very large files. Keep in mind that your Git history is a snapshot of everything which has ever happened. Deleting the files from the current branch and pushing just means you won't add more very large content. The content already in the history would still be there.

To get an idea on how to get started removing the large files from you repo, see this helpful Stack Overflow question:

How to remove/delete a large file from commit history in Git repository?