packable | Extensive pack and unpack capabilities
kandi X-RAY | packable Summary
kandi X-RAY | packable Summary
Extensive pack and unpack capabilities
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Read an IO object from a file
- Lookup a class .
- Create a new filter .
- Creates a PackerPacker .
- Merge merge options .
- Unpack a string into a string .
- Sets the given block .
- Write to disk .
packable Key Features
packable Examples and Code Snippets
Community Discussions
Trending Discussions on packable
QUESTION
I'm trying to improve performance of a git repository that is being used almost exclusively by me to version a scientific computing project. The project's simulation software blasts teeny (less than 100KB) plaintext files into fairly deep directories, representing separate, relatively economical simulation results. I point out that these are economical to indicate that I can create many thousands of them over the course of a short amount of time, which means this is just going to keep getting worse. These simulations are run as batches, which can mean that individual commits can include several hundred MB of data, all in the form of these deep sub-trees populated with teeny text files. The institutional computing cluster I am running this on uses a 33TB RAID6 array of platter drives to store all my group's data (if it matters, this drive doesn't have a ton of headroom by percentage at the moment--about 1.6 TB).
I'm reasonably sure this is bad performance on the RAID6 array's part, because when I run a top-level git add .
it can take tens of minutes, even if only a few files have changed. Committing is just as bad. Pushing, once things are committed usually still takes minutes, but is a bit faster (and the slow part of the push is not the part where it sends the data over the network). Doing all of this in an interactive session where I've requested extra cores also speeds things up, but it can still take minutes to finish adding new simulation results. When I do the same on my laptop, which has a modern NVME-PCIE SSD in it, these operations take seconds.
So, any advice? I looked at git lfs, but am not convinced this would help me a ton because the pointers it would create are not a million times smaller than the files they'd be pointing to, which is the normal use case. If people still think it'd help I guess I can give that a try. Also, if it matters, the cluster's linux is old (of course) so: git version 1.8.3.1
...
Happy to add more context if needed. EDIT git count-objects -vH
returns:
ANSWER
Answered 2021-Apr-02 at 02:41As @CodeCaster pointed out, the git on my cluster was indeed ancient and this was in part the source of the problem. I'm not totally convinced that the raid array on my school's cluster isn't just slow somehow, but after updating to a more recent git my pulls, pushes, adds and commits have all become far less painful. They've gone from taking tens of minutes to a handful of seconds (which is more the speed I'm used to).
For what it's worth, this SO answer is what convinced me to try to upgrade git (again, thanks @CodeCaster). As @torek has pointed out, the repos are backwards compatible, so there have been no issues handling my repo that was being handled by a git from 2015 with a git from this year.
If anyone reading this concludes that it would be annoying for them to pursue this solution because they don't have root on their shared infrastructure, my approach was to use conda to install a different git in the conda environment I was working with anyway. As of this post conda install -c conda-forge git
in a clean miniconda3 env will get you git 2.30.2, which is plenty current. The most recent performance update mentioned in the other SO post is in version 2.24. I suppose there are other avenues to a local git installation, but in a scientific computing environment where there's usually a local conda available for a user without too much trouble this seemed like the easiest path to a newer version.
QUESTION
I am attempting to port some Python code over to Go. The Python code has a State class which has 2 abstract classes Packable and Hashable which unmarshal/marshal and then hash/digest the data from the class into bytes.
I was thinking of porting it over to Go by doing the following, having 2 interfaces, Hasher and Packer, and then embedding these in my State struct like below.
...ANSWER
Answered 2020-Jun-11 at 16:50This would only work as written if the implementations of Hasher
and Packer
don't need to know anything about the type they're embedded in (which they won't), in which case they should probably just be plain functions.
It seems like what you really want here is to define methods on State
so that the State
type implements Hasher
and Packer
, in which case you should not embed those interfaces. Just define the methods on State
correctly.
Interfaces are demonstrated in the Tour of Go.
QUESTION
I seem to have got a stuck git repo. It hangs on all basic add,commit commands, git push returns everything as up to date. From other posts I have done git gc and git fsck/ I figure the basic debug step is git status, so:
...ANSWER
Answered 2020-Apr-19 at 16:24An infinite loop on git status has been seen before (hbons/SparkleShare issue 1170 as an example)
The next step, with a Git 2.25+, would be to enable trace2, which I present here:
QUESTION
I used to have one folder per repository and push to gitlab. Later I wanted to group all these codes into a folder (Wrapper). I have remove the .git in each of these folders
...ANSWER
Answered 2020-Mar-16 at 05:48In addition of "How to remove/delete a large file from commit history in Git repository?", the best practice is to use the new tool git filter-repo
which replaces BFG and git filter-branch
.
QUESTION
I have a Series which consists of a list of some random products. This is what it looks like if I print the describe:
...ANSWER
Answered 2019-Dec-27 at 00:33It seems there has been previous issues regarding how pandas value_counts()
deals with tied values, in an inconsistent way.
As for idxmax()
the documentation states clearly:
If multiple values equal the maximum, the first row label with that value is returned.
I am afraid the amount of information you provide is not enough for me to generate a full example with your data but here is an attempt:
QUESTION
I'm having trouble with switching my local SugarCRM instance from a gitlab repo to a new github repo. Locally, the project is on a vagrant VM running on a Windows 10 host. I run git commands from the host machine. I created a new private github repo, set it as the new remote, and generated a new public/private ssh key pair on the host machine in C:\Users\.ssh. When I try to push the project to the remote repo for the first time, I get
...ANSWER
Answered 2019-Aug-09 at 19:31EDIT/UPDATE: What actually did the trick for OP was git update-git-for-windows
. See comments for details!
Things I'd try in such situations:
- Make sure my current user owns all the current git-internal files, as lacking permissions can interfere with various git operations.
- Linux:
sudo chown -R --changes "$USER" .git
- Windows: open main folder permissions and let the permissions get rolled down via the Advanced Security Settings dialog (using
☑ Replace all child object permissions with inheritable permissions from this object
)
- Linux:
- On Windows: Make sure the path isn't long, as the Windows filesystem NTFS will mess up git if paths get longer than 255 chars/bytes. Keep in mind that the regular git object paths alone (e.g.
.git/objects/fe/73f087d8c25fbc1159a50434ef9bf15773e7ac
) have a length of 55 characters. Also that special unicode characters can consume more than 1 byte of the file path. Also files in the repo that exceed the maximum path lengths can become problematic for some operations. In worst case just move the repo to e.g.C:\repo\
or something in similar length and try there. - if even that fails and the repo seems completely broken, I'd try cleaning up the repo by cloning it locally
git clone ./repo newrepo
and retry the push from within the new repo clone
QUESTION
Before I ran git gc
, I had a few thousand loose objects:
ANSWER
Answered 2019-Apr-01 at 15:06If you're sure no other Git command is running in that repository at the time you run git gc
on that repository, you can add --prune=all
. The default is --prune=2.weeks.ago
, which gives those other running commands 14 days to complete their work; you can use --prune=1.day.ago
to give them less time, for instance.
You can also configure gc.pruneExpire
: if not set, it defaults to 2.weeks.ago
which is what produces the above default. As j6t notes, the gc.pruneExpire
setting variant of this is now
rather than all
. It's unwise to set now
here though: an automatic git gc
will use this value and will run in the background, while other Git operations run.
Note that if you have a version of Git >= 2.5 but below 2.15.0, a reduced gc.pruneExpire
can break your added work-trees in less time than the default two weeks. The bug is that git gc
fails to use the HEAD
and index of the added work-tree as a starting point for the reachability traversal of the object DAG. As a result, git gc
can remove added blobs that have not yet been committed, and, if you have a worktree with a detached HEAD, even some commits. The best fix for this is to upgrade to 2.15.0 or later, since even the 2-week default isn't necessarily enough.
QUESTION
I would like to find out how to test same spark code. Googling around I have found the spark-tetsting-base. Well, now I would like to try it out but I am not able to run it with Maven.
First, I have scala code which is packable with mvn package
and has the following project structure:
ANSWER
Answered 2017-Jun-15 at 00:38The test cannot be run due to the folder structure. Your folder structure
QUESTION
My repository is 1GB in size according to BitBucket.
I have commits going all the way back to the start of 2017.
How to I truncate my history? For example, I have a certain tag XXX and I would like to delete all commits from XXX back to the initial commit. So that the repository is smaller.
I tried this on my PC:
...ANSWER
Answered 2018-Dec-20 at 10:24Without rewriting history (e.g., rebasing), you cannot remove/alter history. Git by design, uses a chaining of all commits. So, altering an old commit will cause to modify the complete history and all following SHA1s (commit ids). Maybe you still have lots of remote branches on BitBucket (which are not included in your main master branch), Git won't throw away commit if they are referenced from a branch/tag. - For that reason it is important that you force push all branches and (or delete all unured) tags after the rebasing or altering history - otherwise they still point to the old history (which cannot be deleted because some tags/branches points to it).
Big repositories often result from the fact, that the commits contain binaries. if those can be generated using the source, those should not be included in the commits. - If you want to publish them, some platforms provide the option the attach files to tags (oftentimes called releases).
QUESTION
I was attempting to push a bunch of new files to my private Azure Devops source control repository using git and I get the following error:
error: unable to parse remote unpack status: ng refs/heads/dirtypiece/gdpr An error occurred while communicating with the remote host. The error code is 0x800703E5.
I also get this error when attempting to push a 12 MB video file up, but don't encounter the error when attempting to push up a small set of changes (like a single file).
Here's the full command and output I see for reference:
"C:\Program Files (x86)\Git\bin\git.exe" push -u --recurse-submodules=check --progress "origin" refs/heads/dirtypiece/gdpr:refs/heads/dirtypiece/gdpr Counting objects: 3622, done. Delta compression using up to 8 threads. Compressing objects: 100% (1602/1602), done. Writing objects: 100% (3622/3622), 11.53 MiB | 1.91 MiB/s, done. Total 3622 (delta 1926), reused 3545 (delta 1888) remote: We noticed you're using an older version of Git. For the best experience, upgrade to a newer version. error: unable to parse remote unpack status: ng refs/heads/dirtypiece/gdpr An error occurred while communicating with the remote host. The error code is 0x800703E5. Everything up-to-date Done
I don't remember ever hitting anything like this in the past when I've attempted to push up a large number of files (always seemed to push fine). I was thinking it might be a temporary throttling or resource constraint issue on the Azure Devops servers, but the issue has been happening for 2 days now.
I also checked the size of my repository to see if I was hitting a 10 GB limit or something like that, but it appears to be within tolerance:
$ git count-objects -vH count: 2248 size: 5.85 MiB in-pack: 123703 packs: 19 size-pack: 759.69 MiB prune-packable: 14 garbage: 0 size-garbage: 0 bytes
Has anyone encountered this issue before or know how to get around it? I've definitely pushed up more files and larger files in the past so am confused as to why this started happening.
Thanks for any advice!
...ANSWER
Answered 2018-Sep-25 at 18:02I encountered this exact same error yesterday on macOS when trying to push some binary files to the repo (images, audio). Scripts and other small text files were fine.
After a few hours of hair-pulling, I tried switching the remote URL from HTTPS to SSH and the error is gone.
Before doing that, however, I also tried pushing from a different machine running Windows 10 and there was no error.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install packable
On a UNIX-like operating system, using your system’s package manager is easiest. However, the packaged Ruby version may not be the newest one. There is also an installer for Windows. Managers help you to switch between multiple Ruby versions on your system. Installers can be used to install a specific or multiple Ruby versions. Please refer ruby-lang.org for more information.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page