crypto-export | CSV exports of cryptocurrency exchange data using APIs | CSV Processing library
kandi X-RAY | crypto-export Summary
kandi X-RAY | crypto-export Summary
This script is a simple tool to export detailed exchange transactions (deposits, withdrawals, fills, etc.) from a growing set of cryptocurrency exchanges in order to simplify analysis for portfolio management and tax calculations. Specifically, it creates a set of uniform comma-separated value (CSV) files which can be imported into the CoinTracking website (shameless affiliate link below):. CoinTracking is a phenomenal tool which I use daily, but it has some odd quirks when it comes to exchange imports--not to mention the fact that API-based imports require a paid account with them. No hard feelings on that front, and I actually have a paid account, but I've found the API imports lacking when they really don't need to be. For example, Coinbase and GDAX provide API-based methods to get all the data you could ever want, but CoinTracking doesn't seem to use it all. Alas. Furthermore, the CSV exports from many exchanges (when they are available at all) often leave out important information, or format the data very strangely, or take a ridiculous amount of effort to accomplish. YES, I'M LOOKING AT YOU, GDAX. Seriously, Like 50 clicks and 10 emails just to get all of the CSV data for a common variety of trading pair transactions? Ugh. However, it's not hard to work around these limitations by writing a "bridge" script, so that's what I've done. As a bonus, you can use the CSV data that this script exports in any way you choose, even if you don't use CoinTracking at all. It gives you an easy way to gather transaction details from multiple sources into a collection of files that all have exactly the same format, readable in Excel or Python or anything else that can handle comma-separated values.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of crypto-export
crypto-export Key Features
crypto-export Examples and Code Snippets
Community Discussions
Trending Discussions on CSV Processing
QUESTION
I am currently working on a spring based API which has to transform csv data and to expose them as json. it has to read big CSV files which will contain more than 500 columns and 2.5 millions lines each. I am not guaranteed to have the same header between files (each file can have a completly different header than another), so I have no way to create a dedicated class which would provide mapping with the CSV headers. Currently the api controller is calling a csv service which reads the CSV data using a BufferReader.
The code works fine on my local machine but it is very slow : it takes about 20 seconds to process 450 columns and 40 000 lines. To improve speed processing, I tried to implement multithreading with Callable(s) but I am not familiar with that kind of concept, so the implementation might be wrong.
Other than that the api is running out of heap memory when running on the server, I know that a solution would be to enhance the amount of available memory but I suspect that the replace() and split() operations on strings made in the Callable(s) are responsible for consuming a large amout of heap memory.
So I actually have several questions :
#1. How could I improve the speed of the CSV reading ?
#2. Is the multithread implementation with Callable correct ?
#3. How could I reduce the amount of heap memory used in the process ?
#4. Do you know of a different approach to split at comas and replace the double quotes in each CSV line ? Would StringBuilder be of any healp here ? What about StringTokenizer ?
Here below the CSV method
...ANSWER
Answered 2022-Jan-29 at 02:56I don't think that splitting this work onto multiple threads is going to provide much improvement, and may in fact make the problem worse by consuming even more memory. The main problem is using too much heap memory, and the performance problem is likely to be due to excessive garbage collection when the remaining available heap is very small (but it's best to measure and profile to determine the exact cause of performance problems).
The memory consumption would be less from the replace and split operations, and more from the fact that the entire contents of the file need to be read into memory in this approach. Each line may not consume much memory, but multiplied by millions of lines, it all adds up.
If you have enough memory available on the machine to assign a heap size large enough to hold the entire contents, that will be the simplest solution, as it won't require changing the code.
Otherwise, the best way to deal with large amounts of data in a bounded amount of memory is to use a streaming approach. This means that each line of the file is processed and then passed directly to the output, without collecting all of the lines in memory in between. This will require changing the method signature to use a return type other than List. Assuming you are using Java 8 or later, the Stream API can be very helpful. You could rewrite the method like this:
QUESTION
I have a JSON that I want to insert into BQ. The column data type is STRING. Here is the sample JSON value.
...ANSWER
Answered 2021-Jun-02 at 06:55I think there is an issue with how you escape the double quotes.
I could reproduce the issue you describe, and fixed it by escaping the double quotes with " instead of a backslash \:
QUESTION
I'm sorry if this has been asked before. It probably has, but I just have not been able to find it. On with the question:
I often have loops which are initialized with certain conditions that affect or (de)activate certain behaviors inside them, but do not drastically change the general loop logic. These conditions do not change through the loop's operation, but have to be checked every iteration anyways. Is there a way to optimized said loop in a pythonic way to avoid doing the same check every single time? I understand this would be a compiler's job in any compiled language, but there ain't no compiler here.
Now, for a specific example, imagine I have a function that parses a CSV file with a format somewhat like this, where you do not know in advance the columns that will be present on it:
...ANSWER
Answered 2021-Apr-23 at 11:36Your code seems right to me, performance-wise.
You are doing your checks at the beginning of the loop:
QUESTION
I am attempting to create an program to scrape xml files. I'm experimenting with go because of it's goroutines. I have several thousand files, so some type of multiprocessing is almost a necessity...
I got a program to successfully run, and convert xml to csv(as a test, not quite the end result), on a test set of files, but when run with the full set of files, it gives this:
...ANSWER
Answered 2021-Apr-21 at 15:25I apologize for not including the correct error. as the comments pointed out i was doing something dumb and creating a routine for every file. Thanks to JimB for correcting me, and torek for providing a solution and this link. https://gobyexample.com/worker-pools
QUESTION
I am processing CSV and using the following code to process a single line.
...ANSWER
Answered 2020-Jul-31 at 21:54The fastest way to do something is to not do it at all.
If you can ensure that your source string s will outlive the use of the returned vector, you could replace your std::vector with std::vector which would point to the beginning of each substring. You then replace your identified delimiters with zeroes.
[EDIT] I have not moved up to C++17, so no string_view for me :)
NOTE: typical CSV is different from what you imply; it doesn't use escape for the comma, but surrounds entries with comma in it with double quotes. But I assume you know your data.
Implementation:
QUESTION
I am using regex for CSV processing where data can be in Quotes, or no quotes. But if there is just a comma at the starting column, it skips it.
Here is the regex I am using:
(?:,"|^")(""|[\w\W]*?)(?=",|"$)|(?:,(?!")|^(?!"))([^,]*?|)(?=$|,)
Now the example data I am using is:
,"data",moredata,"Data"
Which should have 4 matches ["","data","moredata","Data"], but it always skips the first comma. It is fine if there is quotes on the first column, or it is not blank, but if it is empty with no quotes, it ignores it.
Here is a sample code I am using for testing purposes, it is written in Dart:
...ANSWER
Answered 2020-May-11 at 22:02Investigating your expression
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install crypto-export
Install Python (I use WinPython 3.6)
Install the coinbase and gdax modules using pip install
Clone/download the crypto_export.py script from this repo
Modify and rename the crypto_export.conf.sample file with your API credentials
Run python crypto_export.py to perform the export
Import each exchange's records into CoinTracking using custom CSV rules (see last section)
Installing the Crypto Export script is simple: just clone or download (zip) the repository, or even copy and paste the raw script content into a new .py file on your computer.
Navigate to the installation folder that you chose, e.g. C:\Python\WinPython-64bit-3.5.4.1Qt5
Run the WinPython Command Prompt.exe application to enter the command line environment
Type pip install coinbase gdax to install the libraries from official sources
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
