tsfile | columnar file format designed for time-series data | Time Series Database library
kandi X-RAY | tsfile Summary
kandi X-RAY | tsfile Summary
THIS REPO HAS MOVED TO TsFile is a columnar file format designed for time-series data, which supports efficient compression and query. It is easy to integrate TsFile with your IOT big data processing frameworks.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Test program
- Converts a global time filter into a global time filter
- Convert a global time filter into a query filter by using the select series list
- Execute a QueryExpression
- Test entry point
- Flush all row groups to the output stream
- Ends the TimeSeriesMetaData
- Add a new StringContainer from the header
- Adds a string to the container
- Gets value from timestamp
- Convert a JSON Schema to a map of Measure Descriptors
- Finds CompressionType by short name
- Compares this pair with the specified object
- Decode int
- Join a list of byte arrays
- Returns the next value
- Read all values from the target page list
- Read a long value from an input stream
- Writes test file
- Clone a string container
- Construct data point with data type
- Loads the configuration file
- Read a double
- Convert a string to a TSRecord
- Reads a float from the given stream
- Encodes all values in the list and writes it to the OutputStream
tsfile Key Features
tsfile Examples and Code Snippets
Community Discussions
Trending Discussions on tsfile
QUESTION
2022-03-09 10:17:10,158 [pool-12-IoTDB-RPC-Client-1] WARN o.a.i.d.u.ErrorHandlingUtils:61 - Status code: INTERNAL_SERVER_ERROR(500), operation: "select count(tenantId$s) from root.benchmark.type.model_850_7.device_model850_15". executeStatement failed java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Invalid input: 95 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeStatement(TSServiceImpl.java:637) at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeStatement.getResult(TSIService.java:2853) at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeStatement.getResult(TSIService.java:2833) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.IllegalArgumentException: Invalid input: 95 at org.apache.iotdb.tsfile.file.metadata.enums.MetadataIndexNodeType.deserialize(MetadataIndexNodeType.java:70) at org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode.deserializeFrom(MetadataIndexNode.java:107) at org.apache.iotdb.tsfile.read.TsFileSequenceReader.getMetadataAndEndOffset(TsFileSequenceReader.java:980) at org.apache.iotdb.tsfile.read.TsFileSequenceReader.getLeafMetadataIndexPair(TsFileSequenceReader.java:471) at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readTimeseriesMetadata(TsFileSequenceReader.java:443) at org.apache.iotdb.db.engine.cache.TimeSeriesMetadataCache.get(TimeSeriesMetadataCache.java:194) at org.apache.iotdb.db.utils.FileLoaderUtils.loadTimeSeriesMetadata(FileLoaderUtils.java:168)
...ANSWER
Answered 2022-Mar-11 at 02:03You can do this if you have another disk:
Add the path of the second disk to data_dir in the iOTDB configuration file
Move some TS files from the first disk to the second disk
QUESTION
I configured TsFile synchronization. Schema already synchronizes, while the data is not synchronized. Have no idea why it would happen.enter image description here
...ANSWER
Answered 2022-Mar-10 at 09:05Does the TsFile on the sending end have a TsFile resource? If not, the file is not sealed. Currently, only sealed files can be synchronized.
QUESTION
Apache IoTDB compiling error :maven-download-plugin has wget (get-thrift-executable) EOFException
ran "mvn clean package -pl distribution -am -DskipTests" and then received the error. [INFO] -------------------< org.apache.iotdb:iotdb-thrift >-------------------- [INFO] Building rpc-thrift 0.12.5-SNAPSHOT [4/17] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ iotdb-thrift --- [INFO] Deleting /Users/qiaojialin/Documents/git_workspace/incubator-iotdb/thrift/target [INFO] [INFO] --- maven-checkstyle-plugin:3.0.0:check (validate) @ iotdb-thrift --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce-maven-version) @ iotdb-thrift --- [INFO] Skipping Rule Enforcement. [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce-version-convergence) @ iotdb-thrift --- [INFO] Skipping Rule Enforcement. [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (vulnerability-checks) @ iotdb-thrift --- [INFO] Skipping Rule Enforcement. [INFO] [INFO] --- spotless-maven-plugin:2.4.2:check (spotless-check) @ iotdb-thrift --- [INFO] [INFO] --- download-maven-plugin:1.3.0:wget (get-thrift-executable) @ iotdb-thrift --- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Apache IoTDB Project Parent POM 0.12.5-SNAPSHOT: [INFO] [INFO] Apache IoTDB Project Parent POM .................... SUCCESS [ 8.961 s] [INFO] TsFile ............................................. SUCCESS [ 19.869 s] [INFO] sql-antlr .......................................... SUCCESS [ 4.801 s] [INFO] rpc-thrift ......................................... FAILURE [ 0.301 s] [INFO] rpc-thrift-cluster ................................. SKIPPED [INFO] rpc-thrift-sync .................................... SKIPPED [INFO] Service-rpc ........................................ SKIPPED [INFO] IoTDB Jdbc ......................................... SKIPPED [INFO] IoTDB Server ....................................... SKIPPED [INFO] IoTDB Session ...................................... SKIPPED [INFO] IoTDB Cli .......................................... SKIPPED [INFO] IoTDB Grafana ...................................... SKIPPED [INFO] cluster ............................................ SKIPPED [INFO] Compile Tools ...................................... SKIPPED [INFO] Tools: Thrift ...................................... SKIPPED [INFO] Client for cpp ..................................... SKIPPED [INFO] IoTDB Distribution ................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 34.350 s [INFO] Finished at: 2022-03-02T10:31:28+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal com.googlecode.maven-download-plugin:download-maven-plugin:1.3.0:wget (get-thrift-executable) on project iotdb-thrift: IO Error: EOFException -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :iotdb-thrift
...ANSWER
Answered 2022-Mar-04 at 05:43delete directory".m2/repository/.cache/download-maven-plugin". Then compile again. It will work.
QUESTION
I'm currently inserting data into IoTDB and I see some small TsFiles is merging into the large files. I would like to know how often the TsFiles be merged and how the merge process is triggered. Thanks!
...ANSWER
Answered 2022-Feb-22 at 09:09In the 0.12.x IoTDB iotdb-engine.properties
file, there is a configuration merge_fileSelection_time_budget=30000
, which means if there are enough amount of TsFiles of one level existed (seq_file_num_in_each_level=6 or unseq_file_num_in_each_level=10), the merge task will start. After 30000ms
then, IoTDB will check if there still are enough TsFiles existed and start the next merge task.
QUESTION
How to convert TsFile into CSV when paths are unknown in IoTDB? The CSV should contain path, time, type, value, etc.
...ANSWER
Answered 2022-Jan-13 at 02:04Currently, we do not have export CSV tools for TsFile, but you could use export tools for IoTDB.
Step 1. Load TsFile in a clean IoTDB: https://iotdb.apache.org/UserGuide/Master/System-Tools/Load-External-Tsfile.html
Step 2. Use export-csv and use https://iotdb.apache.org/zh/UserGuide/Master/System-Tools/Load-External-Tsfile.html
or select * from root align by device
: https://iotdb.apache.org/UserGuide/Master/System-Tools/CSV-Tool.html
QUESTION
ANSWER
Answered 2021-Nov-07 at 14:20In latest version, IoTDB will print the actual size of plan when it exceeds the wal_buffer_size. For example, Log cannot fit into the buffer, please increase wal_buffer_size to 16777216
.
You can check server log and set wal_buffer_size in iotdb-engine.properties
.
QUESTION
I've tried to locale the TSFiles to HDFS, the config as below:
...ANSWER
Answered 2021-Oct-27 at 05:55the compaction of IoTDB doesn't adapt storage in HDFS yet. Please close the compaction by setting compaction_strategy=NO_COMPACTION
and enable_unseq_compaction=false
when using HDFS.
QUESTION
We are using Apache IoTDB Server in Version 0.11.2 in a scenario and observe a data directory / tsfiles that are way bigger than they should be (about 130 sensors with 4 Million double values for each sensor but the files are about 200gb).
Are there known issues or do you have any ideas what could cause this is how to track that down?
The only thing we could think off could be some merge artefacts as we do write many datapoints out of order so merging has to happen frequently.
Are there any ideas or tools on how to debug / inspect the tsfiles to get an idea whats happening here?
Any help or hints is appreciated!
...ANSWER
Answered 2021-Oct-06 at 06:55this may be due to the compaction strategy.
You could fix this in two ways (no need at the same time):
(1) upgrade to 0.12.2 version
(2) open the configuration in iotdb-engine.properties: force_full_merge=true
The reason is:
The unsequenced data compaction in the 0.11.2 version has two strategies.
E.g.,
Chunks in a sequence TsFile: [1,3], [4,5]
Chunks in a unsequence TsFile: [2]
(I use [1,3] to indicate the timestamp of 2 data points in a Chunk)
(1) When using full merge(rewrite all data): we get a tidy sequence file: [1,2,3],[4,5]
(2) However, to speed up the compaction, we use append merge by default, when we get a sequence TsFile: [1,3],[4,5],[1,2,3]. In this TsFile, [1,3] does not have metadata at the end of the File, it is garbage data.
So, if you have lots of out-of-order data merged frequently, this will happen (get a verrry big TsFile).
The big TsFile will be tidy after a new compaction.
You could also use TsFileSketchTool or example/tsfile/TsFileSequenceRead to see the structure of the TsFile.
QUESTION
the template TR not displayed, and generates an error : NG0303: Can't bind to 'ngForOf' since it isn't a known property of 'tr'. I don't understand why. I am in Angular 11 and navigator Chrone
Error in the console html :
...
ANSWER
Answered 2021-May-21 at 21:35You should add the component to the declarations array:
QUESTION
I have a scenario where I have select dropdownlist(classification array). if my selecList has multiple values then ill display as a dropdown. if it has only one element in the array then by default it should be shown in selectList dropdown with that element selected by default. I need to write a condition where if classification element length is less orequal to 1 then show the element selected by default in matselect or display the list. Could you guys advise
...ANSWER
Answered 2020-Oct-20 at 16:00I think so you could set has selected value on constructor. I'm assuming you are using reactive forms. (it looks like in your code example).
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install tsfile
You can use tsfile like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the tsfile component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page