csv-schema | Schema Language for CSV | CSV Processing library

by digital-preservation HTML Version: 1.1 License: MPL-2.0

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | csv-schema Summary

csv-schema is a HTML library typically used in Utilities, CSV Processing applications. csv-schema has no bugs, it has no vulnerabilities, it has a Weak Copyleft License and it has low support. You can download it from GitHub.

A Schema Language for CSV (Comma Separated Value) files. This repository holds the code for creating the CSV Schema specification document, which is then published as HTML. The Schema language is formally expressed in EBNF. You can find the the documentation and latest published specification here:

Support

Quality

Security

License

Reuse

Support

csv-schema has a low active ecosystem.

It has 72 star(s) with 28 fork(s). There are 13 watchers for this library.

It had no major release in the last 12 months.

There are 22 open issues and 6 have been closed. On average issues are closed in 90 days. There are 1 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of csv-schema is 1.1

Quality

csv-schema has 0 bugs and 0 code smells.

Security

csv-schema has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

csv-schema code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

csv-schema is licensed under the MPL-2.0 License. This license is Weak Copyleft.

Weak Copyleft licenses have some restrictions, but you can use them in commercial projects.

Reuse

csv-schema releases are available to install and integrate.

It has 8397 lines of code, 0 functions and 45 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of csv-schema

Get all kandi verified functions for this library.

csv-schema Key Features

No Key Features are available at this moment for csv-schema.

csv-schema Examples and Code Snippets

No Code Snippets are available at this moment for csv-schema.

Community Discussions

Trending Discussions on csv-schema

CSV to json with dynamic schema using NiFi

QUESTION

CSV to json with dynamic schema using NiFi

Asked 2018-Sep-05 at 06:18

I am getting a CSV file from a 3rd party. Schema for this file is dynamic, the only thing I can be certain of is,

each column with data will also have header name.
file will always have a header.
header name will always be a string of alphabets with no spaces and dots. (so, kind of "clean").
values should be treated as strings, as I am not sure what they will be sending.

Now to use this type of data in my system, I am thinking of using MongoDB as staging area. As no. of columns, or order of columns, or columns name are not constant from one load to another. I think MongoDB will serve a good staging area.

I read about ConvertRecord processor, which is ideal for CSV to JSON converter, but I don't have a schema. I just want each row to go as a document, with header name as a key and value as value.

How should I go about it? Also this file is going to be in some 25-30 GB range, so I do not want to bring down my system.

I thought of doing it by my own processor (in Java), and I was able to get what I am looking for, but it seems to be taking too much time, and it kind of doesn't look optimal.

Let me know, if this can be achieved via existing processor?

Thanks, Rakesh

Updated on : 09/05/2018

a2bd0551-0165-1000-7c6a-a32ca4db047ccsv_to_json_no_schema_v191bc4a66-704c-3a2f-0000-000000000000defb04c4-c15c-3a07-0000-0000000000001 GB10000defb04c4-c15c-3a07-0000-000000000000bb6c25ae-f2b6-386a-0000-000000000000PROCESSOR0 sec1successdefb04c4-c15c-3a07-0000-000000000000eb6cd54a-e1f1-3871-0000-000000000000PROCESSOR0ad804e3c-f233-3556-0000-000000000000defb04c4-c15c-3a07-0000-0000000000001 GB10000defb04c4-c15c-3a07-0000-00000000000064b15a56-8a5f-3297-0000-000000000000PROCESSOR0 sec1invaliddefb04c4-c15c-3a07-0000-000000000000bb6c25ae-f2b6-386a-0000-000000000000PROCESSOR0c30bd123-c436-36ce-0000-000000000000defb04c4-c15c-3a07-0000-0000000000001 GB10000defb04c4-c15c-3a07-0000-0000000000008a0e37da-acd2-3d72-0000-000000000000PROCESSOR0 sec1validdefb04c4-c15c-3a07-0000-000000000000bb6c25ae-f2b6-386a-0000-000000000000PROCESSOR0247d2139-26b7-31fe-0000-000000000000defb04c4-c15c-3a07-0000-0000000000001 GB10000defb04c4-c15c-3a07-0000-0000000000001297bea9-b30f-3f45-0000-000000000000PROCESSOR0 sec1failuredefb04c4-c15c-3a07-0000-0000000000008a0e37da-acd2-3d72-0000-000000000000PROCESSOR045e5403f-99f7-3ddf-0000-000000000000defb04c4-c15c-3a07-0000-0000000000001 GB10000defb04c4-c15c-3a07-0000-0000000000009f8f32f7-130c-35bd-0000-000000000000PROCESSOR0 sec1successdefb04c4-c15c-3a07-0000-0000000000008a0e37da-acd2-3d72-0000-000000000000PROCESSOR088b0195a-34b2-34f0-0000-000000000000defb04c4-c15c-3a07-0000-000000000000nifi-record-serialization-services-narorg.apache.nifi1.6.0Schema Write StrategySchema Write Strategyschema-access-strategyschema-access-strategyschema-registryorg.apache.nifi.schemaregistry.services.SchemaRegistryschema-registryschema-nameschema-nameschema-versionschema-versionschema-branchschema-branchschema-textschema-textDate FormatDate FormatTime FormatTime FormatTimestamp FormatTimestamp FormatPretty Print JSONPretty Print JSONsuppress-nullssuppress-nullsJsonRecordSetWriterfalseSchema Write Strategyno-schemaschema-access-strategyschema-registryschema-nameschema-versionschema-branchschema-textDate FormatTime FormatTimestamp FormatPretty Print JSONsuppress-nullsENABLEDorg.apache.nifi.json.JsonRecordSetWriterc3e80a29-498b-36d4-0000-000000000000defb04c4-c15c-3a07-0000-000000000000nifi-record-serialization-services-narorg.apache.nifi1.6.0schema-access-strategyschema-access-strategyschema-registryorg.apache.nifi.schemaregistry.services.SchemaRegistryschema-registryschema-nameschema-nameschema-versionschema-versionschema-branchschema-branchschema-textschema-textcsv-reader-csv-parsercsv-reader-csv-parserDate FormatDate FormatTime FormatTime FormatTimestamp FormatTimestamp FormatCSV FormatCSV FormatValue SeparatorValue SeparatorSkip Header LineSkip Header Lineignore-csv-headerignore-csv-headerQuote CharacterQuote CharacterEscape CharacterEscape CharacterComment MarkerComment MarkerNull StringNull StringTrim FieldsTrim Fieldscsvutils-character-setcsvutils-character-setCSVReaderfalseschema-access-strategyschema-registryschema-nameschema-versionschema-branchschema-textcsv-reader-csv-parserDate FormatTime FormatTimestamp FormatCSV FormatValue SeparatorSkip Header Linetrueignore-csv-headertrueQuote CharacterEscape CharacterComment MarkerNull StringTrim Fieldscsvutils-character-setENABLEDorg.apache.nifi.csv.CSVReader8a0e37da-acd2-3d72-0000-000000000000defb04c4-c15c-3a07-0000-0000000000000.0227.99996948242188nifi-standard-narorg.apache.nifi1.6.0WARN1record-readerorg.apache.nifi.serialization.RecordReaderFactoryrecord-readerrecord-writerorg.apache.nifi.serialization.RecordSetWriterFactoryrecord-writerALLfalse30 secrecord-readerc3e80a29-498b-36d4-0000-000000000000record-writer88b0195a-34b2-34f0-0000-00000000000000 secTIMER_DRIVEN1 secConvertRecordfalsefailurefalsesuccessSTOPPEDorg.apache.nifi.processors.standard.ConvertRecord9f8f32f7-130c-35bd-0000-000000000000defb04c4-c15c-3a07-0000-00000000000011.0483.0nifi-standard-narorg.apache.nifi1.6.0WARN1Log LevelLog LevelLog PayloadLog PayloadAttributes to LogAttributes to Logattributes-to-log-regexattributes-to-log-regexAttributes to IgnoreAttributes to Ignoreattributes-to-ignore-regexattributes-to-ignore-regexLog prefixLog prefixcharacter-setcharacter-setALLfalse30 secLog LevelinfoLog PayloadfalseAttributes to Logattributes-to-log-regex.*Attributes to Ignoreattributes-to-ignore-regexLog prefixcharacter-setUTF-800 secTIMER_DRIVEN1 secLogAttributetruesuccessSTOPPEDorg.apache.nifi.processors.standard.LogAttributebb6c25ae-f2b6-386a-0000-000000000000defb04c4-c15c-3a07-0000-000000000000670.0225.0nifi-standard-narorg.apache.nifi1.6.0WARN1validate-csv-schemavalidate-csv-schemavalidate-csv-headervalidate-csv-headervalidate-csv-delimitervalidate-csv-delimitervalidate-csv-quotevalidate-csv-quotevalidate-csv-eolvalidate-csv-eolvalidate-csv-strategyvalidate-csv-strategyALLfalse30 secvalidate-csv-schemaNotNull,ParseInt(),Optional(ParseInt()),Nullvalidate-csv-headertruevalidate-csv-delimiter,validate-csv-quote"validate-csv-eol\nvalidate-csv-strategyLine by line validation00 secTIMER_DRIVEN1 secValidateCsvfalseinvalidfalsevalidSTOPPEDorg.apache.nifi.processors.standard.ValidateCsveb6cd54a-e1f1-3871-0000-000000000000defb04c4-c15c-3a07-0000-000000000000688.00.0nifi-standard-narorg.apache.nifi1.6.0WARN1File SizeFile SizeBatch SizeBatch SizeData FormatData FormatUnique FlowFilesUnique FlowFilesgenerate-ff-custom-textgenerate-ff-custom-textcharacter-setcharacter-setschema.nameschema.nameALLfalse30 secFile Size0BBatch Size1Data FormatTextUnique FlowFilesfalsegenerate-ff-custom-textname,age,int_val,address Rakesh Prasad,0,99,"address 12 33333, 444441" rakesh Prasad1,1,,"address 12 33333, 444442" rakesh Prasad2,2,55,"address 12 33333, 444443" rakesh Prasad3,,33,"address 12 33333, 444444"character-setUTF-8schema.nameempData01 dayTIMER_DRIVEN1 secGenerateFlowFilefalsesuccessSTOPPEDorg.apache.nifi.processors.standard.GenerateFlowFile1297bea9-b30f-3f45-0000-000000000000defb04c4-c15c-3a07-0000-000000000000450.0539.0nifi-standard-narorg.apache.nifi1.6.0WARN1Log LevelLog LevelLog PayloadLog PayloadAttributes to LogAttributes to Logattributes-to-log-regexattributes-to-log-regexAttributes to IgnoreAttributes to Ignoreattributes-to-ignore-regexattributes-to-ignore-regexLog prefixLog prefixcharacter-setcharacter-setALLfalse30 secLog LevelinfoLog PayloadfalseAttributes to Logattributes-to-log-regex.*Attributes to Ignoreattributes-to-ignore-regexLog prefixcharacter-setUTF-800 secTIMER_DRIVEN1 secLogAttributetruesuccessSTOPPEDorg.apache.nifi.processors.standard.LogAttribute64b15a56-8a5f-3297-0000-000000000000defb04c4-c15c-3a07-0000-000000000000837.0482.0000305175781nifi-standard-narorg.apache.nifi1.6.0WARN1Log LevelLog LevelLog PayloadLog PayloadAttributes to LogAttributes to Logattributes-to-log-regexattributes-to-log-regexAttributes to IgnoreAttributes to Ignoreattributes-to-ignore-regexattributes-to-ignore-regexLog prefixLog prefixcharacter-setcharacter-setALLfalse30 secLog LevelinfoLog PayloadfalseAttributes to Logattributes-to-log-regex.*Attributes to Ignoreattributes-to-ignore-regexLog prefixcharacter-setUTF-800 secTIMER_DRIVEN1 secLogAttributetruesuccessSTOPPEDorg.apache.nifi.processors.standard.LogAttribute09/05/2018 01:32:27 EDT

...

ANSWER

Answered 2018-Sep-04 at 13:04

You can use ConvertRecord with a CSV Reader and in the CSV Reader choose "Use String Fields From Header" for the Schema Access Strategy. This will create a schema dynamically from the header.

Source https://stackoverflow.com/questions/52160637

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install csv-schema

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: