hadoop-examples | Examples codes demonstrating features in Hadoop eco-system
kandi X-RAY | hadoop-examples Summary
kandi X-RAY | hadoop-examples Summary
Examples codes demonstrating features in Hadoop eco-system.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Writes the value of a key
- Convert a date string to a valid date
- Entry point for wordcount2
- Run a job
- Main entry point
- Main entry point for testing
- Reads the air temperature from the file
- Maps a key - value pair to the output
- Reduces the number of elements in the context
- Reduce the specified values to the specified key
- Reduces the values by key
- Main method for testing
hadoop-examples Key Features
hadoop-examples Examples and Code Snippets
Community Discussions
Trending Discussions on hadoop-examples
QUESTION
the question is related to a terasort example. Is there any parameter to change the amount of output records using terasort? The input generated with teragen is 65'536'000 but we are requested to run terasort and output 10'000'000 records. This request is part of a practice with Cloudera distribution, not a real case but benchmark on implementation practice. Teragen:
time hadoop jar opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/hadoop-0.20-mapreduce/hadoop-examples.jar teragen -Dmapreduce.job.maps=12 -Ddfs.blocksize=33554432 -Dmapreduce.map.memory.mb=512 -Dyarn.app.mapreduce.am.containerlauncher.threadpool-initial-size=512 65536000 /user/haley/tgen
Result:
...ANSWER
Answered 2018-Jan-03 at 11:59Is there any parameter to change the amount of output records using terasort?
As far as I understand the source code of TeraSort.java
, it seems to implement a custom partitioner, partitioning and sorting the full input. So there is no parameter to change that behavior.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install hadoop-examples
You can use hadoop-examples like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the hadoop-examples component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page