mapR | Combining the power of Hadoop and R

Acording to this answer: https://stackoverflow.com/a/51236918/16651073 tomcat falls back to default logging if it can resolve the location

Can you try to save the properties without the spaces.

Like this: logging.file.name=application.logs

This is a good question because it highlights the way that some environments impose limits that violate the assumptions external software may hold.

And just for reference, MapR was acquired by HPE so a MapR cluster is now an HPE Ezmeral Data Fabric cluster. I am still training myself to say that.

Anyway, the accepted method for a generic program in language X to communicate with the Ezmeral Data Fabric (the filesystem formerly known as MapR FS) is to mount the file system and just talk to it using file APIs like open/read/write and such. This applies to Go, Python, C, Julia or whatever. Inside Kubernetes, the normal way to do this mount is to use a CSI driver that has some kind of operator working in the background. That operator isn't particularly magical ... it just does what is needful. In the case of data fabric, the operator mounts the data fabric using NFS or FUSE and then bind mounts[1] part of that into the pod's awareness.

But this question is cool because it precludes all of that. If you can't install an operator, then this other stuff is just a dead letter.

There are three alternative approaches that may work.

1. NFS mounts were included in Kubernetes as a native capability before the CSI plugin approach was standardized. It might still be possible to use that on a very vanilla Kubernetes cluster and that could give access to the data cluster.

2. It is possible to integrate a container into your pod that does the necessary FUSE mount in an unprivileged way. This will be kind of painful because you would have to tease apart the FUSE driver from the data fabric install and get it to work. That would let you see the data fabric inside the pod. Even then, there is no guarantee Kubernetes or the OS will allow this to work.

3. There is an unpublished Go file system client that users the low level data fabric API directly. We don't yet release that separately. For more information on that, folks should ping me directly (my contact info is everywhere ... email to ted.dunning hpe.com or gmail.com works)

4. The data fabric allows you to access data via S3. With the 7.0 release of Ezmeral Data Fabric, this capability is heavily revamped to give massive performance especially since you can scale up the number of gateways essentially without limit (I have heard numbers like 3-5GB/s per stateless connection to a gateway, but YMMV). This will require the least futzing and should give plenty of performance. You can even access files as if they were S3 objects.

[1] https://unix.stackexchange.com/questions/198590/what-is-a-bind-mount#:~:text=A%20bind%20mount%20is%20an,the%20same%20as%20the%20original.

To get rid of the first error I'd found (and posted about in the OP), I had to go to the $HIVE_HOME/lib directory and remove this old guava library file: guava-19.0.jar

I had to make sure that the guava library I'd copied from the Hadoop library was there: guava-27.0-jre.jar

On the next attempt I got a different error:

I had to do the inevitable and add this to my build.sbt:

Can you try changing the DDL to use serde rather than TxtInputFormat.

If nothing else helps, as a workaround you can drop/create table and recover partitions. The table is EXTERNAL and drop will not affect the data.

(1) Drop table

Solution to your first problem - --hcatalog-database mydb --hcatalog-table airquality and remove --export dir parameter.

Sqoop export cannot replace data. Pls issue a sqoop eval statement before loading main table to truncate it.

When you say;

I created this path '/MyCluster/apps/application_logs/node1' using mkdir

I assume that you have created it in local file system. The path parameter needs to be created in MapR file system.

To do that, you can either create the dir by interacting with MapR file system directly as follows;

context.write(Text, Text) and job.setOutputValueClass(IntWritable.class); disagree with one another. Make them consistent and it should work.