jsalparser | Apache log files

 by   hiramsoft Java Updated: 1 year ago - Current License: Proprietary

Download this library from

Build Applications

kandi X-RAY | jsalparser REVIEW AND RATINGS

JSalParser - parse AWS S3, CloudFront, and Apache log files in Java

kandi-support
Support

  • jsalparser has a low active ecosystem.
  • It has 11 star(s) with 6 fork(s).
  • It had no major release in the last 12 months.
  • On average issues are closed in 174 days.
  • It has a neutral sentiment in the developer community.

quality kandi
Quality

  • jsalparser has 0 bugs and 0 code smells.

security
Security

  • jsalparser has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • jsalparser code analysis shows 0 unresolved vulnerabilities.
  • There are 0 security hotspots that need review.

license
License

  • jsalparser has a Proprietary License.
  • Proprietary licenses can be open source with a non SPDX compliant license, or non open source licenses, and you need to review them closely before use.

build
Reuse

  • jsalparser releases are not available. You will need to build from source code and install.
  • Build file is available. You can build the component from source.
  • jsalparser saves you 296 person hours of effort in developing the same functionality from scratch.
  • It has 713 lines of code, 115 functions and 11 files with 0 % test coverage
  • It has medium code complexity. Code complexity directly impacts maintainability of the code.
Top functions reviewed by kandi - BETA

Coming Soon for all Libraries!

Currently covering the most popular Java, JavaScript and Python libraries. See a SAMPLE HERE.
kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.

jsalparser Key Features

JSalParser - parse AWS S3, CloudFront, and Apache log files in Java

jsalparser examples and code snippets

  • default
  • Failed to create Blob container in Microsoft Azure

default

### Parse an S3 log line-by-line

```Java
	String content = "1f000000000c6c88eb9dd89c000000000b35b0000000a5 www.example.com [27/Aug/2014:20:20:05 +0000] 192.168.0.1 - BFE596E2F4D94C8F WEBSITE.GET.OBJECT media/example.jpg \"GET /media/example.jpg HTTP/1.1\" 304 - - 27553 202 - \"http://www.example.com/page.html\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53\" -";
	List<S3LogEntry> entries = JSalParser.parseS3Log(content);

	long TenMegabytes = 10000000L;

	for(int i=0;i<entries.size();i++) {
		S3LogEntry entry = entries.get(i);

		// Notice how the numbers are numbers, no additional parsing needed
		if(entry.getObjectSize() > TenMegabytes)
		{
			System.out.println(entry.getTime());

			// getTime() returns a Java Date object,
			// so Java prints:
			// 2014-08-27T20:20:05.000+00:00
		}
	}
```

### Parse a CloudFront log gzip file using a visitor for streaming efficiency

```Java
	// Assume gzipFileStream is some kind of java.io.InputStream
	// You got it either from a FileInputStream (local file on disk), S3, or anywhere else that returns InputStreams

	java.util.zip.GZIPInputStream gzipInputStream = new java.util.zip.GZIPInputStream(gzipFileStream);

	// Process records inline by passing a visitor to effectively get "streaming" log processing
	// The only two things you need are an InputStream and a visitor
	// JSalParser is Thread-Safe
	JSalParser.parseCloudFrontLog(gzipInputStream, new ICloudFrontLogVisitor() {
		int count = 0;
		@Override
		public void accept(CloudFrontWebLogEntry entry) {
			System.out.print("Processing entry #" + (count++) + " from " + entry.getDateTime() + " ");
			// Date is returned as a Java Date object.

			// Numbers are surfaced as Ints and Longs
			if(entry.getServerToClientStatus() == 200)
			{
				System.out.println("OK");
			}
			else
			{
				System.out.println("NOT_OK");
			}

			// You will get:
			/***********
			 Processing entry #0 from 2014-08-28T04:48:38.000Z OK
			 Processing entry #1 from 2014-08-28T04:48:38.000Z OK
			 Processing entry #2 from 2014-08-28T04:49:23.000Z NOT_OK
			 Processing entry #3 from 2014-08-28T04:48:37.000Z OK
			 Processing entry #4 from 2014-08-28T04:48:38.000Z NOT_OK
			 Processing entry #5 from 2014-08-28T04:48:38.000Z OK
			 ***********/
		}
	});
```

Both S3 and CloudFront support accepting Strings and InputStream objects.

Both S3 and CloudFront support the visitor pattern for streaming efficiency.

Most Common Scenarios
--------
These are the most common scenarios for working with S3 or CloudFront log files.

### How do I read logs directly from AWS S3?

The Java AWS SDK ( http://aws.amazon.com/sdk-for-java/ ) is the best way to read and write objects on S3.  The quick-and-dirty way is to construct an AmazonS3 instance and call getObject().

Some pseudo-code:

```Java
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import com.amazonaws.services.s3.model.S3Object;

public void quickAndDirty(){

	AmazonS3 s3 = //... left as an exercise

	S3Object obj = s3.getObject("my-bucket-name", "/the-log-file-key-name");
	S3ObjectInputStream content = obj.getObjectContent();
	// S3ObjectInputStream is a subclass of InputStream

	List<S3LogEntry> entries = JSalParser.parseS3Log(content);
	// ... continue on like normal
}
```

### Read directly from a gzip file:

Java has a built-in GZIPInputStream class ( java.util.zip.GZIPInputStream ) that is a subclass of InputStream.

If the files are local, pass in a FileInputStream instance:

```Java
import java.io;

File gzipFile = new File("path-to-your-local-file.gz");
FileInputStream gzipFileStream = new FileInputStream(gzipFile);
java.util.zip.GZIPInputStream gzipInputStream = new java.util.zip.GZIPInputStream(gzipFileStream);

List<CloudFrontWebLogEntry> entries = JSalParser.parseCloudFrontLog(gzipInputStream);
// .. continue on like normal
```

Or, combine with the AWS SDK to read directly from S3

```Java
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import com.amazonaws.services.s3.model.S3Object;

AmazonS3 s3 = //... left as an exercise

S3Object obj = s3.getObject("my-bucket-name", "/the-log-file-key-name.gz");
S3ObjectInputStream s3Stream = obj.getObjectContent();
java.util.zip.GZIPInputStream gzipInputStream = new java.util.zip.GZIPInputStream(s3Stream);

List<CloudFrontWebLogEntry> entries = JSalParser.parseCloudFrontLog(gzipInputStream);
// .. continue on like normal
```

### How do I process log files as a stream?

JSalParser takes as an optional, second parameter either an ICloudFrontLogVisitor or an IS3LogVisitor instance.
As the parser fully reads each entry, it calls the respective "accept" methods on the visitors.
Your code is free to implement any business logic as needed.

Using the visitor means the parser only has to store a couple log lines in memory at a time,
in theory making it so one could process extremely large files.

### What considerations should I know about CloudFront log files?

CloudFront log files have two header lines, and these header lines describe the "schema" of the log file.
S3 log files, in contrast, have no header file, and the schema is defined in documentation.

Therefore in order to process any CloudFront log file we must first process the header entry.
If you send a CloudFront log file without headers you will get only an array of untyped values.

### How do I use this with Apache HTTP Server log files?

Send the Apache HTTP Server log files to the JSalParser.parseS3Log* family of methods.

### What happens if the parser encounters something it doesn't recognize?

The parser puts "extra" stuff (values it doesn't expect) into the "extras" list.
Order is preserved, but all values are processed as Strings.

### What happens if a value is missing?

The POJO has null or 0 as appropriate.  I usually program in Scala and am much more used to Options,
but alas this is the best choice for Java 7.

### What happened to JODA DateTime?

Java 8 came out and a helpful contributor removed the non-Java dependency.

### What version of Java?
Java 7.  No real reason, just that's the lowest JDK I have available to test.

The only "advanced" feature is generics.

Maven

Failed to create Blob container in Microsoft Azure

import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;
<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>12.0.0</version>
</dependency>
// Create a BlobServiceClient object which will be used to create a container client
BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().connectionString("DefaultEndpointsProtocol=https;AccountName=0730bowmanwindow;AccountKey=xxxxxx;EndpointSuffix=core.windows.net").buildClient();

//Create a unique name for the container
String containerName = "quickstartblobs";

// Create the container and return a container client object
BlobContainerClient containerClient = blobServiceClient.createBlobContainer(containerName);
-----------------------
import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;
<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>12.0.0</version>
</dependency>
// Create a BlobServiceClient object which will be used to create a container client
BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().connectionString("DefaultEndpointsProtocol=https;AccountName=0730bowmanwindow;AccountKey=xxxxxx;EndpointSuffix=core.windows.net").buildClient();

//Create a unique name for the container
String containerName = "quickstartblobs";

// Create the container and return a container client object
BlobContainerClient containerClient = blobServiceClient.createBlobContainer(containerName);
-----------------------
import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;
<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>12.0.0</version>
</dependency>
// Create a BlobServiceClient object which will be used to create a container client
BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().connectionString("DefaultEndpointsProtocol=https;AccountName=0730bowmanwindow;AccountKey=xxxxxx;EndpointSuffix=core.windows.net").buildClient();

//Create a unique name for the container
String containerName = "quickstartblobs";

// Create the container and return a container client object
BlobContainerClient containerClient = blobServiceClient.createBlobContainer(containerName);

COMMUNITY DISCUSSIONS

Top Trending Discussions on jsalparser
  • Failed to create Blob container in Microsoft Azure
Top Trending Discussions on jsalparser

QUESTION

Failed to create Blob container in Microsoft Azure

Asked 2020-May-29 at 14:16

I followed the tutorial of Azure Blob storage service here and supplied the connection string from my storage account to create a container but it did not work as the container is not created and the application runs until I terminate it manually. I used azure-blob-storage API version 12.4.0.

Here is the code:

// Create a BlobServiceClient object which will be used to create a container client
BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().connectionString(connectStr).buildClient();

//Create a unique name for the container
String containerName = "quickstartblobs" + java.util.UUID.randomUUID();

// Create the container and return a container client object
BlobContainerClient containerClient = blobServiceClient.createBlobContainer(containerName);

Here is my pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>aaaa</groupId>
    <artifactId>ddada</artifactId>
    <version>1.0-SNAPSHOT</version>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>11</source>
                    <target>11</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
            <version>1.11.327</version>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.5</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.6</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage -->
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-storage</artifactId>
            <version>1.108.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
        <dependency>
            <groupId>de.hpi.cloudraid</groupId>
            <artifactId>erasure</artifactId>
            <version>1.0.3</version>
        </dependency>
        <dependency>
            <groupId>com.hiramsoft.commons</groupId>
            <artifactId>jsalparser</artifactId>
            <version>0.2-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-csv</artifactId>
            <version>1.7</version>
        </dependency>
        <dependency>
            <groupId>com.google.googlejavaformat</groupId>
            <artifactId>google-java-format</artifactId>
            <version>1.7</version>
        </dependency>
        <dependency>
            <groupId>com.google.crypto.tink</groupId>
            <artifactId>tink</artifactId>
            <version>1.3.0-rc3</version>
        </dependency>
        <dependency>
            <groupId>dev.morphia.morphia</groupId>
            <artifactId>core</artifactId>
            <version>1.5.8</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.10</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.dataformat</groupId>
            <artifactId>jackson-dataformat-csv</artifactId>
            <version>2.11.0.rc1</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.11.0.rc1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.opencsv/opencsv -->
        <dependency>
            <groupId>com.opencsv</groupId>
            <artifactId>opencsv</artifactId>
            <version>5.1</version>
        </dependency>
        <dependency>
            <groupId>com.hierynomus</groupId>
            <artifactId>sshj</artifactId>
            <version>0.27.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.azure/azure-storage-blob -->
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-storage-blob</artifactId>
            <version>12.4.0</version>
        </dependency>
    </dependencies>

</project>

It also showed another error message like this:

May 29, 2020 3:39:49 PM io.netty.channel.AbstractChannelHandlerContext invokeExceptionCaught
WARNING: An exception 'java.lang.NoSuchMethodError: 'java.lang.String io.netty.handler.codec.http.HttpUtil.formatHostnameForHttp(java.net.InetSocketAddress)'' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.NoSuchMethodError: 'java.lang.String io.netty.handler.codec.http.HttpUtil.formatHostnameForHttp(java.net.InetSocketAddress)'
    at reactor.netty.http.client.HttpClientConnect$HttpClientHandler.resolveHostHeaderValue(HttpClientConnect.java:593)
    at reactor.netty.http.client.HttpClientConnect$HttpClientHandler.requestWithBody(HttpClientConnect.java:552)
    at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.lambda$onStateChange$0(HttpClientConnect.java:429)
    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
    at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:430)
    at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:473)
    at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:525)
    at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:434)
    at reactor.netty.channel.ChannelOperationsHandler.channelActive(ChannelOperationsHandler.java:62)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelActive(CombinedChannelDuplexHandler.java:414)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
    at io.netty.channel.CombinedChannelDuplexHandler.channelActive(CombinedChannelDuplexHandler.java:213)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192)
    at reactor.netty.tcp.SslProvider$SslReadHandler.userEventTriggered(SslProvider.java:731)
    at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
    at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
    at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
    at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1480)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1318)
    at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1170)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1195)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.base/java.lang.Thread.run(Thread.java:830)

Any reasons why it did not work?

ANSWER

Answered 2020-May-29 at 14:16

Update:

Thanks to Ihsan Haikal for sharing. If anyone who faces similar problems cannot create a container using the following method, please check the content of pom.xml The problem should not be the code but the configuration part.

Original Answer:

With this steps I can create container on my storage:

First, import:

import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;

And this is the code in my pom.xml:

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>12.0.0</version>
</dependency>

Second, use code:

// Create a BlobServiceClient object which will be used to create a container client
BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().connectionString("DefaultEndpointsProtocol=https;AccountName=0730bowmanwindow;AccountKey=xxxxxx;EndpointSuffix=core.windows.net").buildClient();

//Create a unique name for the container
String containerName = "quickstartblobs";

// Create the container and return a container client object
BlobContainerClient containerClient = blobServiceClient.createBlobContainer(containerName);

You should get connection string from this place:

enter image description here

Then, it creates the container:

enter image description here

You can check what different between you and me.:)

Source https://stackoverflow.com/questions/62060421

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

VULNERABILITIES

No vulnerabilities reported

INSTALL jsalparser

You can use jsalparser like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the jsalparser component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

SUPPORT

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Implement jsalparser faster with kandi.

  • Use the support, quality, security, license, reuse scores and reviewed functions to confirm the fit for your project.
  • Use the, Q & A, Installation and Support guides to implement faster.

Discover Millions of Libraries and
Pre-built Use Cases on kandi