zos | Autonomous operating system | Robotics library

Funny enough, this was on the local machine while I got OutOfMemoryError during the zip generation.

In testing environment, I got OutOfMemoryError during the retrieval of the documents. So Hibernate was complaining too. This was with a step before the generation. Probably this happened since local machine has 16GB and testing env only 1GB.

So the solution was build based on the following steps:

1. retrieve the files in batches with Hibernate, and (flush/clean) transactional entityManager, in order to force Hibernate to not keep in memory all the files. The Batch size was: 50 documents.
2. Adapt the code for the zip4j compression with Aws multipart upload, in order to compress and upload only one batch of files, and reset the buffers afterwards, to avoid OutOfMemory.

Step2 was designed and adapted based on: https://www.bennadel.com/blog/3971-generate-and-incrementally-stream-a-zip-archive-to-amazon-s3-using-multipart-uploads-in-lucee-cfml-5-3-7-47.htm

So the code from the initial question became as follows:

Don't build a memory structure when you can stream the same data directly - as huge structures may hit OutOfMemoryError on the toString() or getBytes() calls.

If you are dealing with platform line separators you can write out rows like this:

Ended up encoding byte array to base64 string then sent it as

You haven't closed the ZipOutputStream correctly - the call to zos.close(); must occur before writing the bytes, not afterwards.

You can get much neater handling of the output if using try with resources block and Files.copy:

After much research i was able to solve 70% of the problems. Others can't be solved given the nature of how an ZipOutputStream & ZipFile reads the data

Problem 1: Incorrect values returned by getSize() & getCompressedSize()

1) During Writing

I was blind to have not seen this earlier but ZipOutputStream already does compression for us and i was double compressing it by using my own inflater so i removed that code and i realized that you must specify these values only when you are using the method as STORED. else they are computed for you from the data. So refracting my zip writing code this is how it looks like

As a general rule of thumb when designing test cases you want to consider both the expected behaviour if everything goes right and the expected behaviours for each way that things could go wrong.

For your particular problem, when everything goes right you should expect your method to compress the file given as argument, upload it and delete the file from the temp directory. You can do so by spying on your static GcpCloudStorageUtil method. Spies are instances of a class which retain their usual behaviour unless specified and which method calls you can monitor. You can read more on the topic here and here (for static method spying).

It's hard to give a thorough answer without access to the rest of your code but I'd suggest :

• creating a non empty file
• calling the createZip method on this file
• verifying that the GcpCloudStorageUtil.uploadFileToGcsBucket method was called
• verifying that the uploaded file content matches the file you created
• verifying that the temp directory is now empty

There are also a number of way that things could go wrong. For instance the file given as argument might not exist. You want to write a test case which ensures that a FileNotFoundException is thrown in this case. This article covers different ways to do this using JUnit4 or JUnit5.

Similarly you could design a test case where you mock the GcpCloudStorageUtil.uploadFileToGcsBucket and force it to throw an exception then verify that the expected exception was thrown. Mocking a method means forcing it to behave a certain way (here, throw an exception). Again you can read more on the topic here.

Edit : Here is a possible test class.

Switching the ftp transfer mode to block mode should do what you want:

I'm unaware of a programmatic interface but you can specify how you want RDWs handled so you don't have to parse the current state. This can be done with the SITE command from the client to the mainframe. See reference in this question

Note: The settings in FTP.DATA will determine the default behaviour. Review this reference in the documentation.

SITE RDW

Specifies that Variable Record Descriptors (RDW) are treated as if they were part of the record and not discarded during ftp transmission of VB or VBS data sets in other than block mode. RDW information is stored in a binary halfword. Transfer files in binary to avoid translation problems that can occur if you transfer this binary field in EBCDIC or ASCII.

SITE NORDW

Specifies that Variable Record Descriptors (RDW) are discarded during ftp transmission of VB or VBS data sets in other than block mode.

Only the exact name of the function "main" is important fot the compiler to know where the program starts.