gridengine | lightweight Python library for distributed computing | Job Scheduling library

Short anwer:

The image viewhelper does not automatically apply any dimensions defined in the constants. f:image is part of ext:fluid, the constants you set are part of ext:fluid_styled_content. The image viewhelper always needs a width or a maxWidth argument if you want to set one. You need to pass your constants to the frontend.

Long answer:

You can see, how it is done in fluid_styled_content:

Have a look into: typo3/sysext/fluid_styled_content/Configuration/TypoScript/ContentElement/Image.typoscript A DataProcessor passes this constants to the GalleryProcessor

Most of the setup issues went away when installing the Debian version. need version 8 since debian is ver 8 & only debian master works ( gridengine-master_8.1.9+dfsg-8_amd64.deb).

The script adds a class Queue like this:

Thank you Dominic you were on to something. I originally checked ~/.bash* files as well as /etc/bash* and /etc/profile but after your comment, I found several scripts in /etc/profile.d/ that we being executed, and one of them set the PATH explicitly without appending. I added $PATH back in there and now scl enable is working as expected!

You aren't processing your data in chunks at all.

With data1 = pd.read_csv('...', chunksize=10000, iterator=True), data1 becomes a pandas.io.parser.TextFileReader, which is an iterator that yields chunks of 10000 rows of your CSV data as a DataFrame.

But then pd.concat consumes this entire iterator, and so attempts to load the whole CSV into memory, defeating the purpose of using chunksize and iterator entirely.

Properly using chunksize and iterator

In order to process your data in chunks, you have to iterate over the actual DataFrame chunks yielded by the iterator read.csv provides.

Unfortunately I'm not aware of a way to do what you ask - I think that the reservation is created once at the time that the job is submitted, not progressively as slots become free. If you haven't already seen the design document for the Resource Reservation feature, it's worth a look to get oriented to the feature.

Instead, I'm going to suggest some strategies for confidently setting job runtimes. The main problem when none of your jobs have runtimes is that Grid Engine can't reserve space infinitely in the future, so even if you set some really rough runtimes (within an order of magnitude of the true runtime), you may get some positive results.

1. If you've run a similar job previously, one simple rule of thumb is to set max runtime to 150% of the typical or maximum runtime of the job, based on historical trends. Use qacct or parse the accounting file to get hard data. Of course, tweak that percentage to whatever suits your risk threshold.
2. Another rule of thumb is to set the max runtime not based on the job's true runtime, but based on a sense around "after this date, the results won't be useful" or "if it takes this long, something's definitely wrong". If you need an answer by Friday, there's no sense in setting the runtime limit for three months out. Similarly, if you're running md5sum on typically megabyte-sized files, there's no sense in setting a 1-day runtime limit; those jobs ought to only take a few seconds or minutes, and if it's really taking a long time, then something is broken.
3. If you really must allow true indefinite-length jobs, then one option is to divide your cluster into infinite and finite queues. Jobs specifying a finite runtime will be able to use both queues, while infinite jobs will have fewer resources available; this will incentivize users to work a little harder at picking runtimes, without forcing them to do so.

Finally, be sure that the multi-slot jobs are submitted with the -R y qsub flag to enable the resource reservation system. This could go in the system default sge_request file, but that's generally not recommended as it can reduce scheduling performance:

Since reservation scheduling performance consumption is known to grow with the number of pending jobs, use of -R y option is recommended only for those jobs actually queuing for bottleneck resources.