noise | Noise generation library for Java , based on the libnoise | Generator Utils library

by SpongePowered Java Version: 1.0.0 License: MIT

kandi X-RAY | noise Summary

noise is a Java library typically used in Generator, Generator Utils, Unity applications. noise has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub.

Noise generation library for Java, based on the libnoise C++ library by Jason Bevins. It is used to generate coherent noise, a type of smoothly-changing noise. It can also generate Perlin noise, ridged multifractal noise, and other types of coherent noise.

Support

Quality

Security

License

Reuse

kandi-support Support

noise has a highly active ecosystem.
It has 94 star(s) with 16 fork(s). There are 27 watchers for this library.
It had no major release in the last 12 months.
There are 6 open issues and 10 have been closed. On average issues are closed in 344 days. There are 2 open pull requests and 0 closed requests.
It has a positive sentiment in the developer community.
The latest version of noise is 1.0.0

noise Support

Best in #Generator Utils

Average in #Generator Utils

noise Support

Best in #Generator Utils

Average in #Generator Utils

quality kandi Quality

noise has 0 bugs and 0 code smells.

noise Quality

Best in #Generator Utils

Average in #Generator Utils

noise Quality

Best in #Generator Utils

Average in #Generator Utils

security Security

noise has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
noise code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

noise Security

Best in #Generator Utils

Average in #Generator Utils

noise Security

Best in #Generator Utils

Average in #Generator Utils

license License

noise is licensed under the MIT License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

noise License

Best in #Generator Utils

Average in #Generator Utils

noise License

Best in #Generator Utils

Average in #Generator Utils

build Reuse

noise releases are available to install and integrate.
Build file is available. You can build the component from source.
Installation instructions are not available. Examples and code snippets are available.
noise saves you 869 person hours of effort in developing the same functionality from scratch.
It has 1989 lines of code, 242 functions and 39 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

noise Reuse

Best in #Generator Utils

Average in #Generator Utils

noise Reuse

Best in #Generator Utils

Average in #Generator Utils

Top functions reviewed by kandi - BETA

kandi has reviewed noise and discovered the below as its top functions. This is intended to give you an instant insight into noise implemented functionality, and help decide if they suit your requirements.

Compute the simple noise gradient .

Produces a string containing the value of the given taglet .

Sets the rotation angle .

Find the insertion point for the given input value .

Sets the control points .

Compute the weighted weights for each element .

Sets the bounds .

Sets the source module at the given index .

Convert the int range to a int range .

Returns the maximum value of the octet .

Get all kandi verified functions for this library.

noise Key Features

Source

Issues

Community Discord

noise Examples and Code Snippets

See all related Code Snippets

Community Discussions

Trending Discussions on noise

iOS 15: Web Audio playback stops working in Safari after locking screen for a few minutes
Xcode log "Writing analzed variants"
Picking out peaks that fit a pattern
OpenVino converted model not returning same score values as original model (Sigmoid)
I have a dataset in which i have two columns with time in it but the dat
In ASP.NET Core 6, why language folders needed in Published output for System.Private.ServiceModel.resources.dll and how to eliminate them?
Is there a better way to remove parts with consecutive zeros that have length equal or above threshold?
Get simple name of type in Julia?
Perl anchored regex performance
What does Windows Desktop (5) mean in context of .net 5?

Trending Discussions on noise

QUESTION

iOS 15: Web Audio playback stops working in Safari after locking screen for a few minutes

Asked 2022-Mar-28 at 11:26

This seems to be an issue after I upgraded my iPod Touch to iOS 15 (15.0.1).

When running the example below, it works fine on the first load, playing the sound as many times as I want. However, if I lock the screen on my iPod Touch, then return after a couple minutes, the sound no longer plays. To troubleshoot I set up a state change listener on the AudioContext instance and verified that Safari sets the state back to running after it was set to interrupted when the screen was locked. Yet, the sound does not play.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>audio-test</title>
    </head>
    <body>
        <script>

            const AudioContext = window.AudioContext || window.webkitAudioContext;
            const audioContext = new AudioContext();

            // Create AudioBuffer and fill with two seconds of white noise.
            const channels = 2;
            const frameCount = audioContext.sampleRate * 2.0;
            const audioBuffer = audioContext.createBuffer(channels, frameCount, audioContext.sampleRate);
            for (var channel = 0; channel < channels; channel++) {
                var nowBuffering = audioBuffer.getChannelData(channel);
                for (var i = 0; i < frameCount; i++) {
                    nowBuffering[i] = Math.random() * 2 - 1;
                }
            }

            const button = document.createElement('button');
            button.textContent = 'Play Audio';
            document.body.append(button);

            button.addEventListener('click', () => {
                const currentSourceNode = new AudioBufferSourceNode(audioContext, {
                    buffer: audioBuffer,
                });
                currentSourceNode.connect(audioContext.destination);
                currentSourceNode.start();
            });

        </script>
    </body>
</html>

ANSWER

Answered 2021-Oct-12 at 19:34

Strangely, if I add an audio element to the HTML that points to some mp3 file, which isn't even referenced in the code at all, then locking the screen for a while then returning to the page no longer affects the audio playback.

Updated code:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>audio-test</title>
    </head>
    <body>
        <script>

            const AudioContext = window.AudioContext || window.webkitAudioContext;
            const audioContext = new AudioContext();

            // Create AudioBuffer and fill with two seconds of white noise.
            const channels = 2;
            const frameCount = audioContext.sampleRate * 2.0;
            const audioBuffer = audioContext.createBuffer(channels, frameCount, audioContext.sampleRate);
            for (var channel = 0; channel < channels; channel++) {
                var nowBuffering = audioBuffer.getChannelData(channel);
                for (var i = 0; i < frameCount; i++) {
                    nowBuffering[i] = Math.random() * 2 - 1;
                }
            }

            const button = document.createElement('button');
            button.textContent = 'Play Audio';
            document.body.append(button);

            button.addEventListener('click', () => {
                const currentSourceNode = new AudioBufferSourceNode(audioContext, {
                    buffer: audioBuffer,
                });
                currentSourceNode.connect(audioContext.destination);
                currentSourceNode.start();
            });

        </script>
        <audio src="https://s3-us-west-2.amazonaws.com/s.cdpn.io/858/outfoxing.mp3" crossorigin="anonymous"></audio>
    </body>
</html>

Source https://stackoverflow.com/questions/69502340

QUESTION

Picking out peaks that fit a pattern

Asked 2022-Jan-11 at 20:27

I've got data with time (seconds) on the x axis and intensity (in relative fluorescent units, or rfu) on the y-axis. It's generated by watching fragments of DNA pass a camera - the bigger the DNA fragment the bigger the time. There are 23 fragments of known size (in DNA base pair units, bp), and therefore there should be 23 peaks. As I know the size of the DNA fragments in bp, I want to recalibrate the x-axis from time (seconds) to base pairs (bp) using a linear model.

Unfortunately there is quite a lot of noise in the data that produces spurious peaks. The only way to confidently tell the true ones from the false ones is that the false ones don't fit the expected pattern in DNA base pairs.

I've provided data from one sample at this link in a data frame called demo. Unfortunately it's too large to paste below.

https://1drv.ms/t/s!AvBi5ipmBYfrhf0v_kvWuN2foLyBgg?e=RWfdXZ

I can pick out all the peaks as follows.

library(ggplot2)
library(ggpmisc)
ggplot(demo, aes(x=time, y=rfu)) +
geom_line(size=0.1) +
stat_peaks(col = "red", span=5, ignore_threshold=0.1) +
theme_bw()

However, these are the expected DNA fragment sizes (in bp):

ladder <- c(50, 75, 100, 125, 150, 200, 250, 300, 350, 400, 450, 475, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000)

And here is a plot with the correct peaks picked out:

Is there any way to get a peak finder such as stat_peak or ggpmisc:::find_peaks to take an pattern of peaks into account when selecting peaks to achieve the second plot??

Addendum: The false peaks may be within the same rfu range as the true peaks, so peak height can't be used to exclude them (see example from a different sample below)

ANSWER

Answered 2022-Jan-04 at 18:39

Before plotting, doing some data manipulation to pull out the maximum value for each of the 23 DNA fragment groups with base R max function, and adding the max plot with additional geom_ layer for the max values.

Here is small reprex example that plots the max value for each group with "red".

data.frame(x = 1:9) %>% 
     mutate(y = rep(LETTERS[1:3], c(3, 3, 3) )) %>% 
     group_by(y) %>% 
     mutate(max = max(x), .groups = "drop") %>% 
ggplot(aes(x = y, y = x)) + 
     geom_point() + 
     geom_point(aes(x = y, y = max), col = "red")

Source https://stackoverflow.com/questions/70582716

QUESTION

OpenVino converted model not returning same score values as original model (Sigmoid)

Asked 2022-Jan-05 at 06:06

I've converted a Keras model for use with OpenVino. The original Keras model used sigmoid to return scores ranging from 0 to 1 for binary classification. After converting the model for use with OpenVino, the scores are all near 0.99 for both classes but seem slightly lower for one of the classes.

For example, test1.jpg and test2.jpg (from opposite classes) yield scores of 0.00320357 and 0.9999, respectively.

With OpenVino, the same images yield scores of 0.9998982 and 0.9962392, respectively.

Edit* One suspicion is that the input array is still accepted by the OpenVino model but is somehow changed in shape or "scrambled" and therefore is never a match for class one? In other words, if you fed it random noise, the score would also always be 0.9999. Maybe I'd have to somehow get the OpenVino model to accept the original shape (1,180,180,3) instead of (1,3,180,180) so I don't have to force the input into a different shape than the one the original model accepted? That's weird though because I specified the shape when making the xml and bin for openvino:

python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo_tf.py --saved_model_dir /Users/.../Desktop/.../model13 --output_dir /Users/.../Desktop/... --input_shape=\[1,180,180,3]

However, I know from error messages that the inference engine is expecting (1,3,180,180) for some unknown reason. Could that be the problem? The other suspicion is something wrong with how the original model was frozen. I'm exploring different ways to freeze the original model (keras model converted to pb) in case the problem is related to that.

I checked to make sure the Sigmoid activation function is being used in the OpenVino implementation (same activation as the Keras model) and it looks like it is. Why, then, are the values not the same? Any help would be much appreciated.

The code for the OpenVino inference is:

import openvino
from openvino.inference_engine import IECore, IENetwork 
from skimage import io
import sys
import numpy as np
import os

def loadNetwork(model_xml, model_bin):

    ie = IECore() 

    network =  ie.read_network(model=model_xml, weights=model_bin)

    input_placeholder_key = list(network.input_info)[0]
    input_placeholder = network.input_info[input_placeholder_key]

    output_placeholder_key = list(network.outputs)[0]
    output_placeholder = network.outputs[output_placeholder_key]

    return network, input_placeholder_key, output_placeholder_key

batch_size = 1
channels = 3
IMG_HEIGHT = 180
IMG_WIDTH = 180

#loadNetwork('saved_model.xml','saved_model.bin')

image_path = 'test.jpg'

def load_source(path_to_image):
    image = io.imread(path_to_image)
    img = np.resize(image,(180,180))
    return img

img_new = load_source('test2.jpg')

#Batch?

def classify(image):
    device = 'CPU'
    network, input_placeholder_key, output_placeholder_key = loadNetwork('saved_model.xml','saved_model.bin')
    ie = IECore() 
    exec_net = ie.load_network(network=network, device_name=device)
    res = exec_net.infer(inputs={input_placeholder_key: image})
    print(res)
    res = res[output_placeholder_key]
    return res

result = classify(img_new)
print(result)
result = result[0]
top_result = np.argmax(result)
print(top_result)
print(result[top_result])

And the result:

{'StatefulPartitionedCall/model/dense/Sigmoid': array([[0.9962392]], dtype=float32)}
[[0.9962392]]
0
0.9962392

ANSWER

Answered 2022-Jan-05 at 06:06

Generally, Tensorflow is the only network with the shape NHWC while most others use NCHW. Thus, the OpenVINO Inference Engine satisfies the majority of networks and uses the NCHW layout. Model must be converted to NCHW layout in order to work with Inference Engine.

The conversion of the native model format into IR involves the process where the Model Optimizer performs the necessary transformation to convert the shape to the layout required by the Inference Engine (N,C,H,W). Using the --input_shape parameter with the correct input shape of the model should suffice.

Besides, most TensorFlow models are trained with images in RGB order. In this case, inference results using the Inference Engine samples may be incorrect. By default, Inference Engine samples and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the sample or demo application or reconvert your model using the Model Optimizer tool with --reverse_input_channels argument.

I suggest you validate this by inferring your model with the Hello Classification Python Sample instead since this is one of the official samples provided to test the model's functionality.

You may refer to this "Intel Math Kernel Library for Deep Neural Network" for deeper explanation regarding the input shape.

Source https://stackoverflow.com/questions/70546922

QUESTION

I have a dataset in which i have two columns with time in it but the dat

Asked 2021-Dec-22 at 17:00

Unnamed: 0     Created Date             Closed Date            Agency   Agency Name  Complaint Type Descriptor  Location Type   Incident Zip    Address Type    City    Landmark    Status  Borough
2869    2869    10/30/2013 09:14:47 AM  10/30/2013 10:48:51 AM  NYPD    New York City Police Department Illegal Parking Double Parked Blocking Traffic  Street/Sidewalk 11217.0 PLACENAME   BROOKLYN    BARCLAYS CENTER Closed  BROOKLYN
23571   23571   10/25/2013 02:33:54 PM  10/25/2013 03:36:36 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 10000   PLACENAME   NEW YORK    CENTRAL PARK    Closed  MANHATTAN
41625   41625   10/22/2013 09:33:56 PM  10/24/2013 05:37:24 PM  TLC Taxi and Limousine Commission   For Hire Vehicle Complaint  Car Service Company Complaint   Street  11430   PLACENAME   JAMAICA J F K AIRPORT   Closed  QUEENS
44331   44331   10/22/2013 07:25:35 AM  10/25/2013 10:40:35 AM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  11430   PLACENAME   JAMAICA J F K AIRPORT   Closed  QUEENS
46913   46913   10/21/2013 05:03:26 PM  10/23/2013 09:59:23 AM  DPR Department of Parks and Recreation  Dead Tree   Dead/Dying Tree Street  11215   PLACENAME   BROOKLYN    BARTEL PRITCHARD SQUARE Closed  BROOKLYN
47459   47459   10/21/2013 02:56:08 PM  10/29/2013 06:17:10 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  10031   PLACENAME   NEW YORK    CITY COLLEGE    Closed  MANHATTAN
48465   48465   10/21/2013 10:44:10 AM  10/21/2013 11:17:47 AM  NYPD    New York City Police Department Illegal Parking Posted Parking Sign Violation   Street/Sidewalk 11434   PLACENAME   JAMAICA PS 37   Closed  QUEENS
51837   51837   10/20/2013 04:36:12 PM  10/20/2013 06:35:49 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 10031.0 PLACENAME   NEW YORK    JACKIE ROBINSON PARK    Closed  MANHATTAN
51848   51848   10/20/2013 04:26:03 PM  10/20/2013 06:34:47 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 10031.0 PLACENAME   NEW YORK    JACKIE ROBINSON PARK    Closed  MANHATTAN
54089   54089   10/19/2013 03:45:47 PM  10/19/2013 04:10:11 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 10000.0 PLACENAME   NEW YORK    CENTRAL PARK    Closed  MANHATTAN
54343   54343   10/19/2013 01:27:43 PM  10/28/2013 08:42:12 AM  DOT Department of Transportation    Street Condition    Rough, Pitted or Cracked Roads  Street  10003.0 PLACENAME   NEW YORK    UNION SQUARE PARK   Closed  MANHATTAN
55140   55140   10/19/2013 02:02:28 AM  10/19/2013 02:19:55 AM  NYPD    New York City Police Department Noise - Vehicle Car/Truck Music Street/Sidewalk 11368.0 PLACENAME   CORONA  WORLDS FAIR MARINA  Closed  QUEENS
57789   57789   10/18/2013 11:55:44 AM  10/23/2013 02:42:14 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  11369.0 PLACENAME   EAST ELMHURST   LA GUARDIA AIRPORT  Closed  QUEENS
63119   63119   10/17/2013 06:52:37 AM  10/25/2013 06:49:59 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  11430.0 PLACENAME   JAMAICA J F K AIRPORT   Closed  QUEENS
66242   66242   10/16/2013 01:56:24 PM  10/22/2013 03:09:11 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  11369   PLACENAME   EAST ELMHURST   LA GUARDIA AIRPORT  Closed  QUEENS
66758   66758   10/16/2013 11:52:43 AM  10/16/2013 04:35:34 PM  NYPD    New York City Police Department Vending Unlicensed  Park/Playground 10036   PLACENAME   NEW YORK    BRYANT PARK Closed  MANHATTAN
66786   66786   10/16/2013 11:42:23 AM  10/18/2013 04:57:04 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Insurance Information Requested Street  10003   PLACENAME   NEW YORK    BETH ISRAEL MED CENTER  Closed  MANHATTAN
66809   66809   10/16/2013 11:36:54 AM  10/16/2013 12:34:23 PM  NYPD    New York City Police Department Traffic Congestion/Gridlock Street/Sidewalk 11430   PLACENAME   JAMAICA J F K AIRPORT   Closed  QUEENS
67465   67465   10/16/2013 09:14:35 AM  10/16/2013 12:43:06 PM  NYPD    New York City Police Department Traffic Drag Racing Street/Sidewalk 11367   PLACENAME   FLUSHING    QUEENS COLLEGE  Closed  QUEENS
72424   72424   10/15/2013 12:22:00 AM  10/21/2013 12:16:15 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  11217   PLACENAME   BROOKLYN    BARCLAYS CENTER Closed  BROOKLYN
75531   75531   10/14/2013 10:59:20 AM  10/14/2013 03:09:51 PM  NYPD    New York City Police Department Vending In Prohibited Area  Park/Playground 10000   PLACENAME   NEW YORK    CENTRAL PARK    Closed  MANHATTAN
77918   77918   10/13/2013 03:16:03 PM  10/13/2013 03:25:45 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 10000   PLACENAME   NEW YORK    CENTRAL PARK    Closed  MANHATTAN
78048   78048   10/13/2013 01:06:02 PM  10/21/2013 10:20:21 AM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  11369   PLACENAME   EAST ELMHURST   LA GUARDIA AIRPORT  Closed  QUEENS
78352   78352   10/13/2013 05:14:33 AM  10/16/2013 01:42:42 PM  TLC Taxi and Limousine Commission   For Hire Vehicle Complaint  Car Service Company Complaint   Street  11217   PLACENAME   BROOKLYN    BARCLAYS CENTER Closed  BROOKLYN
78383   78383   10/13/2013 03:50:02 AM  10/13/2013 05:03:13 AM  NYPD    New York City Police Department Noise - Vehicle Car/Truck Music Street/Sidewalk 11368   PLACENAME   CORONA  WORLDS FAIR MARINA  Closed  QUEENS
79078   79078   10/12/2013 09:53:17 PM  10/13/2013 02:52:07 AM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 10011   PLACENAME   NEW YORK    WASHINGTON SQUARE PARK  Closed  MANHATTAN
84489   84489   10/10/2013 07:16:16 PM  10/10/2013 10:29:16 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 11215   PLACENAME   BROOKLYN    PROSPECT PARK   Closed  BROOKLYN
84518   84518   10/10/2013 07:02:29 PM  10/10/2013 10:29:16 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 11215   PLACENAME   BROOKLYN    PROSPECT PARK   Closed  BROOKLYN
84688   84688   10/10/2013 05:39:19 PM  10/10/2013 10:29:17 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 11215   PLACENAME   BROOKLYN    PROSPECT PARK   Closed  BROOKLYN
84695   84695   10/10/2013 05:37:04 PM  10/10/2013 10:30:19 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 11215   PLACENAME   BROOKLYN    PROSPECT PARK   Closed  BROOKLYN
88812   88812   10/09/2013 09:17:15 PM  10/23/2013 02:15:21 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  11430   PLACENAME   JAMAICA J F K AIRPORT   Closed  QUEENS
89205   89205   10/09/2013 06:01:48 PM  10/09/2013 09:04:26 PM  NYPD    New York City Police Department Vending Unlicensed  Park/Playground 10000   PLACENAME   NEW YORK    CENTRAL PARK    Closed  MANHATTAN
89382   89382   10/09/2013 04:53:01 PM  10/18/2013 08:35:02 AM  DOT Department of Transportation    Public Toilet   Damaged Door    Sidewalk    11238   PLACENAME   BROOKLYN    GRAND ARMY PLAZA    Closed  BROOKLYN
89734   89734   10/09/2013 03:13:23 PM  10/09/2013 05:10:45 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 10036   PLACENAME   NEW YORK    BRYANT PARK Closed  MANHATTAN
93990   93990   10/08/2013 06:14:15 PM  10/09/2013 04:00:59 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  10003   PLACENAME   NEW YORK    BETH ISRAEL MED CENTER  Closed  MANHATTAN
99407   99407   10/07/2013 03:56:11 PM  10/08/2013 07:04:14 AM  DPR Department of Parks and Recreation  Overgrown Tree/Branches Traffic Sign or Signal Blocked  Street  11430.0 PLACENAME   JAMAICA J F K AIRPORT   Closed  QUEENS
99847   99847   10/07/2013 02:33:21 PM  10/09/2013 02:36:42 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  10036.0 PLACENAME   NEW YORK    PORT AUTH 42 STREET Closed  MANHATTAN
100073  100073  10/07/2013 01:36:02 PM  10/09/2013 09:56:55 AM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  10024.0 PLACENAME   NEW YORK    MUSEUM NATURAL HIST Closed  MANHATTAN
101013  101013  10/07/2013 10:05:18 AM  10/09/2013 03:36:23 PM  TLC Taxi and Limousine Commission   Taxi Complaint  Driver Complaint    Street  10017.0 PLACENAME   NEW YORK    GRAND CENTRAL TERM  Closed  MANHATTAN
104020  104020  10/06/2013 02:58:47 PM  10/07/2013 12:11:16 PM  TLC Taxi and Limousine Commission   For Hire Vehicle Complaint  Car Service Company Complaint   Street  11430.0 PLACENAME   JAMAICA JFK Closed  QUEENS
106118  106118  10/05/2013 03:24:47 PM  10/05/2013 04:20:34 PM  NYPD    New York City Police Department Noise - Park    Loud Music/Party    Park/Playground 10000.0 PLACENAME   NEW YORK    CENTRAL PARK    Closed  MANHATTAN
106499  106499  10/05/2013 11:52:13 AM  10/07/2013 08:00:28 AM  DOT Department of Transportation    Public Toilet   Dirty/Graffiti  Sidewalk    11369.0 PLACENAME   EAST ELMHURST   LA GUARDIA AIRPORT  Closed  QUEENS

this is how my data looks like. What I want to do here is to extract date from created date and closed date I have tried .extract method but I am new to this so, it didn't work. I want to calculate housrs from created date and closed which i can do like this:

pd.Timedelta(task3['Closed Date'] - task3['Created Date']).seconds / 60.0

In the end I want the output Find average completion time in hours for top-10 most frequent complaints. Also calculate how many data points you have for each complaint types. Do this analysis only for closed complaints. The sample output I want is as follows:

                         mean                   count
complaint Type           closing_time_hours     closing_time_hours
Blocked Driveway         3.00                   4581
DOF Literature Request   30.16                  5481
General Construction     66.38                  798
Heating                  54.88                  6704
Illegal Parking          3.08                   3336
Nonconst                 65                     100 
Paint-Plaster            49                     3281 
Plumbing                 65                     666   
Strret Condition         81                     2610    
Street Light Condition   90                     4207

these mean and count values are randomly generated The sample output can be produced by groupby function once the hours are extracted from the data. dictionary format

{'Unnamed: 0': {2869: 2869,
  23571: 23571,
  41625: 41625,
  44331: 44331,
  46913: 46913,
  47459: 47459,
  48465: 48465,
  51837: 51837,
  51848: 51848,
  54089: 54089,
  54343: 54343,
  55140: 55140,
  57789: 57789,
  63119: 63119,
  66242: 66242,
  66758: 66758,
  66786: 66786,
  66809: 66809,
  67465: 67465,
  72424: 72424,
  75531: 75531,
  77918: 77918,
  78048: 78048,
  78352: 78352,
  78383: 78383,
  79078: 79078,
  84489: 84489,
  84518: 84518,
  84688: 84688,
  84695: 84695,
  88812: 88812,
  89205: 89205,
  89382: 89382,
  89734: 89734,
  93990: 93990,
  99407: 99407,
  99847: 99847,
  100073: 100073,
  101013: 101013,
  104020: 104020,
  106118: 106118,
  106499: 106499},
 'Created Date': {2869: '10/30/2013 09:14:47 AM',
  23571: '10/25/2013 02:33:54 PM',
  41625: '10/22/2013 09:33:56 PM',
  44331: '10/22/2013 07:25:35 AM',
  46913: '10/21/2013 05:03:26 PM',
  47459: '10/21/2013 02:56:08 PM',
  48465: '10/21/2013 10:44:10 AM',
  51837: '10/20/2013 04:36:12 PM',
  51848: '10/20/2013 04:26:03 PM',
  54089: '10/19/2013 03:45:47 PM',
  54343: '10/19/2013 01:27:43 PM',
  55140: '10/19/2013 02:02:28 AM',
  57789: '10/18/2013 11:55:44 AM',
  63119: '10/17/2013 06:52:37 AM',
  66242: '10/16/2013 01:56:24 PM',
  66758: '10/16/2013 11:52:43 AM',
  66786: '10/16/2013 11:42:23 AM',
  66809: '10/16/2013 11:36:54 AM',
  67465: '10/16/2013 09:14:35 AM',
  72424: '10/15/2013 12:22:00 AM',
  75531: '10/14/2013 10:59:20 AM',
  77918: '10/13/2013 03:16:03 PM',
  78048: '10/13/2013 01:06:02 PM',
  78352: '10/13/2013 05:14:33 AM',
  78383: '10/13/2013 03:50:02 AM',
  79078: '10/12/2013 09:53:17 PM',
  84489: '10/10/2013 07:16:16 PM',
  84518: '10/10/2013 07:02:29 PM',
  84688: '10/10/2013 05:39:19 PM',
  84695: '10/10/2013 05:37:04 PM',
  88812: '10/09/2013 09:17:15 PM',
  89205: '10/09/2013 06:01:48 PM',
  89382: '10/09/2013 04:53:01 PM',
  89734: '10/09/2013 03:13:23 PM',
  93990: '10/08/2013 06:14:15 PM',
  99407: '10/07/2013 03:56:11 PM',
  99847: '10/07/2013 02:33:21 PM',
  100073: '10/07/2013 01:36:02 PM',
  101013: '10/07/2013 10:05:18 AM',
  104020: '10/06/2013 02:58:47 PM',
  106118: '10/05/2013 03:24:47 PM',
  106499: '10/05/2013 11:52:13 AM'},
 'Closed Date': {2869: '10/30/2013 10:48:51 AM',
  23571: '10/25/2013 03:36:36 PM',
  41625: '10/24/2013 05:37:24 PM',
  44331: '10/25/2013 10:40:35 AM',
  46913: '10/23/2013 09:59:23 AM',
  47459: '10/29/2013 06:17:10 PM',
  48465: '10/21/2013 11:17:47 AM',
  51837: '10/20/2013 06:35:49 PM',
  51848: '10/20/2013 06:34:47 PM',
  54089: '10/19/2013 04:10:11 PM',
  54343: '10/28/2013 08:42:12 AM',
  55140: '10/19/2013 02:19:55 AM',
  57789: '10/23/2013 02:42:14 PM',
  63119: '10/25/2013 06:49:59 PM',
  66242: '10/22/2013 03:09:11 PM',
  66758: '10/16/2013 04:35:34 PM',
  66786: '10/18/2013 04:57:04 PM',
  66809: '10/16/2013 12:34:23 PM',
  67465: '10/16/2013 12:43:06 PM',
  72424: '10/21/2013 12:16:15 PM',
  75531: '10/14/2013 03:09:51 PM',
  77918: '10/13/2013 03:25:45 PM',
  78048: '10/21/2013 10:20:21 AM',
  78352: '10/16/2013 01:42:42 PM',
  78383: '10/13/2013 05:03:13 AM',
  79078: '10/13/2013 02:52:07 AM',
  84489: '10/10/2013 10:29:16 PM',
  84518: '10/10/2013 10:29:16 PM',
  84688: '10/10/2013 10:29:17 PM',
  84695: '10/10/2013 10:30:19 PM',
  88812: '10/23/2013 02:15:21 PM',
  89205: '10/09/2013 09:04:26 PM',
  89382: '10/18/2013 08:35:02 AM',
  89734: '10/09/2013 05:10:45 PM',
  93990: '10/09/2013 04:00:59 PM',
  99407: '10/08/2013 07:04:14 AM',
  99847: '10/09/2013 02:36:42 PM',
  100073: '10/09/2013 09:56:55 AM',
  101013: '10/09/2013 03:36:23 PM',
  104020: '10/07/2013 12:11:16 PM',
  106118: '10/05/2013 04:20:34 PM',
  106499: '10/07/2013 08:00:28 AM'},
 'Agency': {2869: 'NYPD',
  23571: 'NYPD',
  41625: 'TLC',
  44331: 'TLC',
  46913: 'DPR',
  47459: 'TLC',
  48465: 'NYPD',
  51837: 'NYPD',
  51848: 'NYPD',
  54089: 'NYPD',
  54343: 'DOT',
  55140: 'NYPD',
  57789: 'TLC',
  63119: 'TLC',
  66242: 'TLC',
  66758: 'NYPD',
  66786: 'TLC',
  66809: 'NYPD',
  67465: 'NYPD',
  72424: 'TLC',
  75531: 'NYPD',
  77918: 'NYPD',
  78048: 'TLC',
  78352: 'TLC',
  78383: 'NYPD',
  79078: 'NYPD',
  84489: 'NYPD',
  84518: 'NYPD',
  84688: 'NYPD',
  84695: 'NYPD',
  88812: 'TLC',
  89205: 'NYPD',
  89382: 'DOT',
  89734: 'NYPD',
  93990: 'TLC',
  99407: 'DPR',
  99847: 'TLC',
  100073: 'TLC',
  101013: 'TLC',
  104020: 'TLC',
  106118: 'NYPD',
  106499: 'DOT'},
 'Agency Name': {2869: 'New York City Police Department',
  23571: 'New York City Police Department',
  41625: 'Taxi and Limousine Commission',
  44331: 'Taxi and Limousine Commission',
  46913: 'Department of Parks and Recreation',
  47459: 'Taxi and Limousine Commission',
  48465: 'New York City Police Department',
  51837: 'New York City Police Department',
  51848: 'New York City Police Department',
  54089: 'New York City Police Department',
  54343: 'Department of Transportation',
  55140: 'New York City Police Department',
  57789: 'Taxi and Limousine Commission',
  63119: 'Taxi and Limousine Commission',
  66242: 'Taxi and Limousine Commission',
  66758: 'New York City Police Department',
  66786: 'Taxi and Limousine Commission',
  66809: 'New York City Police Department',
  67465: 'New York City Police Department',
  72424: 'Taxi and Limousine Commission',
  75531: 'New York City Police Department',
  77918: 'New York City Police Department',
  78048: 'Taxi and Limousine Commission',
  78352: 'Taxi and Limousine Commission',
  78383: 'New York City Police Department',
  79078: 'New York City Police Department',
  84489: 'New York City Police Department',
  84518: 'New York City Police Department',
  84688: 'New York City Police Department',
  84695: 'New York City Police Department',
  88812: 'Taxi and Limousine Commission',
  89205: 'New York City Police Department',
  89382: 'Department of Transportation',
  89734: 'New York City Police Department',
  93990: 'Taxi and Limousine Commission',
  99407: 'Department of Parks and Recreation',
  99847: 'Taxi and Limousine Commission',
  100073: 'Taxi and Limousine Commission',
  101013: 'Taxi and Limousine Commission',
  104020: 'Taxi and Limousine Commission',
  106118: 'New York City Police Department',
  106499: 'Department of Transportation'},
 'Complaint Type': {2869: 'Illegal Parking',
  23571: 'Noise - Park',
  41625: 'For Hire Vehicle Complaint',
  44331: 'Taxi Complaint',
  46913: 'Dead Tree',
  47459: 'Taxi Complaint',
  48465: 'Illegal Parking',
  51837: 'Noise - Park',
  51848: 'Noise - Park',
  54089: 'Noise - Park',
  54343: 'Street Condition',
  55140: 'Noise - Vehicle',
  57789: 'Taxi Complaint',
  63119: 'Taxi Complaint',
  66242: 'Taxi Complaint',
  66758: 'Vending',
  66786: 'Taxi Complaint',
  66809: 'Traffic',
  67465: 'Traffic',
  72424: 'Taxi Complaint',
  75531: 'Vending',
  77918: 'Noise - Park',
  78048: 'Taxi Complaint',
  78352: 'For Hire Vehicle Complaint',
  78383: 'Noise - Vehicle',
  79078: 'Noise - Park',
  84489: 'Noise - Park',
  84518: 'Noise - Park',
  84688: 'Noise - Park',
  84695: 'Noise - Park',
  88812: 'Taxi Complaint',
  89205: 'Vending',
  89382: 'Public Toilet',
  89734: 'Noise - Park',
  93990: 'Taxi Complaint',
  99407: 'Overgrown Tree/Branches',
  99847: 'Taxi Complaint',
  100073: 'Taxi Complaint',
  101013: 'Taxi Complaint',
  104020: 'For Hire Vehicle Complaint',
  106118: 'Noise - Park',
  106499: 'Public Toilet'},
 'Descriptor': {2869: 'Double Parked Blocking Traffic',
  23571: 'Loud Music/Party',
  41625: 'Car Service Company Complaint',
  44331: 'Driver Complaint',
  46913: 'Dead/Dying Tree',
  47459: 'Driver Complaint',
  48465: 'Posted Parking Sign Violation',
  51837: 'Loud Music/Party',
  51848: 'Loud Music/Party',
  54089: 'Loud Music/Party',
  54343: 'Rough, Pitted or Cracked Roads',
  55140: 'Car/Truck Music',
  57789: 'Driver Complaint',
  63119: 'Driver Complaint',
  66242: 'Driver Complaint',
  66758: 'Unlicensed',
  66786: 'Insurance Information Requested',
  66809: 'Congestion/Gridlock',
  67465: 'Drag Racing',
  72424: 'Driver Complaint',
  75531: 'In Prohibited Area',
  77918: 'Loud Music/Party',
  78048: 'Driver Complaint',
  78352: 'Car Service Company Complaint',
  78383: 'Car/Truck Music',
  79078: 'Loud Music/Party',
  84489: 'Loud Music/Party',
  84518: 'Loud Music/Party',
  84688: 'Loud Music/Party',
  84695: 'Loud Music/Party',
  88812: 'Driver Complaint',
  89205: 'Unlicensed',
  89382: 'Damaged Door',
  89734: 'Loud Music/Party',
  93990: 'Driver Complaint',
  99407: 'Traffic Sign or Signal Blocked',
  99847: 'Driver Complaint',
  100073: 'Driver Complaint',
  101013: 'Driver Complaint',
  104020: 'Car Service Company Complaint',
  106118: 'Loud Music/Party',
  106499: 'Dirty/Graffiti'},
 'Location Type': {2869: 'Street/Sidewalk',
  23571: 'Park/Playground',
  41625: 'Street',
  44331: 'Street',
  46913: 'Street',
  47459: 'Street',
  48465: 'Street/Sidewalk',
  51837: 'Park/Playground',
  51848: 'Park/Playground',
  54089: 'Park/Playground',
  54343: 'Street',
  55140: 'Street/Sidewalk',
  57789: 'Street',
  63119: 'Street',
  66242: 'Street',
  66758: 'Park/Playground',
  66786: 'Street',
  66809: 'Street/Sidewalk',
  67465: 'Street/Sidewalk',
  72424: 'Street',
  75531: 'Park/Playground',
  77918: 'Park/Playground',
  78048: 'Street',
  78352: 'Street',
  78383: 'Street/Sidewalk',
  79078: 'Park/Playground',
  84489: 'Park/Playground',
  84518: 'Park/Playground',
  84688: 'Park/Playground',
  84695: 'Park/Playground',
  88812: 'Street',
  89205: 'Park/Playground',
  89382: 'Sidewalk',
  89734: 'Park/Playground',
  93990: 'Street',
  99407: 'Street',
  99847: 'Street',
  100073: 'Street',
  101013: 'Street',
  104020: 'Street',
  106118: 'Park/Playground',
  106499: 'Sidewalk'},
 'Incident Zip': {2869: '11217.0',
  23571: '10000',
  41625: '11430',
  44331: '11430',
  46913: '11215',
  47459: '10031',
  48465: '11434',
  51837: '10031.0',
  51848: '10031.0',
  54089: '10000.0',
  54343: '10003.0',
  55140: '11368.0',
  57789: '11369.0',
  63119: '11430.0',
  66242: '11369',
  66758: '10036',
  66786: '10003',
  66809: '11430',
  67465: '11367',
  72424: '11217',
  75531: '10000',
  77918: '10000',
  78048: '11369',
  78352: '11217',
  78383: '11368',
  79078: '10011',
  84489: '11215',
  84518: '11215',
  84688: '11215',
  84695: '11215',
  88812: '11430',
  89205: '10000',
  89382: '11238',
  89734: '10036',
  93990: '10003',
  99407: '11430.0',
  99847: '10036.0',
  100073: '10024.0',
  101013: '10017.0',
  104020: '11430.0',
  106118: '10000.0',
  106499: '11369.0'},
 'Address Type': {2869: 'PLACENAME',
  23571: 'PLACENAME',
  41625: 'PLACENAME',
  44331: 'PLACENAME',
  46913: 'PLACENAME',
  47459: 'PLACENAME',
  48465: 'PLACENAME',
  51837: 'PLACENAME',
  51848: 'PLACENAME',
  54089: 'PLACENAME',
  54343: 'PLACENAME',
  55140: 'PLACENAME',
  57789: 'PLACENAME',
  63119: 'PLACENAME',
  66242: 'PLACENAME',
  66758: 'PLACENAME',
  66786: 'PLACENAME',
  66809: 'PLACENAME',
  67465: 'PLACENAME',
  72424: 'PLACENAME',
  75531: 'PLACENAME',
  77918: 'PLACENAME',
  78048: 'PLACENAME',
  78352: 'PLACENAME',
  78383: 'PLACENAME',
  79078: 'PLACENAME',
  84489: 'PLACENAME',
  84518: 'PLACENAME',
  84688: 'PLACENAME',
  84695: 'PLACENAME',
  88812: 'PLACENAME',
  89205: 'PLACENAME',
  89382: 'PLACENAME',
  89734: 'PLACENAME',
  93990: 'PLACENAME',
  99407: 'PLACENAME',
  99847: 'PLACENAME',
  100073: 'PLACENAME',
  101013: 'PLACENAME',
  104020: 'PLACENAME',
  106118: 'PLACENAME',
  106499: 'PLACENAME'},
 'City': {2869: 'BROOKLYN',
  23571: 'NEW YORK',
  41625: 'JAMAICA',
  44331: 'JAMAICA',
  46913: 'BROOKLYN',
  47459: 'NEW YORK',
  48465: 'JAMAICA',
  51837: 'NEW YORK',
  51848: 'NEW YORK',
  54089: 'NEW YORK',
  54343: 'NEW YORK',
  55140: 'CORONA',
  57789: 'EAST ELMHURST',
  63119: 'JAMAICA',
  66242: 'EAST ELMHURST',
  66758: 'NEW YORK',
  66786: 'NEW YORK',
  66809: 'JAMAICA',
  67465: 'FLUSHING',
  72424: 'BROOKLYN',
  75531: 'NEW YORK',
  77918: 'NEW YORK',
  78048: 'EAST ELMHURST',
  78352: 'BROOKLYN',
  78383: 'CORONA',
  79078: 'NEW YORK',
  84489: 'BROOKLYN',
  84518: 'BROOKLYN',
  84688: 'BROOKLYN',
  84695: 'BROOKLYN',
  88812: 'JAMAICA',
  89205: 'NEW YORK',
  89382: 'BROOKLYN',
  89734: 'NEW YORK',
  93990: 'NEW YORK',
  99407: 'JAMAICA',
  99847: 'NEW YORK',
  100073: 'NEW YORK',
  101013: 'NEW YORK',
  104020: 'JAMAICA',
  106118: 'NEW YORK',
  106499: 'EAST ELMHURST'},
 'Landmark': {2869: 'BARCLAYS CENTER',
  23571: 'CENTRAL PARK',
  41625: 'J F K AIRPORT',
  44331: 'J F K AIRPORT',
  46913: 'BARTEL PRITCHARD SQUARE',
  47459: 'CITY COLLEGE',
  48465: 'PS 37',
  51837: 'JACKIE ROBINSON PARK',
  51848: 'JACKIE ROBINSON PARK',
  54089: 'CENTRAL PARK',
  54343: 'UNION SQUARE PARK',
  55140: 'WORLDS FAIR MARINA',
  57789: 'LA GUARDIA AIRPORT',
  63119: 'J F K AIRPORT',
  66242: 'LA GUARDIA AIRPORT',
  66758: 'BRYANT PARK',
  66786: 'BETH ISRAEL MED CENTER',
  66809: 'J F K AIRPORT',
  67465: 'QUEENS COLLEGE',
  72424: 'BARCLAYS CENTER',
  75531: 'CENTRAL PARK',
  77918: 'CENTRAL PARK',
  78048: 'LA GUARDIA AIRPORT',
  78352: 'BARCLAYS CENTER',
  78383: 'WORLDS FAIR MARINA',
  79078: 'WASHINGTON SQUARE PARK',
  84489: 'PROSPECT PARK',
  84518: 'PROSPECT PARK',
  84688: 'PROSPECT PARK',
  84695: 'PROSPECT PARK',
  88812: 'J F K AIRPORT',
  89205: 'CENTRAL PARK',
  89382: 'GRAND ARMY PLAZA',
  89734: 'BRYANT PARK',
  93990: 'BETH ISRAEL MED CENTER',
  99407: 'J F K AIRPORT',
  99847: 'PORT AUTH 42 STREET',
  100073: 'MUSEUM NATURAL HIST',
  101013: 'GRAND CENTRAL TERM',
  104020: 'JFK',
  106118: 'CENTRAL PARK',
  106499: 'LA GUARDIA AIRPORT'},
 'Status': {2869: 'Closed',
  23571: 'Closed',
  41625: 'Closed',
  44331: 'Closed',
  46913: 'Closed',
  47459: 'Closed',
  48465: 'Closed',
  51837: 'Closed',
  51848: 'Closed',
  54089: 'Closed',
  54343: 'Closed',
  55140: 'Closed',
  57789: 'Closed',
  63119: 'Closed',
  66242: 'Closed',
  66758: 'Closed',
  66786: 'Closed',
  66809: 'Closed',
  67465: 'Closed',
  72424: 'Closed',
  75531: 'Closed',
  77918: 'Closed',
  78048: 'Closed',
  78352: 'Closed',
  78383: 'Closed',
  79078: 'Closed',
  84489: 'Closed',
  84518: 'Closed',
  84688: 'Closed',
  84695: 'Closed',
  88812: 'Closed',
  89205: 'Closed',
  89382: 'Closed',
  89734: 'Closed',
  93990: 'Closed',
  99407: 'Closed',
  99847: 'Closed',
  100073: 'Closed',
  101013: 'Closed',
  104020: 'Closed',
  106118: 'Closed',
  106499: 'Closed'},
 'Borough': {2869: 'BROOKLYN',
  23571: 'MANHATTAN',
  41625: 'QUEENS',
  44331: 'QUEENS',
  46913: 'BROOKLYN',
  47459: 'MANHATTAN',
  48465: 'QUEENS',
  51837: 'MANHATTAN',
  51848: 'MANHATTAN',
  54089: 'MANHATTAN',
  54343: 'MANHATTAN',
  55140: 'QUEENS',
  57789: 'QUEENS',
  63119: 'QUEENS',
  66242: 'QUEENS',
  66758: 'MANHATTAN',
  66786: 'MANHATTAN',
  66809: 'QUEENS',
  67465: 'QUEENS',
  72424: 'BROOKLYN',
  75531: 'MANHATTAN',
  77918: 'MANHATTAN',
  78048: 'QUEENS',
  78352: 'BROOKLYN',
  78383: 'QUEENS',
  79078: 'MANHATTAN',
  84489: 'BROOKLYN',
  84518: 'BROOKLYN',
  84688: 'BROOKLYN',
  84695: 'BROOKLYN',
  88812: 'QUEENS',
  89205: 'MANHATTAN',
  89382: 'BROOKLYN',
  89734: 'MANHATTAN',
  93990: 'MANHATTAN',
  99407: 'QUEENS',
  99847: 'MANHATTAN',
  100073: 'MANHATTAN',
  101013: 'MANHATTAN',
  104020: 'QUEENS',
  106118: 'MANHATTAN',
  106499: 'QUEENS'}}

ANSWER

Answered 2021-Dec-22 at 17:00

You can first start by converting your date columns to datetime type using pd.to_datetime():

for c in ['Created Date', 'Closed Date']:
    df[c] = pd.to_datetime(df[c])
    #df[c+'_date'] = df[c].dt.date # to extract the date (for created + closed)
    #df[c+'_time'] = df[c].dt.time # to extract the time (for created + closed)

Then you can calculate the difference in time between the two as a new column (as hours) .astype('timedelta64[h]'), and then calculate a grouped mean:

df['difference in time'] = (df['Closed Date'] - df['Created Date']).astype('timedelta64[h]')
print(df.groupby('Complaint Type').agg({'difference in time':'mean'}))

returns:

                            difference in time
Complaint Type                                
Dead Tree                            40.000000
For Hire Vehicle Complaint           48.333333
Illegal Parking                       0.500000
Noise - Park                          1.916667
Noise - Vehicle                       0.500000
Overgrown Tree/Branches              15.000000
Public Toilet                       125.500000
Street Condition                    211.000000
Taxi Complaint                      125.461538
Traffic                               1.500000
Vending                               3.666667

Source https://stackoverflow.com/questions/70447276

QUESTION

Is there a better way to remove parts with consecutive zeros that have length equal or above threshold?

Asked 2021-Nov-27 at 16:37

Problem statement:

As stated by the title, I want to remove parts from an 1D array that have consecutive zeros and length equal or above a threshold.

My solution:

I produced the solution shown in the following MRE:

import numpy as np

THRESHOLD = 4

a = np.array((1,1,0,1,0,0,0,0,1,1,0,0,0,1,0,0,0,0,0,1))

print("Input: " + str(a))

# Find the indices of the parts that meet threshold requirement
gaps_above_threshold_inds = np.where(np.diff(np.nonzero(a)[0]) - 1 >= THRESHOLD)[0]

# Delete these parts from array
for idx in gaps_above_threshold_inds:
    a = np.delete(a, list(range(np.nonzero(a)[0][idx] + 1, np.nonzero(a)[0][idx + 1])))
    
print("Output: " + str(a))

Output:

Input:  [1 1 0 1 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 1]
Output: [1 1 0 1 1 1 0 0 0 1 1]

Question:

Is there a less complicated and more efficient way to do this on a numpy array?

Edit:

Based on @mozway comments, I'm editing my question providing some more information.

Basically, the problem domain is:

I have 1D signals of length ~20.000 samples
Some parts of the signals have been zeroed due to noise
The rest of the signal has non-zero values, in the range ~[50, 250]
Leading and trailing zeros have been removed

My goal is to remove the zero parts above a length threshold as I have already said.

More detailed questions:

As far as numpy efficient handling is concerned, is there a better solution from the one above?
As far as efficient signal processing techniques are concerned, is there more suitable way to achieve the desired goal than using numpy?

Comments on answers:

Regarding my first concern about efficient numpy handling, @mathfux's solution is really great and basically what I was looking for. That's why I accepted this one.

However, the approach by @Jérôme Richard answers my second question and it presents a really high performance solution; really useful if the dataset is extremely big.

Thanks for your great answers!

ANSWER

Answered 2021-Nov-26 at 15:17

A quite efficient method is to use itertools.groupby+itertools.chain:

from itertools import groupby, chain
a2 = np.array(list(chain(*(l for k,g in groupby(a)
                           if len(l:=list(g))<THRESHOLD or k))))

output:

array([1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1])

This works relatively fast, for instance on 1 million items:

# A = np.random.randint(2, size=1000000)
%%timeit
np.array(list(chain(*(l for k,g in groupby(a)
                      if len(l:=list(g))<THRESHOLD or k))))

# 254 ms ± 3.03 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Source https://stackoverflow.com/questions/70125699

QUESTION

Perl anchored regex performance

Asked 2021-Nov-09 at 17:56

Problem and Data

At the bottom of this post is the entire script from which this NYTProf data was generated. The script builds a hash and then attempts to delete keys that contain certain bad pattern. Running the code through NYTProf generates the following

delete @$hash{ grep { /\Q$bad_pattern\E/ } sort keys %$hash };
# spent  7.29ms making    2 calls to main::CORE:sort, avg 3.64ms/call
# spent   808µs making 7552 calls to main::CORE:match, avg 107ns/call
# spent   806µs making 7552 calls to main::CORE:regcomp, avg 107ns/call

There are over 7,000 calls being made to main::CORE:match and main::CORE:regcomp. The assumption is that this is a sufficient amount of calls to reduce noise levels.

Moving on! The bad patterns only need to be deleted if they appear at the beginning of a key. Sounds great! Adding a ^ to anchor the regex should improve performance. However, NYTProf generates the following. NYTprof has been run many times and this is quite consistent

delete @$hash{ grep { /^\Q$bad_pattern\E/ } sort keys %$hash };
# spent  7.34ms making    2 calls to main::CORE:sort, avg 3.67ms/call
# spent  1.62ms making 7552 calls to main::CORE:regcomp, avg 214ns/call
# spent   723µs making 7552 calls to main::CORE:match, avg 96ns/call

Questions

The anchored regex nearly doubles the amount of time spent in these main::CORE:* methods. But an anchored regex should improve performance. What is unique about this dataset that causes the anchored regex to take so much additional time?

Entire Script

use strict;
use Devel::NYTProf;

my @states = qw(KansasCity MississippiState ColoradoMountain IdahoInTheNorthWest AnchorageIsEvenFurtherNorth);
my @cities = qw(WitchitaHouston ChicagoDenver);
my @streets = qw(DowntownMainStreetInTheCity CenterStreetOverTheHill HickoryBasketOnTheWall);
my @seasoncode = qw(8000S 8000P 8000F 8000W);
my @historycode = qw(7000S 7000P 7000F 7000W 7000A 7000D 7000G 7000H);
my @sides = qw(left right up down);

my $hash;
for my $state (@states) {
    for my $city (@cities) {
        for my $street (@streets) {
            for my $season (@seasoncode) {
                for my $history (@historycode) {
                    for my $side (@sides) {
                        $hash->{$state . '[0].' . $city . '[1].' . $street . '[2].' . $season . '.' . $history . '.' . $side} = 1;
                    }
                }
            }
        }
    }
}

sub CleanseHash {
    my @bad_patterns = (
        'KansasCity[0].WitchitaHouston[1].DowntownMainStreetInTheCity[2]',
        'ColoradoMountain[0].ChicagoDenver[1].HickoryBasketOnTheWall[2].8000F'
    );

    for my $bad_pattern (@bad_patterns) {
        delete @$hash{ grep { /^\Q$bad_pattern\E/ } sort keys %$hash };
    }
}

DB::enable_profile();
CleanseHash();
DB::finish_profile();

ANSWER

Answered 2021-Nov-06 at 18:58

It's very unlikely you can optimise the regex engine. If performance is your goal, though, you can concentrate on other parts of the code. For example, try this:

for my $bad_pattern (@bad_patterns) {
    my $re = qr/^\Q$bad_pattern\E/;
    delete @$hash{ grep /$re/, sort keys %$hash };
}

On my machine, it runs much faster (regardless of the presence of the anchor), because the expression form of grep doesn't have to create a scope and the complex compilation of the regex happens just once for each bad pattern.

Source https://stackoverflow.com/questions/69866849

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install noise

You can download it from GitHub.
You can use noise like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the noise component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .