kandi background
Explore Kits

Algorithms | A collection of algorithms and data structures | Learning library

 by   williamfiset Java Version: Current License: MIT

 by   williamfiset Java Version: Current License: MIT

Download this library from

kandi X-RAY | Algorithms Summary

Algorithms is a Java library typically used in Institutions, Learning, Administration, Public Services, Tutorial, Learning, Example Codes applications. Algorithms has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. You can download it from GitHub.
Algorithms and data structures are fundamental to efficient code and good software design. Creating and designing excellent algorithms is required for being an exemplary programmer. This repository's goal is to demonstrate how to correctly implement common data structures and algorithms in the simplest and most elegant ways.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • Algorithms has a medium active ecosystem.
  • It has 10744 star(s) with 2860 fork(s). There are 371 watchers for this library.
  • It had no major release in the last 12 months.
  • There are 50 open issues and 93 have been closed. On average issues are closed in 259 days. There are 19 open pull requests and 0 closed requests.
  • It has a neutral sentiment in the developer community.
  • The latest version of Algorithms is current.
Algorithms Support
Best in #Learning
Average in #Learning
Algorithms Support
Best in #Learning
Average in #Learning

quality kandi Quality

  • Algorithms has no bugs reported.
Algorithms Quality
Best in #Learning
Average in #Learning
Algorithms Quality
Best in #Learning
Average in #Learning

securitySecurity

  • Algorithms has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
Algorithms Security
Best in #Learning
Average in #Learning
Algorithms Security
Best in #Learning
Average in #Learning

license License

  • Algorithms is licensed under the MIT License. This license is Permissive.
  • Permissive licenses have the least restrictions, and you can use them in most projects.
Algorithms License
Best in #Learning
Average in #Learning
Algorithms License
Best in #Learning
Average in #Learning

buildReuse

  • Algorithms releases are not available. You will need to build from source code and install.
  • Build file is available. You can build the component from source.
  • Installation instructions are not available. Examples and code snippets are available.
Algorithms Reuse
Best in #Learning
Average in #Learning
Algorithms Reuse
Best in #Learning
Average in #Learning
Top functions reviewed by kandi - BETA

kandi has reviewed Algorithms and discovered the below as its top functions. This is intended to give you an instant insight into Algorithms implemented functionality, and help decide if they suit your requirements.

  • Returns a textual representation of the tree displayed by the given node .
  • Consolidates the Fibonacci heap
  • Encodes a tree into a string .
  • Run the algorithm .
  • Intersect two line segments .
  • Finds the closest point in the given array .
  • Computes the shortest path in the graph .
  • Calculates intersection of two circles .
  • Returns the intersection of two lines .
  • Performs a reduction on an equation .

Algorithms Key Features

A collection of algorithms and data structures

Running with Gradle (recommended)

copy iconCopydownload iconDownload
./gradlew run -Palgorithm=<algorithm-subpackage>.<algorithm-class>

Create a classes folder

copy iconCopydownload iconDownload
cd Algorithms
mkdir classes

Compile the algorithm

copy iconCopydownload iconDownload
javac -sourcepath src/main/java -d classes src/main/java/ <relative-path-to-java-source-file>

Run the algorithm

copy iconCopydownload iconDownload
java -cp classes <class-fully-qualified-name>

Example

copy iconCopydownload iconDownload
$ javac -d classes -sourcepath src/main/java src/main/java/com/williamfiset/algorithms/search/BinarySearch.java
$ java -cp classes com.williamfiset.algorithms.search.BinarySearch

Python: Converting a list of strings into pandas data-frame with two columns a and b, corresponding to odd and even strings respectively

copy iconCopydownload iconDownload
import pandas as pd

index = [list[i] for i in range(0, len(list), 2)]
data = [list[i + 1] for i in range(0, len(list), 2)]

df = pd.DataFrame(data, index=index)\
    .reset_index()\
    .rename(columns={'index': 'MR', 0: 'Rest'})
-----------------------
df=pd.DataFrame(lst)  #here lst is your list...Don't assign anything to list function
c=df.index%2==0  #checking if the index is even bcz the values are in consicutive order 
out=pd.concat((df.loc[c,0].str.strip(':').reset_index(drop=True),df[~c].reset_index()),axis=1).drop('index',1)
#seperating out rows and concat them so that they can become new columns
out.columns=['MR','Rest']  #Finally renaming the columns
df=pd.DataFrame(lst)  #here lst is your list...Don't assign anything to list function
c=df.index%2==0  #checking if the index is even bcz the values are in consicutive order
out=pd.DataFrame([df.loc[c,0].str.strip(':').tolist(),df.loc[~c,0].tolist()],index=['MR','Rest']).T
out=out.drop_duplicates(subset=['MR'])
#removing duplicated entries on basis of MR column
-----------------------
df=pd.DataFrame(lst)  #here lst is your list...Don't assign anything to list function
c=df.index%2==0  #checking if the index is even bcz the values are in consicutive order 
out=pd.concat((df.loc[c,0].str.strip(':').reset_index(drop=True),df[~c].reset_index()),axis=1).drop('index',1)
#seperating out rows and concat them so that they can become new columns
out.columns=['MR','Rest']  #Finally renaming the columns
df=pd.DataFrame(lst)  #here lst is your list...Don't assign anything to list function
c=df.index%2==0  #checking if the index is even bcz the values are in consicutive order
out=pd.DataFrame([df.loc[c,0].str.strip(':').tolist(),df.loc[~c,0].tolist()],index=['MR','Rest']).T
out=out.drop_duplicates(subset=['MR'])
#removing duplicated entries on basis of MR column
-----------------------
df=pd.DataFrame(lst)  #here lst is your list...Don't assign anything to list function
c=df.index%2==0  #checking if the index is even bcz the values are in consicutive order 
out=pd.concat((df.loc[c,0].str.strip(':').reset_index(drop=True),df[~c].reset_index()),axis=1).drop('index',1)
#seperating out rows and concat them so that they can become new columns
out.columns=['MR','Rest']  #Finally renaming the columns
df=pd.DataFrame(lst)  #here lst is your list...Don't assign anything to list function
c=df.index%2==0  #checking if the index is even bcz the values are in consicutive order
out=pd.DataFrame([df.loc[c,0].str.strip(':').tolist(),df.loc[~c,0].tolist()],index=['MR','Rest']).T
out=out.drop_duplicates(subset=['MR'])
#removing duplicated entries on basis of MR column

Encrypt in JS front end and decrypt in python backend using AES GCM

copy iconCopydownload iconDownload
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import base64
#import os

#key = AESGCM.generate_key(bit_length=256)    
#nonce = os.urandom(12)
key = base64.b64decode('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE=') # fix for testing, AES-256
nonce = base64.b64decode('MDEyMzQ1Njc4OTAx') # fix for testing, 12 bytes

plaintext = b'The quick brown fox jumps over the lazy dog'
aad = b'the aad' # aad = None without additional authenticated data

aesgcm = AESGCM(key)
ciphertext = aesgcm.encrypt(nonce, plaintext, aad)
print('Ciphertext (B64): ' + base64.b64encode(ciphertext).decode('utf8'))
decrypted = aesgcm.decrypt(nonce, ciphertext, aad)
print('Decrypted:        ' + decrypted.decode('utf8'))
Output
Ciphertext (B64): JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Decrypted:        The quick brown fox jumps over the lazy dog
(async () => {      
    var key = Base64.toUint8Array('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE='); // fix for testing, AES-256
    var nonce = Base64.toUint8Array('MDEyMzQ1Njc4OTAx'); // fix for testing, 12 bytes

    var plaintext = new TextEncoder().encode("The quick brown fox jumps over the lazy dog");
    var aad = new TextEncoder().encode('the aad');
                
    var keyImported = await await crypto.subtle.importKey(
        "raw",
        key,
        { name: "AES-GCM" },
        true,
        ["decrypt", "encrypt"]
    );
                
    var ciphertext = await await crypto.subtle.encrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        plaintext
    );
    console.log('Ciphertext (Base64):\n', Base64.fromUint8Array(new Uint8Array(ciphertext)).replace(/(.{48})/g,'$1\n'));
              
    var decrypted = await await crypto.subtle.decrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        ciphertext
    );
    console.log('Plaintext:\n', new TextDecoder().decode(decrypted).replace(/(.{48})/g,'$1\n'));
})();
<script src="https://cdn.jsdelivr.net/npm/js-base64@3.2.4/base64.min.js"></script>
Ciphertext (Base64):
 JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Plaintext:
 The quick brown fox jumps over the lazy dog
-----------------------
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import base64
#import os

#key = AESGCM.generate_key(bit_length=256)    
#nonce = os.urandom(12)
key = base64.b64decode('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE=') # fix for testing, AES-256
nonce = base64.b64decode('MDEyMzQ1Njc4OTAx') # fix for testing, 12 bytes

plaintext = b'The quick brown fox jumps over the lazy dog'
aad = b'the aad' # aad = None without additional authenticated data

aesgcm = AESGCM(key)
ciphertext = aesgcm.encrypt(nonce, plaintext, aad)
print('Ciphertext (B64): ' + base64.b64encode(ciphertext).decode('utf8'))
decrypted = aesgcm.decrypt(nonce, ciphertext, aad)
print('Decrypted:        ' + decrypted.decode('utf8'))
Output
Ciphertext (B64): JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Decrypted:        The quick brown fox jumps over the lazy dog
(async () => {      
    var key = Base64.toUint8Array('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE='); // fix for testing, AES-256
    var nonce = Base64.toUint8Array('MDEyMzQ1Njc4OTAx'); // fix for testing, 12 bytes

    var plaintext = new TextEncoder().encode("The quick brown fox jumps over the lazy dog");
    var aad = new TextEncoder().encode('the aad');
                
    var keyImported = await await crypto.subtle.importKey(
        "raw",
        key,
        { name: "AES-GCM" },
        true,
        ["decrypt", "encrypt"]
    );
                
    var ciphertext = await await crypto.subtle.encrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        plaintext
    );
    console.log('Ciphertext (Base64):\n', Base64.fromUint8Array(new Uint8Array(ciphertext)).replace(/(.{48})/g,'$1\n'));
              
    var decrypted = await await crypto.subtle.decrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        ciphertext
    );
    console.log('Plaintext:\n', new TextDecoder().decode(decrypted).replace(/(.{48})/g,'$1\n'));
})();
<script src="https://cdn.jsdelivr.net/npm/js-base64@3.2.4/base64.min.js"></script>
Ciphertext (Base64):
 JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Plaintext:
 The quick brown fox jumps over the lazy dog
-----------------------
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import base64
#import os

#key = AESGCM.generate_key(bit_length=256)    
#nonce = os.urandom(12)
key = base64.b64decode('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE=') # fix for testing, AES-256
nonce = base64.b64decode('MDEyMzQ1Njc4OTAx') # fix for testing, 12 bytes

plaintext = b'The quick brown fox jumps over the lazy dog'
aad = b'the aad' # aad = None without additional authenticated data

aesgcm = AESGCM(key)
ciphertext = aesgcm.encrypt(nonce, plaintext, aad)
print('Ciphertext (B64): ' + base64.b64encode(ciphertext).decode('utf8'))
decrypted = aesgcm.decrypt(nonce, ciphertext, aad)
print('Decrypted:        ' + decrypted.decode('utf8'))
Output
Ciphertext (B64): JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Decrypted:        The quick brown fox jumps over the lazy dog
(async () => {      
    var key = Base64.toUint8Array('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE='); // fix for testing, AES-256
    var nonce = Base64.toUint8Array('MDEyMzQ1Njc4OTAx'); // fix for testing, 12 bytes

    var plaintext = new TextEncoder().encode("The quick brown fox jumps over the lazy dog");
    var aad = new TextEncoder().encode('the aad');
                
    var keyImported = await await crypto.subtle.importKey(
        "raw",
        key,
        { name: "AES-GCM" },
        true,
        ["decrypt", "encrypt"]
    );
                
    var ciphertext = await await crypto.subtle.encrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        plaintext
    );
    console.log('Ciphertext (Base64):\n', Base64.fromUint8Array(new Uint8Array(ciphertext)).replace(/(.{48})/g,'$1\n'));
              
    var decrypted = await await crypto.subtle.decrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        ciphertext
    );
    console.log('Plaintext:\n', new TextDecoder().decode(decrypted).replace(/(.{48})/g,'$1\n'));
})();
<script src="https://cdn.jsdelivr.net/npm/js-base64@3.2.4/base64.min.js"></script>
Ciphertext (Base64):
 JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Plaintext:
 The quick brown fox jumps over the lazy dog
-----------------------
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import base64
#import os

#key = AESGCM.generate_key(bit_length=256)    
#nonce = os.urandom(12)
key = base64.b64decode('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE=') # fix for testing, AES-256
nonce = base64.b64decode('MDEyMzQ1Njc4OTAx') # fix for testing, 12 bytes

plaintext = b'The quick brown fox jumps over the lazy dog'
aad = b'the aad' # aad = None without additional authenticated data

aesgcm = AESGCM(key)
ciphertext = aesgcm.encrypt(nonce, plaintext, aad)
print('Ciphertext (B64): ' + base64.b64encode(ciphertext).decode('utf8'))
decrypted = aesgcm.decrypt(nonce, ciphertext, aad)
print('Decrypted:        ' + decrypted.decode('utf8'))
Output
Ciphertext (B64): JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Decrypted:        The quick brown fox jumps over the lazy dog
(async () => {      
    var key = Base64.toUint8Array('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE='); // fix for testing, AES-256
    var nonce = Base64.toUint8Array('MDEyMzQ1Njc4OTAx'); // fix for testing, 12 bytes

    var plaintext = new TextEncoder().encode("The quick brown fox jumps over the lazy dog");
    var aad = new TextEncoder().encode('the aad');
                
    var keyImported = await await crypto.subtle.importKey(
        "raw",
        key,
        { name: "AES-GCM" },
        true,
        ["decrypt", "encrypt"]
    );
                
    var ciphertext = await await crypto.subtle.encrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        plaintext
    );
    console.log('Ciphertext (Base64):\n', Base64.fromUint8Array(new Uint8Array(ciphertext)).replace(/(.{48})/g,'$1\n'));
              
    var decrypted = await await crypto.subtle.decrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        ciphertext
    );
    console.log('Plaintext:\n', new TextDecoder().decode(decrypted).replace(/(.{48})/g,'$1\n'));
})();
<script src="https://cdn.jsdelivr.net/npm/js-base64@3.2.4/base64.min.js"></script>
Ciphertext (Base64):
 JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Plaintext:
 The quick brown fox jumps over the lazy dog
-----------------------
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import base64
#import os

#key = AESGCM.generate_key(bit_length=256)    
#nonce = os.urandom(12)
key = base64.b64decode('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE=') # fix for testing, AES-256
nonce = base64.b64decode('MDEyMzQ1Njc4OTAx') # fix for testing, 12 bytes

plaintext = b'The quick brown fox jumps over the lazy dog'
aad = b'the aad' # aad = None without additional authenticated data

aesgcm = AESGCM(key)
ciphertext = aesgcm.encrypt(nonce, plaintext, aad)
print('Ciphertext (B64): ' + base64.b64encode(ciphertext).decode('utf8'))
decrypted = aesgcm.decrypt(nonce, ciphertext, aad)
print('Decrypted:        ' + decrypted.decode('utf8'))
Output
Ciphertext (B64): JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Decrypted:        The quick brown fox jumps over the lazy dog
(async () => {      
    var key = Base64.toUint8Array('MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDE='); // fix for testing, AES-256
    var nonce = Base64.toUint8Array('MDEyMzQ1Njc4OTAx'); // fix for testing, 12 bytes

    var plaintext = new TextEncoder().encode("The quick brown fox jumps over the lazy dog");
    var aad = new TextEncoder().encode('the aad');
                
    var keyImported = await await crypto.subtle.importKey(
        "raw",
        key,
        { name: "AES-GCM" },
        true,
        ["decrypt", "encrypt"]
    );
                
    var ciphertext = await await crypto.subtle.encrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        plaintext
    );
    console.log('Ciphertext (Base64):\n', Base64.fromUint8Array(new Uint8Array(ciphertext)).replace(/(.{48})/g,'$1\n'));
              
    var decrypted = await await crypto.subtle.decrypt(
        { name: "AES-GCM", iv: nonce, additionalData: aad }, // { name: "AES-GCM", iv: nonce } without additional authenticated data
        keyImported,
        ciphertext
    );
    console.log('Plaintext:\n', new TextDecoder().decode(decrypted).replace(/(.{48})/g,'$1\n'));
})();
<script src="https://cdn.jsdelivr.net/npm/js-base64@3.2.4/base64.min.js"></script>
Ciphertext (Base64):
 JOetStCANhPISvQ6G6IcRBauqbtC8fzRooblayHqkqSPKzLbidx/gBWfLNzBC+ZpcAGnGnHXaI7CB1U=
Plaintext:
 The quick brown fox jumps over the lazy dog

Nodejs Restify - Valide Jwt id_token with jwks

copy iconCopydownload iconDownload
// authUser.js

module.exports = function authUser(req, res, next) {
    const token = getJwtToken(req)
    jwt.verify(token, publicKey, { algorithms: ['RS256'] }, function(err, decoded) {
        if (err) {
          res.send(401, 'Unauthorized')
          return next(false)
        }
        return next()
    })
}

server.get("/secure", authUser, function (req, res, next) {
  res.send(200, "secure route response");
  return next();
});

server.get("/", function (req, res, next) {
  res.send(200, "public route response");
  return next();
});
-----------------------
// authUser.js

module.exports = function authUser(req, res, next) {
    const token = getJwtToken(req)
    jwt.verify(token, publicKey, { algorithms: ['RS256'] }, function(err, decoded) {
        if (err) {
          res.send(401, 'Unauthorized')
          return next(false)
        }
        return next()
    })
}

server.get("/secure", authUser, function (req, res, next) {
  res.send(200, "secure route response");
  return next();
});

server.get("/", function (req, res, next) {
  res.send(200, "public route response");
  return next();
});
-----------------------
serve.use(authUser)
-----------------------
 async function authUser(req, res, next) {
  try{    
    req.isLogged=false
    console.log(req.params);
    let isvalid = await validAccessToken(req);
    console.log(">>> isValid : " + isvalid);
    if(isvalid===true){
      //token verifié on passe en preview false
      req.isLogged=true
      return next()
    }else{
      console.log("Unauthorized");
      req.isLogged=false
      return next()                
    }
  }catch(error){

    res.send(401, 'Unauthorized ' + error)
    return next()
  }
}
server.use(openid.authUser);
const preview = ! req.isLogged ;
-----------------------
 async function authUser(req, res, next) {
  try{    
    req.isLogged=false
    console.log(req.params);
    let isvalid = await validAccessToken(req);
    console.log(">>> isValid : " + isvalid);
    if(isvalid===true){
      //token verifié on passe en preview false
      req.isLogged=true
      return next()
    }else{
      console.log("Unauthorized");
      req.isLogged=false
      return next()                
    }
  }catch(error){

    res.send(401, 'Unauthorized ' + error)
    return next()
  }
}
server.use(openid.authUser);
const preview = ! req.isLogged ;
-----------------------
 async function authUser(req, res, next) {
  try{    
    req.isLogged=false
    console.log(req.params);
    let isvalid = await validAccessToken(req);
    console.log(">>> isValid : " + isvalid);
    if(isvalid===true){
      //token verifié on passe en preview false
      req.isLogged=true
      return next()
    }else{
      console.log("Unauthorized");
      req.isLogged=false
      return next()                
    }
  }catch(error){

    res.send(401, 'Unauthorized ' + error)
    return next()
  }
}
server.use(openid.authUser);
const preview = ! req.isLogged ;

How to write an efficient algorithm that generates a list of unique random numbers?

copy iconCopydownload iconDownload
function createRandomList() {
  let i=0, numbers=[]
  while(i++<5000) numbers.push(i);
  numbers.sort(() => (Math.random() > .5) ? 1 : -1);
  return numbers
}

console.log(createRandomList())
-----------------------
let arr = [1,2,3,4 ..., 5000]

for (let i = arr.length-1; i > 0; i--){
  let r = Math.floor(Math.random() * (i + 1));
  let x = a[i]; a[i] = a[r]; a[r] = x;
}
-----------------------
function shuffle(array) {
  var m = array.length, t, i;

  // While there remain elements to shuffle…
  while (m) {

    // Pick a remaining element…
    i = Math.floor(Math.random() * m--);

    // And swap it with the current element.
    t = array[m];
    array[m] = array[i];
    array[i] = t;
  }

  return array;
}

const sequence = [1,2,3,4,5,6,7,8,9] // or gen this for any length you choose
let randomNonRepeatingSequence = shuffle(sequence)
console.log(randomNonRepeatingSequence)

Web cryptography implement HKDF for the output of ECDH

copy iconCopydownload iconDownload
(async () => {
    await deriveKey();
})();

async function deriveKey() {

    //
    // Key import
    //
    var server_x509 =  `-----BEGIN PUBLIC KEY-----
                        MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEd7fej9GYVI7Vt6x5B6XhruHvmE/rnzIj
                        HmpxP8PKfnfWgrJbyG2cgQc3mf9uusqk1FKImA86rx2+avK8+7xIK9wxuF3x2KQq
                        nxNp7bUBit3phyhp72Nt/QLXmZHcDKXL
                        -----END PUBLIC KEY-----`;
    var client_pkcs8 = `-----BEGIN PRIVATE KEY-----
                        MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBjr4EGktNtx+3xErsC
                        MzldruzzfAEEO8Oth1/3b8sNfrqRsAgMnB/oVy024I+15wOhZANiAASbTF7LLedW
                        dik6nH8JX8WeU0R1ZRlqq0EAZ/t+UrFcSOaVJSOx5jMJ3nrqwuk2DnobDqFwXH6t
                        ZMsZHh4NFZ+bCVeHJRqy4SCZvQFB/xcksF29p1v14XHYI/XKMGyLLx4=
                        -----END PRIVATE KEY-----`;

    var client_private_key = b64_to_ab(client_pkcs8.replace('-----BEGIN PRIVATE KEY-----', '').replace('-----END PRIVATE KEY-----', ''));
    var server_public_key = b64_to_ab(server_x509.replace('-----BEGIN PUBLIC KEY-----', '').replace('-----END PUBLIC KEY-----', ''));
    var privateKey = await window.crypto.subtle.importKey( 
        'pkcs8', 
        client_private_key,
        { name: "ECDH", namedCurve: "P-384" },
        true, 
        ["deriveKey", "deriveBits"] 
    );
    var publicKey = await window.crypto.subtle.importKey(
        "spki", 
        server_public_key,
        { name: "ECDH", namedCurve: "P-384" },
        true, 
        [] 
    );
    
    //
    // Determine shared secret
    //
    var sharedSecret = await window.crypto.subtle.deriveBits(
        { name: "ECDH", namedCurve: "P-384", public: publicKey },
        privateKey, 
        384 
    );
    console.log("Shared secret:\n", ab_to_b64(sharedSecret).replace(/(.{48})/g,'$1\n'));
    
    //
    // Derive key from shared secret via HKDF
    //
    var sharedSecretKey = await window.crypto.subtle.importKey(
        "raw", 
        sharedSecret, 
        { name: "HKDF" }, 
        false, 
        ["deriveKey", "deriveBits"]
    );
    var derived_key = await crypto.subtle.deriveBits(
        { name: "HKDF", hash: "SHA-256", salt: new Uint8Array([]), info: new Uint8Array([]) }, 
        sharedSecretKey, 
        256
    );
    console.log("Derived key:\n", ab_to_b64(derived_key).replace(/(.{48})/g,'$1\n'))
}; 

function b64_to_ab(base64_string){
    return Uint8Array.from(atob(base64_string), c => c.charCodeAt(0));
}

function ab_to_b64(arrayBuffer){
    return btoa(String.fromCharCode.apply(null, new Uint8Array(arrayBuffer)));
}
Shared secret:
 xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF
Derived key:
 Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import serialization
import base64

def deriveKey():

  server_pkcs8 = b'''-----BEGIN PRIVATE KEY-----
MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBReGpDVmoVTzxNbJx6
aL4L9z1EdB91eonAmAw7mKDocLfCJITXZPUAmM46c6AipTmhZANiAAR3t96P0ZhU
jtW3rHkHpeGu4e+YT+ufMiMeanE/w8p+d9aCslvIbZyBBzeZ/266yqTUUoiYDzqv
Hb5q8rz7vEgr3DG4XfHYpCqfE2nttQGK3emHKGnvY239AteZkdwMpcs=
-----END PRIVATE KEY-----'''

  client_x509 = b'''-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEm0xeyy3nVnYpOpx/CV/FnlNEdWUZaqtB
AGf7flKxXEjmlSUjseYzCd566sLpNg56Gw6hcFx+rWTLGR4eDRWfmwlXhyUasuEg
mb0BQf8XJLBdvadb9eFx2CP1yjBsiy8e
-----END PUBLIC KEY-----'''

  client_public_key = serialization.load_pem_public_key(client_x509)
  server_private_key = serialization.load_pem_private_key(server_pkcs8, password=None)
  shared_secret = server_private_key.exchange(ec.ECDH(), client_public_key)
  print('Shared secret: ' + base64.b64encode(shared_secret).decode('utf8')) # Shared secret: xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF

  derived_key = HKDF(
    algorithm=hashes.SHA256(),
    length=32,
    salt=None,
    info=None,
  ).derive(shared_secret) 
  print('Derived key:   ' + base64.b64encode(derived_key).decode('utf8')) # Derived key:   Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=

deriveKey()
Shared secret: xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF
Derived key:   Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=
-----------------------
(async () => {
    await deriveKey();
})();

async function deriveKey() {

    //
    // Key import
    //
    var server_x509 =  `-----BEGIN PUBLIC KEY-----
                        MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEd7fej9GYVI7Vt6x5B6XhruHvmE/rnzIj
                        HmpxP8PKfnfWgrJbyG2cgQc3mf9uusqk1FKImA86rx2+avK8+7xIK9wxuF3x2KQq
                        nxNp7bUBit3phyhp72Nt/QLXmZHcDKXL
                        -----END PUBLIC KEY-----`;
    var client_pkcs8 = `-----BEGIN PRIVATE KEY-----
                        MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBjr4EGktNtx+3xErsC
                        MzldruzzfAEEO8Oth1/3b8sNfrqRsAgMnB/oVy024I+15wOhZANiAASbTF7LLedW
                        dik6nH8JX8WeU0R1ZRlqq0EAZ/t+UrFcSOaVJSOx5jMJ3nrqwuk2DnobDqFwXH6t
                        ZMsZHh4NFZ+bCVeHJRqy4SCZvQFB/xcksF29p1v14XHYI/XKMGyLLx4=
                        -----END PRIVATE KEY-----`;

    var client_private_key = b64_to_ab(client_pkcs8.replace('-----BEGIN PRIVATE KEY-----', '').replace('-----END PRIVATE KEY-----', ''));
    var server_public_key = b64_to_ab(server_x509.replace('-----BEGIN PUBLIC KEY-----', '').replace('-----END PUBLIC KEY-----', ''));
    var privateKey = await window.crypto.subtle.importKey( 
        'pkcs8', 
        client_private_key,
        { name: "ECDH", namedCurve: "P-384" },
        true, 
        ["deriveKey", "deriveBits"] 
    );
    var publicKey = await window.crypto.subtle.importKey(
        "spki", 
        server_public_key,
        { name: "ECDH", namedCurve: "P-384" },
        true, 
        [] 
    );
    
    //
    // Determine shared secret
    //
    var sharedSecret = await window.crypto.subtle.deriveBits(
        { name: "ECDH", namedCurve: "P-384", public: publicKey },
        privateKey, 
        384 
    );
    console.log("Shared secret:\n", ab_to_b64(sharedSecret).replace(/(.{48})/g,'$1\n'));
    
    //
    // Derive key from shared secret via HKDF
    //
    var sharedSecretKey = await window.crypto.subtle.importKey(
        "raw", 
        sharedSecret, 
        { name: "HKDF" }, 
        false, 
        ["deriveKey", "deriveBits"]
    );
    var derived_key = await crypto.subtle.deriveBits(
        { name: "HKDF", hash: "SHA-256", salt: new Uint8Array([]), info: new Uint8Array([]) }, 
        sharedSecretKey, 
        256
    );
    console.log("Derived key:\n", ab_to_b64(derived_key).replace(/(.{48})/g,'$1\n'))
}; 

function b64_to_ab(base64_string){
    return Uint8Array.from(atob(base64_string), c => c.charCodeAt(0));
}

function ab_to_b64(arrayBuffer){
    return btoa(String.fromCharCode.apply(null, new Uint8Array(arrayBuffer)));
}
Shared secret:
 xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF
Derived key:
 Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import serialization
import base64

def deriveKey():

  server_pkcs8 = b'''-----BEGIN PRIVATE KEY-----
MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBReGpDVmoVTzxNbJx6
aL4L9z1EdB91eonAmAw7mKDocLfCJITXZPUAmM46c6AipTmhZANiAAR3t96P0ZhU
jtW3rHkHpeGu4e+YT+ufMiMeanE/w8p+d9aCslvIbZyBBzeZ/266yqTUUoiYDzqv
Hb5q8rz7vEgr3DG4XfHYpCqfE2nttQGK3emHKGnvY239AteZkdwMpcs=
-----END PRIVATE KEY-----'''

  client_x509 = b'''-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEm0xeyy3nVnYpOpx/CV/FnlNEdWUZaqtB
AGf7flKxXEjmlSUjseYzCd566sLpNg56Gw6hcFx+rWTLGR4eDRWfmwlXhyUasuEg
mb0BQf8XJLBdvadb9eFx2CP1yjBsiy8e
-----END PUBLIC KEY-----'''

  client_public_key = serialization.load_pem_public_key(client_x509)
  server_private_key = serialization.load_pem_private_key(server_pkcs8, password=None)
  shared_secret = server_private_key.exchange(ec.ECDH(), client_public_key)
  print('Shared secret: ' + base64.b64encode(shared_secret).decode('utf8')) # Shared secret: xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF

  derived_key = HKDF(
    algorithm=hashes.SHA256(),
    length=32,
    salt=None,
    info=None,
  ).derive(shared_secret) 
  print('Derived key:   ' + base64.b64encode(derived_key).decode('utf8')) # Derived key:   Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=

deriveKey()
Shared secret: xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF
Derived key:   Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=
-----------------------
(async () => {
    await deriveKey();
})();

async function deriveKey() {

    //
    // Key import
    //
    var server_x509 =  `-----BEGIN PUBLIC KEY-----
                        MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEd7fej9GYVI7Vt6x5B6XhruHvmE/rnzIj
                        HmpxP8PKfnfWgrJbyG2cgQc3mf9uusqk1FKImA86rx2+avK8+7xIK9wxuF3x2KQq
                        nxNp7bUBit3phyhp72Nt/QLXmZHcDKXL
                        -----END PUBLIC KEY-----`;
    var client_pkcs8 = `-----BEGIN PRIVATE KEY-----
                        MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBjr4EGktNtx+3xErsC
                        MzldruzzfAEEO8Oth1/3b8sNfrqRsAgMnB/oVy024I+15wOhZANiAASbTF7LLedW
                        dik6nH8JX8WeU0R1ZRlqq0EAZ/t+UrFcSOaVJSOx5jMJ3nrqwuk2DnobDqFwXH6t
                        ZMsZHh4NFZ+bCVeHJRqy4SCZvQFB/xcksF29p1v14XHYI/XKMGyLLx4=
                        -----END PRIVATE KEY-----`;

    var client_private_key = b64_to_ab(client_pkcs8.replace('-----BEGIN PRIVATE KEY-----', '').replace('-----END PRIVATE KEY-----', ''));
    var server_public_key = b64_to_ab(server_x509.replace('-----BEGIN PUBLIC KEY-----', '').replace('-----END PUBLIC KEY-----', ''));
    var privateKey = await window.crypto.subtle.importKey( 
        'pkcs8', 
        client_private_key,
        { name: "ECDH", namedCurve: "P-384" },
        true, 
        ["deriveKey", "deriveBits"] 
    );
    var publicKey = await window.crypto.subtle.importKey(
        "spki", 
        server_public_key,
        { name: "ECDH", namedCurve: "P-384" },
        true, 
        [] 
    );
    
    //
    // Determine shared secret
    //
    var sharedSecret = await window.crypto.subtle.deriveBits(
        { name: "ECDH", namedCurve: "P-384", public: publicKey },
        privateKey, 
        384 
    );
    console.log("Shared secret:\n", ab_to_b64(sharedSecret).replace(/(.{48})/g,'$1\n'));
    
    //
    // Derive key from shared secret via HKDF
    //
    var sharedSecretKey = await window.crypto.subtle.importKey(
        "raw", 
        sharedSecret, 
        { name: "HKDF" }, 
        false, 
        ["deriveKey", "deriveBits"]
    );
    var derived_key = await crypto.subtle.deriveBits(
        { name: "HKDF", hash: "SHA-256", salt: new Uint8Array([]), info: new Uint8Array([]) }, 
        sharedSecretKey, 
        256
    );
    console.log("Derived key:\n", ab_to_b64(derived_key).replace(/(.{48})/g,'$1\n'))
}; 

function b64_to_ab(base64_string){
    return Uint8Array.from(atob(base64_string), c => c.charCodeAt(0));
}

function ab_to_b64(arrayBuffer){
    return btoa(String.fromCharCode.apply(null, new Uint8Array(arrayBuffer)));
}
Shared secret:
 xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF
Derived key:
 Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import serialization
import base64

def deriveKey():

  server_pkcs8 = b'''-----BEGIN PRIVATE KEY-----
MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBReGpDVmoVTzxNbJx6
aL4L9z1EdB91eonAmAw7mKDocLfCJITXZPUAmM46c6AipTmhZANiAAR3t96P0ZhU
jtW3rHkHpeGu4e+YT+ufMiMeanE/w8p+d9aCslvIbZyBBzeZ/266yqTUUoiYDzqv
Hb5q8rz7vEgr3DG4XfHYpCqfE2nttQGK3emHKGnvY239AteZkdwMpcs=
-----END PRIVATE KEY-----'''

  client_x509 = b'''-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEm0xeyy3nVnYpOpx/CV/FnlNEdWUZaqtB
AGf7flKxXEjmlSUjseYzCd566sLpNg56Gw6hcFx+rWTLGR4eDRWfmwlXhyUasuEg
mb0BQf8XJLBdvadb9eFx2CP1yjBsiy8e
-----END PUBLIC KEY-----'''

  client_public_key = serialization.load_pem_public_key(client_x509)
  server_private_key = serialization.load_pem_private_key(server_pkcs8, password=None)
  shared_secret = server_private_key.exchange(ec.ECDH(), client_public_key)
  print('Shared secret: ' + base64.b64encode(shared_secret).decode('utf8')) # Shared secret: xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF

  derived_key = HKDF(
    algorithm=hashes.SHA256(),
    length=32,
    salt=None,
    info=None,
  ).derive(shared_secret) 
  print('Derived key:   ' + base64.b64encode(derived_key).decode('utf8')) # Derived key:   Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=

deriveKey()
Shared secret: xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF
Derived key:   Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=
-----------------------
(async () => {
    await deriveKey();
})();

async function deriveKey() {

    //
    // Key import
    //
    var server_x509 =  `-----BEGIN PUBLIC KEY-----
                        MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEd7fej9GYVI7Vt6x5B6XhruHvmE/rnzIj
                        HmpxP8PKfnfWgrJbyG2cgQc3mf9uusqk1FKImA86rx2+avK8+7xIK9wxuF3x2KQq
                        nxNp7bUBit3phyhp72Nt/QLXmZHcDKXL
                        -----END PUBLIC KEY-----`;
    var client_pkcs8 = `-----BEGIN PRIVATE KEY-----
                        MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBjr4EGktNtx+3xErsC
                        MzldruzzfAEEO8Oth1/3b8sNfrqRsAgMnB/oVy024I+15wOhZANiAASbTF7LLedW
                        dik6nH8JX8WeU0R1ZRlqq0EAZ/t+UrFcSOaVJSOx5jMJ3nrqwuk2DnobDqFwXH6t
                        ZMsZHh4NFZ+bCVeHJRqy4SCZvQFB/xcksF29p1v14XHYI/XKMGyLLx4=
                        -----END PRIVATE KEY-----`;

    var client_private_key = b64_to_ab(client_pkcs8.replace('-----BEGIN PRIVATE KEY-----', '').replace('-----END PRIVATE KEY-----', ''));
    var server_public_key = b64_to_ab(server_x509.replace('-----BEGIN PUBLIC KEY-----', '').replace('-----END PUBLIC KEY-----', ''));
    var privateKey = await window.crypto.subtle.importKey( 
        'pkcs8', 
        client_private_key,
        { name: "ECDH", namedCurve: "P-384" },
        true, 
        ["deriveKey", "deriveBits"] 
    );
    var publicKey = await window.crypto.subtle.importKey(
        "spki", 
        server_public_key,
        { name: "ECDH", namedCurve: "P-384" },
        true, 
        [] 
    );
    
    //
    // Determine shared secret
    //
    var sharedSecret = await window.crypto.subtle.deriveBits(
        { name: "ECDH", namedCurve: "P-384", public: publicKey },
        privateKey, 
        384 
    );
    console.log("Shared secret:\n", ab_to_b64(sharedSecret).replace(/(.{48})/g,'$1\n'));
    
    //
    // Derive key from shared secret via HKDF
    //
    var sharedSecretKey = await window.crypto.subtle.importKey(
        "raw", 
        sharedSecret, 
        { name: "HKDF" }, 
        false, 
        ["deriveKey", "deriveBits"]
    );
    var derived_key = await crypto.subtle.deriveBits(
        { name: "HKDF", hash: "SHA-256", salt: new Uint8Array([]), info: new Uint8Array([]) }, 
        sharedSecretKey, 
        256
    );
    console.log("Derived key:\n", ab_to_b64(derived_key).replace(/(.{48})/g,'$1\n'))
}; 

function b64_to_ab(base64_string){
    return Uint8Array.from(atob(base64_string), c => c.charCodeAt(0));
}

function ab_to_b64(arrayBuffer){
    return btoa(String.fromCharCode.apply(null, new Uint8Array(arrayBuffer)));
}
Shared secret:
 xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF
Derived key:
 Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import serialization
import base64

def deriveKey():

  server_pkcs8 = b'''-----BEGIN PRIVATE KEY-----
MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBReGpDVmoVTzxNbJx6
aL4L9z1EdB91eonAmAw7mKDocLfCJITXZPUAmM46c6AipTmhZANiAAR3t96P0ZhU
jtW3rHkHpeGu4e+YT+ufMiMeanE/w8p+d9aCslvIbZyBBzeZ/266yqTUUoiYDzqv
Hb5q8rz7vEgr3DG4XfHYpCqfE2nttQGK3emHKGnvY239AteZkdwMpcs=
-----END PRIVATE KEY-----'''

  client_x509 = b'''-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEm0xeyy3nVnYpOpx/CV/FnlNEdWUZaqtB
AGf7flKxXEjmlSUjseYzCd566sLpNg56Gw6hcFx+rWTLGR4eDRWfmwlXhyUasuEg
mb0BQf8XJLBdvadb9eFx2CP1yjBsiy8e
-----END PUBLIC KEY-----'''

  client_public_key = serialization.load_pem_public_key(client_x509)
  server_private_key = serialization.load_pem_private_key(server_pkcs8, password=None)
  shared_secret = server_private_key.exchange(ec.ECDH(), client_public_key)
  print('Shared secret: ' + base64.b64encode(shared_secret).decode('utf8')) # Shared secret: xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF

  derived_key = HKDF(
    algorithm=hashes.SHA256(),
    length=32,
    salt=None,
    info=None,
  ).derive(shared_secret) 
  print('Derived key:   ' + base64.b64encode(derived_key).decode('utf8')) # Derived key:   Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=

deriveKey()
Shared secret: xbU6oDHMTYj3O71liM5KEJof3/0P4HlHJ28k7qtdqU/36llCizIlOWXtj8v+IngF
Derived key:   Yh0FkhqrT9XDQqIiSrGv5YmBjCSj9jhR5fF6HusbN1Q=

Issues running OSMnx on conda

copy iconCopydownload iconDownload
pip install -U networkx
-----------------------
conda remove -c conda-forge networkx
conda install -c conda-forge networkx=2.5
conda install -c conda-forge osmnx

Which implementation is better for code readability?

copy iconCopydownload iconDownload
#include <ranges>
#include <vector>
#include <algorithm>

struct Memory {
    using System = std::vector<int>;
    using Group = std::vector<System>;
    std::vector<Group> TrackingZonesShowingLogicalParcels;
};

 void foo(Memory* m_pMemory)
 {
     // Create a view representing all of the zones
     auto all_zones = m_pMemory->TrackingZonesShowingLogicalParcels | std::views::join | std::views::join;

     // use that view as if it was a container.
     std::fill(all_zones.begin(), all_zones.end(), 1);
 }

calculate the percentage increase or decrease in a list of values in java

copy iconCopydownload iconDownload
static double checkMov(int arr[], int n) {
        double var1, var2, result = 0;

        var1 = arr[0];
        var2 = arr[1];

        // Calculate successive change of 1st 2 change
        result = var1 + var2 + ((var1 * var2) / 100);

        // Calculate successive change
        // for rest of the value
        for (int i = 2; i < n; i++) {
            result = result + arr[i] + ((result * arr[i]) / 100);
        }

        return result; // This returns 7.0955315970304415E44 
    }

How can I attach the corresponding HTML ending to each piece of scraped text?

copy iconCopydownload iconDownload
from bs4 import BeautifulSoup, NavigableString
import requests
import re
import pandas as pd
import itertools

#DEFINING TWO FUNCTIONS TO HELP FINDING ONLY THE WANTED CITATIONS:
MATCH_ALL = r'.*'


def like(string):
    """
    Return a compiled regular expression that matches the given
    string with any prefix and postfix, e.g. if string = "hello",
    the returned regex matches r".*hello.*"
    """
    string_ = string
    if not isinstance(string_, str):
        string_ = str(string_)
    regex = MATCH_ALL + re.escape(string_) + MATCH_ALL
    return re.compile(regex, flags=re.DOTALL)


def find_by_text(soup, text, tag, **kwargs):
    """
    Find the tag in soup that matches all provided kwargs, and contains the
    text.

    If no match is found, raise ValueError.
    """
    elements = soup.find_all(tag, **kwargs)
    matches = []
    for element in elements:
        if element.find(text=like(text)):
            matches.append(element)
    if len(matches) == 0:
        raise ValueError("No matching citations were found")
    else:
        return matches

#DEFINING URL LIST:

base_URL = "https://sis1.host.cs.st-andrews.ac.uk/GAP/"
mrn = ["MR4044696", "MR2900886", "MR3169623", "MR4180136"]
url_dict = {}

for i in range(len(mrn)):
    url = (base_URL + mrn[i] + '.html')
    url_dict[url] = mrn[i]
    
print(url_dict)

all_content = []
all_matches = []

#THIS IS THE LOOP WHICH ITERATES THROUGH AND GATHER THE RESULTS:

for url, mrn in url_dict.items():
    page = requests.get(url)
    soup = BeautifulSoup(page.content, 'html.parser')
    match = (find_by_text(soup, 'GAP', 'li'))[0]
    match.insert(0, NavigableString("%s, " %mrn))
    all_matches.append(match)
print(all_matches)
[<li>MR4044696, 
  The GAP Group, GAP – groups, algorithms and programming, version 4.10, Available from http://www.gap-system.org, 2018. 

</li>, <li>MR2900886, 
  The GAP Group, <span class="MathTeX">$GAP$</span><script type="math/tex">GAP</script> groups, algorithms, and programming, version 4.4.12 (2008), http://www.gap-system.org. 

</li>, <li>MR3169623, 
  Distler, A., Mitchell, J. D. (2011). <span class="it">Smallsemi - A Library of Small Semigroups.</span> http://blacklistshorteners.com/, Oct A GAP 4 package [5], Version 0.6.4. 

</li>, <li>MR4180136, 
  The GAP Group, 2019. GAP – Groups, Algorithms, and Programming, Version 4.10.1; https://www.gap-system.org. 

</li>]
-----------------------
from bs4 import BeautifulSoup, NavigableString
import requests
import re
import pandas as pd
import itertools

#DEFINING TWO FUNCTIONS TO HELP FINDING ONLY THE WANTED CITATIONS:
MATCH_ALL = r'.*'


def like(string):
    """
    Return a compiled regular expression that matches the given
    string with any prefix and postfix, e.g. if string = "hello",
    the returned regex matches r".*hello.*"
    """
    string_ = string
    if not isinstance(string_, str):
        string_ = str(string_)
    regex = MATCH_ALL + re.escape(string_) + MATCH_ALL
    return re.compile(regex, flags=re.DOTALL)


def find_by_text(soup, text, tag, **kwargs):
    """
    Find the tag in soup that matches all provided kwargs, and contains the
    text.

    If no match is found, raise ValueError.
    """
    elements = soup.find_all(tag, **kwargs)
    matches = []
    for element in elements:
        if element.find(text=like(text)):
            matches.append(element)
    if len(matches) == 0:
        raise ValueError("No matching citations were found")
    else:
        return matches

#DEFINING URL LIST:

base_URL = "https://sis1.host.cs.st-andrews.ac.uk/GAP/"
mrn = ["MR4044696", "MR2900886", "MR3169623", "MR4180136"]
url_dict = {}

for i in range(len(mrn)):
    url = (base_URL + mrn[i] + '.html')
    url_dict[url] = mrn[i]
    
print(url_dict)

all_content = []
all_matches = []

#THIS IS THE LOOP WHICH ITERATES THROUGH AND GATHER THE RESULTS:

for url, mrn in url_dict.items():
    page = requests.get(url)
    soup = BeautifulSoup(page.content, 'html.parser')
    match = (find_by_text(soup, 'GAP', 'li'))[0]
    match.insert(0, NavigableString("%s, " %mrn))
    all_matches.append(match)
print(all_matches)
[<li>MR4044696, 
  The GAP Group, GAP – groups, algorithms and programming, version 4.10, Available from http://www.gap-system.org, 2018. 

</li>, <li>MR2900886, 
  The GAP Group, <span class="MathTeX">$GAP$</span><script type="math/tex">GAP</script> groups, algorithms, and programming, version 4.4.12 (2008), http://www.gap-system.org. 

</li>, <li>MR3169623, 
  Distler, A., Mitchell, J. D. (2011). <span class="it">Smallsemi - A Library of Small Semigroups.</span> http://blacklistshorteners.com/, Oct A GAP 4 package [5], Version 0.6.4. 

</li>, <li>MR4180136, 
  The GAP Group, 2019. GAP – Groups, Algorithms, and Programming, Version 4.10.1; https://www.gap-system.org. 

</li>]

Resource-efficient finding ONE possible combinations of numbers to reach a given sum

copy iconCopydownload iconDownload
def find_one_subset_sum(inp, sum)
  candidates = inp.filter { |i| i < sum } # don't do this optimize if inp contain negative numbers
  sum_combi_hash = Hash.new do |hash, key|
    if key > 0
      hash[key] = {candidates[key] => [candidates[key]]}
      if hash[key - 1][sum].nil?
        hash[key - 1].each do |s, c|
          hash[key][s] = c
          new_s = s + candidates[key]
          next unless hash[key][new_s].nil?
          hash[key][new_s] = c + [candidates[key]]
          break if new_s == sum
        end
      else
        hash[key][sum] = hash[key - 1][sum]
      end
      hash[key]
    else
      hash[key] = {candidates.first => [candidates.first]}
    end
  end
  sum_combi_hash[candidates.size - 1][sum]
end
-----------------------
def subset_sum((first, *rest), target)
  return (first == target ? [first] : nil) if rest.empty?
  rv = subset_sum(rest, target)
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
arr = [1,2,3,4,5,6]
subset_sum([1,2,3,4,5,6], 15)
  #=> [4,5,6]
arr1 = arr.shuffle
  #=> [5, 3, 6, 4, 2, 1]
subset_sum(arr1, 15)
  #=> [3, 6, 4, 2]
subset_sum([1, 1, 1, 1, 1, 2, 2, 5, 5, 5, 5, 5, 10], 6)
  #=> [1, 5]
def subset_sum((first, *rest), target)
  puts "first = #{first}, rest = #{rest}, target = #{target}"
  if rest.empty?
    puts "  rest is empty, return #{first == target ? [first] : nil}"        
    return first == target ? [first] : nil
  end 
  rv = subset_sum(rest, target)
  puts "  rv #{first} not used"
  puts "  return #{rv}" unless rv.nil?
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
subset_sum(arr1, 15)
first = 5, rest = [3, 6, 4, 2, 1], target = 15
first = 3, rest = [6, 4, 2, 1], target = 15
first = 6, rest = [4, 2, 1], target = 15
first = 4, rest = [2, 1], target = 15
first = 2, rest = [1], target = 15
first = 1, rest = [], target = 15
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 13
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 11
first = 1, rest = [], target = 11
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 9
first = 2, rest = [1], target = 9
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 7
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 5
first = 1, rest = [], target = 5
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 3
  rest is empty, return 
  rv 3 not used
first = 6, rest = [4, 2, 1], target = 12
first = 4, rest = [2, 1], target = 12
first = 2, rest = [1], target = 12
first = 1, rest = [], target = 12
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 10
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 8
first = 1, rest = [], target = 8
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 6
first = 2, rest = [1], target = 6
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 4
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 2
first = 1, rest = [], target = 2
  rest is empty, return 
  rv 2 not used
  rv 5 not used
  return [3, 6, 4, 2]
#=> [3, 6, 4, 2]
a4 = [1,2,3,4]
all = (1..4).each_with_object([]) do |n,a|
  a4.combination(n).each { |aa| a << aa.sum }
end
  #=> [1, 2, 3, 4, 3, 4, 5, 5, 6, 7, 6, 7, 8, 9, 10]
all_uniq = all.uniq
  #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
all.uniq.select { |a| a <= 5 }
  #=> [1, 2, 3, 4, 5]
-----------------------
def subset_sum((first, *rest), target)
  return (first == target ? [first] : nil) if rest.empty?
  rv = subset_sum(rest, target)
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
arr = [1,2,3,4,5,6]
subset_sum([1,2,3,4,5,6], 15)
  #=> [4,5,6]
arr1 = arr.shuffle
  #=> [5, 3, 6, 4, 2, 1]
subset_sum(arr1, 15)
  #=> [3, 6, 4, 2]
subset_sum([1, 1, 1, 1, 1, 2, 2, 5, 5, 5, 5, 5, 10], 6)
  #=> [1, 5]
def subset_sum((first, *rest), target)
  puts "first = #{first}, rest = #{rest}, target = #{target}"
  if rest.empty?
    puts "  rest is empty, return #{first == target ? [first] : nil}"        
    return first == target ? [first] : nil
  end 
  rv = subset_sum(rest, target)
  puts "  rv #{first} not used"
  puts "  return #{rv}" unless rv.nil?
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
subset_sum(arr1, 15)
first = 5, rest = [3, 6, 4, 2, 1], target = 15
first = 3, rest = [6, 4, 2, 1], target = 15
first = 6, rest = [4, 2, 1], target = 15
first = 4, rest = [2, 1], target = 15
first = 2, rest = [1], target = 15
first = 1, rest = [], target = 15
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 13
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 11
first = 1, rest = [], target = 11
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 9
first = 2, rest = [1], target = 9
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 7
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 5
first = 1, rest = [], target = 5
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 3
  rest is empty, return 
  rv 3 not used
first = 6, rest = [4, 2, 1], target = 12
first = 4, rest = [2, 1], target = 12
first = 2, rest = [1], target = 12
first = 1, rest = [], target = 12
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 10
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 8
first = 1, rest = [], target = 8
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 6
first = 2, rest = [1], target = 6
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 4
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 2
first = 1, rest = [], target = 2
  rest is empty, return 
  rv 2 not used
  rv 5 not used
  return [3, 6, 4, 2]
#=> [3, 6, 4, 2]
a4 = [1,2,3,4]
all = (1..4).each_with_object([]) do |n,a|
  a4.combination(n).each { |aa| a << aa.sum }
end
  #=> [1, 2, 3, 4, 3, 4, 5, 5, 6, 7, 6, 7, 8, 9, 10]
all_uniq = all.uniq
  #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
all.uniq.select { |a| a <= 5 }
  #=> [1, 2, 3, 4, 5]
-----------------------
def subset_sum((first, *rest), target)
  return (first == target ? [first] : nil) if rest.empty?
  rv = subset_sum(rest, target)
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
arr = [1,2,3,4,5,6]
subset_sum([1,2,3,4,5,6], 15)
  #=> [4,5,6]
arr1 = arr.shuffle
  #=> [5, 3, 6, 4, 2, 1]
subset_sum(arr1, 15)
  #=> [3, 6, 4, 2]
subset_sum([1, 1, 1, 1, 1, 2, 2, 5, 5, 5, 5, 5, 10], 6)
  #=> [1, 5]
def subset_sum((first, *rest), target)
  puts "first = #{first}, rest = #{rest}, target = #{target}"
  if rest.empty?
    puts "  rest is empty, return #{first == target ? [first] : nil}"        
    return first == target ? [first] : nil
  end 
  rv = subset_sum(rest, target)
  puts "  rv #{first} not used"
  puts "  return #{rv}" unless rv.nil?
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
subset_sum(arr1, 15)
first = 5, rest = [3, 6, 4, 2, 1], target = 15
first = 3, rest = [6, 4, 2, 1], target = 15
first = 6, rest = [4, 2, 1], target = 15
first = 4, rest = [2, 1], target = 15
first = 2, rest = [1], target = 15
first = 1, rest = [], target = 15
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 13
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 11
first = 1, rest = [], target = 11
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 9
first = 2, rest = [1], target = 9
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 7
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 5
first = 1, rest = [], target = 5
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 3
  rest is empty, return 
  rv 3 not used
first = 6, rest = [4, 2, 1], target = 12
first = 4, rest = [2, 1], target = 12
first = 2, rest = [1], target = 12
first = 1, rest = [], target = 12
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 10
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 8
first = 1, rest = [], target = 8
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 6
first = 2, rest = [1], target = 6
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 4
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 2
first = 1, rest = [], target = 2
  rest is empty, return 
  rv 2 not used
  rv 5 not used
  return [3, 6, 4, 2]
#=> [3, 6, 4, 2]
a4 = [1,2,3,4]
all = (1..4).each_with_object([]) do |n,a|
  a4.combination(n).each { |aa| a << aa.sum }
end
  #=> [1, 2, 3, 4, 3, 4, 5, 5, 6, 7, 6, 7, 8, 9, 10]
all_uniq = all.uniq
  #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
all.uniq.select { |a| a <= 5 }
  #=> [1, 2, 3, 4, 5]
-----------------------
def subset_sum((first, *rest), target)
  return (first == target ? [first] : nil) if rest.empty?
  rv = subset_sum(rest, target)
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
arr = [1,2,3,4,5,6]
subset_sum([1,2,3,4,5,6], 15)
  #=> [4,5,6]
arr1 = arr.shuffle
  #=> [5, 3, 6, 4, 2, 1]
subset_sum(arr1, 15)
  #=> [3, 6, 4, 2]
subset_sum([1, 1, 1, 1, 1, 2, 2, 5, 5, 5, 5, 5, 10], 6)
  #=> [1, 5]
def subset_sum((first, *rest), target)
  puts "first = #{first}, rest = #{rest}, target = #{target}"
  if rest.empty?
    puts "  rest is empty, return #{first == target ? [first] : nil}"        
    return first == target ? [first] : nil
  end 
  rv = subset_sum(rest, target)
  puts "  rv #{first} not used"
  puts "  return #{rv}" unless rv.nil?
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
subset_sum(arr1, 15)
first = 5, rest = [3, 6, 4, 2, 1], target = 15
first = 3, rest = [6, 4, 2, 1], target = 15
first = 6, rest = [4, 2, 1], target = 15
first = 4, rest = [2, 1], target = 15
first = 2, rest = [1], target = 15
first = 1, rest = [], target = 15
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 13
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 11
first = 1, rest = [], target = 11
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 9
first = 2, rest = [1], target = 9
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 7
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 5
first = 1, rest = [], target = 5
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 3
  rest is empty, return 
  rv 3 not used
first = 6, rest = [4, 2, 1], target = 12
first = 4, rest = [2, 1], target = 12
first = 2, rest = [1], target = 12
first = 1, rest = [], target = 12
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 10
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 8
first = 1, rest = [], target = 8
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 6
first = 2, rest = [1], target = 6
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 4
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 2
first = 1, rest = [], target = 2
  rest is empty, return 
  rv 2 not used
  rv 5 not used
  return [3, 6, 4, 2]
#=> [3, 6, 4, 2]
a4 = [1,2,3,4]
all = (1..4).each_with_object([]) do |n,a|
  a4.combination(n).each { |aa| a << aa.sum }
end
  #=> [1, 2, 3, 4, 3, 4, 5, 5, 6, 7, 6, 7, 8, 9, 10]
all_uniq = all.uniq
  #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
all.uniq.select { |a| a <= 5 }
  #=> [1, 2, 3, 4, 5]
-----------------------
def subset_sum((first, *rest), target)
  return (first == target ? [first] : nil) if rest.empty?
  rv = subset_sum(rest, target)
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
arr = [1,2,3,4,5,6]
subset_sum([1,2,3,4,5,6], 15)
  #=> [4,5,6]
arr1 = arr.shuffle
  #=> [5, 3, 6, 4, 2, 1]
subset_sum(arr1, 15)
  #=> [3, 6, 4, 2]
subset_sum([1, 1, 1, 1, 1, 2, 2, 5, 5, 5, 5, 5, 10], 6)
  #=> [1, 5]
def subset_sum((first, *rest), target)
  puts "first = #{first}, rest = #{rest}, target = #{target}"
  if rest.empty?
    puts "  rest is empty, return #{first == target ? [first] : nil}"        
    return first == target ? [first] : nil
  end 
  rv = subset_sum(rest, target)
  puts "  rv #{first} not used"
  puts "  return #{rv}" unless rv.nil?
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
subset_sum(arr1, 15)
first = 5, rest = [3, 6, 4, 2, 1], target = 15
first = 3, rest = [6, 4, 2, 1], target = 15
first = 6, rest = [4, 2, 1], target = 15
first = 4, rest = [2, 1], target = 15
first = 2, rest = [1], target = 15
first = 1, rest = [], target = 15
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 13
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 11
first = 1, rest = [], target = 11
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 9
first = 2, rest = [1], target = 9
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 7
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 5
first = 1, rest = [], target = 5
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 3
  rest is empty, return 
  rv 3 not used
first = 6, rest = [4, 2, 1], target = 12
first = 4, rest = [2, 1], target = 12
first = 2, rest = [1], target = 12
first = 1, rest = [], target = 12
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 10
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 8
first = 1, rest = [], target = 8
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 6
first = 2, rest = [1], target = 6
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 4
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 2
first = 1, rest = [], target = 2
  rest is empty, return 
  rv 2 not used
  rv 5 not used
  return [3, 6, 4, 2]
#=> [3, 6, 4, 2]
a4 = [1,2,3,4]
all = (1..4).each_with_object([]) do |n,a|
  a4.combination(n).each { |aa| a << aa.sum }
end
  #=> [1, 2, 3, 4, 3, 4, 5, 5, 6, 7, 6, 7, 8, 9, 10]
all_uniq = all.uniq
  #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
all.uniq.select { |a| a <= 5 }
  #=> [1, 2, 3, 4, 5]
-----------------------
def subset_sum((first, *rest), target)
  return (first == target ? [first] : nil) if rest.empty?
  rv = subset_sum(rest, target)
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
arr = [1,2,3,4,5,6]
subset_sum([1,2,3,4,5,6], 15)
  #=> [4,5,6]
arr1 = arr.shuffle
  #=> [5, 3, 6, 4, 2, 1]
subset_sum(arr1, 15)
  #=> [3, 6, 4, 2]
subset_sum([1, 1, 1, 1, 1, 2, 2, 5, 5, 5, 5, 5, 10], 6)
  #=> [1, 5]
def subset_sum((first, *rest), target)
  puts "first = #{first}, rest = #{rest}, target = #{target}"
  if rest.empty?
    puts "  rest is empty, return #{first == target ? [first] : nil}"        
    return first == target ? [first] : nil
  end 
  rv = subset_sum(rest, target)
  puts "  rv #{first} not used"
  puts "  return #{rv}" unless rv.nil?
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
subset_sum(arr1, 15)
first = 5, rest = [3, 6, 4, 2, 1], target = 15
first = 3, rest = [6, 4, 2, 1], target = 15
first = 6, rest = [4, 2, 1], target = 15
first = 4, rest = [2, 1], target = 15
first = 2, rest = [1], target = 15
first = 1, rest = [], target = 15
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 13
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 11
first = 1, rest = [], target = 11
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 9
first = 2, rest = [1], target = 9
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 7
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 5
first = 1, rest = [], target = 5
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 3
  rest is empty, return 
  rv 3 not used
first = 6, rest = [4, 2, 1], target = 12
first = 4, rest = [2, 1], target = 12
first = 2, rest = [1], target = 12
first = 1, rest = [], target = 12
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 10
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 8
first = 1, rest = [], target = 8
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 6
first = 2, rest = [1], target = 6
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 4
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 2
first = 1, rest = [], target = 2
  rest is empty, return 
  rv 2 not used
  rv 5 not used
  return [3, 6, 4, 2]
#=> [3, 6, 4, 2]
a4 = [1,2,3,4]
all = (1..4).each_with_object([]) do |n,a|
  a4.combination(n).each { |aa| a << aa.sum }
end
  #=> [1, 2, 3, 4, 3, 4, 5, 5, 6, 7, 6, 7, 8, 9, 10]
all_uniq = all.uniq
  #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
all.uniq.select { |a| a <= 5 }
  #=> [1, 2, 3, 4, 5]
-----------------------
def subset_sum((first, *rest), target)
  return (first == target ? [first] : nil) if rest.empty?
  rv = subset_sum(rest, target)
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
arr = [1,2,3,4,5,6]
subset_sum([1,2,3,4,5,6], 15)
  #=> [4,5,6]
arr1 = arr.shuffle
  #=> [5, 3, 6, 4, 2, 1]
subset_sum(arr1, 15)
  #=> [3, 6, 4, 2]
subset_sum([1, 1, 1, 1, 1, 2, 2, 5, 5, 5, 5, 5, 10], 6)
  #=> [1, 5]
def subset_sum((first, *rest), target)
  puts "first = #{first}, rest = #{rest}, target = #{target}"
  if rest.empty?
    puts "  rest is empty, return #{first == target ? [first] : nil}"        
    return first == target ? [first] : nil
  end 
  rv = subset_sum(rest, target)
  puts "  rv #{first} not used"
  puts "  return #{rv}" unless rv.nil?
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
subset_sum(arr1, 15)
first = 5, rest = [3, 6, 4, 2, 1], target = 15
first = 3, rest = [6, 4, 2, 1], target = 15
first = 6, rest = [4, 2, 1], target = 15
first = 4, rest = [2, 1], target = 15
first = 2, rest = [1], target = 15
first = 1, rest = [], target = 15
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 13
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 11
first = 1, rest = [], target = 11
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 9
first = 2, rest = [1], target = 9
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 7
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 5
first = 1, rest = [], target = 5
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 3
  rest is empty, return 
  rv 3 not used
first = 6, rest = [4, 2, 1], target = 12
first = 4, rest = [2, 1], target = 12
first = 2, rest = [1], target = 12
first = 1, rest = [], target = 12
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 10
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 8
first = 1, rest = [], target = 8
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 6
first = 2, rest = [1], target = 6
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 4
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 2
first = 1, rest = [], target = 2
  rest is empty, return 
  rv 2 not used
  rv 5 not used
  return [3, 6, 4, 2]
#=> [3, 6, 4, 2]
a4 = [1,2,3,4]
all = (1..4).each_with_object([]) do |n,a|
  a4.combination(n).each { |aa| a << aa.sum }
end
  #=> [1, 2, 3, 4, 3, 4, 5, 5, 6, 7, 6, 7, 8, 9, 10]
all_uniq = all.uniq
  #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
all.uniq.select { |a| a <= 5 }
  #=> [1, 2, 3, 4, 5]
-----------------------
def subset_sum((first, *rest), target)
  return (first == target ? [first] : nil) if rest.empty?
  rv = subset_sum(rest, target)
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
arr = [1,2,3,4,5,6]
subset_sum([1,2,3,4,5,6], 15)
  #=> [4,5,6]
arr1 = arr.shuffle
  #=> [5, 3, 6, 4, 2, 1]
subset_sum(arr1, 15)
  #=> [3, 6, 4, 2]
subset_sum([1, 1, 1, 1, 1, 2, 2, 5, 5, 5, 5, 5, 10], 6)
  #=> [1, 5]
def subset_sum((first, *rest), target)
  puts "first = #{first}, rest = #{rest}, target = #{target}"
  if rest.empty?
    puts "  rest is empty, return #{first == target ? [first] : nil}"        
    return first == target ? [first] : nil
  end 
  rv = subset_sum(rest, target)
  puts "  rv #{first} not used"
  puts "  return #{rv}" unless rv.nil?
  return rv unless rv.nil?
  return nil if first > target
  return [first] if first == target
  rv = subset_sum(rest, target-first)
  rv.nil? ? nil : [first, *rv]
end
subset_sum(arr1, 15)
first = 5, rest = [3, 6, 4, 2, 1], target = 15
first = 3, rest = [6, 4, 2, 1], target = 15
first = 6, rest = [4, 2, 1], target = 15
first = 4, rest = [2, 1], target = 15
first = 2, rest = [1], target = 15
first = 1, rest = [], target = 15
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 13
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 11
first = 1, rest = [], target = 11
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 9
first = 2, rest = [1], target = 9
first = 1, rest = [], target = 9
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 7
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 5
first = 1, rest = [], target = 5
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 3
  rest is empty, return 
  rv 3 not used
first = 6, rest = [4, 2, 1], target = 12
first = 4, rest = [2, 1], target = 12
first = 2, rest = [1], target = 12
first = 1, rest = [], target = 12
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 10
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 8
first = 1, rest = [], target = 8
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 6 not used
first = 4, rest = [2, 1], target = 6
first = 2, rest = [1], target = 6
first = 1, rest = [], target = 6
  rest is empty, return 
  rv 2 not used
first = 1, rest = [], target = 4
  rest is empty, return 
  rv 4 not used
first = 2, rest = [1], target = 2
first = 1, rest = [], target = 2
  rest is empty, return 
  rv 2 not used
  rv 5 not used
  return [3, 6, 4, 2]
#=> [3, 6, 4, 2]
a4 = [1,2,3,4]
all = (1..4).each_with_object([]) do |n,a|
  a4.combination(n).each { |aa| a << aa.sum }
end
  #=> [1, 2, 3, 4, 3, 4, 5, 5, 6, 7, 6, 7, 8, 9, 10]
all_uniq = all.uniq
  #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
all.uniq.select { |a| a <= 5 }
  #=> [1, 2, 3, 4, 5]

Community Discussions

Trending Discussions on Algorithms
  • Apache Beam SIGKILL
  • Python: Converting a list of strings into pandas data-frame with two columns a and b, corresponding to odd and even strings respectively
  • Algorithm calculations on perfromance and size
  • How can I limit and sort on document ID in firestore?
  • Encrypt in JS front end and decrypt in python backend using AES GCM
  • Nodejs Restify - Valide Jwt id_token with jwks
  • How to write an efficient algorithm that generates a list of unique random numbers?
  • Web cryptography implement HKDF for the output of ECDH
  • &quot; samples: %r&quot; % [int(l) for l in lengths]) ValueError: Found input variables with inconsistent numbers of samples: [219870, 0, 0]
  • Issues running OSMnx on conda
Trending Discussions on Algorithms

QUESTION

Apache Beam SIGKILL

Asked 2021-Jun-15 at 13:51

The Question

How do I best execute memory-intensive pipelines in Apache Beam?

Background

I've written a pipeline that takes the Naemura Bird dataset and converts the images and annotations to TF Records with TF Examples of the required format for the TF object detection API.

I tested the pipeline using DirectRunner with a small subset of images (4 or 5) and it worked fine.

The Problem

When running the pipeline with a bigger data set (day 1 of 3, ~21GB) it crashes after a while with a non-descriptive SIGKILL. I do see a memory peak before the crash and assume that the process is killed because of a too high memory load.

I ran the pipeline through strace. These are the last lines in the trace:

[pid 53702] 10:00:09.105069 poll([{fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}, {fd=17, events=POLLIN}, {fd=18, events=POLLIN}, {fd=19, events=POLLIN}, {fd=20, events=POLLIN}], 11, 100) = 0 (Timeout)
[pid 53702] 10:00:09.205826 poll([{fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}, {fd=17, events=POLLIN}, {fd=18, events=POLLIN}, {fd=19, events=POLLIN}, {fd=20, events=POLLIN}], 11, 100 <unfinished ...>
[pid 53534] 10:00:09.259806 mmap(NULL, 63082496, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3aa43d7000
[pid 53694] 10:00:09.297140 <... clock_nanosleep resumed>NULL) = 0
[pid 53694] 10:00:09.297273 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=200000000},  <unfinished ...>
[pid 53702] 10:00:09.306409 <... poll resumed>) = 0 (Timeout)
[pid 53702] 10:00:09.306478 poll([{fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}, {fd=17, events=POLLIN}, {fd=18, events=POLLIN}, {fd=19, events=POLLIN}, {fd=20, events=POLLIN}], 11, 100) = 0 (Timeout)
[pid 53702] 10:00:09.406866 poll([{fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}, {fd=17, events=POLLIN}, {fd=18, events=POLLIN}, {fd=19, events=POLLIN}, {fd=20, events=POLLIN}], 11, 100 <unfinished ...>
[pid 53710] 10:03:55.844910 <... futex resumed>) = ?
[pid 53709] 10:03:57.797618 <... futex resumed>) = ?
[pid 53708] 10:03:57.797737 <... futex resumed>) = ?
[pid 53707] 10:03:57.797793 <... futex resumed>) = ?
[pid 53706] 10:03:57.797847 <... futex resumed>) = ?
[pid 53705] 10:03:57.797896 <... futex resumed>) = ?
[pid 53704] 10:03:57.797983 <... futex resumed>) = ?
[pid 53703] 10:03:57.798035 <... futex resumed>) = ?
[pid 53702] 10:03:57.798085 +++ killed by SIGKILL +++
[pid 53701] 10:03:57.798124 <... futex resumed>) = ?
[pid 53700] 10:03:57.798173 <... futex resumed>) = ?
[pid 53699] 10:03:57.798224 <... futex resumed>) = ?
[pid 53698] 10:03:57.798272 <... futex resumed>) = ?
[pid 53697] 10:03:57.798321 <... accept4 resumed> <unfinished ...>) = ?
[pid 53694] 10:03:57.798372 <... clock_nanosleep resumed> <unfinished ...>) = ?
[pid 53693] 10:03:57.798426 <... futex resumed>) = ?
[pid 53660] 10:03:57.798475 <... futex resumed>) = ?
[pid 53641] 10:03:57.798523 <... futex resumed>) = ?
[pid 53640] 10:03:57.798572 <... futex resumed>) = ?
[pid 53639] 10:03:57.798620 <... futex resumed>) = ?
[pid 53710] 10:03:57.798755 +++ killed by SIGKILL +++
[pid 53709] 10:03:57.798792 +++ killed by SIGKILL +++
[pid 53708] 10:03:57.798828 +++ killed by SIGKILL +++
[pid 53707] 10:03:57.798864 +++ killed by SIGKILL +++
[pid 53706] 10:03:57.798900 +++ killed by SIGKILL +++
[pid 53705] 10:03:57.798937 +++ killed by SIGKILL +++
[pid 53704] 10:03:57.798973 +++ killed by SIGKILL +++
[pid 53703] 10:03:57.799008 +++ killed by SIGKILL +++
[pid 53701] 10:03:57.799044 +++ killed by SIGKILL +++
[pid 53700] 10:03:57.799079 +++ killed by SIGKILL +++
[pid 53699] 10:03:57.799116 +++ killed by SIGKILL +++
[pid 53698] 10:03:57.799152 +++ killed by SIGKILL +++
[pid 53697] 10:03:57.799187 +++ killed by SIGKILL +++
[pid 53694] 10:03:57.799245 +++ killed by SIGKILL +++
[pid 53693] 10:03:57.799282 +++ killed by SIGKILL +++
[pid 53660] 10:03:57.799318 +++ killed by SIGKILL +++
[pid 53641] 10:03:57.799354 +++ killed by SIGKILL +++
[pid 53640] 10:03:57.799390 +++ killed by SIGKILL +++
[pid 53639] 10:03:57.910349 +++ killed by SIGKILL +++
10:03:57.910381 +++ killed by SIGKILL +++

ANSWER

Answered 2021-Jun-15 at 13:51

Multiple things could cause this behaviour, because the pipeline runs fine with less Data, analysing what has changed could lead us to a resolution.

Option 1 : clean your input data

The third line of the logs you provide might indicate that you're processing unclean data in your bigger pipeline mmap(NULL, could mean that | "Get Content" >> beam.Map(lambda x: x.read_utf8()) is trying to read a null value.

Is there an empty file somewhere ? Are your files utf8 encoded ?

Option 2 : use smaller files as input

I'm guessing using the fileio.ReadMatches() will try to load into memory the whole file, if your file is bigger than your memory, this could lead to errors. Can you split your data into smaller files ?

Option 3 : use a bigger infrastructure

If files are too big for your current machine with a DirectRunner you could try to use an on-demand infrastructure using another runner on the Cloud such as DataflowRunner

Source https://stackoverflow.com/questions/67684186

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install Algorithms

You can download it from GitHub.
You can use Algorithms like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the Algorithms component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

This repository is contribution friendly :smiley:. If you'd like to add or improve an algorithm, your contribution is welcome! Please be sure to checkout the Wiki for instructions.

DOWNLOAD this Library from

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Share this Page

share link
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

  • © 2022 Open Weaver Inc.