Popular New Releases in Python
youtube-dl
youtube-dl 2021.12.17
models
TensorFlow Official Models 2.7.1
thefuck
transformers
v4.18.0: Checkpoint sharding, vision models
flask
Popular Libraries in Python
by public-apis python
184682 MIT
A collective list of free APIs
by donnemartin python
143449 NOASSERTION
Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
by TheAlgorithms python
117097 MIT
All Algorithms implemented in Python
by jackfrued python
114192
Python - 100天从新手到大师
by ytdl-org python
108335 Unlicense
Command-line program to download videos from YouTube.com and other video sites
by vinta python
102379 NOASSERTION
A curated list of awesome Python frameworks, libraries, software and resources
by tensorflow python
73392 NOASSERTION
Models and examples built with TensorFlow
by nvbn python
65678 MIT
Magnificent app which corrects your previous console command.
by django python
63447 NOASSERTION
The Web framework for perfectionists with deadlines.
Trending New libraries in Python
by ultralytics python
25236 GPL-3.0
YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite
by yt-dlp python
22499 Unlicense
A youtube-dl fork with additional features and fixes
by babysor python
20425 NOASSERTION
🚀AI拟声: 5秒内克隆您的声音并生成任意语音内容 Clone a voice in 5 seconds to generate arbitrary speech in real-time
by beurtschipper python
19784 NOASSERTION
Recovers passwords from pixelized screenshots
by PaddlePaddle python
19581 Apache-2.0
Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)
by TencentARC python
17269 NOASSERTION
GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.
by github python
16816 CC-BY-4.0
Documentation for GitHub Copilot
by mingrammer python
16552 MIT
:art: Diagram as Code for prototyping cloud system architectures
by jina-ai python
14316 Apache-2.0
Cloud-native neural search framework for 𝙖𝙣𝙮 kind of data
Top Authors in Python
1
4429 Libraries
28
2
3923 Libraries
0
3
2232 Libraries
604
4
1263 Libraries
3038
5
723 Libraries
0
6
690 Libraries
1
7
628 Libraries
0
8
627 Libraries
23550
9
550 Libraries
33409
10
512 Libraries
20636
1
4429 Libraries
28
2
3923 Libraries
0
3
2232 Libraries
604
4
1263 Libraries
3038
5
723 Libraries
0
6
690 Libraries
1
7
628 Libraries
0
8
627 Libraries
23550
9
550 Libraries
33409
10
512 Libraries
20636
Trending Kits in Python
Python is an object-oriented programming language. Python can do anything almost any other language can do, at comparable speeds. This kit has some simple to use exercises in Python to help someone new to learn how to program in Python and get started with their journey.
For a detailed tutorial on installing & executing the solution as well as learning resources including training & certification opportunities, please visit the OpenWeaver Community
Python Repositories with Basic Example Exercises
Basic python CLI programs as examples. This list has programs useful for someone who is a beginner and also someone willing to go advance level.
Some Python Games for Practice
Support
If you need help using this kit, you may reach us at the OpenWeaver Community.
AI fake news detector helps detect fake news through binary classification methods. It helps build experiences by controlling the flow of disinformation. It's built on top of various powerful machine learning libraries. The tool works by training a neural network to spot fake articles based on their text content. When you run your own data through the tool, it gives you back a list of articles that it thinks are likely to be fake. You can then train the model further or decide if those results are acceptable or not. In addition to identifying fake news, this model can also be trained to identify real news. This allows you to compare the model's performance across different domains (e.g., politics vs. sports). The following installer and deployment instructions will walk you through the steps of creating an AI fake news detector by using fakenews-detection, jupyter, vscode, and pandas. We will use fake news detection libraries (having fully modifiable source code) to customize and build a simple classifier that can detect fake news articles. kandi kit provides you with a fully deployable AI Fake News Detector. Source code included so that you can customize it for your requirement.
With this kit, you can
1. Use a pre-trained model for detecting fake news.
2. Train the model on your custom dataset.
3. Expose the fake news detection as an API
Add-on on examples are also included as given below
1. Use web scraper to automatically make your training dataset.
2. Visualise training and prediction data for useful insights.
Instructions to Run
Follow the below instructions to run the solution.
1. Locate and open the FakeNewsDetection-starter.ipynb notebook from the Jupyter Notebook browser window.
2. Execute cells in the notebook by selecting Cell --> Run All from Menu bar
3. Once all the cells of the notebook are executed, the prediction result will be written to the file 'fake_news_test_output.csv'
Training with your dataset:
1. Add news articles to a csv file under a column name 'news_text'.
2. Add corresponding labels as 'real' or 'fake' denoting whether a news article is real or not.
3. You can refer to the file 'fake_news_train.csv' for an example.
4. Set the variable for training file in the notebook under Variables section.
Testing with your dataset:
1. Add news articles to a csv file under a column name 'news_text'.
2. You can refer to the file 'fake_news_test.csv' for an example.
3. Set the variable for testing file in the notebook under Variables section.
You can execute the cells of notebook by selecting Cell from the menu bar.
For any support, you can reach us at FAQ & Support
Libraries useful for this solution
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers. Jupyter Notebook is used for our development.
Exploratory Data Analysis
For extensive analysis and exploration of data, and to deal with arrays, these libraries are used. They are also used for performing scientific computation and data manipulation.
Text mining
Libraries in this group are used for analysis and processing of unstructured natural language. The data, as in its original form aren't used as it has to go through processing pipeline to become suitable for applying machine learning techniques and algorithms.
Machine Learning
Machine learning libraries and frameworks here are helpful in providing state-of-the-art solutions using Machine learning.
Data Visualization
The patterns and relationships are identified by representing data visually and below libraries are used for generating visual plots of the data.
Troubleshooting
1. If you encounter any error related to MS Visual C++, please install MS Visual Build tools
2.While running batch file, if you encounter Windows protection alert, select More info --> Run anyway.
3.During kit installer, if you encounter Windows security alert, click Allow.
4. If you encounter Memory Error, check if the available memory is sufficient and it is proportion to the size of the data being used. For our dataset, the minimum required memory is 8GB.
If your computer doesn't support standard commands from windows 10, you can follow the instructions below to finish the kit installation.
1. Click here to install python
2. Click here to download the repository
3. Extract the zip file and navigate to the directory 'fakenews-detection-main'
4. Open terminal in the extracted directory 'fakenews-detection-main'
5. Install dependencies by executing the command 'pip install -r requirements.txt'
6. Run the command ‘jupyter notebook’ and select the notebook ‘FakeNewsdetection-starter.ipynb’ on the browser window.
Support
For any support, you can reach us at FAQ & Support
Deepfake detection is identifying manipulated or synthetic media content using machine learning algorithms and computer vision techniques. It detects anomalies in facial and body movements, and other visual artifacts.
In this kit, we build a Deepfake Detection Engine using the popular Facenet_pytorch is a Python library that provides implementations of deep learning models for face recognition tasks. It includes pre-trained models such as
- MTCNN (Multi-Task Cascaded Convolutional Networks) for face detection and alignment, and
- InceptionResnetV1 for detecting whether an image is fake or real.
We use these two models to detect and recognize faces in images with high accuracy. The library is built on top of PyTorch, a popular open-source machine learning framework, and provides an easy-to-use API for face recognition tasks
Libraries used in this solution
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers.
Jupyter Notebook is used for our development.
Machine Learning
Machine learning libraries and frameworks here are helpful in providing state-of-the-art solutions using Machine learning
Kit Solution Source
API Integration
Support
For any support, you can reach us at OpenWeaver Community Support
Generative artificial intelligence (AI) describes algorithms that help in creating/generating new content, including audio, code, images, text and videos.
In this kit, we build a real-time Voice-to-Image Generator using the concept of Generative AI. It is carried out in two steps:
- Voice-to-text conversion - The speech is captured in real-time through the microphone and converted to text using state-of-the-art Opensource AI models from OpenAI and Whisper libraries.
- Text to Image Generation - The converted text is provided as input to the state-of-the-art Image Generation models like Dalle-2, and the image is thus generated.
Libraries used in this solution
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers.
Jupyter Notebook is used for our development.
Machine Learning
Machine learning libraries and frameworks here are helpful in providing state-of-the-art solutions using Machine learning
Kit Solution Source
UI App Integration
Support
For any support, you can reach us at OpenWeaver Community Support
This Predictive Analytics kit provides an analytical view of students’ performance in mathematics and predicts grades to be scored in the final test.
The key features of this solution are:
- Analysis of grades of students
- Visualisation of patterns
- Prediction of grade in the final test
For a detailed tutorial on installing & executing the solution as well as learning resources including training & certification opportunities, please visit the OpenWeaver Community
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers. Jupyter Notebook is used for our development.
Data Mining
Our solution integrates data from various sources, and we have used below libraries for exploring patterns in these data and understanding correlation between the features.
Data Visualisation
The patterns and relationships are identified by representing data visually and below libraries are used for that.
Machine learning
Below libraries and model collections helps to create the machine learning models for the core prediction of use case in our solution.
Support
If you need help using this kit, you may reach us at the OpenWeaver Community.
The use case of AI Course Recommender System is to provide personalized recommendation to the user based on their interest, course they can take and their current knowledge. This system will be able to recommend course based on user’s interest, current knowledge, analytical view of students’ performance in mathematics and recommends if a student can consider math subject for his/ her higher education. The recommended course will be based on the information of user’s profile, analysis of grades of students, visualization of patterns, prediction of grade in final test, and some rules that were set by their instructor. Using machine learning algorithms, we can train our model on a set of data and then predict the ratings for new items. This is all done in Python using numpy, pandas, matplotlib, scikit-learn and seaborn. kandi kit provides you with a fully deployable AI Course Recommender System. Source code included so that you can customize it for your requirement.
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers.
Data Mining
Our solution integrates data from various sources, and we have used below libraries for exploring patterns in these data and understanding correlation between the features.
Data Visualisation
The patterns and relationships are identified by representing data visually and below libraries are used.
Machine learning
Below libraries and model collections helps to create the machine learning models for the core prediction of use case in our solution.
Federated Learning can train machine learning models on data from different hospitals, banks and autonomous vehicles without sharing sensitive data. But how do you create a Federated learning application? The answer is the kandi 1-click solution kit for Credit-risk-federated-learning.
Certainly, Federated Learning can be applied in the credit risk scenario to improve credit risk models' accuracy without compromising customer data privacy.
Banks collect and centralize customer data to train their credit risk models in the traditional approach. However, this approach can be challenging due to regulatory compliance, data privacy, and security concerns. Federated Learning addresses these challenges by allowing banks to train their credit risk models on customer data without transferring it to a centralized location.
This fully editable source code builds your Credit risk federated learning in minutes. The entire solution is available as a package to download from the source code repository.
Federated Learning in credit risk scenarios can have several benefits, including:
- Improved accuracy: Federated Learning allows banks to train models on a larger and more diverse dataset, leading to better accuracy.
- Data privacy: Federated Learning ensures that sensitive customer data is kept private and secure, which is critical in the context of credit risk.
- Regulatory compliance: Federated Learning can help banks comply with regulations around data privacy and security.
Troubleshooting
- Install the Microsoft Visual C++ Redistributable for Visual Studio 2022 in case the kit doesn't successfully run on your windows system.
- In case, step 1 doesn't solve your issue, set up Microsoft build Tools.
For a detailed tutorial on installing & executing the solution as well as learning resources including training & certification opportunities, please visit the OpenWeaver Community
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers. Jupyter Notebook is used for our development.
Data Pre-processing
Numpy and Pandas are powerful tools for data preprocessing in machine learning. They provide tools for handling missing data, feature scaling, one-hot encoding, data normalization, and transformation.
These tools can help you to prepare your data for machine learning and improve the performance of your models.
Machine learning
Scikit-learn is a powerful and versatile machine learning library in Python that provides a wide range of tools and algorithms for building and training machine learning models. It is widely used in academia and industry for various machine learning applications.
Federated Learning Framework
Flower is an open-source framework for Federated Learning that provides tools and APIs to simplify the development and deployment of Federated Learning models. Flower is designed to make it easier for developers to implement Federated Learning in their applications by providing a flexible and scalable platform for building and training models.
Kit Solution Source
Support
If you need help using this kit, you may reach us at the OpenWeaver Community.
Large Language Models are foundation models that utilize deep learning in natural language processing and natural language generation tasks. Typically these models are trained on billions of parameters with a huge corpus of data.
GPT4all provides an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue. GPT4All is a 7B parameter LLM trained using a Low-Rank Adaptation (LoRA) method, yielding 430k post-processed instances, on a vast curated corpus of over 800k high-quality assistant interactions.
In this kit, we will use GPT4All to create a content generator, similar to ChatGPT, without the need for API keys and Internet to create content.
Libraries used in this solution
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers.
Jupyter Notebook is used for our development.
Machine Learning
Machine learning libraries and frameworks here are helpful in providing state-of-the-art solutions using Machine learning
Kit Solution Source
API Integration
Support
For any support, you can reach us at OpenWeaver Community Support
Angry Birds python game is a Finnish action-based media franchise. It is created by Rovio Entertainment.
The game series focuses on a flock of birds referred to by the same name who try to save their eggs from the green-colored pigs. Angry Birds game is written in python using Pygame and Pymunk which are open-source modules specifically intended to help you make games and other multimedia applications. Pygame can be used to load background images, sounds, and buttons, which makes the UI interactions more efficient. Pymunk is best when you need 2d physics from python which can be used for demos or simulations. Pymunk is basically built on top of the 2D physics library Chipmunk.
Troubleshooting
- While running batch file, if you encounter Windows protection alert, select More info --> Run anyway
- During kit installer, if you encounter Windows security alert, click Allow
For a detailed tutorial on installing & executing the solution as well as learning resources including training & certification opportunities, please visit the OpenWeaver Community
Development Environment
VSCode is used for development and debugging. VSCode is used to get a typical experience of IDE for developers.
Gaming Libraries
Pygame helps in providing computer graphics and audio libraries.
Pymunk is a easy-to-use pythonic 2d physics library that can be used whenever you need 2d rigid body physics from Python.
Support
If you need help using this kit, you may reach us at the OpenWeaver Community.
A Tower Defense Game between Humans and Aliens. Kill as much as aliens as you can to upgrade and discover new humans.
Development Environment
VSCode is used for development and debugging. VSCode is used to get a typical experience of IDE for developers.
Gaming Libraries
Pygame helps in providing computer graphics and audio libraries.
Pymunk is an easy-to-use pythonic 2d physics library that can be used whenever you need 2d rigid body physics from Python.
Troubleshooting
- While running batch file, if you encounter Windows protection alert, select More info --> Run anyway
- During kit installer, if you encounter Windows security alert, click Allow
Support
If you need help to use this kit, you can email us at kandi.support@openweaver.com or direct message us on Twitter Message @OpenWeaverInc.
Bias is prevalent in every aspect of our lives. Our brains are hardwired to categorize things we encounter in order to make sense of the complicated world around us. However, biases can cause us to form prejudices against others, which allows for egregious inequalities to form between different demographics.
While bias comes in many forms, bias words in writing is one form. Implicit bias in letter writing or evaluations negatively affects individuals at every stage of their career.
In this challenge, we are inviting to build a solution for detecting bias in writings such as letter of recommendations, Job Descriptions etc with respect to gender and race for promoting equity. You can choose any topic of your choice. The sample solution kit helps to detect gender bias.
Instruction to Run
Follow below instructions to run the solution.
- Locate and open the gender-bias.ipynb notebook from the Jupyter Notebook browser window.
- Execute cells in the notebook by selecting Cell --> Run All from Menu bar
For running it with your text,
- Open letterofRecW file from the location data/input from gender-bias.ipynb location.
- Update text in the letterofRecW file.
- Execute cells in the notebook by selecting Cell --> Run All from Menu bar.
- Output will be stored in a file gender-biased-words.txt in the location data/output. Output text is in json format. Output data format is: name - detector name. e.g. "Terms biased towards women" summary - summary of the detected bias flags - flag the detected bias words. e.g. "leader" You can additionally create your own detectors for race and dictionary dataset as well as other enhancements for additional score.
Troubleshooting
- While running batch file, if you encounter Windows protection alert, select More info --> Run anyway
- During kit installer, if you encounter Windows security alert, click Allow
For a detailed tutorial on installing & executing the solution as well as learning resources including training & certification opportunities, please visit the OpenWeaver Community
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers. Jupyter Notebook is used for our development.
Text Mining
Libraries in this group are used for analysis and processing of unstructured natural language. The data, as in its original form aren't used as it has to go through processing pipeline to become suitable for applying machine learning techniques and algorithms.
Testing
The libraries listed here can be used for unit testing as well as integration testing
Support
If you need help using this kit, you may reach us at the OpenWeaver Community.
The next word predictor is an exciting feature that helps you type faster on your mobile phone. It predicts the next word in the context you want to type. It is a very useful tool for people who type often and make mistakes while typing. It can be leveraged for auto-suggestion features in messenger and search engine apps.
The next word predictor makes it easy for readers to understand what exactly you are trying for them to read about.
- Next word predictor is a very useful feature as it increases the readability of your content as well as makes it more understandable for readers.
- Saves time by reducing the number of typos and grammatical errors in your content.
- Modify source code to customize as per your requirements.
Instructions to Run
Follow the below instructions to run the solution.
- Locate and open the 'Next Word Predictor.ipynb' notebook from the Jupyter Notebook browser window.
- Execute cells in the notebook by selecting Cell --> Run All from the Menu bar.
- Once all the cells of the notebook are executed, the last interactive cell (Customisation) will be active, there we can give the input data or we can give the input text in the variable 'text_seq' under the variable section.
Input
text_seq = "I'm gonna make him an offer he can't"
Output
['refuse', 'resist', 'take', 'deny', 'get']
Troubleshooting
- If you encounter any error related to MS Visual C++, please install MS Visual Build tools
- While running batch file, if you encounter Windows protection alert, select More info --> Run anyway.
- During kit installer, if you encounter Windows security alert, click Allow.
- If you encounter Memory Error, check if the available memory is sufficient and it is proportion to the size of the data being used. For our dataset, the minimum required memory is 8GB.
If your computer doesn't support standard commands from windows 10, you can follow the instructions below to finish the kit installation.
- Install python
- Download the repository
- Extract the zip file and navigate to the directory 'next-word-prediction-main'
- Open terminal in the extracted directory 'next-word-prediction-main'
- Install dependencies by executing the command 'pip install -r requirements.txt'
- Run the command ‘jupyter notebook’ and select the notebook ‘Next Word Predictor.ipynb’ on the browser window.
For a detailed tutorial on installing & executing the solution as well as learning resources including training & certification opportunities, please visit the OpenWeaver Community
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web-based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers. Jupyter Notebook is used for our development.
Exploratory Data Analysis
For extensive analysis and exploration of data, and to deal with arrays, these libraries are used. They are also used for performing scientific computation and data manipulation.
Text Mining
Libraries in this group are used for analysis and processing of unstructured natural language.
Machine Learning
The library offers state-of-the-art pre-trained models for Natural Language Processing (NLP).
Support
If you need help using this kit, you may reach us at the OpenWeaver Community.
AI-powered emoji detectors can help increase engagement with their customers. It will help them to build strong relationships with their customers. The emoji detector will help you in analyzing your audience and their preferences so that you can deliver the right content. You can also use the technology to provide customer support to your customers by providing customized answers.
One of the most important aspects of AI-Powered Emoji Detector is that it will help you in detecting any kind of emotions and expressions on your face OR hand gestures from a web camera. It will help in detecting whether you are happy, sad, or angry, and so on. This technology is also used for predicting different kinds of expressions like happiness, fear, sadness, etc.
For a detailed tutorial on installing & executing the solution as well as learning resources including training & certification opportunities, please visit the OpenWeaver Community
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers. Jupyter Notebook is used for our development.
Image Preparation and Processing
These libraries help in preparing data by annotating and labelling images. Also processes images for running machine learning algorithm. We use opencv library for capturing frames from live streaming videocam.
Data Analysis/Manipulation
These libraries help in analyzing data and doing data manipulations.
Machine Learning
Below libraries and model collections helps to create the machine learning models for the core recognition use cases in our solution.
Utilities
The below utility library helps in storing huge amounts of numerical data and manipulate that data easily from NumPy.
Support
If you need help using this kit, you may reach us at the OpenWeaver Community.
Real-time object tracking system is a technology used to track objects in real time. It can be used for security purposes or for commercial purposes. Tracking can be done for video formats and live streaming webcam.
The real-time object tracking system has many applications, such as in retail stores, airports, stadiums and other places where security is important. The system can be used to monitor customer activity in stores, track inventory and detect shoplifting. It can also be used to increase safety in public places by monitoring the movements of pedestrians or vehicles.
For a detailed tutorial on installing & executing the solution as well as learning resources including training & certification opportunities, please visit the OpenWeaver Community
Development Environment
VSCode and Jupyter Notebook can be used for development and debugging. Jupyter Notebook is a web-based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers.
Object Detection and Tracking
The following libraries have a set of pre-trained models which could be used to identify objects and track them from live streaming videos.
Machine Learning Libraries
The following libraries could be used to create machine learning models which focus on the vision, extraction of data, image processing, and more. Thus making it handy for the users.
Support
If you need help using this kit, you may reach us at the OpenWeaver Community.
Disease predictor is a way to recognize patient’s health by applying data mining and machine learning techniques on patient treatment history.
Symptoms, Diagnosis for Personalized Healthcare Services for a Predictive Analytic Perspective. Pandas library is used in this kandi kit to predict the probability of disease. The kit has used pandas to load datasets and visualize the data, NumPy to implement our algorithm, and sklearn-pandas to build our model.
In this project we will be using Pandas and Scikit-Learn to create a model that predicts whether or not a patient has a disease based on their demographics and lab results. We will also be using Jupyter Notebook to write code interactively so that we can see how our model performs when we change various parameters such as the number of features, amount of training data, etc.
kandi kit provides you with a fully deployable Disease Predictor. Source code included so that you can customize it for your requirement.
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers. Jupyter Notebook is used for our development.
Exploratory Data Analysis
For extensive analysis and exploration of data, and to deal with arrays, these libraries are used. They are also used for performing scientific computation and data manipulation.
Data Visualization
The patterns and relationships are identified by representing data visually and below libraries are used for generating visual plots of the data.
Support
If you need help to use this kit, you can email us at kandi.support@openweaver.com or direct message us on Twitter Message @OpenWeaverInc .
Super Mario is frequently cited as one of the greatest video games of all time. This kit is a classic remake of Super Mario Bros. developed in python. This kit includes only level 1-1.
Development Environment
VSCode is used for development, debugging and are used to get a typical experience of IDE for developers.
Gaming Libraries
Pygame helps in providing computer graphics and audio libraries. PyTMX helps to load maps for games.
Support
If you need help to use this kit, you can email us at kandi.support@openweaver.com or direct message us on Twitter Message @OpenWeaverInc.
Python Machine Learning libraries help develop supervised and unsupervised learning, data pre-processing, feature extraction tools, and deep learning.
Following are the top use cases of these shortlisted libraries for Python Machine Learning,
- Pre-processing of data that includes data cleaning and feature engineering tasks such as normalization, imputation, missing value treatment, and outlier detection.
- Model selecting and optimizing, such as cross-validation, hyperparameter tuning, and model selection metrics.
- Visualizations to understand data and results. This includes visualizing data distributions, feature importance, and model performance.
- Developing algorithms, including supervised learning algorithms (e.g. linear regression, logistic regression, support vector machines, decision trees, random forests, and neural networks) as well as unsupervised learning algorithms (e.g. clustering, dimensionality reduction, and anomaly detection).
- Calculating performance metrics such as accuracy, precision, recall, and F1 score.
The following is a list of the 18 most popular open-source Python libraries for Machine Learning,
keras:
- It provides a high-level API for building and training deep neural networks.
- Keras allows you to define and incorporate custom layers and loss functions.
- Configure Keras to run on top of deep learning frameworks like TensorFlow, etc.
Scikit-Learn:
- It is an essential library in the field of machine learning and data science.
- It provides tools for cross-validation, hyperparameter tuning, and model selection.
- The library runs on top of other scientific Python libraries like NumPy and SciPy.
Pandas:
- It is a popular Python library for data manipulation and analysis.
- It offers tools for data cleaning. This includes handling missing values, data alignment, and data type conversion.
- It supports time series data, making it valuable for financial analysis and forecasting.
YOLOv5:
- "You Only Look Once version 5," is a popular computer vision model for object detection.
- It is popular for its real-time object detection capabilities.
- It has improved upon the accuracy of its predecessors while maintaining its speed.
Ray:
- It is an open-source distributed computing framework used in Python.
- It enables you to parallelize and distribute Python applications.
- It helps with low-latency, high-throughput computing tasks.
ML-From-Scratch:
- This helps you gain a deep understanding of the underlying algorithms and mathematics.
- This allows you to customize it for your specific problem and data. This makes it more effective and efficient.
- Building models from scratch provides insight into optimization techniques.
examples:
- It helps in AI, ML, DL, Pytorch, TensorFlow applications.
- This library in PyTorch is essential for working with computer vision tasks.
- You can access pre-trained models like ResNet, VGG, and AlexNet through "torchvision.models".
Paddle:
- It is an open-source deep learning platform developed by Baidu.
- It is a powerful deep learning framework, like TensorFlow and PyTorch.
- It focuses on simplicity and efficiency.
rasa:
- It is an open-source Python library designed for building conversational AI apps.
- It provides tools for creating and managing conversational flows.
- It supports many languages and can helps in a global context.
horovod:
- It is a popular library in Python used for distributed deep learning.
- It enables you to scale your DL models to many GPUs and even across many machines.
- It supports various deep learning frameworks like TensorFlow, PyTorch, and MXNet.
mlflow:
- It is an open-source platform for managing the end-to-end machine learning lifecycle.
- It allows you to log and compare experiments.
- It provides tools for packaging models in a standard format.
imgaug:
- It is an important tool for image augmentation. It is especially used in machine learning and computer vision tasks.
- It allows you to customize augmentation pipelines to suit your specific needs.
- It works well with other popular libraries like OpenCV and NumPy.
ChatterBot:
- It provides a framework and pre-built components. That makes it easier to create chatbots.
- This library often includes NLP capabilities. This allows chatbots to understand and generate human-like text responses.
- These libraries offer options for customizing the behavior and responses of chatbots.
nni:
- NNI handles distributed training, making it suitable for large-scale experiments.
- NNI is important for streamlining and improving the machine learning model development process.
- It automates and optimizes ML model selection and hyperparameter tuning.
numpy-ml:
- It is a fundamental library in the Python ecosystem. It is especially used in the context of machine learning and data science.
- It is open-source and has a large and active community.
- It is crucial for performing efficient numerical and array-based operations.
tpot:
- It is a Python library for automated machine learning (AutoML).
- This includes feature selection, data preprocessing, and the choice of models.
- It employs techniques like cross-validation to reduce the risk of overfitting.
autokeras:
- It is an open-source library for automated machine learning (AutoML).
- It simplifies the process of building and training machine learning models.
- It is accessible to both beginners and experienced ML practitioners.
pattern:
- It is often referred to as a design pattern library.
- It is a collection of reusable solutions to common software design problems.
- These patterns help developers create more efficient, maintainable, and scalable code.
FAQ
1. What is scikit-learn?
It is an ML library for Python. That provides simple and efficient tools for data analysis and modeling. It offers a wide range of algorithms for classification, regression, clustering, and more.
2. What is PyTorch?
PyTorch is an open-source machine learning library. It is developed by Facebook's AI Research lab. It helps with deep learning and provides dynamic computation graphs. This makies it popular among researchers.
3. What is Keras?
Keras is an open-source deep learning API. That runs on top of other deep learning frameworks like TensorFlow and Theano. It's designed to be and allows for rapid prototyping of neural networks.
4. How do I install these libraries?
You can install these libraries using Python's package manager, pip. For example, you can install scikit-learn with pip install scikit-learn. Also, install TensorFlow with pip install tensorflow, and PyTorch with pip install torch.
5. What is the difference between a tensor and an array in TensorFlow?
In TensorFlow, a tensor is a multi-dimensional array. This array can be placed on GPU for accelerated computation. It is like NumPy arrays but optimized for deep learning tasks.
This is related to water treatment and purification.
DataViz Group
This helps with DaaViz
Cyber Data Analytics Group
This is for Cyber Data Analytics Group
zeronet group
this is zeronet group
Automation Group
this is part of Automation Group
AI has been to build intelligent agents that can understand the vision and language inputs and communicate with humans through natural language.
Vision and language, two of the most fundamental methods for humans to perceive the world, are also two key cornerstones of AI. A longstanding goal of AI has been to build intelligent agents that can understand the world through vision and language inputs, and communicate with humans through natural language.
In order to achieve this goal, vision-language pre-training has emerged as an effective approach, where deep neural network models are pre-trained on large scale image-text datasets to improve performance on downstream vision-language tasks, such as image-text retrieval, image captioning, and visual question answering.
Image Captioning and Visual Question and Answering involves the usage of Large Multimodal Models (LMMs). Multimodal Learning seeks to allow computers to represent real-world objects and concepts using multiple data streams. We make use of one such model - Saleforce's BLIP (Bootstrapping Language-Image Pre-training)
Libraries used in this solution
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers.
Jupyter Notebook is used for our development.
Machine Learning
Machine learning libraries and frameworks here are helpful in providing state-of-the-art solutions using Machine learning
Kit Solution Source
App User Interface
Support
For any support, you can reach us at OpenWeaver Community Support
Image Restoration is the process of converting a noisy/blur/low quality image to a better-quality image in terms of resolution, color, noise, and blur.
Image Restoration techniques involve Deblurring, colorization, super resolution enhancement and other popular techniques as well.
This kit provides the solution for
- Restoring old images to colored images
- Upscale and enhance image resolution
The Image colorization is done by a technique called NoGAN from an open source project called "deoldify". Deoldify is a shading rendering model for images and videos that enables the restoration of native color to black and white photos and videos.
The Image Upscaling and enhancement is done through a technique called GFPGAN which uses Generative Adversarial Networks comprising of Deep learning techniques for Face/Image restoration.
Libraries used in this solution
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers.
Jupyter Notebook is used for our development.
Machine Learning
Machine learning libraries and frameworks here are helpful in providing state-of-the-art solutions using Machine learning
App User Interface
Kit Solution Source
Support
For any support, you can reach us at OpenWeaver Community Support
Trending Discussions on Python
Python/Docker ImportError: cannot import name 'json' from itsdangerous
Why is it faster to compare strings that match than strings that do not?
Why is `np.sum(range(N))` very slow?
Error while downloading the requirements using pip install (setup command: use_2to3 is invalid.)
Repeatedly removing the maximum average subarray
WARNING: Running pip as the 'root' user
How do I calculate square root in Python?
pip-compile raising AssertionError on its logging handler
ImportError: cannot import name 'url' from 'django.conf.urls' after upgrading to Django 4.0
How did print(*a, a.pop(0)) change?
QUESTION
Python/Docker ImportError: cannot import name 'json' from itsdangerous
Asked 2022-Mar-31 at 12:49I am trying to get a Flask and Docker application to work but when I try and run it using my docker-compose up
command in my Visual Studio terminal, it gives me an ImportError called ImportError: cannot import name 'json' from itsdangerous
. I have tried to look for possible solutions to this problem but as of right now there are not many on here or anywhere else. The only two solutions I could find are to change the current installation of MarkupSafe and itsdangerous to a higher version: https://serverfault.com/questions/1094062/from-itsdangerous-import-json-as-json-importerror-cannot-import-name-json-fr and another one on GitHub that tells me to essentially change the MarkUpSafe and itsdangerous installation again https://github.com/aws/aws-sam-cli/issues/3661, I have also tried to make a virtual environment named veganetworkscriptenv
to install the packages but that has also failed as well. I am currently using Flask 2.0.0 and Docker 5.0.0 and the error occurs on line eight in vegamain.py.
Here is the full ImportError that I get when I try and run the program:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10
Here are my requirements.txt, vegamain.py, Dockerfile, and docker-compose.yml files:
requirements.txt:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10Flask==2.0.0
11Flask-SQLAlchemy==2.4.4
12SQLAlchemy==1.3.20
13Flask-Migrate==2.5.3
14Flask-Script==2.0.6
15Flask-Cors==3.0.9
16requests==2.25.0
17mysqlclient==2.0.1
18pika==1.1.0
19wolframalpha==4.3.0
20
vegamain.py:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10Flask==2.0.0
11Flask-SQLAlchemy==2.4.4
12SQLAlchemy==1.3.20
13Flask-Migrate==2.5.3
14Flask-Script==2.0.6
15Flask-Cors==3.0.9
16requests==2.25.0
17mysqlclient==2.0.1
18pika==1.1.0
19wolframalpha==4.3.0
20# Veganetwork (C) TetraSystemSolutions 2022
21# all rights are reserved.
22#
23# Author: Trevor R. Blanchard Feb-19-2022-Jul-30-2022
24#
25
26# get our imports in order first
27from flask import Flask # <-- error occurs here!!!
28
29# start the application through flask.
30app = Flask(__name__)
31
32# if set to true will return only a "Hello World" string.
33Debug = True
34
35# start a route to the index part of the app in flask.
36@app.route('/')
37def index():
38 if (Debug == True):
39 return 'Hello World!'
40 else:
41 pass
42
43# start the flask app here --->
44if __name__ == '__main__':
45 app.run(debug=True, host='0.0.0.0')
46
Dockerfile:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10Flask==2.0.0
11Flask-SQLAlchemy==2.4.4
12SQLAlchemy==1.3.20
13Flask-Migrate==2.5.3
14Flask-Script==2.0.6
15Flask-Cors==3.0.9
16requests==2.25.0
17mysqlclient==2.0.1
18pika==1.1.0
19wolframalpha==4.3.0
20# Veganetwork (C) TetraSystemSolutions 2022
21# all rights are reserved.
22#
23# Author: Trevor R. Blanchard Feb-19-2022-Jul-30-2022
24#
25
26# get our imports in order first
27from flask import Flask # <-- error occurs here!!!
28
29# start the application through flask.
30app = Flask(__name__)
31
32# if set to true will return only a "Hello World" string.
33Debug = True
34
35# start a route to the index part of the app in flask.
36@app.route('/')
37def index():
38 if (Debug == True):
39 return 'Hello World!'
40 else:
41 pass
42
43# start the flask app here --->
44if __name__ == '__main__':
45 app.run(debug=True, host='0.0.0.0')
46FROM python:3.9
47ENV PYTHONUNBUFFERED 1
48WORKDIR /app
49COPY requirements.txt /app/requirements.txt
50RUN pip install -r requirements.txt
51COPY . /app
52
docker-compose.yml:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10Flask==2.0.0
11Flask-SQLAlchemy==2.4.4
12SQLAlchemy==1.3.20
13Flask-Migrate==2.5.3
14Flask-Script==2.0.6
15Flask-Cors==3.0.9
16requests==2.25.0
17mysqlclient==2.0.1
18pika==1.1.0
19wolframalpha==4.3.0
20# Veganetwork (C) TetraSystemSolutions 2022
21# all rights are reserved.
22#
23# Author: Trevor R. Blanchard Feb-19-2022-Jul-30-2022
24#
25
26# get our imports in order first
27from flask import Flask # <-- error occurs here!!!
28
29# start the application through flask.
30app = Flask(__name__)
31
32# if set to true will return only a "Hello World" string.
33Debug = True
34
35# start a route to the index part of the app in flask.
36@app.route('/')
37def index():
38 if (Debug == True):
39 return 'Hello World!'
40 else:
41 pass
42
43# start the flask app here --->
44if __name__ == '__main__':
45 app.run(debug=True, host='0.0.0.0')
46FROM python:3.9
47ENV PYTHONUNBUFFERED 1
48WORKDIR /app
49COPY requirements.txt /app/requirements.txt
50RUN pip install -r requirements.txt
51COPY . /app
52version: '3.8'
53services:
54 backend:
55 build:
56 context: .
57 dockerfile: Dockerfile
58 command: 'python vegamain.py'
59 ports:
60 - 8004:5000
61 volumes:
62 - .:/app
63 depends_on:
64 - db
65
66# queue:
67# build:
68# context: .
69# dockerfile: Dockerfile
70# command: 'python -u consumer.py'
71# depends_on:
72# - db
73
74 db:
75 image: mysql:5.7.22
76 restart: always
77 environment:
78 MYSQL_DATABASE: admin
79 MYSQL_USER: root
80 MYSQL_PASSWORD: root
81 MYSQL_ROOT_PASSWORD: root
82 volumes:
83 - .dbdata:/var/lib/mysql
84 ports:
85 - 33069:3306
86
How exactly can I fix this code? thank you!
ANSWER
Answered 2022-Feb-20 at 12:31I was facing the same issue while running docker containers with flask.
I downgraded Flask
to 1.1.4
and markupsafe
to 2.0.1
which solved my issue.
Check this for reference.
QUESTION
Why is it faster to compare strings that match than strings that do not?
Asked 2022-Mar-30 at 11:58Here are two measurements:
1timeit.timeit('"toto"=="1234"', number=100000000)
21.8320042459999968
3timeit.timeit('"toto"=="toto"', number=100000000)
41.4517491540000265
5
As you can see, comparing two strings that match is faster than comparing two strings with the same size that do not match.
This is quite disturbing: During a string comparison, I believed that Python was testing strings character by character, so "toto"=="toto"
should be longer to test than "toto"=="1234"
as it requires four tests against one for the non-matching comparison. Maybe the comparison is hash-based, but in this case, timings should be the same for both comparisons.
Why?
ANSWER
Answered 2022-Mar-30 at 11:57Combining my comment and the comment by @khelwood:
TL;DR:
When analysing the bytecode for the two comparisons, it reveals the 'time'
and 'time'
strings are assigned to the same object. Therefore, an up-front identity check (at C-level) is the reason for the increased comparison speed.
The reason for the same object assignment is that, as an implementation detail, CPython interns strings which contain only 'name characters' (i.e. alpha and underscore characters). This enables the object's identity check.
Bytecode:
1timeit.timeit('"toto"=="1234"', number=100000000)
21.8320042459999968
3timeit.timeit('"toto"=="toto"', number=100000000)
41.4517491540000265
5import dis
6
7In [24]: dis.dis("'time'=='time'")
8 1 0 LOAD_CONST 0 ('time') # <-- same object (0)
9 2 LOAD_CONST 0 ('time') # <-- same object (0)
10 4 COMPARE_OP 2 (==)
11 6 RETURN_VALUE
12
13In [25]: dis.dis("'time'=='1234'")
14 1 0 LOAD_CONST 0 ('time') # <-- different object (0)
15 2 LOAD_CONST 1 ('1234') # <-- different object (1)
16 4 COMPARE_OP 2 (==)
17 6 RETURN_VALUE
18
Assignment Timing:
The 'speed-up' can also be seen in using assignment for the time tests. The assignment (and compare) of two variables to the same string, is faster than the assignment (and compare) of two variables to different strings. Further supporting the hypothesis the underlying logic is performing an object comparison. This is confirmed in the next section.
1timeit.timeit('"toto"=="1234"', number=100000000)
21.8320042459999968
3timeit.timeit('"toto"=="toto"', number=100000000)
41.4517491540000265
5import dis
6
7In [24]: dis.dis("'time'=='time'")
8 1 0 LOAD_CONST 0 ('time') # <-- same object (0)
9 2 LOAD_CONST 0 ('time') # <-- same object (0)
10 4 COMPARE_OP 2 (==)
11 6 RETURN_VALUE
12
13In [25]: dis.dis("'time'=='1234'")
14 1 0 LOAD_CONST 0 ('time') # <-- different object (0)
15 2 LOAD_CONST 1 ('1234') # <-- different object (1)
16 4 COMPARE_OP 2 (==)
17 6 RETURN_VALUE
18In [26]: timeit.timeit("x='time'; y='time'; x==y", number=1000000)
19Out[26]: 0.0745926329982467
20
21In [27]: timeit.timeit("x='time'; y='1234'; x==y", number=1000000)
22Out[27]: 0.10328884399496019
23
Python source code:
As helpfully provided by @mkrieger1 and @Masklinn in their comments, the source code for unicodeobject.c
performs a pointer comparison first and if True
, returns immediately.
1timeit.timeit('"toto"=="1234"', number=100000000)
21.8320042459999968
3timeit.timeit('"toto"=="toto"', number=100000000)
41.4517491540000265
5import dis
6
7In [24]: dis.dis("'time'=='time'")
8 1 0 LOAD_CONST 0 ('time') # <-- same object (0)
9 2 LOAD_CONST 0 ('time') # <-- same object (0)
10 4 COMPARE_OP 2 (==)
11 6 RETURN_VALUE
12
13In [25]: dis.dis("'time'=='1234'")
14 1 0 LOAD_CONST 0 ('time') # <-- different object (0)
15 2 LOAD_CONST 1 ('1234') # <-- different object (1)
16 4 COMPARE_OP 2 (==)
17 6 RETURN_VALUE
18In [26]: timeit.timeit("x='time'; y='time'; x==y", number=1000000)
19Out[26]: 0.0745926329982467
20
21In [27]: timeit.timeit("x='time'; y='1234'; x==y", number=1000000)
22Out[27]: 0.10328884399496019
23int
24_PyUnicode_Equal(PyObject *str1, PyObject *str2)
25{
26 assert(PyUnicode_CheckExact(str1));
27 assert(PyUnicode_CheckExact(str2));
28 if (str1 == str2) { // <-- Here
29 return 1;
30 }
31 if (PyUnicode_READY(str1) || PyUnicode_READY(str2)) {
32 return -1;
33 }
34 return unicode_compare_eq(str1, str2);
35}
36
Appendix:
- Reference answer nicely illustrating how to read the disassembled bytecode output. Courtesy of @Delgan
- Reference answer which nicely describes CPython's string interning. Coutresy of @ShadowRanger
QUESTION
Why is `np.sum(range(N))` very slow?
Asked 2022-Mar-29 at 14:31I saw a video about speed of loops in python, where it was explained that doing sum(range(N))
is much faster than manually looping through range
and adding the variables together, since the former runs in C due to built-in functions being used, while in the latter the summation is done in (slow) python. I was curious what happens when adding numpy
to the mix. As I expected np.sum(np.arange(N))
is the fastest, but sum(np.arange(N))
and np.sum(range(N))
are even slower than doing the naive for loop.
Why is this?
Here's the script I used to test, some comments about the supposed cause of slowing done where I know (taken mostly from the video) and the results I got on my machine (python 3.10.0, numpy 1.21.2):
updated script:
1import numpy as np
2from timeit import timeit
3
4N = 10_000_000
5repetition = 10
6
7def sum0(N = N):
8 s = 0
9 i = 0
10 while i < N: # condition is checked in python
11 s += i
12 i += 1 # both additions are done in python
13 return s
14
15def sum1(N = N):
16 s = 0
17 for i in range(N): # increment in C
18 s += i # addition in python
19 return s
20
21def sum2(N = N):
22 return sum(range(N)) # everything in C
23
24def sum3(N = N):
25 return sum(list(range(N)))
26
27def sum4(N = N):
28 return np.sum(range(N)) # very slow np.array conversion
29
30def sum5(N = N):
31 # much faster np.array conversion
32 return np.sum(np.fromiter(range(N),dtype = int))
33
34def sum5v2_(N = N):
35 # much faster np.array conversion
36 return np.sum(np.fromiter(range(N),dtype = np.int_))
37
38def sum6(N = N):
39 # possibly slow conversion to Py_long from np.int
40 return sum(np.arange(N))
41
42def sum7(N = N):
43 # list returns a list of np.int-s
44 return sum(list(np.arange(N)))
45
46def sum7v2(N = N):
47 # tolist conversion to python int seems faster than the implicit conversion
48 # in sum(list()) (tolist returns a list of python int-s)
49 return sum(np.arange(N).tolist())
50
51def sum8(N = N):
52 return np.sum(np.arange(N)) # everything in numpy (fortran libblas?)
53
54def sum9(N = N):
55 return np.arange(N).sum() # remove dispatch overhead
56
57def array_basic(N = N):
58 return np.array(range(N))
59
60def array_dtype(N = N):
61 return np.array(range(N),dtype = np.int_)
62
63def array_iter(N = N):
64 # np.sum's source code mentions to use fromiter to convert from generators
65 return np.fromiter(range(N),dtype = np.int_)
66
67print(f"while loop: {timeit(sum0, number = repetition)}")
68print(f"for loop: {timeit(sum1, number = repetition)}")
69print(f"sum_range: {timeit(sum2, number = repetition)}")
70print(f"sum_rangelist: {timeit(sum3, number = repetition)}")
71print(f"npsum_range: {timeit(sum4, number = repetition)}")
72print(f"npsum_iterrange: {timeit(sum5, number = repetition)}")
73print(f"npsum_iterrangev2: {timeit(sum5, number = repetition)}")
74print(f"sum_arange: {timeit(sum6, number = repetition)}")
75print(f"sum_list_arange: {timeit(sum7, number = repetition)}")
76print(f"sum_arange_tolist: {timeit(sum7v2, number = repetition)}")
77print(f"npsum_arange: {timeit(sum8, number = repetition)}")
78print(f"nparangenpsum: {timeit(sum9, number = repetition)}")
79print(f"array_basic: {timeit(array_basic, number = repetition)}")
80print(f"array_dtype: {timeit(array_dtype, number = repetition)}")
81print(f"array_iter: {timeit(array_iter, number = repetition)}")
82
83print(f"npsumarangeREP: {timeit(lambda : sum8(N/1000), number = 100000*repetition)}")
84print(f"npsumarangeREP: {timeit(lambda : sum9(N/1000), number = 100000*repetition)}")
85
86# Example output:
87#
88# while loop: 11.493371912998555
89# for loop: 7.385945574002108
90# sum_range: 2.4605720699983067
91# sum_rangelist: 4.509678105998319
92# npsum_range: 11.85120212900074
93# npsum_iterrange: 4.464334709002287
94# npsum_iterrangev2: 4.498494338993623
95# sum_arange: 9.537815956995473
96# sum_list_arange: 13.290120724996086
97# sum_arange_tolist: 5.231948580003518
98# npsum_arange: 0.241889145996538
99# nparangenpsum: 0.21876695199898677
100# array_basic: 11.736577274998126
101# array_dtype: 8.71628468400013
102# array_iter: 4.303306431000237
103# npsumarangeREP: 21.240833958996518
104# npsumarangeREP: 16.690092379001726
105
106
ANSWER
Answered 2021-Oct-16 at 17:42From the cpython source code for sum
sum initially seems to attempt a fast path that assumes all inputs are the same type. If that fails it will just iterate:
1import numpy as np
2from timeit import timeit
3
4N = 10_000_000
5repetition = 10
6
7def sum0(N = N):
8 s = 0
9 i = 0
10 while i < N: # condition is checked in python
11 s += i
12 i += 1 # both additions are done in python
13 return s
14
15def sum1(N = N):
16 s = 0
17 for i in range(N): # increment in C
18 s += i # addition in python
19 return s
20
21def sum2(N = N):
22 return sum(range(N)) # everything in C
23
24def sum3(N = N):
25 return sum(list(range(N)))
26
27def sum4(N = N):
28 return np.sum(range(N)) # very slow np.array conversion
29
30def sum5(N = N):
31 # much faster np.array conversion
32 return np.sum(np.fromiter(range(N),dtype = int))
33
34def sum5v2_(N = N):
35 # much faster np.array conversion
36 return np.sum(np.fromiter(range(N),dtype = np.int_))
37
38def sum6(N = N):
39 # possibly slow conversion to Py_long from np.int
40 return sum(np.arange(N))
41
42def sum7(N = N):
43 # list returns a list of np.int-s
44 return sum(list(np.arange(N)))
45
46def sum7v2(N = N):
47 # tolist conversion to python int seems faster than the implicit conversion
48 # in sum(list()) (tolist returns a list of python int-s)
49 return sum(np.arange(N).tolist())
50
51def sum8(N = N):
52 return np.sum(np.arange(N)) # everything in numpy (fortran libblas?)
53
54def sum9(N = N):
55 return np.arange(N).sum() # remove dispatch overhead
56
57def array_basic(N = N):
58 return np.array(range(N))
59
60def array_dtype(N = N):
61 return np.array(range(N),dtype = np.int_)
62
63def array_iter(N = N):
64 # np.sum's source code mentions to use fromiter to convert from generators
65 return np.fromiter(range(N),dtype = np.int_)
66
67print(f"while loop: {timeit(sum0, number = repetition)}")
68print(f"for loop: {timeit(sum1, number = repetition)}")
69print(f"sum_range: {timeit(sum2, number = repetition)}")
70print(f"sum_rangelist: {timeit(sum3, number = repetition)}")
71print(f"npsum_range: {timeit(sum4, number = repetition)}")
72print(f"npsum_iterrange: {timeit(sum5, number = repetition)}")
73print(f"npsum_iterrangev2: {timeit(sum5, number = repetition)}")
74print(f"sum_arange: {timeit(sum6, number = repetition)}")
75print(f"sum_list_arange: {timeit(sum7, number = repetition)}")
76print(f"sum_arange_tolist: {timeit(sum7v2, number = repetition)}")
77print(f"npsum_arange: {timeit(sum8, number = repetition)}")
78print(f"nparangenpsum: {timeit(sum9, number = repetition)}")
79print(f"array_basic: {timeit(array_basic, number = repetition)}")
80print(f"array_dtype: {timeit(array_dtype, number = repetition)}")
81print(f"array_iter: {timeit(array_iter, number = repetition)}")
82
83print(f"npsumarangeREP: {timeit(lambda : sum8(N/1000), number = 100000*repetition)}")
84print(f"npsumarangeREP: {timeit(lambda : sum9(N/1000), number = 100000*repetition)}")
85
86# Example output:
87#
88# while loop: 11.493371912998555
89# for loop: 7.385945574002108
90# sum_range: 2.4605720699983067
91# sum_rangelist: 4.509678105998319
92# npsum_range: 11.85120212900074
93# npsum_iterrange: 4.464334709002287
94# npsum_iterrangev2: 4.498494338993623
95# sum_arange: 9.537815956995473
96# sum_list_arange: 13.290120724996086
97# sum_arange_tolist: 5.231948580003518
98# npsum_arange: 0.241889145996538
99# nparangenpsum: 0.21876695199898677
100# array_basic: 11.736577274998126
101# array_dtype: 8.71628468400013
102# array_iter: 4.303306431000237
103# npsumarangeREP: 21.240833958996518
104# npsumarangeREP: 16.690092379001726
105
106/* Fast addition by keeping temporary sums in C instead of new Python objects.
107 Assumes all inputs are the same type. If the assumption fails, default
108 to the more general routine.
109*/
110
I'm not entirely certain what is happening under the hood, but it is likely the repeated creation/conversion of C types to Python objects that is causing these slow-downs. It's worth noting that both sum
and range
are implemented in C.
This next bit is not really an answer to the question, but I wondered if we could speed up sum
for python range
s as range
is quite a smart object.
To do this I've used functools.singledispatch
to override the built-in sum
function specifically for the range
type; then implemented a small function to calculate the sum of an arithmetic progression.
1import numpy as np
2from timeit import timeit
3
4N = 10_000_000
5repetition = 10
6
7def sum0(N = N):
8 s = 0
9 i = 0
10 while i < N: # condition is checked in python
11 s += i
12 i += 1 # both additions are done in python
13 return s
14
15def sum1(N = N):
16 s = 0
17 for i in range(N): # increment in C
18 s += i # addition in python
19 return s
20
21def sum2(N = N):
22 return sum(range(N)) # everything in C
23
24def sum3(N = N):
25 return sum(list(range(N)))
26
27def sum4(N = N):
28 return np.sum(range(N)) # very slow np.array conversion
29
30def sum5(N = N):
31 # much faster np.array conversion
32 return np.sum(np.fromiter(range(N),dtype = int))
33
34def sum5v2_(N = N):
35 # much faster np.array conversion
36 return np.sum(np.fromiter(range(N),dtype = np.int_))
37
38def sum6(N = N):
39 # possibly slow conversion to Py_long from np.int
40 return sum(np.arange(N))
41
42def sum7(N = N):
43 # list returns a list of np.int-s
44 return sum(list(np.arange(N)))
45
46def sum7v2(N = N):
47 # tolist conversion to python int seems faster than the implicit conversion
48 # in sum(list()) (tolist returns a list of python int-s)
49 return sum(np.arange(N).tolist())
50
51def sum8(N = N):
52 return np.sum(np.arange(N)) # everything in numpy (fortran libblas?)
53
54def sum9(N = N):
55 return np.arange(N).sum() # remove dispatch overhead
56
57def array_basic(N = N):
58 return np.array(range(N))
59
60def array_dtype(N = N):
61 return np.array(range(N),dtype = np.int_)
62
63def array_iter(N = N):
64 # np.sum's source code mentions to use fromiter to convert from generators
65 return np.fromiter(range(N),dtype = np.int_)
66
67print(f"while loop: {timeit(sum0, number = repetition)}")
68print(f"for loop: {timeit(sum1, number = repetition)}")
69print(f"sum_range: {timeit(sum2, number = repetition)}")
70print(f"sum_rangelist: {timeit(sum3, number = repetition)}")
71print(f"npsum_range: {timeit(sum4, number = repetition)}")
72print(f"npsum_iterrange: {timeit(sum5, number = repetition)}")
73print(f"npsum_iterrangev2: {timeit(sum5, number = repetition)}")
74print(f"sum_arange: {timeit(sum6, number = repetition)}")
75print(f"sum_list_arange: {timeit(sum7, number = repetition)}")
76print(f"sum_arange_tolist: {timeit(sum7v2, number = repetition)}")
77print(f"npsum_arange: {timeit(sum8, number = repetition)}")
78print(f"nparangenpsum: {timeit(sum9, number = repetition)}")
79print(f"array_basic: {timeit(array_basic, number = repetition)}")
80print(f"array_dtype: {timeit(array_dtype, number = repetition)}")
81print(f"array_iter: {timeit(array_iter, number = repetition)}")
82
83print(f"npsumarangeREP: {timeit(lambda : sum8(N/1000), number = 100000*repetition)}")
84print(f"npsumarangeREP: {timeit(lambda : sum9(N/1000), number = 100000*repetition)}")
85
86# Example output:
87#
88# while loop: 11.493371912998555
89# for loop: 7.385945574002108
90# sum_range: 2.4605720699983067
91# sum_rangelist: 4.509678105998319
92# npsum_range: 11.85120212900074
93# npsum_iterrange: 4.464334709002287
94# npsum_iterrangev2: 4.498494338993623
95# sum_arange: 9.537815956995473
96# sum_list_arange: 13.290120724996086
97# sum_arange_tolist: 5.231948580003518
98# npsum_arange: 0.241889145996538
99# nparangenpsum: 0.21876695199898677
100# array_basic: 11.736577274998126
101# array_dtype: 8.71628468400013
102# array_iter: 4.303306431000237
103# npsumarangeREP: 21.240833958996518
104# npsumarangeREP: 16.690092379001726
105
106/* Fast addition by keeping temporary sums in C instead of new Python objects.
107 Assumes all inputs are the same type. If the assumption fails, default
108 to the more general routine.
109*/
110from functools import singledispatch
111
112def sum_range(range_, /, start=0):
113 """Overloaded `sum` for range, compute arithmetic sum"""
114 n = len(range_)
115 if not n:
116 return start
117 return int(start + (n * (range_[0] + range_[-1]) / 2))
118
119sum = singledispatch(sum)
120sum.register(range, sum_range)
121
122def test():
123 """
124 >>> sum(range(0, 100))
125 4950
126 >>> sum(range(0, 10, 2))
127 20
128 >>> sum(range(0, 9, 2))
129 20
130 >>> sum(range(0, -10, -1))
131 -45
132 >>> sum(range(-10, 10))
133 -10
134 >>> sum(range(-1, -100, -2))
135 -2500
136 >>> sum(range(0, 10, 100))
137 0
138 >>> sum(range(0, 0))
139 0
140 >>> sum(range(0, 100), 50)
141 5000
142 >>> sum(range(0, 0), 10)
143 10
144 """
145
146if __name__ == "__main__":
147 import doctest
148 doctest.testmod()
149
I'm not sure if this is complete, but it's definitely faster than looping.
QUESTION
Error while downloading the requirements using pip install (setup command: use_2to3 is invalid.)
Asked 2022-Mar-05 at 07:13version pip 21.2.4 python 3.6
The command:
1pip install -r requirments.txt
2
The content of my requirements.txt
:
1pip install -r requirments.txt
2mongoengine==0.19.1
3numpy==1.16.2
4pylint
5pandas==1.1.5
6fawkes
7
The command is failing with this error
1pip install -r requirments.txt
2mongoengine==0.19.1
3numpy==1.16.2
4pylint
5pandas==1.1.5
6fawkes
7ERROR: Command errored out with exit status 1:
8 command: /Users/*/Desktop/ml/*/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"'; __file__='"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-pip-egg-info-97994d6e
9 cwd: /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/
10 Complete output (1 lines):
11 error in mongoengine setup command: use_2to3 is invalid.
12 ----------------------------------------
13WARNING: Discarding https://*/pypi/packages/mongoengine-0.19.1.tar.gz#md5=68e613009f6466239158821a102ac084 (from https://*/pypi/simple/mongoengine/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
14ERROR: Could not find a version that satisfies the requirement mongoengine==0.19.1 (from versions: 0.15.0, 0.19.1)
15ERROR: No matching distribution found for mongoengine==0.19.1
16
ANSWER
Answered 2021-Nov-19 at 13:30It looks like setuptools>=58
breaks support for use_2to3
:
So you should update setuptools
to setuptools<58
or avoid using packages with use_2to3
in the setup parameters.
I was having the same problem, pip==19.3.1
QUESTION
Repeatedly removing the maximum average subarray
Asked 2022-Feb-28 at 18:19I have an array of positive integers. For example:
1[1, 7, 8, 4, 2, 1, 4]
2
A "reduction operation" finds the array prefix with the highest average, and deletes it. Here, an array prefix means a contiguous subarray whose left end is the start of the array, such as [1]
or [1, 7]
or [1, 7, 8]
above. Ties are broken by taking the longer prefix.
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8
I will repeat the reduction operation until the array is empty:
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8[1, 7, 8, 4, 2, 1, 4]
9^ ^
10[4, 2, 1, 4]
11^ ^
12[2, 1, 4]
13^ ^
14[]
15
Now, actually performing these array modifications isn't necessary; I'm only looking for the list of lengths of prefixes that would be deleted by this process, for example, [3, 1, 3]
above.
What is an efficient algorithm for computing these prefix lengths?
The naive approach is to recompute all sums and averages from scratch in every iteration for an O(n^2)
algorithm-- I've attached Python code for this below. I'm looking for any improvement on this approach-- most preferably, any solution below O(n^2)
, but an algorithm with the same complexity but better constant factors would also be helpful.
Here are a few of the things I've tried (without success):
- Dynamically maintaining prefix sums, for example with a Binary Indexed Tree. While I can easily update prefix sums or find a maximum prefix sum in
O(log n)
time, I haven't found any data structure which can update the average, as the denominator in the average is changing. - Reusing the previous 'rankings' of prefix averages-- these rankings can change, e.g. in some array, the prefix ending at index
5
may have a larger average than the prefix ending at index6
, but after removing the first 3 elements, now the prefix ending at index2
may have a smaller average than the one ending at3
. - Looking for patterns in where prefixes end; for example, the rightmost element of any max average prefix is always a local maximum in the array, but it's not clear how much this helps.
This is a working Python implementation of the naive, quadratic method:
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8[1, 7, 8, 4, 2, 1, 4]
9^ ^
10[4, 2, 1, 4]
11^ ^
12[2, 1, 4]
13^ ^
14[]
15from fractions import Fraction
16def find_array_reductions(nums: List[int]) -> List[int]:
17 """Return list of lengths of max average prefix reductions."""
18
19 def max_prefix_avg(arr: List[int]) -> Tuple[float, int]:
20 """Return value and length of max average prefix in arr."""
21 if len(arr) == 0:
22 return (-math.inf, 0)
23
24 best_length = 1
25 best_average = Fraction(0, 1)
26 running_sum = 0
27
28 for i, x in enumerate(arr, 1):
29 running_sum += x
30 new_average = Fraction(running_sum, i)
31 if new_average >= best_average:
32 best_average = new_average
33 best_length = i
34
35 return (float(best_average), best_length)
36
37 removed_lengths = []
38 total_removed = 0
39
40 while total_removed < len(nums):
41 _, new_removal = max_prefix_avg(nums[total_removed:])
42 removed_lengths.append(new_removal)
43 total_removed += new_removal
44
45 return removed_lengths
46
Edit: The originally published code had a rare error with large inputs from using Python's math.isclose()
with default parameters for floating point comparison, rather than proper fraction comparison. This has been fixed in the current code. An example of the error can be found at this Try it online link, along with a foreword explaining exactly what causes this bug, if you're curious.
ANSWER
Answered 2022-Feb-27 at 22:44This problem has a fun O(n) solution.
If you draw a graph of cumulative sum vs index, then:
The average value in the subarray between any two indexes is the slope of the line between those points on the graph.
The first highest-average-prefix will end at the point that makes the highest angle from 0. The next highest-average-prefix must then have a smaller average, and it will end at the point that makes the highest angle from the first ending. Continuing to the end of the array, we find that...
These segments of highest average are exactly the segments in the upper convex hull of the cumulative sum graph.
Find these segments using the monotone chain algorithm. Since the points are already sorted, it takes O(n) time.
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8[1, 7, 8, 4, 2, 1, 4]
9^ ^
10[4, 2, 1, 4]
11^ ^
12[2, 1, 4]
13^ ^
14[]
15from fractions import Fraction
16def find_array_reductions(nums: List[int]) -> List[int]:
17 """Return list of lengths of max average prefix reductions."""
18
19 def max_prefix_avg(arr: List[int]) -> Tuple[float, int]:
20 """Return value and length of max average prefix in arr."""
21 if len(arr) == 0:
22 return (-math.inf, 0)
23
24 best_length = 1
25 best_average = Fraction(0, 1)
26 running_sum = 0
27
28 for i, x in enumerate(arr, 1):
29 running_sum += x
30 new_average = Fraction(running_sum, i)
31 if new_average >= best_average:
32 best_average = new_average
33 best_length = i
34
35 return (float(best_average), best_length)
36
37 removed_lengths = []
38 total_removed = 0
39
40 while total_removed < len(nums):
41 _, new_removal = max_prefix_avg(nums[total_removed:])
42 removed_lengths.append(new_removal)
43 total_removed += new_removal
44
45 return removed_lengths
46# Lengths of the segments in the upper convex hull
47# of the cumulative sum graph
48def upperSumHullLengths(arr):
49 if len(arr) < 2:
50 if len(arr) < 1:
51 return []
52 else:
53 return [1]
54
55 hull = [(0, 0),(1, arr[0])]
56 for x in range(2, len(arr)+1):
57 # this has x coordinate x-1
58 prevPoint = hull[len(hull) - 1]
59 # next point in cumulative sum
60 point = (x, prevPoint[1] + arr[x-1])
61 # remove points not on the convex hull
62 while len(hull) >= 2:
63 p0 = hull[len(hull)-2]
64 dx0 = prevPoint[0] - p0[0]
65 dy0 = prevPoint[1] - p0[1]
66 dx1 = x - prevPoint[0]
67 dy1 = point[1] - prevPoint[1]
68 if dy1*dx0 < dy0*dx1:
69 break
70 hull.pop()
71 prevPoint = p0
72 hull.append(point)
73
74 return [hull[i+1][0] - hull[i][0] for i in range(0, len(hull)-1)]
75
76
77print(upperSumHullLengths([ 1, 7, 8, 4, 2, 1, 4]))
78
prints:
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8[1, 7, 8, 4, 2, 1, 4]
9^ ^
10[4, 2, 1, 4]
11^ ^
12[2, 1, 4]
13^ ^
14[]
15from fractions import Fraction
16def find_array_reductions(nums: List[int]) -> List[int]:
17 """Return list of lengths of max average prefix reductions."""
18
19 def max_prefix_avg(arr: List[int]) -> Tuple[float, int]:
20 """Return value and length of max average prefix in arr."""
21 if len(arr) == 0:
22 return (-math.inf, 0)
23
24 best_length = 1
25 best_average = Fraction(0, 1)
26 running_sum = 0
27
28 for i, x in enumerate(arr, 1):
29 running_sum += x
30 new_average = Fraction(running_sum, i)
31 if new_average >= best_average:
32 best_average = new_average
33 best_length = i
34
35 return (float(best_average), best_length)
36
37 removed_lengths = []
38 total_removed = 0
39
40 while total_removed < len(nums):
41 _, new_removal = max_prefix_avg(nums[total_removed:])
42 removed_lengths.append(new_removal)
43 total_removed += new_removal
44
45 return removed_lengths
46# Lengths of the segments in the upper convex hull
47# of the cumulative sum graph
48def upperSumHullLengths(arr):
49 if len(arr) < 2:
50 if len(arr) < 1:
51 return []
52 else:
53 return [1]
54
55 hull = [(0, 0),(1, arr[0])]
56 for x in range(2, len(arr)+1):
57 # this has x coordinate x-1
58 prevPoint = hull[len(hull) - 1]
59 # next point in cumulative sum
60 point = (x, prevPoint[1] + arr[x-1])
61 # remove points not on the convex hull
62 while len(hull) >= 2:
63 p0 = hull[len(hull)-2]
64 dx0 = prevPoint[0] - p0[0]
65 dy0 = prevPoint[1] - p0[1]
66 dx1 = x - prevPoint[0]
67 dy1 = point[1] - prevPoint[1]
68 if dy1*dx0 < dy0*dx1:
69 break
70 hull.pop()
71 prevPoint = p0
72 hull.append(point)
73
74 return [hull[i+1][0] - hull[i][0] for i in range(0, len(hull)-1)]
75
76
77print(upperSumHullLengths([ 1, 7, 8, 4, 2, 1, 4]))
78[3, 1, 3]
79
QUESTION
WARNING: Running pip as the 'root' user
Asked 2022-Feb-24 at 01:59I am making simple image of my python Django app in Docker. But at the end of the building container it throws next warning (I am building it on Ubuntu 20.04):
1WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead
2
Why does it throw this warning if I am installing Python requirements inside my image? I am building my image using:
1WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead
2sudo docker build -t my_app:1 .
3
Should I be worried about warning that pip throws, because I know it can break my system?
Here is my Dockerfile:
1WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead
2sudo docker build -t my_app:1 .
3FROM python:3.8-slim-buster
4
5WORKDIR /app
6
7COPY requirements.txt requirements.txt
8
9RUN pip install -r requirements.txt
10
11COPY . .
12
13CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
14
ANSWER
Answered 2021-Aug-29 at 08:12The way your container is built doesn't add a user, so everything is done as root.
You could create a user and install to that users's home directory by doing something like this;
1WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead
2sudo docker build -t my_app:1 .
3FROM python:3.8-slim-buster
4
5WORKDIR /app
6
7COPY requirements.txt requirements.txt
8
9RUN pip install -r requirements.txt
10
11COPY . .
12
13CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
14FROM python:3.8.3-alpine
15
16RUN pip install --upgrade pip
17
18RUN adduser -D myuser
19USER myuser
20WORKDIR /home/myuser
21
22COPY --chown=myuser:myuser requirements.txt requirements.txt
23RUN pip install --user -r requirements.txt
24
25ENV PATH="/home/myuser/.local/bin:${PATH}"
26
27COPY --chown=myuser:myuser . .
28
29CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
30
QUESTION
How do I calculate square root in Python?
Asked 2022-Feb-17 at 03:40I need to calculate the square root of some numbers, for example √9 = 3
and √2 = 1.4142
. How can I do it in Python?
The inputs will probably be all positive integers, and relatively small (say less than a billion), but just in case they're not, is there anything that might break?
Related
- Integer square root in python
- Is there a short-hand for nth root of x in Python?
- Difference between **(1/2), math.sqrt and cmath.sqrt?
- Why is math.sqrt() incorrect for large numbers?
- Python sqrt limit for very large numbers?
- Which is faster in Python: x**.5 or math.sqrt(x)?
- Why does Python give the "wrong" answer for square root? (specific to Python 2)
- calculating n-th roots using Python 3's decimal module
- How can I take the square root of -1 using python? (focused on NumPy)
- Arbitrary precision of square roots
Note: This is an attempt at a canonical question after a discussion on Meta about an existing question with the same title.
ANSWER
Answered 2022-Feb-04 at 19:44math.sqrt()
The math
module from the standard library has a sqrt
function to calculate the square root of a number. It takes any type that can be converted to float
(which includes int
) as an argument and returns a float
.
1>>> import math
2>>> math.sqrt(9)
33.0
4
The power operator (**
) or the built-in pow()
function can also be used to calculate a square root. Mathematically speaking, the square root of a
equals a
to the power of 1/2
.
The power operator requires numeric types and matches the conversion rules for binary arithmetic operators, so in this case it will return either a float
or a complex
number.
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10
(Note: in Python 2, 1/2
is truncated to 0
, so you have to force floating point arithmetic with 1.0/2
or similar. See Why does Python give the "wrong" answer for square root?)
This method can be generalized to nth root, though fractions that can't be exactly represented as a float
(like 1/3 or any denominator that's not a power of 2) may cause some inaccuracy:
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14
Exponentiation works with negative numbers and complex numbers, though the results have some slight inaccuracy:
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14>>> (-25) ** .5 # Should be 5j
15(3.061616997868383e-16+5j)
16>>> 8j ** .5 # Should be 2+2j
17(2.0000000000000004+2j)
18
Note the parentheses on -25
! Otherwise it's parsed as -(25**.5)
because exponentiation is more tightly binding than unary negation.
Meanwhile, math
is only built for floats, so for x<0
, math.sqrt()
will raise ValueError: math domain error
and for complex x
, it'll raise TypeError: can't convert complex to float
. Instead, you can use cmath.sqrt()
, which is more more accurate than exponentiation (and will likely be faster too):
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14>>> (-25) ** .5 # Should be 5j
15(3.061616997868383e-16+5j)
16>>> 8j ** .5 # Should be 2+2j
17(2.0000000000000004+2j)
18>>> import cmath
19>>> cmath.sqrt(-25)
205j
21>>> cmath.sqrt(8j)
22(2+2j)
23
Both options involve an implicit conversion to float
, so floating point precision is a factor. For example:
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14>>> (-25) ** .5 # Should be 5j
15(3.061616997868383e-16+5j)
16>>> 8j ** .5 # Should be 2+2j
17(2.0000000000000004+2j)
18>>> import cmath
19>>> cmath.sqrt(-25)
205j
21>>> cmath.sqrt(8j)
22(2+2j)
23>>> n = 10**30
24>>> square = n**2
25>>> x = square**.5
26>>> x == n
27False
28>>> x - n # how far off are they?
290.0
30>>> int(x) - n # how far off is the float from the int?
3119884624838656
32
Very large numbers might not even fit in a float and you'll get OverflowError: int too large to convert to float
. See Python sqrt limit for very large numbers?
Let's look at Decimal
for example:
Exponentiation fails unless the exponent is also Decimal
:
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14>>> (-25) ** .5 # Should be 5j
15(3.061616997868383e-16+5j)
16>>> 8j ** .5 # Should be 2+2j
17(2.0000000000000004+2j)
18>>> import cmath
19>>> cmath.sqrt(-25)
205j
21>>> cmath.sqrt(8j)
22(2+2j)
23>>> n = 10**30
24>>> square = n**2
25>>> x = square**.5
26>>> x == n
27False
28>>> x - n # how far off are they?
290.0
30>>> int(x) - n # how far off is the float from the int?
3119884624838656
32>>> decimal.Decimal('9') ** .5
33Traceback (most recent call last):
34 File "<stdin>", line 1, in <module>
35TypeError: unsupported operand type(s) for ** or pow(): 'decimal.Decimal' and 'float'
36>>> decimal.Decimal('9') ** decimal.Decimal('.5')
37Decimal('3.000000000000000000000000000')
38
Meanwhile, math
and cmath
will silently convert their arguments to float
and complex
respectively, which could mean loss of precision.
decimal
also has its own .sqrt()
. See also calculating n-th roots using Python 3's decimal module
QUESTION
pip-compile raising AssertionError on its logging handler
Asked 2022-Feb-13 at 12:37I have a dockerfile that currently only installs pip-tools
1FROM python:3.9
2
3RUN pip install --upgrade pip && \
4 pip install pip-tools
5
6COPY ./ /root/project
7
8WORKDIR /root/project
9
10ENTRYPOINT ["tail", "-f", "/dev/null"]
11
I build and open a shell in the container using the following commands:
1FROM python:3.9
2
3RUN pip install --upgrade pip && \
4 pip install pip-tools
5
6COPY ./ /root/project
7
8WORKDIR /root/project
9
10ENTRYPOINT ["tail", "-f", "/dev/null"]
11docker build -t brunoapi_image .
12docker run --rm -ti --name brunoapi_container --entrypoint bash brunoapi_image
13
Then, when I try to run pip-compile
inside the container I get this very weird error (full traceback):
1FROM python:3.9
2
3RUN pip install --upgrade pip && \
4 pip install pip-tools
5
6COPY ./ /root/project
7
8WORKDIR /root/project
9
10ENTRYPOINT ["tail", "-f", "/dev/null"]
11docker build -t brunoapi_image .
12docker run --rm -ti --name brunoapi_container --entrypoint bash brunoapi_image
13root@727f1f38f095:~/project# pip-compile
14Traceback (most recent call last):
15 File "/usr/local/bin/pip-compile", line 8, in <module>
16 sys.exit(cli())
17 File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
18 return self.main(*args, **kwargs)
19 File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
20 rv = self.invoke(ctx)
21 File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
22 return ctx.invoke(self.callback, **ctx.params)
23 File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
24 return __callback(*args, **kwargs)
25 File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
26 return f(get_current_context(), *args, **kwargs)
27 File "/usr/local/lib/python3.9/site-packages/piptools/scripts/compile.py", line 342, in cli
28 repository = PyPIRepository(pip_args, cache_dir=cache_dir)
29 File "/usr/local/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 106, in __init__
30 self._setup_logging()
31 File "/usr/local/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging
32 assert isinstance(handler, logging.StreamHandler)
33AssertionError
34
I have no clue what's going on and I've never seen this error before. Can anyone shed some light into this?
Running on macOS Monterey
ANSWER
Answered 2022-Feb-05 at 16:30It is a bug, you can downgrade using:
pip install "pip<22"
QUESTION
ImportError: cannot import name 'url' from 'django.conf.urls' after upgrading to Django 4.0
Asked 2022-Feb-10 at 21:14After upgrading to Django 4.0, I get the following error when running python manage.py runserver
1 ...
2 File "/path/to/myproject/myproject/urls.py", line 16, in <module>
3 from django.conf.urls import url
4ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)
5
My urls.py is as follows:
1 ...
2 File "/path/to/myproject/myproject/urls.py", line 16, in <module>
3 from django.conf.urls import url
4ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)
5from django.conf.urls
6
7from myapp.views import home
8
9urlpatterns = [
10 url(r'^$', home, name="home"),
11 url(r'^myapp/', include('myapp.urls'),
12]
13
ANSWER
Answered 2022-Jan-10 at 21:38django.conf.urls.url()
was deprecated in Django 3.0, and is removed in Django 4.0+.
The easiest fix is to replace url()
with re_path()
. re_path
uses regexes like url
, so you only have to update the import and replace url
with re_path
.
1 ...
2 File "/path/to/myproject/myproject/urls.py", line 16, in <module>
3 from django.conf.urls import url
4ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)
5from django.conf.urls
6
7from myapp.views import home
8
9urlpatterns = [
10 url(r'^$', home, name="home"),
11 url(r'^myapp/', include('myapp.urls'),
12]
13from django.urls import include, re_path
14
15from myapp.views import home
16
17urlpatterns = [
18 re_path(r'^$', home, name='home'),
19 re_path(r'^myapp/', include('myapp.urls'),
20]
21
Alternatively, you could switch to using path
. path()
does not use regexes, so you'll have to update your URL patterns if you switch to path.
1 ...
2 File "/path/to/myproject/myproject/urls.py", line 16, in <module>
3 from django.conf.urls import url
4ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)
5from django.conf.urls
6
7from myapp.views import home
8
9urlpatterns = [
10 url(r'^$', home, name="home"),
11 url(r'^myapp/', include('myapp.urls'),
12]
13from django.urls import include, re_path
14
15from myapp.views import home
16
17urlpatterns = [
18 re_path(r'^$', home, name='home'),
19 re_path(r'^myapp/', include('myapp.urls'),
20]
21from django.urls import include, path
22
23from myapp.views import home
24
25urlpatterns = [
26 path('', home, name='home'),
27 path('myapp/', include('myapp.urls'),
28]
29
If you have a large project with many URL patterns to update, you may find the django-upgrade library useful to update your urls.py
files.
QUESTION
How did print(*a, a.pop(0)) change?
Asked 2022-Feb-04 at 21:21This code:
1a = [1, 2, 3]
2print(*a, a.pop(0))
3
Python 3.8 prints 2 3 1
(does the pop
before unpacking).
Python 3.9 prints 1 2 3 1
(does the pop
after unpacking).
What caused the change? I didn't find it in the changelog.
Edit: Not just in function calls but also for example in a list display:
1a = [1, 2, 3]
2print(*a, a.pop(0))
3a = [1, 2, 3]
4b = [*a, a.pop(0)]
5print(b)
6
Prints [2, 3, 1]
vs [1, 2, 3, 1]
. And Expression lists says "The expressions are evaluated from left to right" (that's the link to Python 3.8 documentation), so I'd expect the unpacking expression to happen first.
ANSWER
Answered 2022-Feb-04 at 21:21I suspect this may have been an accident, though I prefer the new behavior.
The new behavior is a consequence of a change to how the bytecode for *
arguments works. The change is in the changelog under Python 3.9.0 alpha 3:
bpo-39320: Replace four complex bytecodes for building sequences with three simpler ones.
The following four bytecodes have been removed:
- BUILD_LIST_UNPACK
- BUILD_TUPLE_UNPACK
- BUILD_SET_UNPACK
- BUILD_TUPLE_UNPACK_WITH_CALL
The following three bytecodes have been added:
- LIST_TO_TUPLE
- LIST_EXTEND
- SET_UPDATE
On Python 3.8, the bytecode for f(*a, a.pop())
looks like this:
1a = [1, 2, 3]
2print(*a, a.pop(0))
3a = [1, 2, 3]
4b = [*a, a.pop(0)]
5print(b)
6 1 0 LOAD_NAME 0 (f)
7 2 LOAD_NAME 1 (a)
8 4 LOAD_NAME 1 (a)
9 6 LOAD_METHOD 2 (pop)
10 8 CALL_METHOD 0
11 10 BUILD_TUPLE 1
12 12 BUILD_TUPLE_UNPACK_WITH_CALL 2
13 14 CALL_FUNCTION_EX 0
14 16 RETURN_VALUE
15
while on 3.9, it looks like this:
1a = [1, 2, 3]
2print(*a, a.pop(0))
3a = [1, 2, 3]
4b = [*a, a.pop(0)]
5print(b)
6 1 0 LOAD_NAME 0 (f)
7 2 LOAD_NAME 1 (a)
8 4 LOAD_NAME 1 (a)
9 6 LOAD_METHOD 2 (pop)
10 8 CALL_METHOD 0
11 10 BUILD_TUPLE 1
12 12 BUILD_TUPLE_UNPACK_WITH_CALL 2
13 14 CALL_FUNCTION_EX 0
14 16 RETURN_VALUE
15 1 0 LOAD_NAME 0 (f)
16 2 BUILD_LIST 0
17 4 LOAD_NAME 1 (a)
18 6 LIST_EXTEND 1
19 8 LOAD_NAME 1 (a)
20 10 LOAD_METHOD 2 (pop)
21 12 CALL_METHOD 0
22 14 LIST_APPEND 1
23 16 LIST_TO_TUPLE
24 18 CALL_FUNCTION_EX 0
25 20 RETURN_VALUE
26
In the old bytecode, the code pushes a
and (a.pop(),)
onto the stack, then unpacks those two iterables into a tuple. In the new bytecode, the code pushes a list onto the stack, then does l.extend(a)
and l.append(a.pop())
, then calls tuple(l)
.
This change has the effect of shifting the unpacking of a
to before the pop
call, but this doesn't seem to have been deliberate. Looking at bpo-39320, the intent was to simplify the bytecode instructions, not to change the behavior, and the bpo thread has no discussion of behavior changes.
Community Discussions contain sources that include Stack Exchange Network