Popular New Releases in Python
youtube-dl
youtube-dl 2021.12.17
models
TensorFlow Official Models 2.7.1
thefuck
transformers
v4.18.0: Checkpoint sharding, vision models
flask
Popular Libraries in Python
by public-apis python
184682
MIT
A collective list of free APIs
by donnemartin python
143449
NOASSERTION
Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
by TheAlgorithms python
117097
MIT
All Algorithms implemented in Python
by jackfrued python
114192
Python - 100天从新手到大师
by ytdl-org python
108335
Unlicense
Command-line program to download videos from YouTube.com and other video sites
by vinta python
102379
NOASSERTION
A curated list of awesome Python frameworks, libraries, software and resources
by tensorflow python
73392
NOASSERTION
Models and examples built with TensorFlow
by nvbn python
65678
MIT
Magnificent app which corrects your previous console command.
by django python
63447
NOASSERTION
The Web framework for perfectionists with deadlines.
Trending New libraries in Python
by ultralytics python
25236
GPL-3.0
YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite
by yt-dlp python
22499
Unlicense
A youtube-dl fork with additional features and fixes
by babysor python
20425
NOASSERTION
🚀AI拟声: 5秒内克隆您的声音并生成任意语音内容 Clone a voice in 5 seconds to generate arbitrary speech in real-time
by beurtschipper python
19784
NOASSERTION
Recovers passwords from pixelized screenshots
by PaddlePaddle python
19581
Apache-2.0
Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)
by TencentARC python
17269
NOASSERTION
GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.
by github python
16816
CC-BY-4.0
Documentation for GitHub Copilot
by mingrammer python
16552
MIT
:art: Diagram as Code for prototyping cloud system architectures
by jina-ai python
14316
Apache-2.0
Cloud-native neural search framework for 𝙖𝙣𝙮 kind of data
Top Authors in Python
1
4429 Libraries
28
2
3923 Libraries
0
3
2232 Libraries
604
4
1263 Libraries
3038
5
723 Libraries
0
6
690 Libraries
1
7
628 Libraries
0
8
627 Libraries
23550
9
550 Libraries
33409
10
512 Libraries
20636
1
4429 Libraries
28
2
3923 Libraries
0
3
2232 Libraries
604
4
1263 Libraries
3038
5
723 Libraries
0
6
690 Libraries
1
7
628 Libraries
0
8
627 Libraries
23550
9
550 Libraries
33409
10
512 Libraries
20636
Trending Kits in Python
With this kit, you can
1. Use a pre-trained model for detecting fake news.
2. Train the model on your custom dataset.
3. Expose the fake news detection as an API
Add-on on examples are also included as given below
1. Use web scraper to automatically make your training dataset.
2. Visualise training and prediction data for useful insights.
Instructions to Run
Follow the below instructions to run the solution.
1. Locate and open the FakeNewsDetection-starter.ipynb notebook from the Jupyter Notebook browser window.
2. Execute cells in the notebook by selecting Cell --> Run All from Menu bar
3. Once all the cells of the notebook are executed, the prediction result will be written to the file 'fake_news_test_output.csv'
Training with your dataset:
1. Add news articles to a csv file under a column name 'news_text'.
2. Add corresponding labels as 'real' or 'fake' denoting whether a news article is real or not.
3. You can refer to the file 'fake_news_train.csv' for an example.
4. Set the variable for training file in the notebook under Variables section.
Testing with your dataset:
1. Add news articles to a csv file under a column name 'news_text'.
2. You can refer to the file 'fake_news_test.csv' for an example.
3. Set the variable for testing file in the notebook under Variables section.
You can execute the cells of notebook by selecting Cell from the menu bar.
For any support, you can reach us at FAQ & Support
Libraries useful for this solution
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers. Jupyter Notebook is used for our development.
Exploratory Data Analysis
For extensive analysis and exploration of data, and to deal with arrays, these libraries are used. They are also used for performing scientific computation and data manipulation.
Text mining
Libraries in this group are used for analysis and processing of unstructured natural language. The data, as in its original form aren't used as it has to go through processing pipeline to become suitable for applying machine learning techniques and algorithms.
Machine Learning
Machine learning libraries and frameworks here are helpful in providing state-of-the-art solutions using Machine learning.
Data Visualization
The patterns and relationships are identified by representing data visually and below libraries are used for generating visual plots of the data.
Troubleshooting
1. If you encounter any error related to MS Visual C++, please install MS Visual Build tools
2.While running batch file, if you encounter Windows protection alert, select More info --> Run anyway.
3.During kit installer, if you encounter Windows security alert, click Allow.
4. If you encounter Memory Error, check if the available memory is sufficient and it is proportion to the size of the data being used. For our dataset, the minimum required memory is 8GB.
If your computer doesn't support standard commands from windows 10, you can follow the instructions below to finish the kit installation.
1. Click here to install python
2. Click here to download the repository
3. Extract the zip file and navigate to the directory 'fakenews-detection-main'
4. Open terminal in the extracted directory 'fakenews-detection-main'
5. Install dependencies by executing the command 'pip install -r requirements.txt'
6. Run the command ‘jupyter notebook’ and select the notebook ‘FakeNewsdetection-starter.ipynb’ on the browser window.
Support
For any support, you can reach us at FAQ & Support
The use case of AI Course Recommender System is to provide personalized recommendation to the user based on their interest, course they can take and their current knowledge. This system will be able to recommend course based on user’s interest, current knowledge, analytical view of students’ performance in mathematics and recommends if a student can consider math subject for his/ her higher education. The recommended course will be based on the information of user’s profile, analysis of grades of students, visualization of patterns, prediction of grade in final test, and some rules that were set by their instructor. Using machine learning algorithms, we can train our model on a set of data and then predict the ratings for new items. This is all done in Python using numpy, pandas, matplotlib, scikit-learn and seaborn. kandi kit provides you with a fully deployable AI Course Recommender System. Source code included so that you can customize it for your requirement.
Development Environment
VSCode and Jupyter Notebook are used for development and debugging. Jupyter Notebook is a web based interactive environment often used for experiments, whereas VSCode is used to get a typical experience of IDE for developers.
Data Mining
Our solution integrates data from various sources, and we have used below libraries for exploring patterns in these data and understanding correlation between the features.
Data Visualisation
The patterns and relationships are identified by representing data visually and below libraries are used.
Machine learning
Below libraries and model collections helps to create the machine learning models for the core prediction of use case in our solution.
Describe your solution here. e.g. Our solution is a Mental Health Monitor and Virtual Companion for students in the digital classroom. It acts as a virtual friend to the student and keeps her company during online classes. It also reminds her of class schedules, submissions, and other aspects of the classroom. It monitors for mental wellbeing vitals like class participation, connecting with friends, and sends motivational messages. In future versions, we can add connect with parents, teachers, and other stakeholders to address the well-being and academic excellence in a holistic manner. We have used the below techniques for our solution. 1. Machine learning to train for the virtual conversations for the virtual companion. 2. Data exploration to explore the academic and digital activity data to use in training for predicting behavior patterns
Machine Learning
You may be using multiple libraries for different functions in your solution. Please create one group for each function and add the libraries to each group. e.g. I have created a group for Machine Learning which has the libraries used in my solution. The below libraries helps in capturing the embeddings for the text. The embeddings are vectoral representation of text with their semantics.
Data Exploration
You may be using multiple libraries for different functions in your solution. Please create one group for each function and add the libraries to each group. e.g. I have created a group for the libraries used for Data Exploration in my solution. The data exploration helps in doing extensive analysis of different data types and in assisting to understand the patterns. Pandas is used in our solution for data manipulation and analysis.
We are AI-Wave team and we present Moody, a platform that allows us to monitor and assist student’s mental health through time, by using natural language processing and system recommendation algorithms, day to day student’s information, virtual assistant, emotional tracking and a real time dashboard. Website (MVP) : https://buildwithai-aiwave.netlify.app Solution info : https://docs.google.com/document/d/1AQqLaMC4d99qQ34ylzDyK2JHvoobNgjogHfQld03kQs/edit Youtube video: https://www.youtube.com/watch?v=kEx9w8sUhMA Github: https://github.com/shahbaazkyz/AI-Wave
Development Environment
We use Jupyter for development environment
Machine Learning
All the machine learning libraries used for the solution
Data Manipulation & Adquisition
All the libraries used for data adquisition and manipulation
MVP
The link to the website of the solution is : https://buildwithai-aiwave.netlify.app
This kit is created for Impact of COVID 19 on the mental health of students by team Phoenix
Development Environment
Library related to development and debugging environment
Machine Learning (overall implementation)
Library for using Machine Learning algorithm
Data Manipulation
Libraries for data manipulation
Data Visualization
Libraries for data visualization
This is the Online Class Companion Project by Team Geeky Plats
Development Environments
These are the development environments used for the project.
Libraries Used
The various libraries used in the project
Student well beings help us to predict mental disorder and parent satisfaction based on the the students information. Both are binary classification problem and we used to solved by traditional machine learning algorithms. This is the project from Team Neuron Fire.
Development Environment
We used Jupyter Notebooks by developing and debugging. Jupyter Notebook is a web based interactive environment often used for experiments.
Data Mining
We used pandas and numpy for Data Mining.
Data Visualizations
We used matplotlib and seaborn python libraries for data visualizations.
Machine learning
We used Sklearn python libraries for binary classification problem solve.
We are Team HungryHacks trying to solve food insecurity issues using Data Science and AI.
Group Name 1
Data Preprocessing
Group Name 2
Data Exploration and Visualization
Group Name 6
Front-end Development
Group Name 5
GITHUB Repo
This is the Project present by TeamAlphaPro
Development Environment
We used Jupiter Notebook for development and debugging
Data Mining
We used Numpy and Pandas for Data Mining
Machine Learning
We used scikit-learn for Machine Learning
Development Environment
Vs code : used to write python scripts for data preprocessing and deployment jupyter notebook : used for data analysis and model training streamlit : used for model deployment and web app creation heroku : used for deployment of our web app to make it accessible globally
Data Mining
numpy : used for computational mathematics Pandas : used for loading data and Exploratory data analysis
Data Visualization
Machine Learning
Sklearn : used for training our ML models and evaluation joblib : used to store and load our ML model
Build WIth AI 2021 Challenge 2 Kit Submission
Machine Learning
Libraries used for voice activity detection(VAD), embedding extraction and spectral embedding clustering.
Audio Processing
Libraries used for importing, converting, exporting, playing, and processing audio.
User Interface
Libraries used for user interface.
Miscellaneous
Other libraries used.
Data Summary : Resources that have been shared for the problem statement has info about food items and their description. Also we had order info from both donor and from consumer side orders on daily basis. We have done data cleaning and preprocessing as required. Recommendation system : In order to control the food wastage we have built Recommendation engine using "item-item based collaborative filtering" to recommend the items which expire early and are more in consumption. Data Analysis : We have developed a dashboard on tableau using cleaned datasets and these analysis can be used to match supply-demand of different types of food and to give an overview to the NGO, donors and consumers on how to reduce the food wastage. Use the open source, cloud APIs, or public libraries listed below in your application development based on your technology preferences, such as primary language. The below list also provides a view of the components' rating on different dimensions such as community support availability, security vulnerability, and overall quality, helping you make an informed choice for implementation and maintenance of your application. Please review the components carefully, having a no license alert or proprietary license, and use them appropriately in your applications. Please check the component page for the exact license of the component. You can also get information on the component's features, installation steps, top code snippets, and top community discussions on the component details page. The links to package managers are listed for download, where packages are readily available. Otherwise, build from the respective repositories for use in your application. You can also use the source code from the repositories in your applications based on the respective license types.
analysis
machine learning
this is related to water treatment and purification
DataViz Group
This helps with DaaViz
Cyber Data Analytics Group
This is for Cyber Data Analytics Group
zeronet group
this is zeronet group
Automation Group
this is part of Automation Group
We can create any animation with suitable libraries or combinations of libraries in Python that are well-known for their functionalities and functionalities. For developers looking for options with less complex code with maximum customization options, users can customize their plots and designs depending on their preferences.
Choosing a suitable library plays a key role in any machine learning or data science project, and we should do it properly to avoid other related issues which may arise. Some libraries offer an interactive plot to attract playing with the graph and visualize it uniquely. It will allow you to edit videos, create animations, and create a map or geological animations where we can analyze the geological data.
Here is the list of handpicked 18 best Python Animation Libraries in 2023 which will help you with your animation requirements:
manim - 3b1b:
- Is a Python library for creating mathematical animations and educational videos that Grant Sanderson develops.
- Is an open source library that allows users to create high-quality animations which visualize mathematical concepts like animations of graphs, functions, fractals, and more.
- Uses Python code for creating animations which we can export to animated GIFs or video files.
PythonRobotics:
- Is a Python library for implementing different robotics simulations, visualizations, and algorithms.
- Offers various resources and tools for robotics developers, like algorithms for path planning, localization, motion control, mapping, and many more.
- Includes simulation environments like 2D and 3D simulators, that will allow developers to test their algorithms in virtual environments before deploying them on real robots.
matplotlib:
- Is a comprehensive library for creating animated, interactive visualizations and static in Python.
- Produces publication-quality figures in different interactive environments and hardcopy formats across platforms.
- Can be used in Python Scripts, web application servers, various graphical user interface toolkits, and Python/IPython shells.
manim - ManimCommunity:
- Is an animation engine for explanatory math videos used for programmatically creating precise animations.
- Includes various tools for creating animations like support for vector graphics, 3D objects, and complex mathematical equations.
- Also includes features for creating animations with custom fonts, styles, and colors.
plotly.py:
- Is a Python library used to create interactive data visualizations, built on the plotly JavaScript library that allows developers to create various interactive plots.
- Is designed to be easy to use and includes different resources and tools for creating high-quality visualizations.
- Includes support for complex data structures like pandas DataFrames and offers various customization options for fonts, styles, and colors.
seaborn:
- Is a Python data visualization library based on Matplotlib, offering a high-level interface to create attractive and informative statistical graphics.
- Offers various plotting functions for visualizing various data types like continuous data, data distribution, and categorial data.
- Its the ability to create visually appealing plots with minimal effort and supports the customization of plot elements like axes, titles, legends, and labels.
moviepy:
- Is a Python library for video editing, concatenations, cutting, video composting, title insertions, creation of custom effects, and video processing.
- Has the ability to add audio to video clips easily and offers various filters and audio effects like changing pitch and speed, adding sound effects, and adjusting volume.
- Includes support for creating animations like moving text, images, and shapes and allows users to export their video clips to different file formats.
termtosvg:
- Is a Python library that allows users to record terminal sessions and save them as SVG animations.
- Produces clean-looking and lightweight still frames embeddable on the project page or animations.
- Includes support for recording multiple terminal sessions, allowing users to control the size and speed of the resulting animation.
altair:
- Is a declarative statistical visualization library that can help you spend more time understanding your data and its meaning.
- Offers a simple syntax for creating different visualizations, like line charts, histograms, scatterplots, and bar charts.
- Its declarative syntax lets user's express visualizations as a series of high-level mappings between visual and data properties like color, size, and position.
PathPlanning:
- Is a Python library used for path and motion planning applications designed to be accessible to beginners and experts with a straightforward API.
- Offers various algorithms for computing collision-free paths for drones, mobile robots, and manipulators in 2D and 3D environments.
- Also offers tools for trajectory generation, motion control, and obstacle avoidance and supports simulation and visualization of robot motion.
alive-progress:
- Is a Python library for displaying spinners and progress bars in command-line applications designed to offer a customizable way of showing progress indicators for long-running processes or tasks.
- Supports for pausing and resuming progress indicators, nested spinners, and progress bars.
- Designed to be intuitive and simple with various default settings and a straightforward API for customizing the behavior and appearance of spinners and progress bars.
asciimatics:
- Is a package for helping people create full-screen text UIs on any platform and offers a single cross-platform Python class to do all the low-level console functions.
- Includes cursor positioning, mouse input, screen scraping, colored/styled text, detecting and handling if the console resizes, and keyboard input like Unicode support.
- Is a Python library for creating text-based animations and user interfaces in the terminal.
pygal:
- Is a Python library for creating interactive Scalable Vector Graphics (SVG) graphs and charts.
- Offers various tools for generating customizable charts, graphs, and high-quality for use in presentations, reports, and web applications.
- Includes built-in support for data/time axis labeling, responsive design, and integration with web frameworks and interactive charts elements.
GANimation:
- Is a Python implementation of the GANimation research project, which offers various tools for generating animations from still images using Generative Adversarial Networks (GANs).
- Includes tools for augmenting and preprocessing input data, customizable GAN training parameters and architecture, and support for evaluating and visualizing GAN models.
- Offers various tools for fine-tuning GAN models and generating high-quality animations for various applications.
deep-motion-editing:
- Offers advanced and fundamental functions to work with 3D character animations in deep learning with Pytorch.
- Is a Python implementation of the research project of the same name, which offers tools for editing the motion of human characters in video sequences using deep learning methods.
- Its ability to generate realistic, high-quality animations for various applications offers tools for fine-tuning the deep learning model and editing the generated motions to achieve the desired results.
geoplotlib:
- Is a Python library for creating geographical maps and visualizations and offers an easy-to-use interface for creating maps with different data types, like polygons, heatmaps, lines, and points.
- Includes support for different tile providers and map projections, customizable styling options for data layers like size, transparency, and color.
- Designed for creating interactive maps and visualizations and is suitable for various applications like data analysis, presentation, and exploration.
Linux-Fake-Background-Webcam:
- Is a Python library that will allow users to replace their webcam background with a custom video or image on Linux systems.
- Works by creating a virtual webcam device that can be selected as the input source in video conferencing applications, allowing users to appear as if they are in various environments and locations.
- Includes the ability to control the position and size of the custom background video or image and support for replacing the webcam background with a custom video or audio.
celluloid:
- Is a Python library that offers a simple interface for creating visualizations and animations in Matplotlib
- Designed to make it easy for users to create animations without having to write to deal with low-level details and complex code.
- Includes a simple interface for adding and updating data in the animation, the ability to save the animation as an MP4 or GIF video file, and support for customizing the animation style and appearance.
Use these Python Machine Learning libraries for developing supervised and unsupervised learning algorithms, data pre-processing and feature extraction tools, deep learning frameworks and more.
Following are the top use cases of these shortlisted libraries for Python Machine Learning,
· Pre-processing of data that includes data cleaning and feature engineering tasks such as normalization, imputation, missing value treatment, and outlier detection.
· Model selecting and optimizing, such as cross-validation, hyperparameter tuning, and model selection metrics.
· Visualizations to understand data and results. This includes visualizing data distributions, feature importance, and model performance.
· Developing algorithms, including supervised learning algorithms (e.g. linear regression, logistic regression, support vector machines, decision trees, random forests, and neural networks) as well as unsupervised learning algorithms (e.g. clustering, dimensionality reduction, and anomaly detection).
· Calculating performance metrics such as accuracy, precision, recall, and F1 score.
The following is a list of the 18 most popular open-source Python libraries for Machine Learning,
Dilbert was dropped from hundreds of newspapers over Scott Adams’ racist comments. Multiple researchers have documented over the past few months how ChatGPT can be prompted to provide racist responses.
A three-decade globally famous comic strip has been canceled because of the creator’s racist comments in his YouTube show. ChatGPT, Bing Bot, and many such AI Bots are conversing with millions of users daily and have been documented to provide misleading, inaccurate, and biased responses. How can we hold AI to the same high standards we expect from society, especially when AI is now generative and scaled for global consumer use?
While no silver bullet exists, multiple aspects can make AI more responsible. Having open AI models is a great start. Hugging Face, EleutherAI, and many others are championing an open approach to AI. Openness and collaboration can bring in diverse contributions, reviews, and rigorous testing of AI models and help reduce bias.
NIST’s AI risk management guidelines released recently provide a comprehensive view across the AI lifecycle consisting of collecting and processing Data & Input, the build, and validation of the AI model, its deployment, and monitoring in the context of usage. Acknowledging the possibility of bias, eliminating data capture biases, or unconscious biases when generating synthetic data, designing for counterfactual fairness, and human-in-loop designs can reduce the risk of bias.
Use the below tools for assessment and to improve the fairness and robustness of your models.
Use the below tools for Explainability, Interpretability, and Monitoring.
Google toolkit on Tensorflow for Privacy, Federated Learning, and Explainability.
SPEAKER COUNTING It enhances understanding through automatic speech recognition Beneficial for real - world applications like call-center transcription and meeting transcription analytics Speaker Diarization is a developing field of study, with new approaches being published on a frequent basis. The Problem Not many studies have been done for estimating a large number of speakers. Diarization becomes extremely difficult when the number of speakers is huge. Providing the number of speakers to the diarization system can be advantageous Complete solution Architecture - Machine Learning model - To predict the no. of speakers and the time stamps of the speaker. Web App - Frontend for the user to use the feature. Middleware Flask Api - To connect Frontend and ML Model. We have build a Web App that a user can use to communicate and leverage the advantages of the our Machine learning model. Since the model we build and the web app are build on different platforms, we used REST API as a middleware to connect frontend and model.
ML Model Solution Process
These are used to create our Web UI using node as backend and VueJs as front end. 1. Preprocessing: Denoising -> Speech separation 2. Embedding Extraction: YAMNet sound & classification model 3. Speaker Counting: Machine learning model selection -> Model training -> Model prediction
Data Preprocessing
Technologies used for pre processing the audio data.
Audio Pre Processing
The additional libaries are use to processing the audio which are needed to be fed into the classifier model.
Model Trainning
This libaries are used to create the two classifier models which are then both combined into one.
ChatGPT is creating huge interest across all fields, including application development. You can use ChatGPT in your applications to provide additional intelligence. Here are some cool applications you can leverage ChatGPT for.
Start with Andrej Karpathy’s NanoGPT to get a feel for how GPT works at a smaller scale, even in your laptop.
ChatGPT Prompts is a great repo to help you leverage ChatGPT personas such as a translator, interviewer, JavaScript console, UI developer, IT architect, Excel sheet, stand-up comedian, philosopher, plagiarism checker, doctor, financial analyst, and many more.
If you'd like to use ChatGPT on your desktop try ChatGPT by lencx.
Up your chat game with ChatGPT apps for WhatApp, Telegram and Discord.
If you would like to use ChatGPT to compare with Google results from your browser, or integrate ChatGPT into your IDE or a general-purpose API to integrate with any other application like Telegram and others to create your custom ChatGPT integrations, below are useful repositories. Try them to build and get familiar with ChatGPT use cases.
Here are the best open source Python Telegram Bot libraries for your application. You can use these components to build an intelligent telegram bot that can automatically or on request send text, video, images, documents, and other information and perform different activities.
Using a Python Telegram Bot Library, you can Broadcast, Teach, Collect Leads, Search, Reply, Remind, Play, Connect, etc. To help developers, these bots act as a BotFather. Also, you can implement the pair-to-pair security protocol and end-to-end encryption to ensure that each exchange of messages between the Bot and the user is secure.
We have handpicked the top and trending Python Telegram Bot libraries for your next project below.
python-telegram-bot:
- Used for building a bot in telegram easily by coupling with Python libraries like Flask, Requests, and Viz.
- Provides an asynchronous interface for the Telegram Bot API.
- The library features many high-level classes to make the development straightforward.
Telethon:
- An asynchronous Python 3 MTProto library that helps interact with Telegram’s API as a user through a bot API alternative.
- If we install cryptg, the library will work faster as the encryption and decryption will be done using C instead of Python.
- If we install pillow, larger images will be automatically resized while sending photos to prevent telegram from failing with ‘Invalid Image’ messages.
pyTelegramBotAPI:
- Used as the Python implementation for the telegram bot as it supports both synchronous and asynchronous methods.
- Provides functions like send_message, send_xyz, send_document, etc. And listens to all incoming messages.
- Can have an arbitrary name, but it should have only one parameter: the message.
aiogram:
- Used for building bots quickly and efficiently using the available template and a fully asynchronous framework for Telegram Bot API.
- Used in Bot applications and Automation.
- Can reply into webhook, i.e., making requests in response to any latest updates.
pyrogram:
- Modern, elegant, and asynchronous Telegram MTProto API framework in Python for bots and users.
- Enables you to easily interact with the main Telegram API using a user interface or a bot API alternative using Python.
- Types and methods are all type-hinted, which will enable excellent editor support.
telepot:
- Helps build applications for Telegram Bot API and works on Python 2.7 and Python 3.
- Use telepot.glance() function for extracting the headline information.
- Supports synchronous and asynchronous methods of programming.
mtprotoproxy:
- Fully asynchronous and can process a lot of connections.
- Not just a tool but has an API that can help customize the Telegram MTProto proxy.
- Can be used for logging, limiting access, and creating proxy farms that are hard to filter.
BrainDamage:
- Used in Runtime environments, Docker applications, and Servers.
- Can destroy the active slaves, remove the stub from the host and registry entries, run shell commands on the host, and download files on a host computer.
- Used for Artificial Intelligence and Machine Learning.
informer:
- Used in Docker applications, Bot, and Automation.
- Allows you to masquerade as multiple REAL users on telegram and spy on 500+ Telegram channels per account.
- Is a Telegram Mass Surveillance Bot in Python.
TeleGram-Scraper:
- Used to export competitor groups, channel members, and add them to your own group or channel.
- You can scrape search results and extract the contents produced from those search results.
- Supports telegram premium API.
telebot:
- Provides the best-of-its-kind API for command routing, keyboards, and inline query requests and callbacks.
- Are Concise API, supports Command routing, Middleware, Effortless bot callbacks, and Transparent File API.
- Is a highload-ready solution that has APIs that are easy to memorize and use.
mautrix-telegram:
- A hybrid Matrix- telegram puppeting or relaybot bridging library.
- Has 2-factor authentication enabled for logging in with a bot token.
- Includes a simple script to help migration between different database management systems.
python-aria-mirror-bot:
- A telegram bot for mirroring files on the internet to our Google Drive or Telegram.
- Supports Mirroring direct download links to Google Drive, Upload and download progress, Docker support, Download or upload speed and ETAs, Index Link support, and many more.
- Stops duplicates for all tasks except for qBittorrent and youtube-dl tasks.
TorrentLeech-Gdrive:
- Based on the Pyrogram library and Leecher.
- Supports Telegram file mirroring to the cloud with its unzipping, untar, and unrar.
- Help change the rclone destination configuration on the fly.
Here are the top Python Email Libraries you will need to get built-in support to send emails using the SMTP protocol. The smtplib module will define an SMTP client session object which we can use for sending email to any machine connected to the internet with an ESMTP or SMTP listener daemon.
We can use Python for sending an email in many ways, like an email with an attachment, an HTML/Markdown-formatted email message, or plain text. Many third-party libraries are available in the market, like EmailReplyParser for parsing the response from email threads, EmailMessage module in Python’s Standard Library, Mailgun-py, which is an official and actively maintained library, and Flanker which is a parsing and validation library.
We have handpicked the top 15 Python Email Libraries in 2023 which fulfill all the developer requirements:
GHunt:
- Is an offensive Google Framework that is designed to evolve effectively.
- Is fully asynchronous and uses CLI & modules.
- Will automatically use venvs to avoid dependency conflicts with other projects.
mailinabox:
- Helps us in taking back control of our email by defining an easy-to-deploy, one-click SMTP+everything else server: a mail server in a box.
- Promotes innovation, privacy, and decentralization on the web.
- Has auditable, idempotent, and automated configuration.
Mailpile:
- Is a fast web-mail client, modern, with user-friendly privacy and encryption features.
- Places emphasize providing an elegant user interface and a clean and pleasant user experience.
- Allows email to be organized with tags and lets us automatically configure tag incoming email depending on Bayesian classifiers or static rules.
theHarvester:
- Is a simple yet powerful tool designed for use during the reconnaissance stage of a penetrated or a red team assessment test.
- Performs Open Source Intelligence (OSINT) by gathering help in determining a domain’s external threat landscape.
- Gathers emails, subdomains, URLs, names, and IPs with the help of multiple public resources.
pyWhat:
- Will easily let you identify IP addresses, emails, and more.
- Helps in filtering outputs, exporting, and sorting according to your requirements.
- Supports filters like bug bounties and many more.
sync-engine:
- Offers a RESTful API on top of a powerful email sync platform to make it easier to build apps on top of the email.
- The API service will provide a REST API to help interact with your data.
- The sync engine will automatically begin synchronizing your account with your current provider.
yagmail:
- Is an SMTP/GMAIL client which aims to make sending emails the simplest and easiest task.
- With yagmail, sending emails to many people by offering a list of email strings rather than mentioning a single string is possible.
- Is a wrapper library on top of smtplib that is a standard library to send emails using Python.
mailin:
- Is an SMTP server that listens to emails, parses them, and posts them in json format to the URL you choose.
- Checks incoming email’s spf, spam score, and dkim to tell you the language in which it is written.
- Can be used as a standalone application embedded inside a node application or directly from the command line.
offlineimap:
- Is software that helps download your email mailbox or mailboxes as local Maildirs.
- OfflineIMAP will synchronize both sides using IMAP.
- Allows you to read your email offline without requiring your mail reader to support your IMAP operations.
gmail:
- Is a Pythonic interface for Google’s Gmail with all the necessary tools.
- Will search, archive, delete, read, and send multipart emails, manage labels, and mark as read/unread emails.
- Connects with live Gmail for accessing real-time data.
flanker:
- Is an open source parsing library in Python which contains an address and a MIME parsing library.
- Both can parse addresses, validate them, and convert headers to Unicode, which detects and preserves encodings when possible.
- Stores the message in a single string, tracks changes, and returns unchanged parts during serialization.
inbox.py:
- Is an asynchronous simplest SMTP server.
- Can handle one instance of over one thousand emails per second.
- Is a library for managing email inboxes that provides an API for creating, deleting, reading, and retrieving emails.
django-anymail:
- Let's send and receive an email in Django with your choice of transactional Email Service Provider (ESPs).
- Includes integration to each ESP’s sending APIs into Django’s built-in email package, simplified inline images from HTML email, and ‘Batch Transactional’ sends with your ESP’s merging and templating features.
- Maintains compatibility with all Django versions and is extensively tested on all Python versions which Django supports.
python-o365:
- Aims in making interact with Office 365 and Microsoft Graph in a Pythonic and easy way.
- Helps with access to Calendar, OneDrive, Email, and Contacts.
- Provides automatic handling between local and server datetimes.
SimplyEmail:
- Is an email recon that is easy and quick using the framework built on.
- Uses built-in Parsers for the rawest results.
- Multiprocessing Queue for modules and Results Queue for easy Email data handling.
Trending Discussions on Python
Python/Docker ImportError: cannot import name 'json' from itsdangerous
Why is it faster to compare strings that match than strings that do not?
Why is `np.sum(range(N))` very slow?
Error while downloading the requirements using pip install (setup command: use_2to3 is invalid.)
Repeatedly removing the maximum average subarray
WARNING: Running pip as the 'root' user
How do I calculate square root in Python?
pip-compile raising AssertionError on its logging handler
ImportError: cannot import name 'url' from 'django.conf.urls' after upgrading to Django 4.0
How did print(*a, a.pop(0)) change?
QUESTION
Python/Docker ImportError: cannot import name 'json' from itsdangerous
Asked 2022-Mar-31 at 12:49I am trying to get a Flask and Docker application to work but when I try and run it using my docker-compose up
command in my Visual Studio terminal, it gives me an ImportError called ImportError: cannot import name 'json' from itsdangerous
. I have tried to look for possible solutions to this problem but as of right now there are not many on here or anywhere else. The only two solutions I could find are to change the current installation of MarkupSafe and itsdangerous to a higher version: https://serverfault.com/questions/1094062/from-itsdangerous-import-json-as-json-importerror-cannot-import-name-json-fr and another one on GitHub that tells me to essentially change the MarkUpSafe and itsdangerous installation again https://github.com/aws/aws-sam-cli/issues/3661, I have also tried to make a virtual environment named veganetworkscriptenv
to install the packages but that has also failed as well. I am currently using Flask 2.0.0 and Docker 5.0.0 and the error occurs on line eight in vegamain.py.
Here is the full ImportError that I get when I try and run the program:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10
Here are my requirements.txt, vegamain.py, Dockerfile, and docker-compose.yml files:
requirements.txt:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10Flask==2.0.0
11Flask-SQLAlchemy==2.4.4
12SQLAlchemy==1.3.20
13Flask-Migrate==2.5.3
14Flask-Script==2.0.6
15Flask-Cors==3.0.9
16requests==2.25.0
17mysqlclient==2.0.1
18pika==1.1.0
19wolframalpha==4.3.0
20
vegamain.py:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10Flask==2.0.0
11Flask-SQLAlchemy==2.4.4
12SQLAlchemy==1.3.20
13Flask-Migrate==2.5.3
14Flask-Script==2.0.6
15Flask-Cors==3.0.9
16requests==2.25.0
17mysqlclient==2.0.1
18pika==1.1.0
19wolframalpha==4.3.0
20# Veganetwork (C) TetraSystemSolutions 2022
21# all rights are reserved.
22#
23# Author: Trevor R. Blanchard Feb-19-2022-Jul-30-2022
24#
25
26# get our imports in order first
27from flask import Flask # <-- error occurs here!!!
28
29# start the application through flask.
30app = Flask(__name__)
31
32# if set to true will return only a "Hello World" string.
33Debug = True
34
35# start a route to the index part of the app in flask.
36@app.route('/')
37def index():
38 if (Debug == True):
39 return 'Hello World!'
40 else:
41 pass
42
43# start the flask app here --->
44if __name__ == '__main__':
45 app.run(debug=True, host='0.0.0.0')
46
Dockerfile:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10Flask==2.0.0
11Flask-SQLAlchemy==2.4.4
12SQLAlchemy==1.3.20
13Flask-Migrate==2.5.3
14Flask-Script==2.0.6
15Flask-Cors==3.0.9
16requests==2.25.0
17mysqlclient==2.0.1
18pika==1.1.0
19wolframalpha==4.3.0
20# Veganetwork (C) TetraSystemSolutions 2022
21# all rights are reserved.
22#
23# Author: Trevor R. Blanchard Feb-19-2022-Jul-30-2022
24#
25
26# get our imports in order first
27from flask import Flask # <-- error occurs here!!!
28
29# start the application through flask.
30app = Flask(__name__)
31
32# if set to true will return only a "Hello World" string.
33Debug = True
34
35# start a route to the index part of the app in flask.
36@app.route('/')
37def index():
38 if (Debug == True):
39 return 'Hello World!'
40 else:
41 pass
42
43# start the flask app here --->
44if __name__ == '__main__':
45 app.run(debug=True, host='0.0.0.0')
46FROM python:3.9
47ENV PYTHONUNBUFFERED 1
48WORKDIR /app
49COPY requirements.txt /app/requirements.txt
50RUN pip install -r requirements.txt
51COPY . /app
52
docker-compose.yml:
1veganetworkscript-backend-1 | Traceback (most recent call last):
2veganetworkscript-backend-1 | File "/app/vegamain.py", line 8, in <module>
3veganetworkscript-backend-1 | from flask import Flask
4veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 19, in <module>
5veganetworkscript-backend-1 | from . import json
6veganetworkscript-backend-1 | File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 15, in <module>
7veganetworkscript-backend-1 | from itsdangerous import json as _json
8veganetworkscript-backend-1 | ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)
9veganetworkscript-backend-1 exited with code 1
10Flask==2.0.0
11Flask-SQLAlchemy==2.4.4
12SQLAlchemy==1.3.20
13Flask-Migrate==2.5.3
14Flask-Script==2.0.6
15Flask-Cors==3.0.9
16requests==2.25.0
17mysqlclient==2.0.1
18pika==1.1.0
19wolframalpha==4.3.0
20# Veganetwork (C) TetraSystemSolutions 2022
21# all rights are reserved.
22#
23# Author: Trevor R. Blanchard Feb-19-2022-Jul-30-2022
24#
25
26# get our imports in order first
27from flask import Flask # <-- error occurs here!!!
28
29# start the application through flask.
30app = Flask(__name__)
31
32# if set to true will return only a "Hello World" string.
33Debug = True
34
35# start a route to the index part of the app in flask.
36@app.route('/')
37def index():
38 if (Debug == True):
39 return 'Hello World!'
40 else:
41 pass
42
43# start the flask app here --->
44if __name__ == '__main__':
45 app.run(debug=True, host='0.0.0.0')
46FROM python:3.9
47ENV PYTHONUNBUFFERED 1
48WORKDIR /app
49COPY requirements.txt /app/requirements.txt
50RUN pip install -r requirements.txt
51COPY . /app
52version: '3.8'
53services:
54 backend:
55 build:
56 context: .
57 dockerfile: Dockerfile
58 command: 'python vegamain.py'
59 ports:
60 - 8004:5000
61 volumes:
62 - .:/app
63 depends_on:
64 - db
65
66# queue:
67# build:
68# context: .
69# dockerfile: Dockerfile
70# command: 'python -u consumer.py'
71# depends_on:
72# - db
73
74 db:
75 image: mysql:5.7.22
76 restart: always
77 environment:
78 MYSQL_DATABASE: admin
79 MYSQL_USER: root
80 MYSQL_PASSWORD: root
81 MYSQL_ROOT_PASSWORD: root
82 volumes:
83 - .dbdata:/var/lib/mysql
84 ports:
85 - 33069:3306
86
How exactly can I fix this code? thank you!
ANSWER
Answered 2022-Feb-20 at 12:31I was facing the same issue while running docker containers with flask.
I downgraded Flask
to 1.1.4
and markupsafe
to 2.0.1
which solved my issue.
Check this for reference.
QUESTION
Why is it faster to compare strings that match than strings that do not?
Asked 2022-Mar-30 at 11:58Here are two measurements:
1timeit.timeit('"toto"=="1234"', number=100000000)
21.8320042459999968
3timeit.timeit('"toto"=="toto"', number=100000000)
41.4517491540000265
5
As you can see, comparing two strings that match is faster than comparing two strings with the same size that do not match.
This is quite disturbing: During a string comparison, I believed that Python was testing strings character by character, so "toto"=="toto"
should be longer to test than "toto"=="1234"
as it requires four tests against one for the non-matching comparison. Maybe the comparison is hash-based, but in this case, timings should be the same for both comparisons.
Why?
ANSWER
Answered 2022-Mar-30 at 11:57Combining my comment and the comment by @khelwood:
TL;DR:
When analysing the bytecode for the two comparisons, it reveals the 'time'
and 'time'
strings are assigned to the same object. Therefore, an up-front identity check (at C-level) is the reason for the increased comparison speed.
The reason for the same object assignment is that, as an implementation detail, CPython interns strings which contain only 'name characters' (i.e. alpha and underscore characters). This enables the object's identity check.
Bytecode:
1timeit.timeit('"toto"=="1234"', number=100000000)
21.8320042459999968
3timeit.timeit('"toto"=="toto"', number=100000000)
41.4517491540000265
5import dis
6
7In [24]: dis.dis("'time'=='time'")
8 1 0 LOAD_CONST 0 ('time') # <-- same object (0)
9 2 LOAD_CONST 0 ('time') # <-- same object (0)
10 4 COMPARE_OP 2 (==)
11 6 RETURN_VALUE
12
13In [25]: dis.dis("'time'=='1234'")
14 1 0 LOAD_CONST 0 ('time') # <-- different object (0)
15 2 LOAD_CONST 1 ('1234') # <-- different object (1)
16 4 COMPARE_OP 2 (==)
17 6 RETURN_VALUE
18
Assignment Timing:
The 'speed-up' can also be seen in using assignment for the time tests. The assignment (and compare) of two variables to the same string, is faster than the assignment (and compare) of two variables to different strings. Further supporting the hypothesis the underlying logic is performing an object comparison. This is confirmed in the next section.
1timeit.timeit('"toto"=="1234"', number=100000000)
21.8320042459999968
3timeit.timeit('"toto"=="toto"', number=100000000)
41.4517491540000265
5import dis
6
7In [24]: dis.dis("'time'=='time'")
8 1 0 LOAD_CONST 0 ('time') # <-- same object (0)
9 2 LOAD_CONST 0 ('time') # <-- same object (0)
10 4 COMPARE_OP 2 (==)
11 6 RETURN_VALUE
12
13In [25]: dis.dis("'time'=='1234'")
14 1 0 LOAD_CONST 0 ('time') # <-- different object (0)
15 2 LOAD_CONST 1 ('1234') # <-- different object (1)
16 4 COMPARE_OP 2 (==)
17 6 RETURN_VALUE
18In [26]: timeit.timeit("x='time'; y='time'; x==y", number=1000000)
19Out[26]: 0.0745926329982467
20
21In [27]: timeit.timeit("x='time'; y='1234'; x==y", number=1000000)
22Out[27]: 0.10328884399496019
23
Python source code:
As helpfully provided by @mkrieger1 and @Masklinn in their comments, the source code for unicodeobject.c
performs a pointer comparison first and if True
, returns immediately.
1timeit.timeit('"toto"=="1234"', number=100000000)
21.8320042459999968
3timeit.timeit('"toto"=="toto"', number=100000000)
41.4517491540000265
5import dis
6
7In [24]: dis.dis("'time'=='time'")
8 1 0 LOAD_CONST 0 ('time') # <-- same object (0)
9 2 LOAD_CONST 0 ('time') # <-- same object (0)
10 4 COMPARE_OP 2 (==)
11 6 RETURN_VALUE
12
13In [25]: dis.dis("'time'=='1234'")
14 1 0 LOAD_CONST 0 ('time') # <-- different object (0)
15 2 LOAD_CONST 1 ('1234') # <-- different object (1)
16 4 COMPARE_OP 2 (==)
17 6 RETURN_VALUE
18In [26]: timeit.timeit("x='time'; y='time'; x==y", number=1000000)
19Out[26]: 0.0745926329982467
20
21In [27]: timeit.timeit("x='time'; y='1234'; x==y", number=1000000)
22Out[27]: 0.10328884399496019
23int
24_PyUnicode_Equal(PyObject *str1, PyObject *str2)
25{
26 assert(PyUnicode_CheckExact(str1));
27 assert(PyUnicode_CheckExact(str2));
28 if (str1 == str2) { // <-- Here
29 return 1;
30 }
31 if (PyUnicode_READY(str1) || PyUnicode_READY(str2)) {
32 return -1;
33 }
34 return unicode_compare_eq(str1, str2);
35}
36
Appendix:
- Reference answer nicely illustrating how to read the disassembled bytecode output. Courtesy of @Delgan
- Reference answer which nicely describes CPython's string interning. Coutresy of @ShadowRanger
QUESTION
Why is `np.sum(range(N))` very slow?
Asked 2022-Mar-29 at 14:31I saw a video about speed of loops in python, where it was explained that doing sum(range(N))
is much faster than manually looping through range
and adding the variables together, since the former runs in C due to built-in functions being used, while in the latter the summation is done in (slow) python. I was curious what happens when adding numpy
to the mix. As I expected np.sum(np.arange(N))
is the fastest, but sum(np.arange(N))
and np.sum(range(N))
are even slower than doing the naive for loop.
Why is this?
Here's the script I used to test, some comments about the supposed cause of slowing done where I know (taken mostly from the video) and the results I got on my machine (python 3.10.0, numpy 1.21.2):
updated script:
1import numpy as np
2from timeit import timeit
3
4N = 10_000_000
5repetition = 10
6
7def sum0(N = N):
8 s = 0
9 i = 0
10 while i < N: # condition is checked in python
11 s += i
12 i += 1 # both additions are done in python
13 return s
14
15def sum1(N = N):
16 s = 0
17 for i in range(N): # increment in C
18 s += i # addition in python
19 return s
20
21def sum2(N = N):
22 return sum(range(N)) # everything in C
23
24def sum3(N = N):
25 return sum(list(range(N)))
26
27def sum4(N = N):
28 return np.sum(range(N)) # very slow np.array conversion
29
30def sum5(N = N):
31 # much faster np.array conversion
32 return np.sum(np.fromiter(range(N),dtype = int))
33
34def sum5v2_(N = N):
35 # much faster np.array conversion
36 return np.sum(np.fromiter(range(N),dtype = np.int_))
37
38def sum6(N = N):
39 # possibly slow conversion to Py_long from np.int
40 return sum(np.arange(N))
41
42def sum7(N = N):
43 # list returns a list of np.int-s
44 return sum(list(np.arange(N)))
45
46def sum7v2(N = N):
47 # tolist conversion to python int seems faster than the implicit conversion
48 # in sum(list()) (tolist returns a list of python int-s)
49 return sum(np.arange(N).tolist())
50
51def sum8(N = N):
52 return np.sum(np.arange(N)) # everything in numpy (fortran libblas?)
53
54def sum9(N = N):
55 return np.arange(N).sum() # remove dispatch overhead
56
57def array_basic(N = N):
58 return np.array(range(N))
59
60def array_dtype(N = N):
61 return np.array(range(N),dtype = np.int_)
62
63def array_iter(N = N):
64 # np.sum's source code mentions to use fromiter to convert from generators
65 return np.fromiter(range(N),dtype = np.int_)
66
67print(f"while loop: {timeit(sum0, number = repetition)}")
68print(f"for loop: {timeit(sum1, number = repetition)}")
69print(f"sum_range: {timeit(sum2, number = repetition)}")
70print(f"sum_rangelist: {timeit(sum3, number = repetition)}")
71print(f"npsum_range: {timeit(sum4, number = repetition)}")
72print(f"npsum_iterrange: {timeit(sum5, number = repetition)}")
73print(f"npsum_iterrangev2: {timeit(sum5, number = repetition)}")
74print(f"sum_arange: {timeit(sum6, number = repetition)}")
75print(f"sum_list_arange: {timeit(sum7, number = repetition)}")
76print(f"sum_arange_tolist: {timeit(sum7v2, number = repetition)}")
77print(f"npsum_arange: {timeit(sum8, number = repetition)}")
78print(f"nparangenpsum: {timeit(sum9, number = repetition)}")
79print(f"array_basic: {timeit(array_basic, number = repetition)}")
80print(f"array_dtype: {timeit(array_dtype, number = repetition)}")
81print(f"array_iter: {timeit(array_iter, number = repetition)}")
82
83print(f"npsumarangeREP: {timeit(lambda : sum8(N/1000), number = 100000*repetition)}")
84print(f"npsumarangeREP: {timeit(lambda : sum9(N/1000), number = 100000*repetition)}")
85
86# Example output:
87#
88# while loop: 11.493371912998555
89# for loop: 7.385945574002108
90# sum_range: 2.4605720699983067
91# sum_rangelist: 4.509678105998319
92# npsum_range: 11.85120212900074
93# npsum_iterrange: 4.464334709002287
94# npsum_iterrangev2: 4.498494338993623
95# sum_arange: 9.537815956995473
96# sum_list_arange: 13.290120724996086
97# sum_arange_tolist: 5.231948580003518
98# npsum_arange: 0.241889145996538
99# nparangenpsum: 0.21876695199898677
100# array_basic: 11.736577274998126
101# array_dtype: 8.71628468400013
102# array_iter: 4.303306431000237
103# npsumarangeREP: 21.240833958996518
104# npsumarangeREP: 16.690092379001726
105
106
ANSWER
Answered 2021-Oct-16 at 17:42From the cpython source code for sum
sum initially seems to attempt a fast path that assumes all inputs are the same type. If that fails it will just iterate:
1import numpy as np
2from timeit import timeit
3
4N = 10_000_000
5repetition = 10
6
7def sum0(N = N):
8 s = 0
9 i = 0
10 while i < N: # condition is checked in python
11 s += i
12 i += 1 # both additions are done in python
13 return s
14
15def sum1(N = N):
16 s = 0
17 for i in range(N): # increment in C
18 s += i # addition in python
19 return s
20
21def sum2(N = N):
22 return sum(range(N)) # everything in C
23
24def sum3(N = N):
25 return sum(list(range(N)))
26
27def sum4(N = N):
28 return np.sum(range(N)) # very slow np.array conversion
29
30def sum5(N = N):
31 # much faster np.array conversion
32 return np.sum(np.fromiter(range(N),dtype = int))
33
34def sum5v2_(N = N):
35 # much faster np.array conversion
36 return np.sum(np.fromiter(range(N),dtype = np.int_))
37
38def sum6(N = N):
39 # possibly slow conversion to Py_long from np.int
40 return sum(np.arange(N))
41
42def sum7(N = N):
43 # list returns a list of np.int-s
44 return sum(list(np.arange(N)))
45
46def sum7v2(N = N):
47 # tolist conversion to python int seems faster than the implicit conversion
48 # in sum(list()) (tolist returns a list of python int-s)
49 return sum(np.arange(N).tolist())
50
51def sum8(N = N):
52 return np.sum(np.arange(N)) # everything in numpy (fortran libblas?)
53
54def sum9(N = N):
55 return np.arange(N).sum() # remove dispatch overhead
56
57def array_basic(N = N):
58 return np.array(range(N))
59
60def array_dtype(N = N):
61 return np.array(range(N),dtype = np.int_)
62
63def array_iter(N = N):
64 # np.sum's source code mentions to use fromiter to convert from generators
65 return np.fromiter(range(N),dtype = np.int_)
66
67print(f"while loop: {timeit(sum0, number = repetition)}")
68print(f"for loop: {timeit(sum1, number = repetition)}")
69print(f"sum_range: {timeit(sum2, number = repetition)}")
70print(f"sum_rangelist: {timeit(sum3, number = repetition)}")
71print(f"npsum_range: {timeit(sum4, number = repetition)}")
72print(f"npsum_iterrange: {timeit(sum5, number = repetition)}")
73print(f"npsum_iterrangev2: {timeit(sum5, number = repetition)}")
74print(f"sum_arange: {timeit(sum6, number = repetition)}")
75print(f"sum_list_arange: {timeit(sum7, number = repetition)}")
76print(f"sum_arange_tolist: {timeit(sum7v2, number = repetition)}")
77print(f"npsum_arange: {timeit(sum8, number = repetition)}")
78print(f"nparangenpsum: {timeit(sum9, number = repetition)}")
79print(f"array_basic: {timeit(array_basic, number = repetition)}")
80print(f"array_dtype: {timeit(array_dtype, number = repetition)}")
81print(f"array_iter: {timeit(array_iter, number = repetition)}")
82
83print(f"npsumarangeREP: {timeit(lambda : sum8(N/1000), number = 100000*repetition)}")
84print(f"npsumarangeREP: {timeit(lambda : sum9(N/1000), number = 100000*repetition)}")
85
86# Example output:
87#
88# while loop: 11.493371912998555
89# for loop: 7.385945574002108
90# sum_range: 2.4605720699983067
91# sum_rangelist: 4.509678105998319
92# npsum_range: 11.85120212900074
93# npsum_iterrange: 4.464334709002287
94# npsum_iterrangev2: 4.498494338993623
95# sum_arange: 9.537815956995473
96# sum_list_arange: 13.290120724996086
97# sum_arange_tolist: 5.231948580003518
98# npsum_arange: 0.241889145996538
99# nparangenpsum: 0.21876695199898677
100# array_basic: 11.736577274998126
101# array_dtype: 8.71628468400013
102# array_iter: 4.303306431000237
103# npsumarangeREP: 21.240833958996518
104# npsumarangeREP: 16.690092379001726
105
106/* Fast addition by keeping temporary sums in C instead of new Python objects.
107 Assumes all inputs are the same type. If the assumption fails, default
108 to the more general routine.
109*/
110
I'm not entirely certain what is happening under the hood, but it is likely the repeated creation/conversion of C types to Python objects that is causing these slow-downs. It's worth noting that both sum
and range
are implemented in C.
This next bit is not really an answer to the question, but I wondered if we could speed up sum
for python range
s as range
is quite a smart object.
To do this I've used functools.singledispatch
to override the built-in sum
function specifically for the range
type; then implemented a small function to calculate the sum of an arithmetic progression.
1import numpy as np
2from timeit import timeit
3
4N = 10_000_000
5repetition = 10
6
7def sum0(N = N):
8 s = 0
9 i = 0
10 while i < N: # condition is checked in python
11 s += i
12 i += 1 # both additions are done in python
13 return s
14
15def sum1(N = N):
16 s = 0
17 for i in range(N): # increment in C
18 s += i # addition in python
19 return s
20
21def sum2(N = N):
22 return sum(range(N)) # everything in C
23
24def sum3(N = N):
25 return sum(list(range(N)))
26
27def sum4(N = N):
28 return np.sum(range(N)) # very slow np.array conversion
29
30def sum5(N = N):
31 # much faster np.array conversion
32 return np.sum(np.fromiter(range(N),dtype = int))
33
34def sum5v2_(N = N):
35 # much faster np.array conversion
36 return np.sum(np.fromiter(range(N),dtype = np.int_))
37
38def sum6(N = N):
39 # possibly slow conversion to Py_long from np.int
40 return sum(np.arange(N))
41
42def sum7(N = N):
43 # list returns a list of np.int-s
44 return sum(list(np.arange(N)))
45
46def sum7v2(N = N):
47 # tolist conversion to python int seems faster than the implicit conversion
48 # in sum(list()) (tolist returns a list of python int-s)
49 return sum(np.arange(N).tolist())
50
51def sum8(N = N):
52 return np.sum(np.arange(N)) # everything in numpy (fortran libblas?)
53
54def sum9(N = N):
55 return np.arange(N).sum() # remove dispatch overhead
56
57def array_basic(N = N):
58 return np.array(range(N))
59
60def array_dtype(N = N):
61 return np.array(range(N),dtype = np.int_)
62
63def array_iter(N = N):
64 # np.sum's source code mentions to use fromiter to convert from generators
65 return np.fromiter(range(N),dtype = np.int_)
66
67print(f"while loop: {timeit(sum0, number = repetition)}")
68print(f"for loop: {timeit(sum1, number = repetition)}")
69print(f"sum_range: {timeit(sum2, number = repetition)}")
70print(f"sum_rangelist: {timeit(sum3, number = repetition)}")
71print(f"npsum_range: {timeit(sum4, number = repetition)}")
72print(f"npsum_iterrange: {timeit(sum5, number = repetition)}")
73print(f"npsum_iterrangev2: {timeit(sum5, number = repetition)}")
74print(f"sum_arange: {timeit(sum6, number = repetition)}")
75print(f"sum_list_arange: {timeit(sum7, number = repetition)}")
76print(f"sum_arange_tolist: {timeit(sum7v2, number = repetition)}")
77print(f"npsum_arange: {timeit(sum8, number = repetition)}")
78print(f"nparangenpsum: {timeit(sum9, number = repetition)}")
79print(f"array_basic: {timeit(array_basic, number = repetition)}")
80print(f"array_dtype: {timeit(array_dtype, number = repetition)}")
81print(f"array_iter: {timeit(array_iter, number = repetition)}")
82
83print(f"npsumarangeREP: {timeit(lambda : sum8(N/1000), number = 100000*repetition)}")
84print(f"npsumarangeREP: {timeit(lambda : sum9(N/1000), number = 100000*repetition)}")
85
86# Example output:
87#
88# while loop: 11.493371912998555
89# for loop: 7.385945574002108
90# sum_range: 2.4605720699983067
91# sum_rangelist: 4.509678105998319
92# npsum_range: 11.85120212900074
93# npsum_iterrange: 4.464334709002287
94# npsum_iterrangev2: 4.498494338993623
95# sum_arange: 9.537815956995473
96# sum_list_arange: 13.290120724996086
97# sum_arange_tolist: 5.231948580003518
98# npsum_arange: 0.241889145996538
99# nparangenpsum: 0.21876695199898677
100# array_basic: 11.736577274998126
101# array_dtype: 8.71628468400013
102# array_iter: 4.303306431000237
103# npsumarangeREP: 21.240833958996518
104# npsumarangeREP: 16.690092379001726
105
106/* Fast addition by keeping temporary sums in C instead of new Python objects.
107 Assumes all inputs are the same type. If the assumption fails, default
108 to the more general routine.
109*/
110from functools import singledispatch
111
112def sum_range(range_, /, start=0):
113 """Overloaded `sum` for range, compute arithmetic sum"""
114 n = len(range_)
115 if not n:
116 return start
117 return int(start + (n * (range_[0] + range_[-1]) / 2))
118
119sum = singledispatch(sum)
120sum.register(range, sum_range)
121
122def test():
123 """
124 >>> sum(range(0, 100))
125 4950
126 >>> sum(range(0, 10, 2))
127 20
128 >>> sum(range(0, 9, 2))
129 20
130 >>> sum(range(0, -10, -1))
131 -45
132 >>> sum(range(-10, 10))
133 -10
134 >>> sum(range(-1, -100, -2))
135 -2500
136 >>> sum(range(0, 10, 100))
137 0
138 >>> sum(range(0, 0))
139 0
140 >>> sum(range(0, 100), 50)
141 5000
142 >>> sum(range(0, 0), 10)
143 10
144 """
145
146if __name__ == "__main__":
147 import doctest
148 doctest.testmod()
149
I'm not sure if this is complete, but it's definitely faster than looping.
QUESTION
Error while downloading the requirements using pip install (setup command: use_2to3 is invalid.)
Asked 2022-Mar-05 at 07:13version pip 21.2.4 python 3.6
The command:
1pip install -r requirments.txt
2
The content of my requirements.txt
:
1pip install -r requirments.txt
2mongoengine==0.19.1
3numpy==1.16.2
4pylint
5pandas==1.1.5
6fawkes
7
The command is failing with this error
1pip install -r requirments.txt
2mongoengine==0.19.1
3numpy==1.16.2
4pylint
5pandas==1.1.5
6fawkes
7ERROR: Command errored out with exit status 1:
8 command: /Users/*/Desktop/ml/*/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"'; __file__='"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-pip-egg-info-97994d6e
9 cwd: /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/
10 Complete output (1 lines):
11 error in mongoengine setup command: use_2to3 is invalid.
12 ----------------------------------------
13WARNING: Discarding https://*/pypi/packages/mongoengine-0.19.1.tar.gz#md5=68e613009f6466239158821a102ac084 (from https://*/pypi/simple/mongoengine/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
14ERROR: Could not find a version that satisfies the requirement mongoengine==0.19.1 (from versions: 0.15.0, 0.19.1)
15ERROR: No matching distribution found for mongoengine==0.19.1
16
ANSWER
Answered 2021-Nov-19 at 13:30It looks like setuptools>=58
breaks support for use_2to3
:
So you should update setuptools
to setuptools<58
or avoid using packages with use_2to3
in the setup parameters.
I was having the same problem, pip==19.3.1
QUESTION
Repeatedly removing the maximum average subarray
Asked 2022-Feb-28 at 18:19I have an array of positive integers. For example:
1[1, 7, 8, 4, 2, 1, 4]
2
A "reduction operation" finds the array prefix with the highest average, and deletes it. Here, an array prefix means a contiguous subarray whose left end is the start of the array, such as [1]
or [1, 7]
or [1, 7, 8]
above. Ties are broken by taking the longer prefix.
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8
I will repeat the reduction operation until the array is empty:
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8[1, 7, 8, 4, 2, 1, 4]
9^ ^
10[4, 2, 1, 4]
11^ ^
12[2, 1, 4]
13^ ^
14[]
15
Now, actually performing these array modifications isn't necessary; I'm only looking for the list of lengths of prefixes that would be deleted by this process, for example, [3, 1, 3]
above.
What is an efficient algorithm for computing these prefix lengths?
The naive approach is to recompute all sums and averages from scratch in every iteration for an O(n^2)
algorithm-- I've attached Python code for this below. I'm looking for any improvement on this approach-- most preferably, any solution below O(n^2)
, but an algorithm with the same complexity but better constant factors would also be helpful.
Here are a few of the things I've tried (without success):
- Dynamically maintaining prefix sums, for example with a Binary Indexed Tree. While I can easily update prefix sums or find a maximum prefix sum in
O(log n)
time, I haven't found any data structure which can update the average, as the denominator in the average is changing. - Reusing the previous 'rankings' of prefix averages-- these rankings can change, e.g. in some array, the prefix ending at index
5
may have a larger average than the prefix ending at index6
, but after removing the first 3 elements, now the prefix ending at index2
may have a smaller average than the one ending at3
. - Looking for patterns in where prefixes end; for example, the rightmost element of any max average prefix is always a local maximum in the array, but it's not clear how much this helps.
This is a working Python implementation of the naive, quadratic method:
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8[1, 7, 8, 4, 2, 1, 4]
9^ ^
10[4, 2, 1, 4]
11^ ^
12[2, 1, 4]
13^ ^
14[]
15from fractions import Fraction
16def find_array_reductions(nums: List[int]) -> List[int]:
17 """Return list of lengths of max average prefix reductions."""
18
19 def max_prefix_avg(arr: List[int]) -> Tuple[float, int]:
20 """Return value and length of max average prefix in arr."""
21 if len(arr) == 0:
22 return (-math.inf, 0)
23
24 best_length = 1
25 best_average = Fraction(0, 1)
26 running_sum = 0
27
28 for i, x in enumerate(arr, 1):
29 running_sum += x
30 new_average = Fraction(running_sum, i)
31 if new_average >= best_average:
32 best_average = new_average
33 best_length = i
34
35 return (float(best_average), best_length)
36
37 removed_lengths = []
38 total_removed = 0
39
40 while total_removed < len(nums):
41 _, new_removal = max_prefix_avg(nums[total_removed:])
42 removed_lengths.append(new_removal)
43 total_removed += new_removal
44
45 return removed_lengths
46
Edit: The originally published code had a rare error with large inputs from using Python's math.isclose()
with default parameters for floating point comparison, rather than proper fraction comparison. This has been fixed in the current code. An example of the error can be found at this Try it online link, along with a foreword explaining exactly what causes this bug, if you're curious.
ANSWER
Answered 2022-Feb-27 at 22:44This problem has a fun O(n) solution.
If you draw a graph of cumulative sum vs index, then:
The average value in the subarray between any two indexes is the slope of the line between those points on the graph.
The first highest-average-prefix will end at the point that makes the highest angle from 0. The next highest-average-prefix must then have a smaller average, and it will end at the point that makes the highest angle from the first ending. Continuing to the end of the array, we find that...
These segments of highest average are exactly the segments in the upper convex hull of the cumulative sum graph.
Find these segments using the monotone chain algorithm. Since the points are already sorted, it takes O(n) time.
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8[1, 7, 8, 4, 2, 1, 4]
9^ ^
10[4, 2, 1, 4]
11^ ^
12[2, 1, 4]
13^ ^
14[]
15from fractions import Fraction
16def find_array_reductions(nums: List[int]) -> List[int]:
17 """Return list of lengths of max average prefix reductions."""
18
19 def max_prefix_avg(arr: List[int]) -> Tuple[float, int]:
20 """Return value and length of max average prefix in arr."""
21 if len(arr) == 0:
22 return (-math.inf, 0)
23
24 best_length = 1
25 best_average = Fraction(0, 1)
26 running_sum = 0
27
28 for i, x in enumerate(arr, 1):
29 running_sum += x
30 new_average = Fraction(running_sum, i)
31 if new_average >= best_average:
32 best_average = new_average
33 best_length = i
34
35 return (float(best_average), best_length)
36
37 removed_lengths = []
38 total_removed = 0
39
40 while total_removed < len(nums):
41 _, new_removal = max_prefix_avg(nums[total_removed:])
42 removed_lengths.append(new_removal)
43 total_removed += new_removal
44
45 return removed_lengths
46# Lengths of the segments in the upper convex hull
47# of the cumulative sum graph
48def upperSumHullLengths(arr):
49 if len(arr) < 2:
50 if len(arr) < 1:
51 return []
52 else:
53 return [1]
54
55 hull = [(0, 0),(1, arr[0])]
56 for x in range(2, len(arr)+1):
57 # this has x coordinate x-1
58 prevPoint = hull[len(hull) - 1]
59 # next point in cumulative sum
60 point = (x, prevPoint[1] + arr[x-1])
61 # remove points not on the convex hull
62 while len(hull) >= 2:
63 p0 = hull[len(hull)-2]
64 dx0 = prevPoint[0] - p0[0]
65 dy0 = prevPoint[1] - p0[1]
66 dx1 = x - prevPoint[0]
67 dy1 = point[1] - prevPoint[1]
68 if dy1*dx0 < dy0*dx1:
69 break
70 hull.pop()
71 prevPoint = p0
72 hull.append(point)
73
74 return [hull[i+1][0] - hull[i][0] for i in range(0, len(hull)-1)]
75
76
77print(upperSumHullLengths([ 1, 7, 8, 4, 2, 1, 4]))
78
prints:
1[1, 7, 8, 4, 2, 1, 4]
2Original array: [ 1, 7, 8, 4, 2, 1, 4]
3
4Prefix averages: [1.0, 4.0, 5.3, 5.0, 4.4, 3.8, 3.9]
5
6-> Delete [1, 7, 8], with maximum average 5.3
7-> New array -> [4, 2, 1, 4]
8[1, 7, 8, 4, 2, 1, 4]
9^ ^
10[4, 2, 1, 4]
11^ ^
12[2, 1, 4]
13^ ^
14[]
15from fractions import Fraction
16def find_array_reductions(nums: List[int]) -> List[int]:
17 """Return list of lengths of max average prefix reductions."""
18
19 def max_prefix_avg(arr: List[int]) -> Tuple[float, int]:
20 """Return value and length of max average prefix in arr."""
21 if len(arr) == 0:
22 return (-math.inf, 0)
23
24 best_length = 1
25 best_average = Fraction(0, 1)
26 running_sum = 0
27
28 for i, x in enumerate(arr, 1):
29 running_sum += x
30 new_average = Fraction(running_sum, i)
31 if new_average >= best_average:
32 best_average = new_average
33 best_length = i
34
35 return (float(best_average), best_length)
36
37 removed_lengths = []
38 total_removed = 0
39
40 while total_removed < len(nums):
41 _, new_removal = max_prefix_avg(nums[total_removed:])
42 removed_lengths.append(new_removal)
43 total_removed += new_removal
44
45 return removed_lengths
46# Lengths of the segments in the upper convex hull
47# of the cumulative sum graph
48def upperSumHullLengths(arr):
49 if len(arr) < 2:
50 if len(arr) < 1:
51 return []
52 else:
53 return [1]
54
55 hull = [(0, 0),(1, arr[0])]
56 for x in range(2, len(arr)+1):
57 # this has x coordinate x-1
58 prevPoint = hull[len(hull) - 1]
59 # next point in cumulative sum
60 point = (x, prevPoint[1] + arr[x-1])
61 # remove points not on the convex hull
62 while len(hull) >= 2:
63 p0 = hull[len(hull)-2]
64 dx0 = prevPoint[0] - p0[0]
65 dy0 = prevPoint[1] - p0[1]
66 dx1 = x - prevPoint[0]
67 dy1 = point[1] - prevPoint[1]
68 if dy1*dx0 < dy0*dx1:
69 break
70 hull.pop()
71 prevPoint = p0
72 hull.append(point)
73
74 return [hull[i+1][0] - hull[i][0] for i in range(0, len(hull)-1)]
75
76
77print(upperSumHullLengths([ 1, 7, 8, 4, 2, 1, 4]))
78[3, 1, 3]
79
QUESTION
WARNING: Running pip as the 'root' user
Asked 2022-Feb-24 at 01:59I am making simple image of my python Django app in Docker. But at the end of the building container it throws next warning (I am building it on Ubuntu 20.04):
1WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead
2
Why does it throw this warning if I am installing Python requirements inside my image? I am building my image using:
1WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead
2sudo docker build -t my_app:1 .
3
Should I be worried about warning that pip throws, because I know it can break my system?
Here is my Dockerfile:
1WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead
2sudo docker build -t my_app:1 .
3FROM python:3.8-slim-buster
4
5WORKDIR /app
6
7COPY requirements.txt requirements.txt
8
9RUN pip install -r requirements.txt
10
11COPY . .
12
13CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
14
ANSWER
Answered 2021-Aug-29 at 08:12The way your container is built doesn't add a user, so everything is done as root.
You could create a user and install to that users's home directory by doing something like this;
1WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead
2sudo docker build -t my_app:1 .
3FROM python:3.8-slim-buster
4
5WORKDIR /app
6
7COPY requirements.txt requirements.txt
8
9RUN pip install -r requirements.txt
10
11COPY . .
12
13CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
14FROM python:3.8.3-alpine
15
16RUN pip install --upgrade pip
17
18RUN adduser -D myuser
19USER myuser
20WORKDIR /home/myuser
21
22COPY --chown=myuser:myuser requirements.txt requirements.txt
23RUN pip install --user -r requirements.txt
24
25ENV PATH="/home/myuser/.local/bin:${PATH}"
26
27COPY --chown=myuser:myuser . .
28
29CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
30
QUESTION
How do I calculate square root in Python?
Asked 2022-Feb-17 at 03:40I need to calculate the square root of some numbers, for example √9 = 3
and √2 = 1.4142
. How can I do it in Python?
The inputs will probably be all positive integers, and relatively small (say less than a billion), but just in case they're not, is there anything that might break?
Related
- Integer square root in python
- Is there a short-hand for nth root of x in Python?
- Difference between **(1/2), math.sqrt and cmath.sqrt?
- Why is math.sqrt() incorrect for large numbers?
- Python sqrt limit for very large numbers?
- Which is faster in Python: x**.5 or math.sqrt(x)?
- Why does Python give the "wrong" answer for square root? (specific to Python 2)
- calculating n-th roots using Python 3's decimal module
- How can I take the square root of -1 using python? (focused on NumPy)
- Arbitrary precision of square roots
Note: This is an attempt at a canonical question after a discussion on Meta about an existing question with the same title.
ANSWER
Answered 2022-Feb-04 at 19:44math.sqrt()
The math
module from the standard library has a sqrt
function to calculate the square root of a number. It takes any type that can be converted to float
(which includes int
) as an argument and returns a float
.
1>>> import math
2>>> math.sqrt(9)
33.0
4
The power operator (**
) or the built-in pow()
function can also be used to calculate a square root. Mathematically speaking, the square root of a
equals a
to the power of 1/2
.
The power operator requires numeric types and matches the conversion rules for binary arithmetic operators, so in this case it will return either a float
or a complex
number.
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10
(Note: in Python 2, 1/2
is truncated to 0
, so you have to force floating point arithmetic with 1.0/2
or similar. See Why does Python give the "wrong" answer for square root?)
This method can be generalized to nth root, though fractions that can't be exactly represented as a float
(like 1/3 or any denominator that's not a power of 2) may cause some inaccuracy:
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14
Exponentiation works with negative numbers and complex numbers, though the results have some slight inaccuracy:
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14>>> (-25) ** .5 # Should be 5j
15(3.061616997868383e-16+5j)
16>>> 8j ** .5 # Should be 2+2j
17(2.0000000000000004+2j)
18
Note the parentheses on -25
! Otherwise it's parsed as -(25**.5)
because exponentiation is more tightly binding than unary negation.
Meanwhile, math
is only built for floats, so for x<0
, math.sqrt()
will raise ValueError: math domain error
and for complex x
, it'll raise TypeError: can't convert complex to float
. Instead, you can use cmath.sqrt()
, which is more more accurate than exponentiation (and will likely be faster too):
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14>>> (-25) ** .5 # Should be 5j
15(3.061616997868383e-16+5j)
16>>> 8j ** .5 # Should be 2+2j
17(2.0000000000000004+2j)
18>>> import cmath
19>>> cmath.sqrt(-25)
205j
21>>> cmath.sqrt(8j)
22(2+2j)
23
Both options involve an implicit conversion to float
, so floating point precision is a factor. For example:
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14>>> (-25) ** .5 # Should be 5j
15(3.061616997868383e-16+5j)
16>>> 8j ** .5 # Should be 2+2j
17(2.0000000000000004+2j)
18>>> import cmath
19>>> cmath.sqrt(-25)
205j
21>>> cmath.sqrt(8j)
22(2+2j)
23>>> n = 10**30
24>>> square = n**2
25>>> x = square**.5
26>>> x == n
27False
28>>> x - n # how far off are they?
290.0
30>>> int(x) - n # how far off is the float from the int?
3119884624838656
32
Very large numbers might not even fit in a float and you'll get OverflowError: int too large to convert to float
. See Python sqrt limit for very large numbers?
Let's look at Decimal
for example:
Exponentiation fails unless the exponent is also Decimal
:
1>>> import math
2>>> math.sqrt(9)
33.0
4>>> 9 ** (1/2)
53.0
6>>> 9 ** .5 # Same thing
73.0
8>>> 2 ** .5
91.4142135623730951
10>>> 8 ** (1/3)
112.0
12>>> 125 ** (1/3)
134.999999999999999
14>>> (-25) ** .5 # Should be 5j
15(3.061616997868383e-16+5j)
16>>> 8j ** .5 # Should be 2+2j
17(2.0000000000000004+2j)
18>>> import cmath
19>>> cmath.sqrt(-25)
205j
21>>> cmath.sqrt(8j)
22(2+2j)
23>>> n = 10**30
24>>> square = n**2
25>>> x = square**.5
26>>> x == n
27False
28>>> x - n # how far off are they?
290.0
30>>> int(x) - n # how far off is the float from the int?
3119884624838656
32>>> decimal.Decimal('9') ** .5
33Traceback (most recent call last):
34 File "<stdin>", line 1, in <module>
35TypeError: unsupported operand type(s) for ** or pow(): 'decimal.Decimal' and 'float'
36>>> decimal.Decimal('9') ** decimal.Decimal('.5')
37Decimal('3.000000000000000000000000000')
38
Meanwhile, math
and cmath
will silently convert their arguments to float
and complex
respectively, which could mean loss of precision.
decimal
also has its own .sqrt()
. See also calculating n-th roots using Python 3's decimal module
QUESTION
pip-compile raising AssertionError on its logging handler
Asked 2022-Feb-13 at 12:37I have a dockerfile that currently only installs pip-tools
1FROM python:3.9
2
3RUN pip install --upgrade pip && \
4 pip install pip-tools
5
6COPY ./ /root/project
7
8WORKDIR /root/project
9
10ENTRYPOINT ["tail", "-f", "/dev/null"]
11
I build and open a shell in the container using the following commands:
1FROM python:3.9
2
3RUN pip install --upgrade pip && \
4 pip install pip-tools
5
6COPY ./ /root/project
7
8WORKDIR /root/project
9
10ENTRYPOINT ["tail", "-f", "/dev/null"]
11docker build -t brunoapi_image .
12docker run --rm -ti --name brunoapi_container --entrypoint bash brunoapi_image
13
Then, when I try to run pip-compile
inside the container I get this very weird error (full traceback):
1FROM python:3.9
2
3RUN pip install --upgrade pip && \
4 pip install pip-tools
5
6COPY ./ /root/project
7
8WORKDIR /root/project
9
10ENTRYPOINT ["tail", "-f", "/dev/null"]
11docker build -t brunoapi_image .
12docker run --rm -ti --name brunoapi_container --entrypoint bash brunoapi_image
13root@727f1f38f095:~/project# pip-compile
14Traceback (most recent call last):
15 File "/usr/local/bin/pip-compile", line 8, in <module>
16 sys.exit(cli())
17 File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
18 return self.main(*args, **kwargs)
19 File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
20 rv = self.invoke(ctx)
21 File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
22 return ctx.invoke(self.callback, **ctx.params)
23 File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
24 return __callback(*args, **kwargs)
25 File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
26 return f(get_current_context(), *args, **kwargs)
27 File "/usr/local/lib/python3.9/site-packages/piptools/scripts/compile.py", line 342, in cli
28 repository = PyPIRepository(pip_args, cache_dir=cache_dir)
29 File "/usr/local/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 106, in __init__
30 self._setup_logging()
31 File "/usr/local/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging
32 assert isinstance(handler, logging.StreamHandler)
33AssertionError
34
I have no clue what's going on and I've never seen this error before. Can anyone shed some light into this?
Running on macOS Monterey
ANSWER
Answered 2022-Feb-05 at 16:30It is a bug, you can downgrade using:
pip install "pip<22"
QUESTION
ImportError: cannot import name 'url' from 'django.conf.urls' after upgrading to Django 4.0
Asked 2022-Feb-10 at 21:14After upgrading to Django 4.0, I get the following error when running python manage.py runserver
1 ...
2 File "/path/to/myproject/myproject/urls.py", line 16, in <module>
3 from django.conf.urls import url
4ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)
5
My urls.py is as follows:
1 ...
2 File "/path/to/myproject/myproject/urls.py", line 16, in <module>
3 from django.conf.urls import url
4ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)
5from django.conf.urls
6
7from myapp.views import home
8
9urlpatterns = [
10 url(r'^$', home, name="home"),
11 url(r'^myapp/', include('myapp.urls'),
12]
13
ANSWER
Answered 2022-Jan-10 at 21:38django.conf.urls.url()
was deprecated in Django 3.0, and is removed in Django 4.0+.
The easiest fix is to replace url()
with re_path()
. re_path
uses regexes like url
, so you only have to update the import and replace url
with re_path
.
1 ...
2 File "/path/to/myproject/myproject/urls.py", line 16, in <module>
3 from django.conf.urls import url
4ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)
5from django.conf.urls
6
7from myapp.views import home
8
9urlpatterns = [
10 url(r'^$', home, name="home"),
11 url(r'^myapp/', include('myapp.urls'),
12]
13from django.urls import include, re_path
14
15from myapp.views import home
16
17urlpatterns = [
18 re_path(r'^$', home, name='home'),
19 re_path(r'^myapp/', include('myapp.urls'),
20]
21
Alternatively, you could switch to using path
. path()
does not use regexes, so you'll have to update your URL patterns if you switch to path.
1 ...
2 File "/path/to/myproject/myproject/urls.py", line 16, in <module>
3 from django.conf.urls import url
4ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)
5from django.conf.urls
6
7from myapp.views import home
8
9urlpatterns = [
10 url(r'^$', home, name="home"),
11 url(r'^myapp/', include('myapp.urls'),
12]
13from django.urls import include, re_path
14
15from myapp.views import home
16
17urlpatterns = [
18 re_path(r'^$', home, name='home'),
19 re_path(r'^myapp/', include('myapp.urls'),
20]
21from django.urls import include, path
22
23from myapp.views import home
24
25urlpatterns = [
26 path('', home, name='home'),
27 path('myapp/', include('myapp.urls'),
28]
29
If you have a large project with many URL patterns to update, you may find the django-upgrade library useful to update your urls.py
files.
QUESTION
How did print(*a, a.pop(0)) change?
Asked 2022-Feb-04 at 21:21This code:
1a = [1, 2, 3]
2print(*a, a.pop(0))
3
Python 3.8 prints 2 3 1
(does the pop
before unpacking).
Python 3.9 prints 1 2 3 1
(does the pop
after unpacking).
What caused the change? I didn't find it in the changelog.
Edit: Not just in function calls but also for example in a list display:
1a = [1, 2, 3]
2print(*a, a.pop(0))
3a = [1, 2, 3]
4b = [*a, a.pop(0)]
5print(b)
6
Prints [2, 3, 1]
vs [1, 2, 3, 1]
. And Expression lists says "The expressions are evaluated from left to right" (that's the link to Python 3.8 documentation), so I'd expect the unpacking expression to happen first.
ANSWER
Answered 2022-Feb-04 at 21:21I suspect this may have been an accident, though I prefer the new behavior.
The new behavior is a consequence of a change to how the bytecode for *
arguments works. The change is in the changelog under Python 3.9.0 alpha 3:
bpo-39320: Replace four complex bytecodes for building sequences with three simpler ones.
The following four bytecodes have been removed:
- BUILD_LIST_UNPACK
- BUILD_TUPLE_UNPACK
- BUILD_SET_UNPACK
- BUILD_TUPLE_UNPACK_WITH_CALL
The following three bytecodes have been added:
- LIST_TO_TUPLE
- LIST_EXTEND
- SET_UPDATE
On Python 3.8, the bytecode for f(*a, a.pop())
looks like this:
1a = [1, 2, 3]
2print(*a, a.pop(0))
3a = [1, 2, 3]
4b = [*a, a.pop(0)]
5print(b)
6 1 0 LOAD_NAME 0 (f)
7 2 LOAD_NAME 1 (a)
8 4 LOAD_NAME 1 (a)
9 6 LOAD_METHOD 2 (pop)
10 8 CALL_METHOD 0
11 10 BUILD_TUPLE 1
12 12 BUILD_TUPLE_UNPACK_WITH_CALL 2
13 14 CALL_FUNCTION_EX 0
14 16 RETURN_VALUE
15
while on 3.9, it looks like this:
1a = [1, 2, 3]
2print(*a, a.pop(0))
3a = [1, 2, 3]
4b = [*a, a.pop(0)]
5print(b)
6 1 0 LOAD_NAME 0 (f)
7 2 LOAD_NAME 1 (a)
8 4 LOAD_NAME 1 (a)
9 6 LOAD_METHOD 2 (pop)
10 8 CALL_METHOD 0
11 10 BUILD_TUPLE 1
12 12 BUILD_TUPLE_UNPACK_WITH_CALL 2
13 14 CALL_FUNCTION_EX 0
14 16 RETURN_VALUE
15 1 0 LOAD_NAME 0 (f)
16 2 BUILD_LIST 0
17 4 LOAD_NAME 1 (a)
18 6 LIST_EXTEND 1
19 8 LOAD_NAME 1 (a)
20 10 LOAD_METHOD 2 (pop)
21 12 CALL_METHOD 0
22 14 LIST_APPEND 1
23 16 LIST_TO_TUPLE
24 18 CALL_FUNCTION_EX 0
25 20 RETURN_VALUE
26
In the old bytecode, the code pushes a
and (a.pop(),)
onto the stack, then unpacks those two iterables into a tuple. In the new bytecode, the code pushes a list onto the stack, then does l.extend(a)
and l.append(a.pop())
, then calls tuple(l)
.
This change has the effect of shifting the unpacking of a
to before the pop
call, but this doesn't seem to have been deliberate. Looking at bpo-39320, the intent was to simplify the bytecode instructions, not to change the behavior, and the bpo thread has no discussion of behavior changes.
Community Discussions contain sources that include Stack Exchange Network