Open Source Intelligence - Reusable Libraries
by Ashok Balasubramanian 
Updated: Oct 2, 2021
Solution Kit
Open Source Intelligence has played a pivotal role in key events like tracing Covid-19 origins, MH17 downing, the Boston Marathon bombing, and the Myanmar refugee crisis. Approximately 500 million tweets are published every day, totaling over 200 billion posts in a year. Facebook users upload 350 million photos per day. YouTube users add nearly 720,000 hours of new video every day. Almost all devices are online today in the connected world.
While monitoring messages was exclusive to intelligence agencies, the tons of information available in the public realm today has made it possible for general and security enthusiasts to look for insights that might not have been possible earlier. The U.S. Department of State defines OSINT as "intelligence that is produced from publicly available information and is collected, exploited, and disseminated promptly to an appropriate audience to address a specific intelligence requirement."
Designed correctly, OSINT can reduce risk across a variety of common risks such as weather conditions, disease outbreaks, corporate risk management, data privacy, reputation management, in addition to higher-order tasks like national security and cybersecurity. Do not construe this as legal advice, promotion, or authorization to indulge in any activity whatsoever.
OSINT Framework
The OSINT framework enables gathering information from free tools or resources. The below open source libraries introduce and enable gathering information based on the OSINT Framework.
OSINT-Frameworkby lockfale
5343 Version:CurrentOSINT Framework
sherlockby sherlock-project
41730 Version:Current🔎 Hunt down social media accounts by username across social networks
sherlockby sherlock-project
Python 41730 Version:Current License: Permissive (MIT)
OSINTby sinwindie
2125 Version:Current
License: No License (null)Collections of tools and methods created to aid in OSINT collection
OSINTby sinwindie
Python 2125 Version:Current License: No License
Target Reconnaissance
Recon-ng is a full-featured reconnaissance framework designed with the goal of providing a powerful environment to conduct open source web-based reconnaissance quickly and thoroughly.
recon-ngby lanmaster53
2691 Version:CurrentOpen Source Intelligence gathering tool aimed at reducing the time spent harvesting information from open sources.
recon-ngby lanmaster53
Python 2691 Version:Current License: Strong Copyleft (GPL-3.0)
recon-ng-marketplaceby lanmaster53
99 Version:CurrentOfficial module repository for the Recon-ng Framework.
recon-ng-marketplaceby lanmaster53
Python 99 Version:Current License: Strong Copyleft (GPL-3.0)
Information Collection
theHarvester and similar tools gather emails, names, subdomains, IPs and URLs using multiple public data sources.
theHarvesterby laramies
8676 Version:4.3.0 License: No License (null)E-mails, subdomains and names Harvester - OSINT
theHarvesterby laramies
Python 8676 Version:4.3.0 License: No License
metagoofilby laramies
844 Version:CurrentMetadata harvester
metagoofilby laramies
Python 844 Version:Current License: Strong Copyleft (GPL-2.0)
Track Online Assets
Shodan and Amass enable researchers to see the exposed assets.
shodan-pythonby achillean
2186 Version:1.28.0The official Python library for Shodan
shodan-pythonby achillean
Python 2186 Version:1.28.0 License: Others (Non-SPDX)
My-Shodan-Scriptsby random-robbie
756 Version:CurrentCollection of Scripts for shodan searching stuff.
My-Shodan-Scriptsby random-robbie
Python 756 Version:Current License: Permissive (MIT)
Google Search
Google dorks provides information through the usage of operators, which are otherwise difficult to extract using simple searches.
pagodoby opsdisk
2222 Version:v2.4.0pagodo (Passive Google Dork) - Automate Google Hacking Database scraping and searching
pagodoby opsdisk
Python 2222 Version:v2.4.0 License: Strong Copyleft (GPL-3.0)
BigBountyReconby Viralmaniar
912 Version:v0.1BigBountyRecon tool utilises 58 different techniques using various Google dorks and open source tools to expedite the process of initial reconnaissance on the target organisation.
BigBountyReconby Viralmaniar
C# 912 Version:v0.1 License: Permissive (MIT)
Fast-Google-Dorks-Scanby IvanGlinkin
959 Version:Current License: No License (null)The OSINT project, the main idea of which is to collect all the possible Google dorks search combinations and to find the information about the specific web-site: common admin panels, the widespread file types and path traversal. The 100% automated.
Fast-Google-Dorks-Scanby IvanGlinkin
Shell 959 Version:Current License: No License