Posted on

Open Source OSINT Tools: Unveiling the Power of Command Line

Open Source OSINT CLI tools

Open Source Intelligence (OSINT) tools are akin to powerful flashlights that illuminate the hidden nooks and crannies of the internet. They serve as wizards of data collection, capable of extracting valuable information from publicly accessible resources that anyone can reach. These tools transcend the realm of tech wizards and cyber sleuths, finding utility in the arsenals of journalists, market researchers, and law enforcement professionals alike. They serve as indispensable aides, providing the raw material that shapes pivotal decisions and strategies.

Why Command Line OSINT Tools Shine

Command line OSINT tools hold a special allure in the digital landscape. Picture wielding a magic wand that automates mundane tasks, effortlessly sifts through vast troves of data, and unearths precious insights in mere seconds. That’s precisely the magic these command line tools deliver. Stripped of flashy visuals, they harness the power of simplicity to wield immense capabilities. With just text commands, they unravel complex searches, streamline data organization, and seamlessly integrate with other digital tools. It’s no wonder they’ve become darlings among tech enthusiasts who prize efficiency and adaptability.

Let’s Meet Some Top Open Source Command Line OSINT Tools

Now, let’s dive into some of the most popular open-source command line OSINT tools out there and discover what they can do for you:

Email and Contact Information
      • EmailHarvester: Retrieves domain email addresses from search engines, designed to aid penetration testers in the early stages of their tests.

      • Infoga: Collects email accounts, IP addresses, hostnames, and associated countries from different public sources (search engines, key servers) to assess the security of an email structure.

      • Mailspecter: A newer tool designed to find email addresses and related contact information across the web using custom search techniques, ideal for targeted social engineering assessments.

      • OSINT-SPY: Searches and scans for email addresses, IP addresses, and domain information using a variety of search engines and services.

      • Recon-ng: A full-featured Web Reconnaissance framework written in Python, designed to perform information gathering quickly and thoroughly from online sources.

      • SimplyEmail: Gathers and organizes email addresses from websites and search engines, allowing for an in-depth analysis of a target’s email infrastructure.

      • Snovio: An API-driven tool for email discovery and verification, which can be utilized for building lead pipelines and conducting cold outreach efficiently.

      • theHarvester: Gathers emails, subdomains, hosts, employee names, open ports, and banners from different public sources like search engines and social networks.

Network and Device Information
      • Angry IP Scanner: A fast and easy-to-use network scanner that scans IP addresses and ports, featuring additional capabilities like NetBIOS information, web server detection, and more.

      • ARP-Scan: Uses ARP packets to identify hosts on a local network segment, ideal for discovering physical devices on a LAN.

      • Censys CLI: Provides command-line access to query the Censys database, offering detailed information on all devices and hosts visible on the internet.

      • Driftnet: Monitors network traffic and extracts images from TCP streams, offering insights into the visual content being transmitted over a network.

      • EtherApe: A graphical network monitoring tool for Unix systems that displays network activity with color-coded protocols, operating through a command-line interface for setup and management.

      • hping: A command-line TCP/IP packet assembler/analyzer useful for tasks such as network testing, firewall testing, and manual path MTU discovery.

      • Masscan: Known as the fastest Internet port scanner, ideal for scanning entire internet subnets or the entire internet at unparalleled speeds.

      • Netdiscover: An ARP reconnaissance tool used for scanning networks to discover connected devices, useful during the initial phase of penetration testing or red-teaming.

      • Nikto: An open-source web server scanner that conducts extensive tests against web servers, checking for dangerous files and outdated software.

      • Nmap: The essential network scanning tool for network discovery and security auditing, capable of identifying devices, services, operating systems, and packet types.

      • Shodan CLI: Command-line access to the Shodan search engine, providing insights into global internet exposure and potential vulnerabilities of internet-connected devices.

      • tcpdump: A robust packet analyzer that captures and displays TCP/IP and other packets being transmitted or received over a network.

      • Wireshark CLI (Tshark): The command-line version of Wireshark for real-time packet capturing and analysis, providing detailed insights into network traffic.

      • ZMap: An open-source network scanner optimized for performing internet-wide scans and surveys quickly and efficiently.

Document and Metadata Analysis
      • Metagoofil: Extracts metadata of public documents (.pdf, .doc, .xls, etc.) available on target websites, revealing details about the software used to create them and other hidden information.

      • ExifTool: A robust tool to read, write, and edit meta information in a wide array of file types, particularly effective for extracting metadata from digital photographs and documents.

      • Binwalk: Specializes in analyzing, reverse engineering, and extracting firmware images and executable files, helping to uncover hidden metadata and compressed components.

      • Foremost: Originally developed for law enforcement use, Foremost can carve files based on their headers, footers, and internal data structures, making it an excellent tool for recovering hidden information from formatted or damaged media.

      • Pdf-parser: A tool that parses the contents of PDF files to reveal its structure, objects, and metadata, providing deeper insights into potentially manipulated documents or hidden data.

      • Pdfid: Scans PDF files to identify suspicious elements, such as certain keywords or obfuscated JavaScript often used in malicious documents.

      • Bulk Extractor: A program that scans disk images, file systems, and directories of files to extract valuable metadata such as email addresses, credit card numbers, URLs, and other types of information.

Domain and IP Analysis
      • Altdns: Generates permutations, alterations, and mutations of subdomains and then resolves them, crucial for uncovering hidden subdomains that are not easily detectable.

      • Amass: Conducts network mapping of attack surfaces and discovers external assets using both open-source information gathering and active reconnaissance techniques.

      • DNSdumpster: Leverages data from DNSdumpster.com to map out domain DNS data into detailed reports, providing visual insights into a domain’s DNS structure.

      • DNSrecon: Performs DNS enumeration to find misconfigurations and collect comprehensive information about DNS records, enhancing domain security analysis.

      • Dig (Domain Information Groper): A versatile DNS lookup tool that queries DNS name servers for detailed information about host addresses, mail exchanges, and name servers, widely used for DNS troubleshooting.

      • dnsenum: Utilizes scripts that combine tools such as whois, host, and dig to gather extensive information from a domain, enriching DNS analysis.

      • dnsmap: Bursts and brute-forces subdomains using wordlists to uncover additional domains and subdomains associated with a target domain, aiding in depth penetration testing.

      • Fierce: Scans domains to quickly discover IPs, subdomains, and other critical data necessary for network security assessments, using several tactics for effective domain probing.

      • Gobuster: Brute-forces URIs (directories and files) in web applications and DNS subdomains using a wordlist, essential for discovering hidden resources during security assessments.

      • MassDNS: A high-performance DNS resolver designed for bulk lookups and reconnaissance, particularly useful in large-scale DNS enumeration tasks.

      • Nmap Scripting Engine (NSE) for DNS: Utilizes Nmap’s scripting capabilities to query DNS servers about hostnames and gather detailed domain information, adding depth to network security assessments.

      • Sn1per: Integrates various CLI OSINT tools to automate detailed reconnaissance of domains, enhancing penetration testing efforts with automated scanning.

      • SSLScan: Tests SSL/TLS configurations of web servers to quickly identify supported SSL/TLS versions and cipher suites, assessing vulnerabilities in encrypted data transmissions.

      • Sublist3r: Enumerates subdomains of websites using OSINT techniques to aid in the reconnaissance phase of security assessments, identifying potential targets within a domain’s structure.

Website Downloading
      • Aria2: A lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, SFTP, and can handle multiple downloads simultaneously.

      • Cliget: A command-line tool that generates curl/wget commands for downloading files from the browser, capturing download operations for reuse in the command line.

      • cURL: Transfers data with URL syntax, supporting a wide variety of protocols including HTTP, HTTPS, FTP, and more, making it a versatile tool for downloading and uploading files.

      • HTTrack (Command Line Version): Downloads entire websites to a local directory, recursively capturing HTML, images, and other files, preserving the original site structure and links.

      • Lynx: A highly configurable text-based web browser used in the command line to access websites, which can be scripted to download text content from websites.

      • Wget: A non-interactive network downloader that supports HTTP, HTTPS, and FTP protocols, often used for downloading large files and complete websites.

      • WebHTTrack: The command-line counterpart of HTTrack that also features a web interface; it allows for comprehensive website downloads and offline browsing.

      • Wpull: A wget-compatible downloader that supports modern web standards and compression formats, aimed at being a powerful tool for content archiving.

User Search Tools
      • Blackbird: An OSINT tool designed to gather detailed information about email addresses, phone numbers, and names from different public sources and social networks. It can be useful for detailed background checks and identity verification.

      • CheckUsernames: Searches for the use of a specific username across over 170 websites, helping determine the user’s online presence on different platforms.

      • Maigret: Collects a dossier on a person by username only, querying a large number of websites for public information as well as checking for data leaks.

      • Namechk: Utilizes a command-line interface to verify the availability of a specific username across hundreds of websites, helping to identify a user’s potential digital footprint.

      • sherlock: Searches for usernames across many social networks and finds accounts registered with that username, providing quick insights into user presence across the web.

      • SpiderFoot: An automation tool that uses hundreds of OSINT sources to collect comprehensive information about any username, alongside IP addresses, domain names, and more, making it invaluable for extensive user search and reconnaissance.

      • UserRecon: Finds and collects usernames across various social networks, allowing for a comprehensive search of a person’s online presence based on a single username.

Breach Lookups
      • Breach-Miner: A tool designed to parse through various public data breach databases, identifying exposure of specific credentials or sensitive information which aids in vulnerability assessment and security enhancement.

      • DeHashed CLI: Provides a command-line interface to search across multiple data breach sources to find if personal details such as emails, names, or phone numbers have been compromised, facilitating proactive security measures.

      • Have I Been Pwned (HIBP) CLI: A command-line interface for the Have I Been Pwned service that checks if an email address has been part of a data breach. This tool is essential for monitoring and safeguarding personal or organizational email addresses against exposure in public breaches.

      • h8mail: Targets email addresses to check for breaches and available social media profiles, passwords, and leaks. It also supports API usage for enhanced searching capabilities.

      • PwnDB: A command-line tool that searches for credentials leaks on public databases, enabling users to find if their data has been exposed in past data breaches and understand the specifics of the exposure.

    •  

Many more tools can be used for OSINT and reconnaissance not listed here.

As we come to the end of our exploration, it’s abundantly clear that the tools we’ve discussed merely scratch the surface of the expansive universe of Open Source Intelligence (OSINT). Think of them as specialized instruments, finely crafted to unearth specific nuggets of data buried within the vast expanse of the internet. Whether you’re safeguarding a network fortress, unraveling the threads of a personal mystery, or charting the terrain of market landscapes, these command-line marvels stand ready to empower your journey through the ever-expanding ocean of public information.

So, armed with these digital compasses and fueled by a spark of curiosity, you’re poised to embark on your very own OSINT odyssey. Prepare to navigate through the shadows, uncovering hidden treasures and illuminating the darkest corners of the digital realm. With each keystroke, you’ll unravel new insights, forge new paths, and redefine what it means to explore the boundless depths of knowledge that await in the digital age. Let these tools be your guiding stars as you chart a course through the uncharted territories of cyberspace, transforming data into wisdom and unlocking the mysteries that lie beyond.

Posted on

Unveiling Recon-ng: The Sleuth’s Digital Toolkit

Recon-ng is a full-featured reconnaissance framework designed with the goal of providing a powerful environment to conduct open source web-based reconnaissance quickly and thoroughly.

In a world brimming with digital shadows and cyber secrets, a tool emerges from the shadows—meet Recon-ng, your ultimate companion in the art of online investigation. Picture yourself as the protagonist in a high-stakes Jack Ryan thriller, where every piece of information could be the key to unraveling complex mysteries. Recon-ng isn’t just a tool; it’s your ally in navigating the labyrinthine alleys of the internet’s vast expanse.

Imagine you’re a digital sleuth, tasked with piecing together clues in a race against time to prevent a cyber-attack or uncover illicit activities. This is where Recon-ng steps into the spotlight. It is a powerful framework engineered to perform Open Source Intelligence (OSINT) gathering with precision and ease. OSINT, for the uninitiated, is the art of collecting data from publicly available sources to be used in an analysis. Think of it as gathering pieces of a puzzle scattered across the internet, from social media platforms to website registrations and beyond.

Recon-ng is designed to streamline the process of data collection. With it, investigators can automate the tedious task of scouring through pages of search results and social media feeds to extract valuable insights. Whether you’re a cybersecurity expert monitoring potential threats, a journalist tracking down leads for a story, or a law enforcement officer investigating a case, Recon-ng serves as your digital magnifying glass.

But why does this matter? In our interconnected world, the ability to quickly and efficiently gather information can be the difference between preventing a catastrophe and reading about it in the morning paper. Recon-ng is more than just a tool—it’s a gateway to understanding the digital fingerprints that we all leave behind. This framework empowers its users to see beyond the surface, connect dots hidden in plain sight, and uncover the stories woven into the fabric of the digital age.

Stay tuned, as this is just the beginning of our journey into the world of Recon-ng. Next, we’ll delve deeper into the mechanics of how it operates, no coding experience is required, just your curiosity and a thirst for the thrill of the hunt.

The Power of Keys: Unlocking the World of Information with API Integration

API keys are akin to specialized gadgets in a Jack Ryan arsenal, indispensable tools that unlock vast reserves of information. These keys serve as passes, granting access to otherwise restricted areas in the vast database landscapes, turning raw data into actionable intelligence.

API keys, or Application Programming Interface keys, are unique identifiers that allow you to interact with external software services. Think of them as special codes that prove your identity and grant permission to access these services without exposing your username and password. In the context of Recon-ng, these keys are crucial—they are the lifelines that connect the framework to a plethora of data sources, enhancing its capability to gather intelligence.

Now, let’s delve into some of the specific API keys that can transform Recon-ng into an even more powerful tool for digital sleuthing:

    1. Bing API Key: This key opens the gates to Microsoft’s Bing Search API, allowing Recon-ng to pull search data directly from one of the world’s major search engines. It’s like having direct access to a global index of information that could be vital for your investigations.
    2. BuiltWith API Key: With this key, Recon-ng can identify what technologies are used to build websites. Knowing the technology stack of a target can provide insights into potential vulnerabilities or the level of sophistication a particular entity possesses.
    3. Censys API Key and Secret: These keys provide access to Censys’ vast database of information about all the devices connected to the internet. Imagine being able to pull up detailed configurations of servers across the globe—vital for cybersecurity reconnaissance.
    4. Flickr API Key: This key allows access to Flickr’s rich database of images and metadata, which can be a goldmine for gathering intelligence about places, events, or individuals based on their digital footprints in photographs.
    5. FullContact API Key: It turns email addresses and other contact information into full social profiles, giving you a broader picture of an individual’s digital presence.
    6. Google and YouTube API Keys: These keys unlock the vast resources of Google searches, YouTube videos, and even specific geographical data through Google Maps, providing a comprehensive suite of tools for online reconnaissance.
    7. Shodan API Key: Often referred to as the “search engine for hackers,” Shodan provides access to information about internet-connected devices. This is crucial for discovering vulnerable devices or systems exposed on the internet.
    8. Twitter API Keys: These allow Recon-ng to tap into the stream of data from Twitter, enabling real-time and historical analysis of tweets which can reveal trends, sentiments, and public discussions related to your targets.

Each key is a token that brings you one step closer to the truth hidden in the digital ether. By integrating these keys, Recon-ng becomes not just a tool, but a formidable gateway to the intelligence needed to crack cases, thwart threats, and uncover hidden narratives in the cyber age. As you proceed in your digital investigation, remember that each piece of data you unlock with these keys adds a layer of depth to your understanding of the digital landscape—a landscape where information is power, and with great power comes great responsibility.

Setting Up Your Recon-ng Command Center

Stepping into the world of Recon-ng for the first time feels like entering a high-tech control room in a Jack Ryan saga. Your mission, should you choose to accept it, involves configuring and mastering this powerful tool to uncover hidden truths in the digital world. Here’s your guide to setting up and navigating through the myriad features of Recon-ng, turning raw data into a map of actionable intelligence.

Initial Configuration and Workspaces

Upon launching Recon-ng, the first task is to establish your operational environment, termed a “workspace”. Each workspace is a separate realm where specific investigations are contained, allowing you to manage multiple investigations without overlap:

    • Create a Workspace:
workspaces create <name>

This command initiates a new workspace. This isolated environment will store all your queries, results, and configurations.

    • Load a Workspace:
workspaces load <name>

This command switches to an existing workspace.

    • Managing Workspaces:
      • View all available workspaces:
workspaces list
      • Remove a workspace:
workspaces remove <name>
API Keys and Global Options

Before diving deep into data collection, it’s crucial to integrate API keys for various data sources. These keys are your passes to access restricted databases and services:

    • Adding API Keys:
options set <key_name> <key_value>

Input your API keys here, such as those for Google, Bing, or Twitter.

    • Adjust Global Settings:
      • Review settings:
options list
      • Modify settings:
options set <option> <value>
    • Modify settings like VERBOSITY or PROXY to tailor how Recon-ng interacts with you and the internet.
Interacting with the Database

Recon-ng’s heart lies in its database, where all harvested data is stored and managed:

    • Database Queries:
db query <SQL_query>

Execute SQL commands directly on the database, exploring or manipulating the stored data.

    • Inserting and Deleting Records:
      • Add initial seeds to your investigation:
db insert
      • Remove records:
db delete
Modules and the Marketplace

The real power of Recon-ng is realized through its modules, each designed to perform specific tasks or retrieve particular types of information:

    • Searching for Modules:
marketplace search <keyword>

or

modules search <specific query>

Discover available modules by their function.

    • Installing Modules:
marketplace install <module>

Install modules; ensure all dependencies are met before activation to avoid errors.

    • Loading and Configuring Modules:
modules load <module_name>

Load a module and then set required options for each module:

options set <option> <value>

Recording and Automation

To streamline repetitive tasks or document your process, Recon-ng offers automation and recording features:

    • Recording Commands:
script record <filename>

Activate command recording, and stop with:

script stop

to save your session’s commands for future automation.

    • Using Resource Files:
script execute <filename>

Automate Recon-ng operations by creating a resource file (*.rc) with a list of commands and executing it.

Analysis and Reporting

Finally, once data collection is complete, turning that data into reports is essential:

    • Recon-web:
./recon-web

Launch the web interface to analyze data, visualize findings, and generate reports in various formats, transitioning from raw data to comprehensive intelligence.

By setting up Recon-ng meticulously, you ensure that each step in your digital investigation is calculated and precise, much like the strategic moves in a Jack Ryan operation. Each command you enter and each piece of intelligence you gather brings you closer to unveiling the mysteries hidden within the vast expanse of the digital world.

Case Study: Reconnaissance on Google.com Using Recon-ng

Imagine the scene: a room filled with screens, each flickering with streams of data. A digital investigator sits, the glow of the display casting a soft light across determined features. The mission? To gather intelligence on one of the internet’s titans, Google.com, using the formidable OSINT tool, Recon-ng. Here’s how our investigator would embark on this digital reconnaissance, complete with the expected syntax and outcomes.

    • Set Up and Workspace Creation

Firstly, the investigator initializes Recon-ng and creates a dedicated workspace for this operation to keep the investigation organized and isolated.

./recon-ng workspaces create google_recon

This step ensures all gathered data is stored separately, preventing any mix-up with other investigations.

    • Loading Necessary Modules

To gather comprehensive information about Google.com, the investigator decides to start with domain and host-related data. The recon/domains-hosts/bing_domain_web module is chosen to query Bing for subdomains:

modules load recon/domains-hosts/bing_domain_web

Upon loading, the module will require a target domain and valid API key for Bing:

options set SOURCE google.com options set API_KEY <your_bing_api_key>
    • Running the Module and Gathering Data

With the module configured, it’s time to run it and observe the data flowing in:

run

Expected Results: The module queries Bing’s search engine to find subdomains associated with google.com. The expected output would typically list various subdomains such as mail.google.com, maps.google.com, docs.google.com, etc., revealing different services provided under the main domain.

    • Exploring Further with Additional Modules

To deepen the reconnaissance, additional modules can be employed. For instance, using recon/domains-contacts/whois_pocs to gather point of contact information from WHOIS records:

modules load recon/domains-contacts/whois_pocs options set SOURCE google.com run

Expected Results: This module would typically return contact information associated with the domain registration, including names, emails, or phone numbers, which are useful for understanding the administrative structure of the domain.

    • Analyzing and Reporting

After gathering sufficient data, the investigator would use the reporting tools to compile the information into a comprehensive report:

modules load reporting/html options set CREATOR "Investigator's Name" options set CUSTOMER "Internal Review" options set FILENAME google_report.html run

Expected Results: This action creates an HTML report summarizing all gathered data. It includes sections for each module run, displaying domains, subdomains, contact details, and other relevant information about google.com.

This case study demonstrates a methodical approach to using Recon-ng for detailed domain reconnaissance. By sequentially loading and running relevant modules, an investigator can compile a significant amount of data about a target domain. Each step in the process adds layers of information, fleshing out a detailed picture of the target’s digital footprint, essential for security assessments, competitive analysis, or investigative journalism. As always, it’s crucial to conduct such reconnaissance ethically and within the boundaries of the law.

Navigating the Digital Maze with Recon-ng

As we draw the curtains on our digital odyssey with Recon-ng, it’s evident that this tool is much more than a mere software application—it’s a comprehensive suite for digital sleuthing that arms you with the capabilities to navigate through the complex web of information that is the internet today.

Beyond Basic Data Gathering

While we’ve delved into some of the capabilities of Recon-ng, such as extracting domain information and integrating powerful API keys, Recon-ng’s toolkit stretches even further. This versatile tool can also be utilized for:

    • Geolocation Tracking: Trace the geographic footprint of IP addresses, potentially pinpointing the physical locations associated with digital activities.
    • Email Harvesting: Collect email addresses associated with a specific domain. This can be crucial for building contact lists or understanding the communication channels of a target organization.
    • Vulnerability Identification: Identify potential security vulnerabilities in the digital infrastructure of your targets, allowing for proactive security assessments.

These features enhance the depth and breadth of investigations, providing a richer, more detailed view of the digital landscape surrounding a target.

Empowering Modern Investigators

Whether you are a cybersecurity defender, a market analyst, or an investigative journalist, Recon-ng equips you with the tools to unearth the hidden connections that matter. It’s about transforming raw data into insightful, actionable information.

A Call to Ethical Exploration

However, with great power comes great responsibility. As you wield Recon-ng to peel back layers of digital information, it’s paramount to operate within legal frameworks and ethical guidelines. The goal is to enlighten, not invade; to protect, not exploit.

The Future Awaits

As technology evolves, so too will Recon-ng, continuously adapting to the ever-changing digital environment. Its community-driven development ensures that new features and improvements will keep pace with the needs of users across various fields.

In this age of information, where data is both currency and compass, Recon-ng stands as your essential guide through the digital shadows. It’s not just about finding data—it’s about making sense of it, connecting the dots in a world where every byte could be the key to unlocking new vistas of understanding.

Embrace the journey, for each query typed and each module loaded is a step closer to mastering the digital realm with Recon-ng. So, gear up, set your sights, and let the digital expedition begin