As far back as World War II, highly trained agents in the intelligence community have monitored open-source information such as radio broadcasts, newspapers and market fluctuations. Today, given the number and variety of easily accessible data sources, nearly anyone can participate in open-source intelligence gathering.

Some of the public sources from which OSINT researchers collect data points include:

• Internet search engines such as Google, DuckDuckGo, Yahoo, Bing and Yandex.

• Print and online news media including newspapers, magazines and news sites.

• Social media accounts on platforms such as Facebook, X, Instagram and LinkedIn.

• Online forums, blogs and Internet Relay Chats (IRC).

• The dark web, an encrypted area of the internet that is not indexed by search engines.

• Online directories of phone numbers, email addresses and physical addresses.

• Public records including births, deaths, court documents and business filings.

• Government records such as meeting transcripts, budgets, speeches and press releases issued by local, state and federal/national governments.

• Academic research including papers, theses and journals.

• Technical data such as IP addresses, APIs, open ports and web page metadata.

However, before data collection from OSINT sources begin, a clear objective should be established. For example, security professionals who use OSINT first determine which insights they seek to uncover, and which public data will yield the desired results.

After the public information is collected, it must then be processed to filter out unnecessary or redundant data. Security teams can then analyze the refined data and create an actionable intelligence report.

Threat actors often use OSINT to uncover sensitive information they can leverage to exploit vulnerabilities in computer networks.

This may include personal details about an organization’s employees, partners and vendors that are easily accessible on social media and company websites. Or technical information such as credentials, security gaps or encryption keys that may appear in the source code of web pages or cloud applications. There are also public websites that publish compromising information such as stolen logins and passwords from data breaches.

Cybercriminals are able to use this public data for a variety of nefarious purposes.

For example, they could use personal information from social networks to create tailored phishing emails that convince readers to click on a malicious link. Or conduct a Google search with specific commands that reveal security weaknesses in a web application, a practice called “Google dorking.” They may also evade detection during a hacking attempt after reviewing a company’s public assets that describe their cybersecurity defense strategies.

For these reasons, many organizations conduct OSINT assessments of the public sources of information related to their systems, applications and human resources.

The findings can be used to locate unauthorized leaks of proprietary or sensitive data, evaluate information security, and identify vulnerabilities such as unpatched software, misconfigurations or open ports. Organizations may also conduct penetration testing of their systems and networks using the same OSINT data that are publicly accessible by cybercriminals and hackers.

Often, the information collected during an OSINT assessment is combined with non-public data to create a more comprehensive threat intelligence report. Frequent updates to OSINT cybersecurity assessments can help organizations mitigate the risk of data breaches, ransomware, malware and other cyberattacks.

Because of the vast amount of public information available, it is often impractical to manually collect, sort and analyze OSINT data. Specialized open-source intelligence tools can help manage and automate data tasks for a variety of OSINT use cases.

Some OSINT analysis tools use artificial intelligence and machine learning to detect which information is valuable and relevant, and which is insignificant or unrelated. Among the more popular OSINT tools are:

• Osintframework.com (link resides outside ibm.com) – An extensive directory of free, online OSINT tools and resources hosted on the developer platform GitHub. Both hackers and cybersecurity professionals can use this directory as a starting point to drill down into the specific functionality they seek in an OSINT tool.

• Maltego (link resides outside ibm.com) – A real-time data mining solution for Windows, Mac and Linux platforms that provides graphic representations of data patterns and connections. With its ability to profile and track the online activities of individuals, this tool can be useful to both cybersecurity professionals and threat actors.

• Spiderfoot (link resides outside ibm.com) – A data source integration tool for information such as email addresses, phone numbers, IP addresses, subdomains and more. Ethical hackers may use this resource to investigate publicly available information that could pose a threat to an organization or an individual.

• Shodan (link resides outside ibm.com) – A search engine for internet-connected devices that can also provide information on metadata and open ports. Because this tool can identify security vulnerabilities for millions of devices, it can be useful to both cybersecurity professionals and cybercriminals.

• Babel X (link resides outside ibm.com) – A multilingual, AI-enabled search tool capable of searching the world wide web and dark web in more than 200 languages. Security teams within an organization may use this tool to search for sensitive or proprietary information that may be posted on the dark web or in a foreign country.

• Metasploit (link resides outside ibm.com) – A penetration testing tool that can identify security vulnerabilities in networks, systems and applications. Both cybersecurity professionals and hackers find value in this tool because it can expose the specific weaknesses that may enable a successful cyberattack.