Digging for Gold: An Introduction to OSINT
- Alex Pack
- May 16, 2023
- 9 min read
An Introduction to OSINT for Counter-Terrorism Researchers and Academics
Recently, I had the opportunity to lecture to interns at the International Institute for Counter-Terrorism on the intelligence cycle, open-source intelligence, and how individuals can get started on counterterrorism research. Having given the lecture, it felt appropriate to share some of the ideas. While I will not release the original presentation, many of the points are covered in this blog post.
Introduction
In today's rapidly evolving digital landscape, Open-Source Intelligence (OSINT) has become an invaluable tool for researchers and analysts, particularly in the field of counter-terrorism. This post delves into the world of OSINT, its importance, the intelligence cycle, and how to effectively leverage search engines and techniques for your investigations.
What is OSINT?
Open-Source Intelligence (OSINT) is a form of intelligence collection that involves finding, selecting, and acquiring information from publicly available sources and analyzing it to produce actionable intelligence. In the intelligence community, the term "open" refers to overt, publicly available sources (as opposed to covert or clandestine sources).
OSINT can include a wide variety of sources, including:
Media: Newspapers, magazines, radio, television
Public Government Data: Government reports, official data such as budgets and demographics, hearings, legislative debates, press conferences, speeches, marine and aeronautical safety warnings, environmental impact statements, and contract awards
Academic Literature: Information acquired from journals, conferences, symposia, academic papers, dissertations, and theses
Professional Literature: Technical reports, preprints, patents, working papers, business documents, unpublished works, and newsletters
Web Sources: Blogs, forums, social networks, video-sharing sites, wikis, and other public data available online
The key principle of open-source intelligence is that the information is available to anyone who can search for and retrieve it. In this digital age, where the amount of data produced every day is vast, OSINT practitioners need to be able to effectively and efficiently find, collect, organize, analyze, and make conclusions from a variety of sources.
Benefits of OSINT
For counterterrorism researchers and academics, the primary benefit of OSINT is that it uses publicly available sources to gather data, which is then analyzed to create a comprehensive intelligence product. While government intelligence analysts will typically have access to other forms of intelligence, such as human intelligence (HUMINT) or signals intelligence (SIGINT), to form their intelligence picture, academics and researchers are limited in the tools that are available. The democratization of publically accessible data, its minimal (if any cost), and the relatively low risk of OSINT have offered significant opportunities to researchers and academics to conduct research and better assess the security environment.
Challenges of OSINT
Despite these advantages of OSINT, there are still disadvantages. Specifically, the vast amount of data available can, at times, be overwhelming to process, thus making it more difficult to identify the intelligence picture or recognize the important trends. The central skill of an OSINT practitioner is being able to discriminate valuable information from background noise by analyzing, evaluating, and interpreting that information in context, often corroborating it with information from other sources.
The Intelligence Cycle
Before going further, it is important to mention the intelligence cycle. While the intelligence cycle is generally used by analysts and decision-makers in the intelligence community, it is also applicable to academic researchers and analysts using OSINT. The intelligence cycle can be broken down into five overlapping phases:
Planning & Direction - Defining the intelligence requirements and formulating a plan for data collection. This stage involves determining what information is needed and establishing priorities, timelines, and strategies for obtaining the data.
Collection - During this stage, researchers explore different sources and platforms to collect relevant and reliable information using a combination of manual and automated techniques.
Processing & Exploitation - Converting raw data into a structured format for further analysis. This may involve filtering and organizing the data, as well as translating or transcribing content if necessary. During this stage, researchers can also leverage newly collected information to reassess their collection plan.
Analysis & Production - Evaluating the processed information, identifying patterns, and generating actionable insights. Researchers use various analytical techniques and tools to interpret the data, assess its credibility, and draw conclusions that address the intelligence requirements set during the Planning & Direction phase.
Dissemination & Evaluation - Sharing the intelligence with relevant stakeholders and assessing its impact and effectiveness. This stage ensures that the insights generated are communicated to the appropriate parties while also assessing whether the goals set during the Planning & Direction phase were reached. This ultimately allows the researchers to improve their OSINT investigations/research in the future.
Applying the Intelligence Cycle to OSINT
The intelligence cycle can be tailored to OSINT by focusing on open sources for data collection and leveraging advanced search techniques during the planning and direction phase. By following this process, researchers can ensure their OSINT efforts are targeted, systematic, and, ultimately, produce valuable findings.
For example, during the planning and direction stage, researchers might identify specific social media platforms, websites, or forums as potential sources of information. They can then use advanced search operators to refine their queries and obtain precise results during the collection phase. Throughout the processing and exploitation stages, researchers should focus on converting the collected data into a structured format that can be more easily analyzed. Finally, during the analysis and production stage, researchers can draw on their expertise to assess the credibility of the information and its implications for counter-terrorism efforts.
OSINT & Prospecting: Analagous Activities
When it comes to OSINT research and analysis, one could liken it to the process of sifting for gold using a funnel and a sieve. In the world of OSINT, the "gold" is the valuable, actionable intelligence or results of your research, while the riverbed filled with sand and water symbolizes the vast expanse of publicly available information that you need to sort through.
The Funnel
Like a gold prospector uses a funnel to channel the raw material from a riverbed, an OSINT analyst begins by casting a wide net over the sea of public information. This phase aligns with the initial stages of OSINT collection. The wide-open top of the funnel represents the collection of large amounts of raw data from many different sources - social media, news articles, databases, and more. As the funnel narrows towards the bottom, it mirrors how the analyst begins to filter and focus on specific information that seems relevant to their focus areas or topics. This narrowing action in the funnel is analogous to the "Planning & Direction" and the initial "Collection" stages of the intelligence cycle, where data is identified and gathered based on its relevance to the set objectives.
The Sieve
After the funnel has been used to manage the flow of information, a sieve comes into play. The sieve represents the 'Processing & Exploitation' and 'Analysis & Production' stages of the intelligence cycle. Like a sieve is used to separate gold nuggets from the sand, the analyst filters the gathered data, discarding irrelevant or less valuable information. The valuable nuggets of intelligence are akin to the actionable insights that meet the defined intelligence requirements.
The mesh holes in the sieve could be compared to the analytical frameworks, methodologies, and tools that the analyst uses to identify valuable insights from the data. Whereas the gold prospector shakes the sieve to drop away the less valuable sand, dirt, and mud away from the gold, the OSINT research needs to examine the data from different angles and use different frameworks to systematically separate critical information from noise. Also, like the prospector may have to sift their materials through multiple rounds of funnel and sieve to find the gold nuggets, OSINT researchers may need to conduct multiple rounds of collection and analysis to refine the data into valuable insights. This comparison highlights the precision and patience needed for successful OSINT research.
The Collection Process: From Search Engines to Dorking
The collection process is a crucial step in the intelligence cycle and arguably one of the most significant phases in OSINT. This is where your search for information begins. It's worth mentioning that the search engines you choose to use can significantly impact your research. Search engines like Google, Bing, and Baidu are excellent tools for initiating your OSINT research. Each one has its own strengths and weaknesses, and savvy researchers know how to leverage each to their advantage.
For instance, Google has a broad reach and a sophisticated algorithm that provides highly relevant results, making it an ideal tool for general searches. Baidu is the leading search engine in China, making it an invaluable tool when searching for information from or about China. While Yandex, another search engine, specializes in reverse image searching, making it an ideal place to start when trying to engage in location identification. Knowing the strengths and weaknesses of each search engine can help you decide which one to use based on your specific intelligence requirements.
Once you've chosen your search engine, it's time to dive into the data. But with the vast amount of information available on the internet, how do you find the most relevant results? This is where "Google Dorking" or advanced search operators come into play. These operators allow you to refine your search queries and narrow down the results. For example, you can use quotation marks to search for an exact phrase, the minus sign to exclude a word from your search, or the site: operator to search within a specific website. These operators can significantly improve the efficiency of your search and ensure that you're finding the most relevant information. However, always remember that different search engines may use different search operators, so be sure to familiarize yourself with the ones used by the search engine you're using.
Choosing the Right Browser: Interoperability & Privacy
As you venture deeper into the world of OSINT, the choice of the browser becomes a significant consideration. The browser you use for OSINT research should ideally be separate from the one you use for personal activities. This separation is not only a good security practice but also helps keep your research and personal digital lives organized.
Different browsers offer varying degrees of functionality, privacy, and interoperability with apps and plug-ins. It's essential to understand the trade-offs associated with each to choose the best one for your needs.
On one end of the spectrum, browsers like Google Chrome offer exceptional functionality and interoperability. They are compatible with a multitude of apps and plug-ins, many of which have been developed specifically for OSINT research. However, these browsers are also known to track your activities extensively, potentially leading to data leakage about your research activities. Therefore, while these browsers may offer a high level of convenience and functionality, they may not be the best choice if privacy is a major concern for you.
On the other end of the spectrum, browsers like Tor are designed with privacy and anonymity as their primary focus. These browsers go to great lengths to protect your activities from being tracked. However, this high level of privacy comes at the cost of functionality and interoperability. Many apps and plug-ins that are useful for OSINT research are not compatible with these browsers. As such, while these browsers are excellent for maintaining privacy, they may limit the tools and techniques available for your OSINT investigations.
In the middle of the spectrum lies Mozilla Firefox. Firefox offers a balanced blend of interoperability and privacy, making it a popular choice for many OSINT researchers, including myself. Firefox is compatible with a wide variety of apps and plug-ins useful for OSINT, and it provides more privacy protections than browsers like Google Chrome. Finally, Firefox allows for extensive customization, enabling you to tweak its settings to suit your specific needs.
Ultimately, the choice of browser for OSINT research is not one-size-fits-all. It depends on your specific needs, the nature of your research, and your personal preferences. Some researchers might prioritize functionality and interoperability and opt for a browser like Google Chrome, while others might prioritize privacy and choose a browser like Tor. The key is to understand the trade-offs and make an informed decision that best serves your OSINT research.
Beginning Your OSINT Journey
This post has provided an introduction to Open-Source Intelligence (OSINT), highlighting its importance for counter-terrorism researchers and academics. It has explored what OSINT is, why it matters, and how it's collected and processed through the intelligence cycle. It has also delved into how to effectively use search engines and advanced search operators for OSINT investigations and how to choose the ideal browser for your research.
However, it is important to note that the world of OSINT is vast and complex, and this post merely scratches the surface. There are many more aspects to consider, including various OSINT sub-disciplines such as Image Intelligence (IMINT) and Geospatial Intelligence (GEOINT), which offer their own unique tools and techniques.
As you embark on your journey into the world of OSINT, keep in mind a quote attributed to Allen Dulles, Fifth Director of the CIA: "A proper analysis of the intelligence obtainable by these overt, normal, and aboveboard means would supply us with over 80 percent, I should estimate, of the information required for the guidance of our national policy."
Keep developing your skills, keep learning, and keep digging for gold. The world of OSINT is vast, and you never know what insights you might uncover.
Disclaimer:
While the primary author of this article is fully human, some sections were sprinkled with a dash of AI magic, courtesy of a popular, friendly AI assistant. But don't worry, it wasn't anything like that scene from "2001: A Space Odyssey." No computers were harmed, no rogue AIs took control, and the author still had to do the heavy lifting of thinking, researching, and typing out the majority of the article.
So, if you found a particular turn of phrase charming or a sentence structure intriguing, there's a chance it was born from a harmonious collaboration of human and artificial intelligence. However, if you spot any typos or grammatical errors, those are all human.
In a nutshell, this blog post was brought to you by the power of human expertise, a dash of AI wizardry, and several all-nighters. Now, back to your regularly scheduled programming of OSINT wisdom!
Comments