MILITARY VALUE OF OPEN SOURCE INTELLIGENCE IN THE INDIAN CONTEXT: A MYTH OR REALITY?

Ninety per cent of intelligence comes from open source. The ten per cent, the clandestine work, is just more dramatic. The real intelligence hero is Sherlock Holmes, not James Bond.

— Lieutenant General Sam Wilson, former Director, Defence Intelligence Agency, United States¹

Cutting Through the Fog of War

In 2018, the United States Central Intelligence Agency (CIA) was taken by surprise when the classified locations of underground US military bases in Syria and Djibouti were disclosed in the open domain by a 20-year-old Australian student, Nathan Ruser. He came across the popular running app Strava, which had released data of the runners’ heatmaps. These heatmaps provide visualisation of the trails and intensity of activity hotspots in different colours on the map, which is publicly available for all runners to see. This was considered harmless, but upon closer inspection Nathan could identify the runners in the middle of random deserts in the Middle East and North Africa as American soldiers by their Strava usernames and groups. They were running distinct 10 km laps daily. Thus, in a matter of minutes, while sitting in his dormitory in Australia, Nathan managed to identify the locations of highly classified American military installations.²

Similarly, it was a Twitter user by the name of Dr Jeffrey Lewis who was the first to break the news about the Russian invasion of Ukraine, early in the morning of 24 February 2022. He used the freely downloadable Google Earth application to overlay commercially available high-resolution satellite imagery of the previous day, showing a large Russian military unit with tanks and armoured personnel carriers (APCs) on Google Maps, indicating real-time traffic jams on the road from the city of Belgorod in Russia to the Ukrainian border. He could accurately analyse the cause of the traffic jams as the movement of a Russian armour formation towards Ukraine.³

Perhaps the most alarming exploitation of open-source information is to track military activities using the European Space Agency’s civilian satellite constellation Sentinel-1. This satellite constellation provides updated mapping, every six days, of the earth’s surface in the public domain.⁴ It uses synthetic aperture radar (SAR). Unlike commercial satellites, Sentinel-1 SAR radar can track military activities at night as well as through cloud cover. Automatic change detection based on the time-series analysis of these radar images would have operational use in military applications.⁵ It also has applications in the maritime domain to detect and track warships at sea.⁶ A well-regarded OSINT enthusiast, H. I. Sutton, also claims that the Sentinel-1 SAR radar transmits on 5.4 GHz and therefore other military radars transmitting on similar frequencies can be mapped on the ground or detected at sea using an open-source technical tool called the 5 GHz Interference Tracker.⁷

These recent examples illustrate the growing significance of OSINT as a discipline of military intelligence (MI) collection and analysis among the other more traditional disciplines, that is human intelligence (HUMINT), signals intelligence (SIGINT), imagery intelligence (IMINT), measurement and signature intelligence (MASINT), and so on. The Russia–Ukraine war has strengthened the view among military planners that the ‘crowd-sourcing’ of intelligence can conspicuously reduce the fog of war.⁸ In an age of information explosion, each individual with a phone, PC and fast internet connection is like a sensor on the ground in the real world. This populates the internet with information and, when this information is analysed with sharp and critical thinking, away from ‘ground zero,’ it provides the big picture and insights that could reliably support covert intelligence collection efforts.

OSINT 101

OSINT is colloquially considered as intelligence derived from open source or, in other words, publicly available information. NATO’s handbook on it segregates publicly available information into four categories:⁹

• Open-Source Data (OSD). This is raw data collected from a primary source such as recorded conversation tapes, geospatial data in the form of satellite imageries and maps, records of electronic transmission, situation reports, and so on.
• Open-Source Information (OSIF). This is open-source data organised as generic information through an elementary filtering and validation process. It includes news articles, social media posts, academic papers, and so on, and helps in gathering insights, tracking events, identifying trends and understanding narratives.
• Open-Source Intelligence (OSINT). This is the systematic analysis of OSIF to gain valuable insights to meet the information requirements for the military decision-making process.
• Validated OSINT. This is the OSINT that has been verified and authenticated by other means, such as confirmation by multiple sources including classified sources. It minimises the risk of deception by adversaries.

OSINT Misconceptions

There are a few misconceptions about OSINT that impede its comprehensive implementation and optimal exploitation by the intelligence communities. The most common belief is that classified information is more reliable and accurate since the information has been collected through covert means despite the efforts by the enemy to deny it. This belief, however, overlooks the possibility that the enemy may have provided false and misleading information using deception. OSINT, on the other hand, utilises publicly available information which has a high chance of being more accurate and reliable as it involves multiple stakeholders such as government organisations, journalists, academic researchers, and so on. Any discrepancies or irregularities in this publicly available information are likely to be highlighted by other stakeholders. This by no means suggests that OSINT is superior to classified intelligence. However, it should not be dismissed as less credible.¹⁰

Secondly, there exists a belief that anyone with access to the internet can generate OSINT. Many intelligence analysts can indeed carry out rudimentary internet searches and exploit the basic OSINT tools. However, the internet is only one among the many sources of OSINT, the others being news and media comprising print, podcasts and TV broadcasts, grey literature from inter-government institutions and think tanks, satellite imagery, social media as well as the dark web, since OSINT includes them, with much content behind paywalls.¹¹ An OSINT analyst would be capable of performing sophisticated web searches and possess multiple skill sets, including technological proficiency to optimally exploit OSINT tools such as advanced geospatial tools and social media analysis, research skills to undertake systematic analysis of inputs from multiple OSINT sources, and an area specialisation to contextualise the available information and derive meaningful insights from it.

Strengths and Challenges of OSINT

OSINT is economical and is not resource-intensive. With the availability of personal computers, high-speed internet and access to public libraries, it is ubiquitous. Unlike other disciplines of intelligence, OSINT enables the sifting of voluminous information in a short time. It covers a wide spectrum of issues of interest, unlike classified means which need to be very focused. It allows an event to be studied and analysed over a wider timespan to enable the reconstruction of a past or recent past from a wider information pool. It can also track events in real time, such as the political, security and economic consequences of a terrorist attack. OSINT is a comparatively low-risk discipline of information collection for intelligence functions and is an important tool to assess the reliability and validity of the information obtained from classified sources.¹²

The most formidable challenge to OSINT is information overload. This is primarily because of the democratic nature of the internet. Therefore, OSINT operations require considerable analytical abilities, insights and subject knowledge to identify reliable information among heaps of information.¹³ This challenge is further compounded by sustained disinformation campaigns by vested interests of governments as well as non-state actors.

OSINT Application in the Armed Forces

OSINT, when given due weightage in military operations, has the potential to be both a force multiplier and a resource multiplier. It is not just a cost-efficient substitute for classified intelligence disciplines such as HUMINT, IMINT, SIGINT and MASINT. It complements classified intelligence at both stages, that is planning and conduct of military operations.¹⁴

• During out-of-area contingency operations, the immediate availability of intelligence on an area of interest can be a major handicap. During such contingencies, OSINT provides the operations staff with initial planning inputs and helps firm up the request for information (RFI) for more conventional classified intelligence disciplines.
• OSINT is also significant for optimising classified intelligence collection resources. With the identification of reliable commercial OSINT sources, not only are there considerable savings in terms of operations expenditure but such steps also allow niche classified intelligence capabilities to be focused on mission-critical gaps. One example is that analysed satellite imagery from commercial sources is a much cheaper and faster option than direct-owned resources during the initial planning phase of an operation.
• Most importantly, the information from OSINT could be given a low-security classification and thus can be freely shared with the other departments of the government as well as with friendly countries without revealing one’s capabilities and limitations in intelligence, surveillance and reconnaissance (ISR).

The most valued intelligence is usually the product of dynamic and continuous fusion of information from all available sources, both from open sources and that collected through technical and HUMINT efforts. Here, open-source intelligence can provide the red flags and alarms for major military, scientific, political and economic events as well as help improve the accuracy of analyses with background contexts.

This aspect was adequately highlighted by the Kargil Review Committee (KRC) constituted in 1999. The KRC report alluded to the importance of OSINT when it says, “The intelligence agencies in other countries have developed large ‘White wings’ of high-quality analysts for in-house analysis. They also contract studies with university departments and think tanks with area specialisations. That is sadly neglected in India.”¹⁵ The KRC report also mentions that “one of the most realistic assessments of Kashmir developments, as they unfolded during Pakistan’s proxy war, was Operation Topac, a war game, written by a team of retired Army officers in 1989.”¹⁶ Topac, mentioned in the KRC report, was an anticipated Pakistan plan for subversion of the J&K government, which was published in the July 1989 edition of the magazine Indian Defence Review (IDR). In hindsight, there is a strong possibility that if the R&AW and the Indian Army had considered the possibility of violation of the LoC by the Pakistani Army — as war-gamed by Topac — more resources in terms of commercial satellite imageries of the area, air effort from the Aviation Research Centre (ARC) for Winter Air Surveillance Operations (WASO) and HUMINT could have been requisitioned. Possibly then, the Pakistani build-up could have been detected earlier.

At the same time, however, open-source intelligence can be inherently misleading and biased, or perhaps even a deliberate deception by government agencies. This, therefore, requires that the information quality of the OSINT is carefully evaluated for the reliability of the source, that is confidence in the given source based on past reporting, and the credibility of the information, or confirmation with earlier reports or confirming other reports. This OSINT evaluation is similar to the information-quality evaluation of classified inputs.¹⁷ It is, in fact, contradictory information from open-source intelligence and classified inputs that are the red flags for the intelligence analyst to re-evaluate his analysis.

To cite an example, the inability of US intelligence agencies to detect Indian nuclear tests in May 1998 was called the “intelligence failure of the decade” by Richard Shelby, Chairman of the US Senate Intelligence Committee. Subsequent investigations into the causes of the intelligence failure highlighted that though the intent to carry out nuclear tests was displayed by the Indian political leadership as well as the Atomic Energy Commission Chairman, along with the Pokhran nuclear test site being under US satellite surveillance, intelligence agencies failed to detect India’s nuclear test preparation. A senior US administration official was quoted as saying, “We were all lulled into thinking that India would not do anything regarding a nuclear test. We made the mistake of assuming they would act rationally.”¹⁸

OSINT in the Indian Intelligence Set-up

Existing Indian Intelligence Structure. The higher intelligence set-up in India comprises the National Security Council (NSC) under the Prime Minister. Its members include the cabinet ministers for Home, Defence, External Affairs and Finance.¹⁹

Intelligence agencies under the control of these ministries include the Cabinet Secretariat exercising control over the Research and Analysis Wing (R&AW), Aviation Research Centre (ARC) and National Technical Research Organisation (NTRO). R&AW is responsible for all types of external intelligence, including political, economic, military and technological domains. ARC provides support to R&AW in terms of aerial reconnaissance, imagery analysis, communication and electronic intelligence.²⁰ NTRO is responsible for the design and setting up of technical facilities comprising COMINT, ELINT, IMINT and CYBERINT.²¹

• The Ministry of Defence (MoD) exercises control over the Defence Intelligence Agency (DIA), which is the nodal intelligence agency and synergises the functioning of the Services’ intelligence directorates. In addition, the tri-service agencies, that is the Signal Intelligence (SI) Directorate and the Defence Imagery Processing and Analysis Centre (DIPAC), also function under DIA.²²
• The Ministry of Home Affairs (MHA) exercises control over the Intelligence Bureau (IB), which is responsible for internal security. In addition, the MHA also exercises control over field intelligence units of the Central Armed Police Forces (CAPF). The Ministry of Finance (MoF) also has an Economic Intelligence Council overseeing government agencies responsible for economic intelligence and economic offences in India.²³

OSINT and Indian Intelligence. Two instances of intelligence failure often quoted in the context of the Indian intelligence setup are the previously mentioned Kargil intrusions of 1998 and the Mumbai attacks of 26 November 2008. In both these cases, the Kargil Review Committee and the Ram Pradhan Committee, constituted to study causes of intelligence failures, have highlighted that while information on enemy intent was available, it was the inability to connect the dots which resulted in intelligence failure.²⁴ The G. C. Saxena Task Force, constituted in May 2000 to examine the recommendations of the Kargil Review Committee related to the intelligence apparatus, has reportedly made observations on the degradation of HUMINT. The Naresh Chandra Task Force, constituted in June 2011 to holistically review national security, has recommended setting up “a dedicated new agency or department to generate intelligence based on information from open source and this agency must not be limited to providing advice and inputs to only the government but also to the Indian private sector.”²⁵

India’s Efforts at Automating Intelligence Processes

In the year 2022, almost 6.6 billion smartphone users, 70 per cent of the world population as internet users, 5,500 active satellites and an equally large number of apps, surveillance cameras and sensors have contributed to the information explosion in the open domain.²⁶ This increasing volume, velocity and variety of data flow from remote sensing technologies to social media posts necessitate that Indian intelligence agencies explore the application of advanced technologies to process huge amounts of data and derive useful intelligence from them.

Fig 1: Total Data Volume World wide 2010-2025

Source: https://www.statista.com/statistics/871513/worldwide-data-created/, accessed on 30 March 2023

Indian efforts to automate intelligence processes through Big Data were prompted after the 26/11 Mumbai attacks. A key planner of the attacks, David Coleman Headley, visited India through Pakistan and West Asia multiple times. However, these unusual travel itineraries were not detected by the systems in place for border and immigration controls, nor by background-check collation systems. In other words, there was a failure in connecting the dots to reveal a pattern.²⁷ The National Intelligence Grid (NATGRID) was then conceptualised in 2009, for the aggregation of several databases with a Google-search-like feature to automate intelligence processes, with C-DAC, Pune as the technology partner. However, NATGRID is an aggregation of classified databases and is accessible to only a few government agencies. It has overrun several deadlines and is still a work in progress.

The Way Ahead for OSINT in India

The most obvious lesson universally learned from the ongoing Russia–Ukraine war is that the fog of war is reducing with the proliferation of open-source information and the participation of stakeholders from the general public. Many ‘cottage-industry’ analysts seem to have emerged and are making extensive use of easily accessed technologies that leverage multi-dimensional, publicly available information. The investigative group Bellingcat, created by OSINT enthusiast Elliot Higgins, is one such group. Bellingcat has been credited for many insightful investigations on the Ukraine war, though the most notable investigations by the group are the shooting down of Malaysia Airlines Flight MH17 on 17 July 2014 over eastern Ukraine, where independent investigators attributed the crash to a hit by a Buk surface-to-air missile.²⁸ The suspected poisoning of former Russian double agent Sergei Skripal in Salisbury, England, on 4 March 2018, was attributed to Russian citizens whose identity was established through multiple leaked government databases and personal information.²⁹

The Indian intelligence setup today faces myriad challenges. At the macro level, national security policymakers and operational leaders across the lines of statecraft face complex, diverse and interconnected threats. These include an increasing threat from China; the need to leverage India’s growing cooperative relationship with the United States; and an expanding Indo-Pacific strategy to push for collaboration in niche defence technologies.³⁰ India observes or engages with extra-regional forces in the IOR and at the same time reassures the IOR rim countries of India as a cooperative security partner. These challenges are compounded by a lack of common understanding of the strategic needs of India among the departments of government and the general public.

At a functional level, intelligence collection by agencies can be ad hoc and disjointed as the exact intelligence requirements from the policymakers and operational leadership levels, in the form of requests for information (RFIs), may not be forthcoming. Moreover, silos within organisations tend to over-classify information that then prevents or restricts its availability for developing peer-reviewed assessments, and even operationally.³¹ At the micro level, the Indian intelligence setup suffers from a sufficient degree of analytical rigour and the ability to connect with area experts for peer review. Additionally, a shortage of language experts in some critical languages impacts and inhibits collection, collation and analytical activities.³²

In the present Indian intelligence setup, NTRO, which is primarily responsible for TECHINT and cyber operations, also monitors open-source information.³³ This does not augur well with the exponential rise of open-source data which needs to be filtered, evaluated and validated before being turned into credible and reliable information, as well as the increasing importance of OSINT in intelligence analysis and assessments. There is, therefore, a pressing need to mainstream OSINT in the intelligence analysis set-up to develop a common understanding of emerging threats and India’s strategic needs among various stakeholders in national security. Secondly, it is to rationalise the open-source information flow to intelligence analysts having access to classified intelligence inputs. These requirements necessitate the raising of an Open-Source Intelligence Centre at the national level, as envisioned by the Naresh Chandra Task Force.

Open-Source Intelligence Centre

The primary aim of the Open-Source Intelligence Centre would be to maximise the open-source information utility for policymaking needs and reduce uncertainty in decision-making. The broad approach would be to interface processed open-source information for validation through human feedback and produce analytically sound intelligence. This OSINT centre should collaborate with professionals from diverse fields and leverage technological advancements in Big Data, artificial intelligence (AI) and machine learning (ML) to validate open-source intelligence. At the conceptual level, the guiding principles for the OSINT centre would be:

• Shared knowledge. It should be based on the principle of shared knowledge and act as a coordination and distribution hub for open-source information between national security policy organisations, intelligence agencies, various ministries as well as other national security experts as required. The centre should encourage a common understanding of India’s strategic needs among the government departments and the general public by focusing on political ideologies, culture, military developments, intentions, evolving strategic interests, science and technology advancements, economy, and so on, of the countries of interest.
• Normalisation of OSINT usage. The hesitation of the intelligence community to accept open-source information of much value could be overcome by the OSINT centre by engaging a diverse workforce who already have a voice in national security policymaking. This workforce will comprise academicians, researchers, journalists, subject-matter experts, and so on, supported by in-service intelligence analysts and military personnel for validating open-source information. The aim should be to normalise the usage of OSINT as foundational intelligence for developing classified assessments.
• Near real-time analysis. Leveraging niche technologies for developing OSINT tools that are tailor-made to meet the requirements of intelligence agencies. This could be real-time battle-damage assessments through harnessing social media inputs, or tracking the movement of military formations by integrating satellite imagery and geotagged social media photographs with Google Maps, and so on. Further, tools based on sentiment analysis or tracking of live events on social media, news channels, publications, and so on, could be designed to provide predictive insights on military affairs, science and technological developments and geo-strategic issues.

At the functional level, the OSINT centre would aim to leverage advanced technologies for the automation of the OSINT process using Big Data, artificial intelligence (AI) and robot process automation (RPA). The automated OSINT process at the centre will involve four stages:

• Information collection. This phase involves defining the key search words for intelligence topics of interest and identifying the sources from which these topics will be gleaned. The source system of the proposed OSINT centre is set in such a way that the information collection is systematic and recurrent at defined intervals or hours of the day.
• Information processing. The automatically collected information could be in different formats and therefore, at the processing stage, all information — language, text and image — is converted into the same format. This processed information is also indexed by assigning attributes and metadata such as subject, place, event, and so on, for quick and efficient query searches.
• Information analysis. The analysis stage aims to validate the processed information by application of AI-based algorithms such as machine learning modelled on the human brain and reasoning, as well as natural language processing (NLP) to understand written and spoken words. The objective of AI applications in the analysis is to decode the dissemination pattern of the processed information in print and social media and establish correlation through clues about the author and the source based on information such as who, what, where, why, how, and so on.

• Information dissemination. The system-analysed information product can be automatically created at a pre-set frequency, say as of 0800h daily. These products could be sentiment analysis of news on various issues or a summary of a review of information by several sources and authors. The metadata attached to such information products can automatically correlate with the individuals and departments who will benefit, and therefore the information product is disseminated automatically on the network, by email, or as hard copies.

Validation of Open-Source Information. The automated OSINT process at the centre will now involve human intervention to generate the final validated open-source intelligence products. The automatically generated open-source information product, when disseminated to the OSINT analyst or a subject-matter expert, would be evaluated for information quality (IQ) of the product on various attributes such as accuracy, objectivity, currency, relevance, and so on. At this final stage, the credibility of the source and the reliability of the information will be assigned by the OSINT analyst before the product is distributed to government departments, including intelligence agencies relevant to national security.

Ideally, this validated open-source intelligence product will now be the foundational intelligence to direct the intelligence efforts from other domains such as HUMINT, SIGINT, IMINT, COMINT, and so on, or provide multiple hypotheses to the classified intelligence analysts to develop a credible all-source intelligence product.

Conclusion

The steep data curve, lessons learnt from the Russia–Ukraine conflict, and technological advancements for harnessing huge open-source data through automation of the processes for collection, processing and analysis, have necessitated the evaluation of opportunities that open-source information provides to the intelligence analysts’ communities. The United States National Security Strategy (NSS) has identified the integration of open-source information for exploitation by the intelligence community as a significant tool to sharpen American statecraft.³⁴ Similarly, the United Kingdom too is focused on OSINT by encouraging its usage, which not only cuts the cost but also permits a greater exchange of information for collaboration, in contrast to stove-pipe analysis and dissemination of classified intelligence.³⁵

To conclude, it may be said that Indian national security policy and operations face a complex, diverse and interconnected threat matrix that necessitates leveraging the open-source information explosion through the application of advanced AI- and ML-based OSINT tools. It now appears that the time is ripe to acknowledge OSINT as a vital military intelligence discipline along with HUMINT, SIGINT, IMINT, MASINT, and so on, and earnestly consider what General Anthony Charles Zinni, CinC US Central Command (2001–03), has said: “80% of what I need to know as CINCENT I got from open sources rather than classified reporting. And within the remaining 20%, if I knew what to look for, I found another 16%. At the end of it all, classified intelligence provided me, at best, with 4% of my command knowledge.”³⁶

Endnotes