Welcome Turk Hack Team Members,
In July 2014, Malaysian Airlines Flight MH17, from Amsterdam to Kuala Lumpur, crashed some 50km from the Ukrainian-Russian border. All 298 passengers and crew on board the Boeing 777 lost their lives.
In the aftermath of the incident, separatists from Donetsk claimed to have shot down a Ukrainian transport aircraft, claims they later withdrew. The Ukrainian authorities said separatists had downed the airliner. The countrys president called the incident an act of terrorism.
Several independent and official investigations followed. One of the most comprehensive was conducted by Bellingcat, the investigative and citizen journalism group.
Bellingcat, along with other media organizations, used open source intelligence (OSINT) techniques and methodology to build a timeline of the incident, and to expose Russian claims and counterclaims as fabrications.
What is OSINT?
OSINT is intelligence drawn from publicly available material, according to the CIA. Most intelligence experts extend that definition to mean information intended for public consumption.
OSINT is information that can be accessed without specialist skills or tools, although it can include sources only available to subscribers, such as newspaper content behind a paywall, or subscription journals.
The CIA says that OSINT includes information gathered from the internet, mass media, specialist journals and research, photos, and geospatial information. Most of these sources were used in the Bellingcat MH17 investigation.
OSINT does not require its exponents to hack into systems or use private credentials to access data. Viewing someones public profile on social media is OSINT; using their login details to unearth private information is not. In intelligence agency terms, OSINT is also information drawn from non-classified sources.
When were OSINT techniques first used?
Open source intelligence predates the internet. Governments have long used newspapers, and later broadcasts, to track potential adversaries military, political, or economic plans and activities.
OSINT is low risk, cheap, and often highly effective, as corporate intelligence consultant Cameron Colquhoun has written in a Bellingcat article on the history of OSINT.
As Colquhoun suggests, OSINT fell out of fashion after World War Two, with intelligence agencies instead focusing on the more glamorous and dangerous world of HUMINT human intelligence or spying and SIGINT: signals and electronic intelligence.
But with the rise of the internet and social media, and online tools that can sift through vast amounts of information, OSINT is now more relevant than ever.
Not just for spies: OSINT and cybersecurity
Intelligence agencies use OSINT to track events, equipment such as weapons systems, and people. These are the targets of interest (ToIs).
But hackers use OSINT to identify technical vulnerabilities as well as human targets for phishing and social engineering attacks. As a result, pen testing and security teams deploy similar techniques to find and close down weaknesses.
When explaining OSINT and how damaging it can be to clients, I do like the analogy of putting up a big poster in your front window with all your information on, Liam Follin, penetration tester and web application security consultant at Pentest People, told The Daily Swig.
We know where to look and we have the tools. But the nefarious side of hacking, the black hats, also know where to look.
OSINT helps security teams unearth clues that individuals leave in the open that compromise security. Like using a vulnerability scanner to find flaws in systems, OSINT tools pick up on problem data, such as dates of birth, Social Security numbers, family members or even hobbies that could help attackers compromise an account.
Common OSINT techniques
There is no single playbook for OSINT: most pen testers have their own methods and preferred tools.
This often starts with manual reconnaissance, and reading up on the target subjects, including using non-technical sources such as an organizations annual report, financial filings, and associated news coverage, as well as content on its websites and YouTube and similar services.
Anything you can obtain via online as well as by traditional media research can be used for OSINT. And one of the simplest and yet most effective tools is using search engines, Anita Bielicka, a cyber threat intelligence researcher at Orpheus, told The Daily Swig.
A hacker will also search for information on employees, cross checking against external social media and professional profiles.
If I can identify the likely members of the development team and identify what they have done in public source code repositories, the technology meetups they have participated in, their blog posts, I can start to build a profile for how strong their skillset is and also an understanding of the types of mistakes they are likely to make, Tim Mackey, principal security strategist at the Synopsys Cybersecurity Research Centre, told The Daily Swig.
If I know what mistakes theyve historically made, I can potentially build a targeted attack against that particular weakness in their coding that all becomes part of the corpus of information we can use to define a potential attack.
And although OSINT is often associated with social media and human tooIs, it can equally be used alongside security scanners to find problems with physical assets and IT systems, for example via a Shodan search.
And researchers will look at GitHub, Stack Overflow, Reddit, vendor support forums, and even job listings for insights on a targets technology.
Since almost everyone uses LinkedIn, there is a massive amount of information there, Andreas Georgiou, security consultant at Trustwave Spiderlabs, told The Daily Swig. If you can find out how an email or username is constructed, you can look at past credential leaks, and have a good chance of finding a way into a network.
OSINT in the open examples of open source intelligence
Pentest Peoples Follin recalls an OSINT engagement that found floor plans of a sensitive Iocation online, and another where an online photo contained enough information to copy a keycard. Both could compromise the physical security of an organization.
This shows why OSINT is a valuable tool for raising security awareness, as well as a technical tool for identifying security risks.
Organizations are potentially enabling cyber-attacks against themselves through the information they publish online, James Dale, penetration testing and red team lead at PA Consulting, told The Daily Swig.
OSINT is harvesting data from legitimate sources such as online search engines, websites, and professional social networks. But our cybersecurity experts have conducted client OSINT assessments and discovered information such as versions of software, names of devices used to print docments, and email addresses.
Along with obvious sources, such as a company website and LinkedIn, this information can also be gathered through métadata stored within files created and published by an organization.
Even fairly trivial information can have big security consequences, warns Dale. A pets name, or the version of Office used to create a docment, may seem insignificant, but it can be used to inform a potential cyber-attack, he says.
Other increasingly important OSINT sources are open data feeds and geospatial information, from Google and other mapping tools.
And the use of OSINT can go even deeper, potentially right into the code of a companys web applications.
If a security team can find those OSINT weaknesses first, they can move to close them down.
Social media and OSINT
Although online media and search engines make OSINT quicker and easier, social media has been the most effective medium for gathering information on individuals with a view to defrauding them or stealing their identities.
Social media has made [conducting] OSINT on people super easy, Drew Porter, president and founder at security firm Red Mesa, told The Daily Swig.
"People willingly put their info out there. If a person is in scope, looking at their social media pages for 15 minutes will tell us more about the person then 1 hour of OSINT will most the time.
Hackers, pen testers, and intelligence agencies view social media profiles on sites such as LinkedIn, Facebook, and Instagram as fair game for OSINT. Some firms use scraping tools, although these are against most sites terms, including LinkedIn for example.
This, of course, will not deter malicious actors. And although security teams have to be careful with social media, trawling these platforms can quickly reveal serious security flaws.
Social media analysis (or SOCMINT) is a subsection of OSINT, although its value can be hampered by privacy and platform restrictions, Louise Taggart, manager for cyber threat detection and response, and Kirsten Ward, senior associate for threat intelligence, at PwC UK, told The Daily Swig.
Sources can include social networking sites, professional networking sites, video sharing or vlog sites, or microblogging sites, for example.
Social media platforms can betray a considerable amount of information on a person or organization, whether it is through a seemingly innocent picture, which could be the answer to a secret question, or a colleague wishing someone happy birthday and thereby inadvertently disclosing their date of birth.
Security teams can use social media as an entry point for social engineering, or for physical site penetration. But the breadth of social media, including images and video, means it can all too easily provide malicious actors with information about security systems and IT, often without the business realizing. With no system compromise to detect, OSINT recon stays well below the radar.
Social networks have long been used in pen testing and red team engagements, Jordan Cheal, senior security consultant at Bridewell Consulting, told The Daily Swig.
They can provide a host of information for performing passive reconnaissance or enumerating data that can be weaponized and used in phishing campaigns or for performing password sprays.
Is OSINT legal or ethical?
In the US and the UK, OSINT is legal, but security teams need to stay within a clearly defined framework, which is agreed with their clients in advance of conducting OSINT.
Much will depend on where target information resides. OSINT that gathers information where there is a reasonable expectation of public access a blog post or a LinkedIn profile, for instance is generally considered legal. But where data are password-protected, obtained by deception, or anonymized and aggregated, the legality is less clear-cut.
It often comes down to intent, says Red Mesas Porter. When we are hired to do it for companies or high net worth individuals it is 100% ethical. When someone is doing it to stalk an ex, its not ethical.
At PwC, Taggart and Ward point to the Berkeley Protocol, which sets out a framework for conducting open source investigations into war crimes and human rights violations, as a useful ethical standard for cybersecurity-related OSINT, too.
The right choice?
Should, then, security teams use OSINT?
If they are not already, they are overlooking a vital means of spotting, and removing, sensitive information from the public domain that could be abused by malicious actors to compromise an organization. And, unlike many other methods and tools, OSINT is largely free.
Set against these benefits are the legal and ethical considerations, and the fact that security teams must exercise a certain level of skill and caution to use OSINT effectively, ethically, and legally.
Its also important to bear in mind, that whatever a good guy can find through OSINT, so can the bad guy, warn PwCs Taggart and Ward.
OSINT tools: An expanding list
Pentesters use a wide range of tools for OSINT, with consultants often using their own tools. Some of these, such as Pentest Peoples Athena, are available on GitHub. Here are some other popular OSINT tools:
Ghunt: a tool for finding information associated with a Google ID
Intel Owl: pulls together threat analysis tool feeds into a single API
ReNgine: an open source tool for aggregating recon feeds
Shodan: IoT device search engine used to find unsecured equipment on LANs and other hardware-based weak spots
Social Mapper: developed by Trustwave Spiderlabs, Social Mapper uses facial recognition, as well as usernames, to track targets across platforms
Spiderfoot: OSINT automation tool, available in open source and commercial versions
Sublist3r: Python-based sub-domain enumerator
theHarvester: Helps to determine a company's external threat landscape on the internet by gathering emails, names, subdomains, IPs and URLs
Google Dorking: less a tool than a technique, Google Dorking involves using specialist search terms to find results not visible to natural language search
//Quoted. Thanks for reading.
In July 2014, Malaysian Airlines Flight MH17, from Amsterdam to Kuala Lumpur, crashed some 50km from the Ukrainian-Russian border. All 298 passengers and crew on board the Boeing 777 lost their lives.
In the aftermath of the incident, separatists from Donetsk claimed to have shot down a Ukrainian transport aircraft, claims they later withdrew. The Ukrainian authorities said separatists had downed the airliner. The countrys president called the incident an act of terrorism.
Several independent and official investigations followed. One of the most comprehensive was conducted by Bellingcat, the investigative and citizen journalism group.
Bellingcat, along with other media organizations, used open source intelligence (OSINT) techniques and methodology to build a timeline of the incident, and to expose Russian claims and counterclaims as fabrications.
What is OSINT?
OSINT is intelligence drawn from publicly available material, according to the CIA. Most intelligence experts extend that definition to mean information intended for public consumption.
OSINT is information that can be accessed without specialist skills or tools, although it can include sources only available to subscribers, such as newspaper content behind a paywall, or subscription journals.
The CIA says that OSINT includes information gathered from the internet, mass media, specialist journals and research, photos, and geospatial information. Most of these sources were used in the Bellingcat MH17 investigation.
OSINT does not require its exponents to hack into systems or use private credentials to access data. Viewing someones public profile on social media is OSINT; using their login details to unearth private information is not. In intelligence agency terms, OSINT is also information drawn from non-classified sources.
When were OSINT techniques first used?
Open source intelligence predates the internet. Governments have long used newspapers, and later broadcasts, to track potential adversaries military, political, or economic plans and activities.
OSINT is low risk, cheap, and often highly effective, as corporate intelligence consultant Cameron Colquhoun has written in a Bellingcat article on the history of OSINT.
As Colquhoun suggests, OSINT fell out of fashion after World War Two, with intelligence agencies instead focusing on the more glamorous and dangerous world of HUMINT human intelligence or spying and SIGINT: signals and electronic intelligence.
But with the rise of the internet and social media, and online tools that can sift through vast amounts of information, OSINT is now more relevant than ever.
Not just for spies: OSINT and cybersecurity
Intelligence agencies use OSINT to track events, equipment such as weapons systems, and people. These are the targets of interest (ToIs).
But hackers use OSINT to identify technical vulnerabilities as well as human targets for phishing and social engineering attacks. As a result, pen testing and security teams deploy similar techniques to find and close down weaknesses.
When explaining OSINT and how damaging it can be to clients, I do like the analogy of putting up a big poster in your front window with all your information on, Liam Follin, penetration tester and web application security consultant at Pentest People, told The Daily Swig.
We know where to look and we have the tools. But the nefarious side of hacking, the black hats, also know where to look.
OSINT helps security teams unearth clues that individuals leave in the open that compromise security. Like using a vulnerability scanner to find flaws in systems, OSINT tools pick up on problem data, such as dates of birth, Social Security numbers, family members or even hobbies that could help attackers compromise an account.
Common OSINT techniques
There is no single playbook for OSINT: most pen testers have their own methods and preferred tools.
This often starts with manual reconnaissance, and reading up on the target subjects, including using non-technical sources such as an organizations annual report, financial filings, and associated news coverage, as well as content on its websites and YouTube and similar services.
Anything you can obtain via online as well as by traditional media research can be used for OSINT. And one of the simplest and yet most effective tools is using search engines, Anita Bielicka, a cyber threat intelligence researcher at Orpheus, told The Daily Swig.
A hacker will also search for information on employees, cross checking against external social media and professional profiles.
If I can identify the likely members of the development team and identify what they have done in public source code repositories, the technology meetups they have participated in, their blog posts, I can start to build a profile for how strong their skillset is and also an understanding of the types of mistakes they are likely to make, Tim Mackey, principal security strategist at the Synopsys Cybersecurity Research Centre, told The Daily Swig.
If I know what mistakes theyve historically made, I can potentially build a targeted attack against that particular weakness in their coding that all becomes part of the corpus of information we can use to define a potential attack.
And although OSINT is often associated with social media and human tooIs, it can equally be used alongside security scanners to find problems with physical assets and IT systems, for example via a Shodan search.
And researchers will look at GitHub, Stack Overflow, Reddit, vendor support forums, and even job listings for insights on a targets technology.
Since almost everyone uses LinkedIn, there is a massive amount of information there, Andreas Georgiou, security consultant at Trustwave Spiderlabs, told The Daily Swig. If you can find out how an email or username is constructed, you can look at past credential leaks, and have a good chance of finding a way into a network.
OSINT in the open examples of open source intelligence
Pentest Peoples Follin recalls an OSINT engagement that found floor plans of a sensitive Iocation online, and another where an online photo contained enough information to copy a keycard. Both could compromise the physical security of an organization.
This shows why OSINT is a valuable tool for raising security awareness, as well as a technical tool for identifying security risks.
Organizations are potentially enabling cyber-attacks against themselves through the information they publish online, James Dale, penetration testing and red team lead at PA Consulting, told The Daily Swig.
OSINT is harvesting data from legitimate sources such as online search engines, websites, and professional social networks. But our cybersecurity experts have conducted client OSINT assessments and discovered information such as versions of software, names of devices used to print docments, and email addresses.
Along with obvious sources, such as a company website and LinkedIn, this information can also be gathered through métadata stored within files created and published by an organization.
Even fairly trivial information can have big security consequences, warns Dale. A pets name, or the version of Office used to create a docment, may seem insignificant, but it can be used to inform a potential cyber-attack, he says.
Other increasingly important OSINT sources are open data feeds and geospatial information, from Google and other mapping tools.
And the use of OSINT can go even deeper, potentially right into the code of a companys web applications.
If a security team can find those OSINT weaknesses first, they can move to close them down.
Social media and OSINT
Although online media and search engines make OSINT quicker and easier, social media has been the most effective medium for gathering information on individuals with a view to defrauding them or stealing their identities.
Social media has made [conducting] OSINT on people super easy, Drew Porter, president and founder at security firm Red Mesa, told The Daily Swig.
"People willingly put their info out there. If a person is in scope, looking at their social media pages for 15 minutes will tell us more about the person then 1 hour of OSINT will most the time.
Hackers, pen testers, and intelligence agencies view social media profiles on sites such as LinkedIn, Facebook, and Instagram as fair game for OSINT. Some firms use scraping tools, although these are against most sites terms, including LinkedIn for example.
This, of course, will not deter malicious actors. And although security teams have to be careful with social media, trawling these platforms can quickly reveal serious security flaws.
Social media analysis (or SOCMINT) is a subsection of OSINT, although its value can be hampered by privacy and platform restrictions, Louise Taggart, manager for cyber threat detection and response, and Kirsten Ward, senior associate for threat intelligence, at PwC UK, told The Daily Swig.
Sources can include social networking sites, professional networking sites, video sharing or vlog sites, or microblogging sites, for example.
Social media platforms can betray a considerable amount of information on a person or organization, whether it is through a seemingly innocent picture, which could be the answer to a secret question, or a colleague wishing someone happy birthday and thereby inadvertently disclosing their date of birth.
Security teams can use social media as an entry point for social engineering, or for physical site penetration. But the breadth of social media, including images and video, means it can all too easily provide malicious actors with information about security systems and IT, often without the business realizing. With no system compromise to detect, OSINT recon stays well below the radar.
Social networks have long been used in pen testing and red team engagements, Jordan Cheal, senior security consultant at Bridewell Consulting, told The Daily Swig.
They can provide a host of information for performing passive reconnaissance or enumerating data that can be weaponized and used in phishing campaigns or for performing password sprays.
Is OSINT legal or ethical?
In the US and the UK, OSINT is legal, but security teams need to stay within a clearly defined framework, which is agreed with their clients in advance of conducting OSINT.
Much will depend on where target information resides. OSINT that gathers information where there is a reasonable expectation of public access a blog post or a LinkedIn profile, for instance is generally considered legal. But where data are password-protected, obtained by deception, or anonymized and aggregated, the legality is less clear-cut.
It often comes down to intent, says Red Mesas Porter. When we are hired to do it for companies or high net worth individuals it is 100% ethical. When someone is doing it to stalk an ex, its not ethical.
At PwC, Taggart and Ward point to the Berkeley Protocol, which sets out a framework for conducting open source investigations into war crimes and human rights violations, as a useful ethical standard for cybersecurity-related OSINT, too.
The right choice?
Should, then, security teams use OSINT?
If they are not already, they are overlooking a vital means of spotting, and removing, sensitive information from the public domain that could be abused by malicious actors to compromise an organization. And, unlike many other methods and tools, OSINT is largely free.
Set against these benefits are the legal and ethical considerations, and the fact that security teams must exercise a certain level of skill and caution to use OSINT effectively, ethically, and legally.
Its also important to bear in mind, that whatever a good guy can find through OSINT, so can the bad guy, warn PwCs Taggart and Ward.
OSINT tools: An expanding list
Pentesters use a wide range of tools for OSINT, with consultants often using their own tools. Some of these, such as Pentest Peoples Athena, are available on GitHub. Here are some other popular OSINT tools:
Ghunt: a tool for finding information associated with a Google ID
Intel Owl: pulls together threat analysis tool feeds into a single API
ReNgine: an open source tool for aggregating recon feeds
Shodan: IoT device search engine used to find unsecured equipment on LANs and other hardware-based weak spots
Social Mapper: developed by Trustwave Spiderlabs, Social Mapper uses facial recognition, as well as usernames, to track targets across platforms
Spiderfoot: OSINT automation tool, available in open source and commercial versions
Sublist3r: Python-based sub-domain enumerator
theHarvester: Helps to determine a company's external threat landscape on the internet by gathering emails, names, subdomains, IPs and URLs
Google Dorking: less a tool than a technique, Google Dorking involves using specialist search terms to find results not visible to natural language search
//Quoted. Thanks for reading.
