Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chris Esther: OSINT, Maltego & Python

Chris Esther: OSINT, Maltego & Python

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Chris Esther:
OSINT, Maltego & Python
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2013 - Saturday, 07 Sep 2013 - Track 1
http://nz.pycon.org/

**Audience level**

Intermediate

**Description**

This is an introduction to OSINT, the Maltego OSINT visualisation tool and Canari a Python framework for creating custom transforms for Maltego.

**Abstract**

--- Open source intelligence (OSINT) ---

What is it? Who uses it and why?

--- OSINT Tools ---

An overview of the commercial products and Python tools available.

--- Maltego ---

An overview of the capabilties of commercial yet freely available tool. Including the ability to extend its functionality with scripting and custom transforms.

--- Canari ---

An introduction to the Python powered framework for creating and managing transforms for Maltego.

--- Example & Experience ---

My experiences in writing a custom transform written in Python using the Canari framework.

**YouTube**

http://www.youtube.com/watch?v=HkRuVNRbieU

New Zealand Python User Group

September 07, 2013
Tweet

More Decks by New Zealand Python User Group

Other Decks in Programming

Transcript

  1. Overview • Open Source Intelligence (OSINT) • OSINT Tools •

    Maltego • Canari • Companies Office Example
  2. US Legal Definition Open-source intelligence (OSINT) is intelligence that is

    produced from publicly available information and is collected, exploited, and disseminated in a timely manner to an appropriate audience for the purpose of addressing a specific intelligence requirement. Sec. 931 of the National Defense Authorization Act for Fiscal Year 2006
  3. OSINT != OSS • So what’s the relationship between Open

    Source Intelligence & Open Source Software?
  4. govt.nz ﺍا ﻟ ﺮ ﻗ ﺔ ﺗ ﺤ ﺮ ﻳﯾ

    ﺮ ﺟ ﺒ ﻬﮭ ﺔ Ar-Raqqah Liberation Front ﺍا ﻟ ﺜ ﻮ ﺭر ﻱي ﺍا ﻟ ﻌ ﺴ ﻜ ﺮ ﻱي ﺍا ﻟ ﻤ ﺠ ﻠ ﺲ Military and Revolutionary Council ﺍا ﻟ ﺸ ﻤ ﺎ ﻝل ﺇإ ﻋ ﺼ ﺎ ﺭر ﻟ ﻮ ﺍا ء North Storm Brigade ﺍا ﺣ ﻔ ﺎ ﺩد ﺍا ﻟ ﺮ ﺳ ﻮ ﻝل ﻟ ﻮ ﺍا ء The Prophet Offspring Brigade ﺍا ﻟ ﺮ ﻗ ﺔ ﺛ ﻮ ﺍا ﺭر ﻟ ﻮ ﺍا ء Ar-Raqqah Rebels Brigade ﺍا ﻟ ﺮ ﺻ ﺎ ﻓ ﺔ ﻟ ﻮ ﺍا ء Resafa Brigade ﺑ ﺎ ﺍا ﻟ ﻤ ﻨ ﺘ ﺼ ﺮ ﻟ ﻮ ﺍا ء Al-Muntasir Brigade ﺍا ﻟ ﻨ ﺼ ﺮ ﺭر ﺍا ﻳﯾ ﺎ ﺕت ﻟ ﻮ ﺍا ء Flags of Victory Brigade ﺍا ﻟ ﻘ ﺴ ﺎ ﻡم ﻟ ﻮ ﺍا ء Al-Qassam Brigade ﺍا ﻟ ﻔ ﺮ ﺍا ﺕت ﻓ ﺮ ﺳ ﺎ ﻥن ﻟ ﻮ ﺍا ء Euphrates Knights Brigade ﺍا ﻹ ﺳ ﻼ ﻡم ﺷ ﻴﯿ ﺦ ﻟ ﻮ ﺍا ء Shaykh al-Islām  Brigade ﺍا ﻟ ﻴﯿ ﻤ ﺎ ﻥن ﺑ ﻦ ﺣ ﺬ ﻳﯾ ﻔ ﺔ ﻟ ﻮ ﺍا ء Huthayfa ibn al-Yaman Brigade ﺍا ﻟ ﻔ ﺎ ﺭر ﻭو ﻕق ﻛ ﺘ ﺎ ﺋ ﺐ Al Farooq Phalanges ﻛ ﺘ ﻴﯿ ﺒ ﺔ ﺍا ﻟ ﺒ ﺮ ﺍا ء ﺑ ﻦ ﻣ ﺎ ﻟ ﻚ Al-Baraa ibn Malik Phalange ﺍا ﻷ ﻣ ﺔ ﺃأ ﻣ ﻳﯾ ﻥن ﻛ ﺗ ﻳﯾ ﺑ ﺔ Custodian of Ummah Phalange ﺟ ﺑ ﻝل ﺑ ﻦ ﻣ ﻌ ﺎ ﺫذ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Muadh ibn Jabal Phalange ﺍا ﻟ ﺣ ﻣ ﺯز ﺓة ﻛ ﺗ ﻳﯾ ﺑ ﺔ Al Hamzah Phalange ﺍا ﻟ ﻘ ﺪ ﺱس ﻛ ﺗ ﻳﯾ ﺑ ﺔ Al Quds Phalange ﻛ ﺘ ﻴﯿ ﺒ ﺔ   ﺇإ ﻋ ﺼ ﺎ ﺭر   ﺍا ﻟ ﺜ ﻮ ﺭر ﺓة The Revolution Storm Phalange ﺍا ﻟ ﺸ ﻤ ﺎ ﻝل ﺃأ ﺳ ﻭو ﺩد ﻛ ﺗ ﻳﯾ ﺑ ﺔ North Lions Phalange ﺍا ﻟ ﻌ ﺯز ﻳﯾ ﺯز ﻋ ﺑ ﺩد ﺍا ﺑ ﻥن ﻋ ﻤ ﺮ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Umar ibn Abd al-Aziz Phalange ﺍا ﻟ ﻌ ﺭر ﺑ ﻲ ﺍا ﻟ ﺭر ﺑ ﻳﯾ ﻊ ﻛ ﺘ ﻴﯿ ﺒ ﺔ Arab Spring Phalange ﺍا ﻷ ﺷ ﻬﮭ ﺏب ﺍا ﻟ ﺑ ﺎ ﺯز ﻛ ﺘ ﻴﯿ ﺒ ﺔ al-Baz al-Ashhab Phalange ﺍا ﻟ ﺤ ﻖ ﺃأ ﻧ ﻭو ﺍا ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Lights of Righteousness Phalange ﺍا ﻟ ﻤ ﺆ ﻣ ﻨ ﻴﯿ ﻦ ﺃأ ﻡم ﻋ ﺎ ﺋ ﺷ ﺔ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Aisha bint Abu Bakr Phalange 101 ﻛ ﺘ ﻴﯿ ﺒ ﺔ 101st Phalange ﺍا ﻟ ﺼ ﺪ ﻳﯾ ﻖ ﺑ ﻛ ﺭر ﺃأ ﺑ ﻭو ﻛ ﺗ ﻳﯾ ﺑ ﺔ Abu Bakr as-Șiddiq  Phalange ﺍا ﻟ ﺠ ﺰ ﻳﯾ ﺮ ﺓة ﺍا ﺣ ﺭر ﺍا ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Peninsula Free Phalange ﺍا ﻟ ﺮ ﺣ ﻤ ﻦ ﺻ ﻭو ﺍا ﻋ ﻕق ﻛ ﺗ ﻳﯾ ﺑ ﺔ Ar-Rahman Detonators Phalange ﺍا ﻟ ﺤ ﻖ ﻛ ﺘ ﻴﯿ ﺒ ﺔ The Righteousness Phalange ﷲ ﺳ ﺑ ﻳﯾ ﻝل ﻓ ﻲ ﺍا ﻟ ﺟ ﻬﮭ ﺎ ﺩد ﻛ ﺗ ﻳﯾ ﺑ ﺔ Jihad For Allah Phalange ﷲ ﻭو ﺟ ﻨ ﺪ ﺍا ﻟ ﻣ ﻘ ﺩد ﺱس ﺑ ﻳﯾ ﺕت ﺳ ﺭر ﻳﯾ ﺔ Jerusalem & the Soldiers of Allah Phalange ﺍا ﻟ ﺮ ﻗ ﺔ ﺷ ﻬﮭ ﺩد ﺍا ء ﻛ ﺗ ﻳﯾ ﺑ ﺔ Ar-Raqqah Martyrs Phalange ﺍا ﻟ ﻔ ﺮ ﺍا ﺕت ﺃأ ﺣ ﺭر ﺍا ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Euphrates Free Phalange ﺯز ﻳﯾ ﺪ ﺑ ﻥن ﺍا ﺳ ﺎ ﻣ ﺔ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Usama ibn Zayd Phalange ﺍا ﻟ ﻘ ﺎ ﺭر ﻋ ﺔ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Al-Qaria Phalange ﻛ ﺘ ﻴﯿ ﺒ ﺔ ﺃأ ﺑ ﻮ ﺩد ﺟ ﺎ ﻧ ﺔ Abu Dujana Phalange ﺍا ﻟ ﻘ ﺮ ﻧ ﻲ ﺃأ ﻭو ﻳﯾ ﺱس ﻛ ﺗ ﻳﯾ ﺑ ﺔ Uwais Qarni Phalange ﻣ ﺎ ﻟ ﻚ ﺑ ﻥن ﺃأ ﻧ ﺱس ﻛ ﺗ ﻳﯾ ﺑ ﺔ Anas ibn Malik Phalange ﺍا ﻷ ﻧ ﺼ ﺎ ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ The Supporters Phalange ﺍا ﻟ ﻘ ﺪ ﺱس ﻛ ﺗ ﻳﯾ ﺑ ﺔ Al Quds Phalange ﺍا ﻟ ﺨ ﺎ ﺻ ﺔ ﺍا ﻟ ﻣ ﻬﮭ ﺎ ﻡم ﻛ ﺗ ﻳﯾ ﺑ ﺔ Special Operations Phalange ﺍا ﻟ ﺼ ﻮ ﺍا ﺭر ﻱي ﺫذ ﺍا ﺕت ﻛ ﺗ ﻳﯾ ﺑ ﺔ Dhat Al-Sawari Phalange ﷲ ﺃأ ﻧ ﺻ ﺎ ﺭر – ﺍا ﻟ ﻁط ﺑ ﻘ ﺔ ﺃأ ﺣ ﺭر ﺍا ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Tabaqah Free - Supporters' of Allah Phalange ﺍا ﻟ ﻌ ﺎ ﺩد ﻳﯾ ﺎ ﺕت ﻛ ﺗ ﻳﯾ ﺑ ﺔ Al-Adiyat Phalange ﻋ ﻤ ﺮ ﺍا ﻟ ﺷ ﻳﯾ ﺦ ﻓ ﻬﮭ ﺩد ﺍا ﻟ ﺷ ﻬﮭ ﻳﯾ ﺩد ﻛ ﺗ ﻳﯾ ﺑ ﺔ Martyr Fahad al-Sheikh Omar ﺍا ﻟ ﺴ ﻨ ﺔ ﺃأ ﺳ ﻭو ﺩد – ﷲ ﺣ ﺯز ﺏب ﻛ ﺗ ﻳﯾ ﺑ ﺔ Hezbollah – Sunni Lions Phalange ﻋ ﻠ ﻲ ﺍا ﻹ ﻣ ﺎ ﻡم ﻛ ﺗ ﻳﯾ ﺑ ﺔ Imam Ali Phalange ﺍا ﻟ ﺸ ﺮ ﻳﯾ ﻌ ﺔ ﺃأ ﻧ ﺻ ﺎ ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Supporters of Shariaa Phalange ﺍا ﻟ ﻤ ﻨ ﺘ ﻘ ﻢ ﷲ ﺃأ ﺳ ﺩد ﻛ ﺗ ﻳﯾ ﺑ ﺔ The Revenger Lion of Allah Phalange ﺍا ﻟ ﺮ ﺣ ﻤ ﻦ ﻋ ﺑ ﺎ ﺩد ﻛ ﺗ ﻳﯾ ﺑ ﺔ Rahman Worshipers Phalange ﺍا ﻟ ﻔ ﺮ ﺍا ﺕت ﺳ ﺭر ﺍا ﻳﯾ ﺎ Euphrates Saraya ﺍا ﻟ ﻌ ﻭو ﺍا ﻡم ﺑ ﻥن ﺯز ﺑ ﻳﯾ ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Zubayr ibn al-Awam Phalange ﺍا ﻟ ﻔ ﺮ ﺍا ﺕت ﺃأ ﺣ ﺭر ﺍا ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Euphrates Free Phalange ﺍا ﻷ ﻧ ﺻ ﺎ ﺭر ﻱي ﺃأ ﻳﯾ ﻭو ﺏب ﺃأ ﺑ ﻭو ﻛ ﺗ ﻳﯾ ﺑ ﺔ Abu Ayyub al-Ansari Phalange ﺍا ﻟ ﻴﯿ ﻤ ﺎ ﻥن ﺑ ﻦ ﺣ ﺬ ﻳﯾ ﻔ ﺔ ﻛ ﺘ ﻴﯿ ﺒ ﺔ Huthayfa ibn al-Yaman Phalange ﺳ ﻔ ﻳﯾ ﺎ ﻥن ﺃأ ﺑ ﻲ ﺑ ﻥن ﻣ ﻌ ﺎ ﻭو ﻳﯾ ﺔ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Muʿawiyah  ibn  ʾAbi  Sufyan  Phalange ﺍا ﻟ ﻄ ﺒ ﻴﯿ ﺔ ﺍا ﻟ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Medical Phalange ﺍا ﻷ ﺛ ﺮ ﺃأ ﻫﮬﮪھ ﻝل ﻛ ﺗ ﻳﯾ ﺑ ﺔ Ahl al-Athar Phalange ﺍا ﻟ ﻌ ﻭو ﺍا ﻡم ﺑ ﻥن ﺯز ﺑ ﻳﯾ ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Zubayr ibn al-Awam Phalange ﺍا ﻟ ﻧ ﺻ ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Victory Phalange ﺍا ﻹ ﺳ ﻼ ﻡم ﺃأ ﻧ ﺻ ﺎ ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Islam Supporters Phalange ﻋ ﺛ ﻣ ﺎ ﻥن ﺍا ﺑ ﻥن ﻋ ﻔ ﺎ ﻥن ﻛ ﺗ ﻳﯾ ﺑ ﺔ Uthman ibn Affan Phalange ﺍا ﻟ ﺤ ﻤ ﺰ ﺓة ﻛ ﺗ ﻳﯾ ﺑ ﺔ Hamza Phalange ﺍا ﻟ ﺤ ﺴ ﻴﯿ ﻦ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Hussein Phalange ﺍا ﻟ ﺤ ﺴ ﻦ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Hasan Phalange ﺍا ﻟ ﺤ ﺼ ﻦ ﻛ ﺗ ﻳﯾ ﺑ ﺔ Hosn Phalange ﺍا ﻟ ﺼ ﺪ ﻳﯾ ﻖ ﻛ ﺗ ﻳﯾ ﺑ ﺔ as-Șiddiq  Phalange ﺍا ﻟ ﻘ ﻌ ﻘ ﺎ ﻉع ﻛ ﺗ ﻳﯾ ﺑ ﺔ Al Qa'qaa Phalange ﻛ ﺘ ﻴﯿ ﺒ ﺔ     ﻣ ﺼ ﻌ ﺐ     ﺍا ﺑ ﻦ   ﻋ ﻤ ﻴﯿ ﺮ       Masaab ibn Umayr Phalange ﺍا ﻟ ﻌ ﻮ ﺍا ﻡم ﺑ ﻥن ﺯز ﺑ ﻳﯾ ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Zubayr ibn al-Awam Phalange ﺍا ﻟ ﺨ ﻄ ﺎ ﺏب ﺍا ﺑ ﻥن ﻋ ﻣ ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Umar ibn Al-Khattab Phalange ﺍا ﻟ ﻔ ﺮ ﺍا ﺕت ﺻ ﻘ ﻭو ﺭر ﻛ ﺗ ﻳﯾ ﺑ ﺔ Euphrates Falcons Phalange ﺍا ﻟ ﻔ ﺮ ﺍا ﺕت ﺳ ﺮ ﺍا ﻳﯾ ﺎ Euphrates Saraya ﺳ ﻔ ﻴﯿ ﺎ ﻥن ﺃأ ﺑ ﻲ ﺑ ﻦ ﻣ ﻌ ﺎ ﻭو ﻳﯾ ﺔ ﻛ ﺘ ﻴﯿ ﺒ ﺔ Muʿawiyah  ibn  ʾAbi  Sufyan  Phalange ﺍا ﻟ ﻔ ﺮ ﺍا ﺕت ﺳ ﺮ ﺍا ﻳﯾ ﺎ Euphrates Saraya ﺍا ﻟ ﺪ ﻳﯾ ﻦ ﺻ ﻼ ﺡح ﺍا ﻟ ﻧ ﺎ ﺻ ﺭر ﻛ ﺗ ﺎ ﺋ ﺏب Saladin Phalanges ﺍا ﻟ ﺨ ﺎ ﺻ ﺔ ﺍا ﻟ ﻤ ﻬﮭ ﺎ ﻡم ﻛ ﺘ ﻴﯿ ﺒ ﺔ Special Operations Phalange ﺍا ﻟ ﺪ ﺑ ﺴ ﻲ ﺃأ ﺣ ﺮ ﺍا ﺭر ﻛ ﺘ ﻴﯿ ﺒ ﺔ al Dibsi Free Phalane ﺍا ﻟ ﺮ ﺳ ﺎ ﻟ ﺔ ﻛ ﺗ ﻳﯾ ﺑ ﺔ The Message Phalange ﺍا ﺍا ﻟ ﺴ ﻨ ﺔ ﺃأ ﻧ ﺼ ﺎ ﺭر ﻛ ﺘ ﻴﯿ ﺒ ﺔ Supporters of Sunna Phalange ﺍا ﻟ ﺤ ﻖ ﺟ ﻨ ﻮ ﺩد ﻛ ﺘ ﻴﯿ ﺒ ﺔ Soldiers of Righteousness Phalange ﺍا ﻟ ﻴﯿ ﻤ ﺎ ﻥن ﺑ ﻦ ﺣ ﺬ ﻳﯾ ﻔ ﺔ ﻟ ﻮ ﺍا ء Huthayfa ibn al-Yaman Brigade ﺍا ﻟ ﺪ ﻳﯾ ﻦ ﺻ ﻼ ﺡح ﺍا ﻟ ﻨ ﺎ ﺻ ﺮ ﻛ ﺘ ﺎ ﺋ ﺐ Saladin Phalanges ﺟ ﺒ ﻬﮭ ﺔ ﺍا ﻟ ﻮ ﺣ ﺪ ﺓة ﻭو ﺍا ﻟ ﺘ ﺤ ﺮ ﻳﯾ ﺮ ﺍا ﻹ ﺳ ﻼ ﻣ ﻴﯿ ﺔ Jabhat al-Wahdet al-Tahrir al-Islamiyye ﺍا ﻟ ﺪ ﺑ ﺴ ﻲ ﺃأ ﺣ ﺮ ﺍا ﺭر ﻛ ﺘ ﻴﯿ ﺒ ﺔ al Dibsi Free Phalane ﺍا ﻟ ﺮ ﺳ ﺎ ﻟ ﺔ ﻛ ﺘ ﻴﯿ ﺒ ﺔ The Message Phalange ﺍا ﺍا ﻟ ﺴ ﻨ ﺔ ﺃأ ﻧ ﺼ ﺎ ﺭر ﻛ ﺘ ﻴﯿ ﺒ ﺔ Supporters of Sunna Phalange ﺍا ﻟ ﺤ ﻖ ﺟ ﻨ ﻮ ﺩد ﻛ ﺘ ﻴﯿ ﺒ ﺔ Soldiers of Righteousness Phalange ﺍا ﻟ ﻴﯿ ﻤ ﺎ ﻥن ﺑ ﻦ ﺣ ﺬ ﻳﯾ ﻔ ﺔ ﻛ ﺘ ﻴﯿ ﺒ ﺔ Huthayfa ibn al-Yaman Phalange ﺳ ﻔ ﻴﯿ ﺎ ﻥن ﺃأ ﺑ ﻲ ﺑ ﻦ ﻣ ﻌ ﺎ ﻭو ﻳﯾ ﺔ ﻛ ﺘ ﻴﯿ ﺒ ﺔ Muʿawiyah  ibn  ʾAbi  Sufyan  Phalange ﺍا ﻟ ﻄ ﺒ ﻴﯿ ﺔ ﺍا ﻟ ﻜ ﺘ ﻴﯿ ﺒ ﺔ Medical Phalange ﺍا ﻷ ﺛ ﺮ ﺃأ ﻫﮬﮪھ ﻞ ﻛ ﺘ ﻴﯿ ﺒ ﺔ Ahl al-Athar Phalange ﺍا ﻟ ﻌ ﻮ ﺍا ﻡم ﺑ ﻦ ﺯز ﺑ ﻴﯿ ﺮ ﻛ ﺘ ﻴﯿ ﺒ ﺔ Zubayr ibn al-Awam Phalange ﺍا ﻟ ﻨ ﺼ ﺮ ﻛ ﺘ ﻴﯿ ﺒ ﺔ Victory Phalange ﺍا ﻹ ﺳ ﻼ ﻡم ﺃأ ﻧ ﺼ ﺎ ﺭر ﻛ ﺘ ﻴﯿ ﺒ ﺔ Islam Supporters Phalange
  5. OSINT Process & Tools • Requirement • Acquire • Process

    • Analyse • Visualise • Communicate Named Entity Network Sentiment Translation
  6. i2

  7. Python Libraries • requests • lxml • BeautifulSoup • Natural

    Language Toolkit (NLTK) • SciPy • NetworkX
  8. Canari • Start Project • Define Entities • Create Transform

    • Export Profile • Import Profile into Maltego
  9. class NzcompaniesEntity(Entity): namespace = 'nzcompanies' @EntityField(name='companyname', propname='companyname', displayname='Company Name') @EntityField(name='companynumber',

    propname='companynumber', displayname='Comp. Number') @EntityField(name='xml', propname='xml', displayname='XML') class CompanyAlias(NzcompaniesEntity): pass
  10. @configure( label='To NZ Companies', uuids=[ 'nzcompanies.v1.findcompanies' ], inputs=[ ( 'NZ

    Companies', Phrase ) ]) def dotransform(request, response): es = companies.SearchCompanies() rs = es.query(request.value) for r in rs: e = CompanyAlias(r.companyname) e.companyname = r.companyname e.companynumber = ifnotnone(r.companynumber) e.xml = r.xml response += e return response
  11. Companies Register • Limited Liability Company • Companies Act 1993

    • Register of Directors & Shareholders • http://www.business.govt.nz/companies/ • SOAP & REST(ish) API • MBIE HMAC Authenication