Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chris Esther: OSINT, Maltego & Python

Chris Esther: OSINT, Maltego & Python

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Chris Esther:
OSINT, Maltego & Python
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2013 - Saturday, 07 Sep 2013 - Track 1
http://nz.pycon.org/

**Audience level**

Intermediate

**Description**

This is an introduction to OSINT, the Maltego OSINT visualisation tool and Canari a Python framework for creating custom transforms for Maltego.

**Abstract**

--- Open source intelligence (OSINT) ---

What is it? Who uses it and why?

--- OSINT Tools ---

An overview of the commercial products and Python tools available.

--- Maltego ---

An overview of the capabilties of commercial yet freely available tool. Including the ability to extend its functionality with scripting and custom transforms.

--- Canari ---

An introduction to the Python powered framework for creating and managing transforms for Maltego.

--- Example & Experience ---

My experiences in writing a custom transform written in Python using the Canari framework.

**YouTube**

http://www.youtube.com/watch?v=HkRuVNRbieU

New Zealand Python User Group

September 07, 2013
Tweet

More Decks by New Zealand Python User Group

Other Decks in Programming

Transcript

  1. OSINT, Maltego & Python

    View full-size slide

  2. Preflight
    • whoami => @ChrisEstherLaw
    • questions welcomed... at the end
    • one url to note down... at the end

    View full-size slide

  3. Overview
    • Open Source Intelligence (OSINT)
    • OSINT Tools
    • Maltego
    • Canari
    • Companies Office Example

    View full-size slide

  4. US Legal Definition
    Open-source intelligence (OSINT) is
    intelligence that is produced from publicly
    available information and is collected,
    exploited, and disseminated in a timely
    manner to an appropriate audience for the
    purpose of addressing a specific intelligence
    requirement.
    Sec. 931 of the National Defense Authorization Act for Fiscal Year 2006

    View full-size slide

  5. OSINT != OSS
    • So what’s the relationship between Open
    Source Intelligence & Open Source
    Software?

    View full-size slide

  6. govt.nz
    ﺍا



    ﺔ ﺗ


    ﻳﯾ
    ﺮ ﺟ

    ﻬﮭ

    Ar-Raqqah Liberation Front
    ﺍا



    ﺭر
    ﻱي ﺍا





    ﻱي ﺍا





    Military and Revolutionary Council
    ﺍا




    ﻝل ﺇإ



    ﺭر ﻟ

    ﺍا
    ء
    North Storm Brigade
    ﺍا



    ﺩد ﺍا




    ﻝل ﻟ

    ﺍا
    ء
    The Prophet Offspring Brigade
    ﺍا



    ﺔ ﺛ

    ﺍا
    ﺭر ﻟ

    ﺍا
    ء
    Ar-Raqqah Rebels Brigade
    ﺍا





    ﺔ ﻟ

    ﺍا
    ء
    Resafa Brigade

    ﺎ ﺍا





    ﺮ ﻟ

    ﺍا
    ء
    Al-Muntasir Brigade
    ﺍا



    ﺮ ﺭر
    ﺍا
    ﻳﯾ

    ﺕت ﻟ

    ﺍا
    ء
    Flags of Victory Brigade
    ﺍا




    ﻡم ﻟ

    ﺍا
    ء
    Al-Qassam Brigade
    ﺍا



    ﺍا
    ﺕت ﻓ



    ﻥن ﻟ

    ﺍا
    ء
    Euphrates Knights Brigade
    ﺍا



    ﻡم ﺷ
    ﻴﯿ
    ﺦ ﻟ

    ﺍا
    ء
    Shaykh al-Islām  Brigade
    ﺍا

    ﻴﯿ


    ﻥن ﺑ
    ﻦ ﺣ

    ﻳﯾ

    ﺔ ﻟ

    ﺍا
    ء
    Huthayfa ibn al-Yaman Brigade
    ﺍا



    ﺭر
    ﻭو
    ﻕق ﻛ




    Al Farooq Phalanges


    ﻴﯿ


    ﺍا



    ﺍا
    ء






    Al-Baraa ibn Malik Phalange
    ﺍا


    ﺔ ﺃأ

    ﻳﯾ
    ﻥن ﻛ

    ﻳﯾ


    Custodian of Ummah Phalange


    ﻝل ﺑ
    ﻦ ﻣ


    ﺫذ ﻛ

    ﻳﯾ


    Muadh ibn Jabal Phalange
    ﺍا



    ﺯز
    ﺓة ﻛ

    ﻳﯾ


    Al Hamzah Phalange
    ﺍا



    ﺱس ﻛ

    ﻳﯾ


    Al Quds Phalange


    ﻴﯿ


     
    ﺇإ



    ﺭر
     
    ﺍا



    ﺭر
    ﺓة
    The Revolution Storm Phalange
    ﺍا




    ﻝل ﺃأ

    ﻭو
    ﺩد ﻛ

    ﻳﯾ


    North Lions Phalange
    ﺍا


    ﺯز
    ﻳﯾ
    ﺯز ﻋ

    ﺩد ﺍا

    ﻥن ﻋ

    ﺮ ﻛ

    ﻳﯾ


    Umar ibn Abd al-Aziz Phalange
    ﺍا


    ﺭر

    ﻲ ﺍا

    ﺭر

    ﻳﯾ
    ﻊ ﻛ

    ﻴﯿ


    Arab Spring Phalange
    ﺍا


    ﻬﮭ
    ﺏب ﺍا



    ﺯز ﻛ

    ﻴﯿ


    al-Baz al-Ashhab Phalange
    ﺍا


    ﻖ ﺃأ

    ﻭو
    ﺍا
    ﺭر ﻛ

    ﻳﯾ


    Lights of Righteousness Phalange
    ﺍا





    ﻴﯿ
    ﻦ ﺃأ
    ﻡم ﻋ



    ﺔ ﻛ

    ﻳﯾ


    Aisha bint Abu Bakr Phalange
    101 ﻛ

    ﻴﯿ


    101st Phalange
    ﺍا



    ﻳﯾ
    ﻖ ﺑ

    ﺭر ﺃأ

    ﻭو ﻛ

    ﻳﯾ


    Abu Bakr as-Șiddiq  Phalange
    ﺍا



    ﻳﯾ

    ﺓة ﺍا

    ﺭر
    ﺍا
    ﺭر ﻛ

    ﻳﯾ


    Peninsula Free Phalange
    ﺍا




    ﻦ ﺻ
    ﻭو
    ﺍا

    ﻕق ﻛ

    ﻳﯾ


    Ar-Rahman Detonators Phalange
    ﺍا


    ﻖ ﻛ

    ﻴﯿ


    The Righteousness Phalange
    ﷲ ﺳ

    ﻳﯾ
    ﻝل ﻓ
    ﻲ ﺍا


    ﻬﮭ

    ﺩد ﻛ

    ﻳﯾ


    Jihad For Allah Phalange
    ﷲ ﻭو


    ﺪ ﺍا



    ﺩد
    ﺱس ﺑ
    ﻳﯾ
    ﺕت ﺳ
    ﺭر
    ﻳﯾ

    Jerusalem & the Soldiers of Allah Phalange
    ﺍا



    ﺔ ﺷ
    ﻬﮭ
    ﺩد
    ﺍا
    ء ﻛ

    ﻳﯾ


    Ar-Raqqah Martyrs Phalange
    ﺍا



    ﺍا
    ﺕت ﺃأ

    ﺭر
    ﺍا
    ﺭر ﻛ

    ﻳﯾ


    Euphrates Free Phalange
    ﺯز
    ﻳﯾ
    ﺪ ﺑ
    ﻥن ﺍا



    ﺔ ﻛ

    ﻳﯾ


    Usama ibn Zayd Phalange
    ﺍا



    ﺭر

    ﺔ ﻛ

    ﻳﯾ


    Al-Qaria Phalange


    ﻴﯿ


    ﺃأ


    ﺩد




    Abu Dujana Phalange
    ﺍا




    ﻲ ﺃأ
    ﻭو
    ﻳﯾ
    ﺱس ﻛ

    ﻳﯾ


    Uwais Qarni Phalange



    ﻚ ﺑ
    ﻥن ﺃأ

    ﺱس ﻛ

    ﻳﯾ


    Anas ibn Malik Phalange
    ﺍا




    ﺭر ﻛ

    ﻳﯾ


    The Supporters Phalange
    ﺍا



    ﺱس ﻛ

    ﻳﯾ


    Al Quds Phalange
    ﺍا




    ﺔ ﺍا


    ﻬﮭ

    ﻡم ﻛ

    ﻳﯾ


    Special Operations Phalange
    ﺍا



    ﺍا
    ﺭر
    ﻱي ﺫذ
    ﺍا
    ﺕت ﻛ

    ﻳﯾ


    Dhat Al-Sawari Phalange
    ﷲ ﺃأ



    ﺭر – ﺍا

    ﻁط


    ﺔ ﺃأ

    ﺭر
    ﺍا
    ﺭر ﻛ

    ﻳﯾ


    Tabaqah Free - Supporters' of Allah Phalange
    ﺍا



    ﺩد
    ﻳﯾ

    ﺕت ﻛ

    ﻳﯾ


    Al-Adiyat Phalange


    ﺮ ﺍا


    ﻳﯾ
    ﺦ ﻓ
    ﻬﮭ
    ﺩد ﺍا


    ﻬﮭ
    ﻳﯾ
    ﺩد ﻛ

    ﻳﯾ


    Martyr Fahad al-Sheikh Omar
    ﺍا



    ﺔ ﺃأ

    ﻭو
    ﺩد – ﷲ ﺣ
    ﺯز
    ﺏب ﻛ

    ﻳﯾ


    Hezbollah – Sunni Lions Phalange


    ﻲ ﺍا



    ﻡم ﻛ

    ﻳﯾ


    Imam Ali Phalange
    ﺍا



    ﻳﯾ

    ﺔ ﺃأ



    ﺭر ﻛ

    ﻳﯾ


    Supporters of Shariaa Phalange
    ﺍا





    ﻢ ﷲ ﺃأ

    ﺩد ﻛ

    ﻳﯾ


    The Revenger Lion of Allah Phalange
    ﺍا




    ﻦ ﻋ


    ﺩد ﻛ

    ﻳﯾ


    Rahman Worshipers Phalange
    ﺍا



    ﺍا
    ﺕت ﺳ
    ﺭر
    ﺍا
    ﻳﯾ

    Euphrates Saraya
    ﺍا


    ﻭو
    ﺍا
    ﻡم ﺑ
    ﻥن ﺯز

    ﻳﯾ
    ﺭر ﻛ

    ﻳﯾ


    Zubayr ibn al-Awam Phalange
    ﺍا



    ﺍا
    ﺕت ﺃأ

    ﺭر
    ﺍا
    ﺭر ﻛ

    ﻳﯾ


    Euphrates Free Phalange
    ﺍا




    ﺭر
    ﻱي ﺃأ
    ﻳﯾ
    ﻭو
    ﺏب ﺃأ

    ﻭو ﻛ

    ﻳﯾ


    Abu Ayyub al-Ansari Phalange
    ﺍا

    ﻴﯿ


    ﻥن ﺑ
    ﻦ ﺣ

    ﻳﯾ

    ﺔ ﻛ

    ﻴﯿ


    Huthayfa ibn al-Yaman Phalange


    ﻳﯾ

    ﻥن ﺃأ

    ﻲ ﺑ
    ﻥن ﻣ


    ﻭو
    ﻳﯾ
    ﺔ ﻛ

    ﻳﯾ


    Muʿawiyah  ibn  ʾAbi  Sufyan  Phalange
    ﺍا



    ﻴﯿ
    ﺔ ﺍا



    ﻳﯾ


    Medical Phalange
    ﺍا


    ﺮ ﺃأ
    ﻫﮬﮪھ
    ﻝل ﻛ

    ﻳﯾ


    Ahl al-Athar Phalange
    ﺍا


    ﻭو
    ﺍا
    ﻡم ﺑ
    ﻥن ﺯز

    ﻳﯾ
    ﺭر ﻛ

    ﻳﯾ


    Zubayr ibn al-Awam Phalange
    ﺍا



    ﺭر ﻛ

    ﻳﯾ


    Victory Phalange
    ﺍا



    ﻡم ﺃأ



    ﺭر ﻛ

    ﻳﯾ


    Islam Supporters Phalange




    ﻥن ﺍا

    ﻥن ﻋ


    ﻥن ﻛ

    ﻳﯾ


    Uthman ibn Affan Phalange
    ﺍا




    ﺓة ﻛ

    ﻳﯾ


    Hamza Phalange
    ﺍا



    ﻴﯿ
    ﻦ ﻛ

    ﻳﯾ


    Hussein Phalange
    ﺍا



    ﻦ ﻛ

    ﻳﯾ


    Hasan Phalange
    ﺍا



    ﻦ ﻛ

    ﻳﯾ


    Hosn Phalange
    ﺍا



    ﻳﯾ
    ﻖ ﻛ

    ﻳﯾ


    as-Șiddiq  Phalange
    ﺍا





    ﻉع ﻛ

    ﻳﯾ


    Al Qa'qaa Phalange


    ﻴﯿ


     
     




     
     
    ﺍا


     


    ﻴﯿ

     
     
     
    Masaab
    ibn
    Umayr
    Phalange
    ﺍا



    ﺍا
    ﻡم ﺑ
    ﻥن ﺯز

    ﻳﯾ
    ﺭر ﻛ

    ﻳﯾ


    Zubayr ibn al-Awam Phalange
    ﺍا




    ﺏب ﺍا

    ﻥن ﻋ

    ﺭر ﻛ

    ﻳﯾ


    Umar ibn Al-Khattab Phalange
    ﺍا



    ﺍا
    ﺕت ﺻ

    ﻭو
    ﺭر ﻛ

    ﻳﯾ


    Euphrates Falcons Phalange
    ﺍا



    ﺍا
    ﺕت ﺳ

    ﺍا
    ﻳﯾ

    Euphrates Saraya


    ﻴﯿ

    ﻥن ﺃأ

    ﻲ ﺑ
    ﻦ ﻣ


    ﻭو
    ﻳﯾ
    ﺔ ﻛ

    ﻴﯿ


    Muʿawiyah  ibn  ʾAbi  Sufyan  Phalange
    ﺍا



    ﺍا
    ﺕت ﺳ

    ﺍا
    ﻳﯾ

    Euphrates Saraya
    ﺍا


    ﻳﯾ
    ﻦ ﺻ

    ﺡح ﺍا




    ﺭر ﻛ



    ﺏب
    Saladin Phalanges
    ﺍا




    ﺔ ﺍا


    ﻬﮭ

    ﻡم ﻛ

    ﻴﯿ


    Special Operations Phalange
    ﺍا




    ﻲ ﺃأ


    ﺍا
    ﺭر ﻛ

    ﻴﯿ


    al Dibsi Free Phalane
    ﺍا





    ﺔ ﻛ

    ﻳﯾ


    The Message Phalange
    ﺍا
    ﺍا



    ﺔ ﺃأ



    ﺭر ﻛ

    ﻴﯿ


    Supporters of Sunna Phalange
    ﺍا


    ﻖ ﺟ


    ﺩد ﻛ

    ﻴﯿ


    Soldiers of Righteousness Phalange
    ﺍا

    ﻴﯿ


    ﻥن ﺑ
    ﻦ ﺣ

    ﻳﯾ

    ﺔ ﻟ

    ﺍا
    ء
    Huthayfa ibn al-Yaman Brigade
    ﺍا


    ﻳﯾ
    ﻦ ﺻ

    ﺡح ﺍا




    ﺮ ﻛ




    Saladin Phalanges


    ﻬﮭ
    ﺔ ﺍا




    ﺓة ﻭو
    ﺍا




    ﻳﯾ
    ﺮ ﺍا




    ﻴﯿ

    Jabhat al-Wahdet al-Tahrir al-Islamiyye
    ﺍا




    ﻲ ﺃأ


    ﺍا
    ﺭر ﻛ

    ﻴﯿ


    al Dibsi Free Phalane
    ﺍا





    ﺔ ﻛ

    ﻴﯿ


    The Message Phalange
    ﺍا
    ﺍا



    ﺔ ﺃأ



    ﺭر ﻛ

    ﻴﯿ


    Supporters of Sunna Phalange
    ﺍا


    ﻖ ﺟ


    ﺩد ﻛ

    ﻴﯿ


    Soldiers of Righteousness Phalange
    ﺍا

    ﻴﯿ


    ﻥن ﺑ
    ﻦ ﺣ

    ﻳﯾ

    ﺔ ﻛ

    ﻴﯿ


    Huthayfa ibn al-Yaman Phalange


    ﻴﯿ

    ﻥن ﺃأ

    ﻲ ﺑ
    ﻦ ﻣ


    ﻭو
    ﻳﯾ
    ﺔ ﻛ

    ﻴﯿ


    Muʿawiyah  ibn  ʾAbi  Sufyan  Phalange
    ﺍا



    ﻴﯿ
    ﺔ ﺍا



    ﻴﯿ


    Medical Phalange
    ﺍا


    ﺮ ﺃأ
    ﻫﮬﮪھ
    ﻞ ﻛ

    ﻴﯿ


    Ahl al-Athar Phalange
    ﺍا



    ﺍا
    ﻡم ﺑ
    ﻦ ﺯز

    ﻴﯿ
    ﺮ ﻛ

    ﻴﯿ


    Zubayr ibn al-Awam Phalange
    ﺍا



    ﺮ ﻛ

    ﻴﯿ


    Victory Phalange
    ﺍا



    ﻡم ﺃأ



    ﺭر ﻛ

    ﻴﯿ


    Islam Supporters Phalange

    View full-size slide

  7. OSINT Process & Tools
    • Requirement
    • Acquire
    • Process
    • Analyse
    • Visualise
    • Communicate
    Named Entity
    Network
    Sentiment
    Translation

    View full-size slide

  8. Wynyard Group

    View full-size slide

  9. OSINT OPSEC Tool

    View full-size slide

  10. Python Libraries
    • requests
    • lxml
    • BeautifulSoup
    • Natural Language Toolkit (NLTK)
    • SciPy
    • NetworkX

    View full-size slide

  11. Simple Example

    View full-size slide

  12. Transforms
    Transform
    Entity Entity
    1 1 or M

    View full-size slide

  13. Data Acquistion

    View full-size slide

  14. Data Analysis

    View full-size slide

  15. Local Transforms
    Transform
    Program
    Request
    XML
    Response
    XML
    STDIN STDOUT

    View full-size slide

  16. Frameworks/Libraries
    • PyMaltego
    • Canari
    • TRX

    View full-size slide

  17. Canari
    • Start Project
    • Define Entities
    • Create Transform
    • Export Profile
    • Import Profile into Maltego

    View full-size slide

  18. class NzcompaniesEntity(Entity):
    namespace = 'nzcompanies'
    @EntityField(name='companyname', propname='companyname', displayname='Company Name')
    @EntityField(name='companynumber', propname='companynumber', displayname='Comp. Number')
    @EntityField(name='xml', propname='xml', displayname='XML')
    class CompanyAlias(NzcompaniesEntity):
    pass

    View full-size slide

  19. @configure(
    label='To NZ Companies',
    uuids=[ 'nzcompanies.v1.findcompanies' ],
    inputs=[ ( 'NZ Companies', Phrase ) ])
    def dotransform(request, response):
    es = companies.SearchCompanies()
    rs = es.query(request.value)
    for r in rs:
    e = CompanyAlias(r.companyname)
    e.companyname = r.companyname
    e.companynumber = ifnotnone(r.companynumber)
    e.xml = r.xml
    response += e
    return response

    View full-size slide

  20. Companies Register
    • Limited Liability Company
    • Companies Act 1993
    • Register of Directors & Shareholders
    • http://www.business.govt.nz/companies/
    • SOAP & REST(ish) API
    • MBIE HMAC Authenication

    View full-size slide

  21. Summary
    • Slides => http://j.mp/1cPTBwL

    View full-size slide