How to create your own streaming torrent client with Python (for learning purposes)

A few years ago I created TouchAndGo [0] with Felipe Lerena and we learned a lot! I'll share some of that knowledge with you.
- What's BitTorrent? How does it work?
- Which libs you can use to find magnets links?
- How to handle magnet links;
- use libtorrent (API, tips, mis);
- download fast and be friendly with the torrent mesh.
- stream a video with Python;
- download subtitles;
- integrate all the parts together.

Disclaimer:
This talk is a rehash of one we did in 2014 with L1pe: https://www.youtube.com/watch?v=oMl1UiF_2Ss (The project is not maintained anymore).

[0] https://github.com/touchandgo-devs/touchandgo

Transcript

1. How to create your own streaming torrent client with Python. (for learning purposes) Nicolás Demarchi - @gilgamezh 22/04/2020 - Python Ireland Meetup (online) https://www.meetup.com/pythonireland/events/270053115/
2. Farewell Marcos Mundstock! Les Luthiers
3. About me: Nicolás Demarchi https://py.amsterdam https://python.org.ar
4. Disclaimer • This talk is a rehash of one we gave in 2014 with L1pe. I had to read all the code and translate it last weekend. • I’m not interested in having a philosophical discussion about torrent. This is a tech talk • I want to discuss how to build VOD using OSS
5. What’s Bit Torrent? • It’s a decentralized protocol for file sharing (p2p) • Each client is a Peer that connects to a Tracker to find other Peers (in order) to download Pieces of a Torrent from them. More about it
6. Torrent & Magnet • The first step to share a file using torrent is to generate a .torrent file. This file contains all the information for a Peer to find other Peers who are sharing the same file: hash value, size, filename(s), Tracker IP Address, size of the Pieces • A Magnet link contains the required information to ask and download the .torrent file from other clients.You don’t have to store/download the .torrent file • Each .torrent file has a unique 20-byte SHA-1 identifier.
7. Peer • It’s an instance of a BitTorrent client that transfers data from and to other clients. • Seed: a Peer with 100% of the Pieces. • Leech: a Peer with < 100%.
8. Tracker • It’s what the Peers use to get the initial list of other Peers sharing a file. • A Peer “announces” in a Tracker that it’s ready to interchange a torrent and the availability for it. • It’s the entry point to a Swarm • It’s possible to skip it using DHT
9. Piece • A Piece is the exchange unit of a .torrent. • A .torrent has all the information about the size and amount of Pieces to download • Common size is between 64KB and 4MB • Each Piece has a SHA-1 unique identifier
11. How to get the magnet links https://github.com/harshanas/Py1337x
12. How to download the torrent • tl;dr → libtorrent ◦ BitTorrent C++ implementation ◦ Focus on performance and usability. ◦ Good docs.. ◦ PYTHON BINDING!! ◦ Easy to use. ◦ Available on any respectable OS (...and Windows).
14. • Session: libtorrent principal instance. It contains the main loop that controls all the torrents we are downloading. • Torrent Handle: It handles a particular .torrent. • Torrent Status: Contains all the information about the .torrent
15. Some tips to understand how it works. • status.pieces() exposes a bitmask representing all the Pieces with a state (True if a piece was downloaded) • It’s possible to get/set the priority of a piece using handle.piece_priority() • The download queue is exposed at status.get_downlodad_queue(), status.download_rate() and status.upload_rate()
17. Strategy
18. Piece picker • It’s who manages which Pieces are added to the download queue. • It has different strategies, rare first is the default. In this mode it always sets the highest priority to the pieces with less availability on the swarm. It’s the most solidary. • Each Piece has a priority from 1 to 7 (7 is highest) and a deadline (handle.set_piece_deadline() ). Both options affect the moment the Piece picker adds it to the download queue. • TIP: A Piece with a really low deadline will be downloaded ASAP.
19. Video
20. Torrent Streaming • VOD with BitTorrent. • How? : Downloading a video from the torrent network and serving it over HTTP to a local player. • How to server it over HTTP?: SimpleHTTPServer
22. Subtitles: subliminal and guessit https://pypi.python.org/pypi/subliminal/ Download subtitles in lots of languages using different providers https://pypi.python.org/pypi/guessit GuessIt is a python library that extracts as much information as possible from a video filename.
24. How to put all the parts together? pip install touchandgo https://github.com/touchandgo-devs/touchandgo We did it (in 2014): TouchAndGo.
25. Last but not least This talk and TouchAndGo are really good example of the most important Python piece: The community. 60% of the work was already done ;-)
26. Questions? mail@gilgamezh.me @gilgamezh