Swiss Army Django: Small Footprint ETL

Presented at DjangoCon AU 2023

Transcript

1. Swiss Army Django Small Footprint ETL Noah Kantrowitz - coderanger.net - August 18, 2023 DjangoCon AU 2023 1
2. Noah Kantrowitz —He/him —coderanger.net | cloudisland.nz/@coderanger —Kubernetes (ContribEx) and Python (webmaster@) —SRE/Platform for Geomagical Labs, part of IKEA —We do CV/AR for the home DjangoCon AU 2023 2
3. ETL DjangoCon AU 2023 3
4. Case Study: Farm RPG DjangoCon AU 2023 4
5. Game Data DjangoCon AU 2023 5
6. Why build this? It's fun! And educational! DjangoCon AU 2023 6
7. What is ETL? Extract Transform Load DjangoCon AU 2023 7
8. Web Scrapers DjangoCon AU 2023 8
9. ELT? —Extract, Load, Transform —Keep raw data —Transform it again and again —Cool but not very small DjangoCon AU 2023 9
10. Scrape Responsibly DjangoCon AU 2023 10
11. The shape of things —Extractors: scraper/importer cron jobs —Transforms: parse HTML/XML/JSON —Reshape the data to make it easy to query —Possibly import-time aggregations like averages —Loaders: Django ORM, maybe DRF or Pydantic DjangoCon AU 2023 11
12. Async & Django Yes, you can really use it! More on the ORM later DjangoCon AU 2023 12
13. Why async? —Everything in one box —Fewer services, fewer problems —Portability and development env DjangoCon AU 2023 13
14. Monoliths! Reports of my death have been greatly exaggerated DjangoCon AU 2023 14
15. Writing an async app —Normal models —Async views —Async middleware if needed DjangoCon AU 2023 15
16. Running an async app —Async server not included —python -m uvicorn etl.asgi:application —--reload for development —--access-log --no-server-header —--ssl-keyfile=... --ssl-certfile=... DjangoCon AU 2023 16
17. Background tasks DjangoCon AU 2023 17
18. Task factories async def do_every(fn, n): while True: await fn() await asyncio.sleep(n) DjangoCon AU 2023 18
19. Django integration class MyAppConfig(AppConfig): def ready(self): coro = do_every(my_extractor, 30) asyncio.create_task(coro) DjangoCon AU 2023 19
20. The other shoe —Crashes can happen —Plan for convergence —Think about failure modes —Make task status models if needed DjangoCon AU 2023 20
21. Modeling for failure —create_task(send_email()) —await Emails.objects.acreate(...) —create_task(do_every(send_all_emails)) —await email.adelete() DjangoCon AU 2023 21
22. Async ORM —Mind your a*s —Transactions and sync_to_async —Don't worry about concurrency —Still improving DjangoCon AU 2023 22
23. Async transactions @transaction.atomic def _transaction(): Foo.objects.create() Bar.objects.create() async def my_view(): await sync_to_async(_transaction)() DjangoCon AU 2023 23
24. Async HTTP HTTPX Or AIOHTTP DjangoCon AU 2023 24
25. Examples! DjangoCon AU 2023 25
26. The simple case async def scrape_things(): resp = await client.get("/api/things/") resp.raise_for_status() for row in resp.json(): await Thing.objects.aupdate_or_create( id=row["key"], defaults={"name": row["fullName"]}, ) DjangoCon AU 2023 26
27. DRF serializers item = await Item.objects.filter(id=data["id"])\ .afirst() ser = ItemAPISerializer(instance=item, data=data) await sync_to_async(ser.is_valid)(raise_exception=True) await sync_to_async(ser.save)() DjangoCon AU 2023 27
28. Diff delete seen_ids = [] for row in data: thing = ...aupdate_or_create(...) seen_ids.append(thing.id) await ...exclude(id__in=seen_ids).adelete() DjangoCon AU 2023 28
29. Decorators —autodiscover_modules("tasks") —_registry = {} —register_to=mod def every(n): def decorator(fn): _registry[fn] = n return fn return decorator DjangoCon AU 2023 29
30. Real-er cron —Croniter is amazing —In-memory or database —Loop and check next >= now DjangoCon AU 2023 30
31. We have data Now what? DjangoCon AU 2023 31
32. GraphQL Here be dragons Use only for small Or very cachable DjangoCon AU 2023 32
33. Query Basics query { topLevelField(filter: {field1: "foo"}) { field2 field3 { nested1 } } } DjangoCon AU 2023 33
34. Why GraphQL? query { items { name, image, value requiredFor { quantity, quest { title, image, text DjangoCon AU 2023 34
35. When not to use GraphQL —Big data - it's too slow —Numeric aggregation - "mean where type = foo" —But you can pre-compute —Poorly linked data - no ForeignKeys DjangoCon AU 2023 35
36. Strawberry DjangoCon AU 2023 36
37. Schema @gql.type class Query: items: list[Item] = field() quests: list[Quest] = field() schema = gql.Schema(query=Query) DjangoCon AU 2023 37
38. Model Types @gql.django.type(models.Item) class Item: id: int name: auto image: auto required_for: list[QuestRequired] reward_for: list[QuestReward] DjangoCon AU 2023 38
39. Filters and Orders @gql.django.filter(models.Item) class ItemFilter: id: auto can_mail: auto @gql.django.order(models.Item) class ItemOrder: id: auto name: auto DjangoCon AU 2023 39
40. Performance? query { items { id, name requiredFor { quantity, quest { id, title DjangoCon AU 2023 40
41. DjangoCon AU 2023 41
42. DjangoCon AU 2023 42
43. Strawberry Views Don't use their async view, it doesn't like Django path("graphql", GraphQLView.as_view(schema=schema)) DjangoCon AU 2023 43
44. Site Generators Gatsby Next.js Pelican DjangoCon AU 2023 44
45. Subscriptions —Channels —In-memory? —Channels-Postgres otherwise —ASGI Router DjangoCon AU 2023 45
46. Enough about boring stuff Async for fun? DjangoCon AU 2023 46
47. Discord-py Or Bolt-python for Slack Or Bottom for IRC if you're old DjangoCon AU 2023 47
48. intents = discord.Intents.default() intents.message_content = True client = discord.Client(intents=intents) class BotConfig(AppConfig): def ready(self): create_task(client.start(token)) DjangoCon AU 2023 48
49. Control the loop —client.run() - expects to own the loop —client.start() - cooperates with others DjangoCon AU 2023 49
50. Chat bot @client.event async def on_message(message): if message.author == client.user: return if message.content == "!count": n = await Thing.objects.acount() await message.channel.send( f"There are {n} things") DjangoCon AU 2023 50
51. Notifications async def scrape_things(): # Do the ETL ... channel = client.get_channel(id) await channel.send("Batch complete") DjangoCon AU 2023 51
52. Let's get weirder: SSH async def handle_ssh(proc): proc.stdout.write('Welcome!\n> ') cmd = await proc.stdin.readline() if cmd == 'count': n = await Thing.objects.acount() proc.stdout.write( f"There are {n} things\n") def ready(): create_task( asyncssh.listen(process_factory=handle_ssh)) DjangoCon AU 2023 52
53. The sky's the limit Streamdeck Aioserial Pytradfri DjangoCon AU 2023 53
54. Back to serious Scaling? Let's go DjangoCon AU 2023 54
55. Ingest sharding —Too much data to load? —if hash(url) % 2 == server_id —Adjust your aggregations DjangoCon AU 2023 55
56. CPU bound? —Does the library drop the GIL? —sync_to_async(…, thread_sensitive=False) —ProcessPoolExecutor or aiomultiprocess —(PEP 703 in the future?) DjangoCon AU 2023 56
57. Big still allowed Microservices too DjangoCon AU 2023 57
58. In review ETL systems are useful for massaging data Async Django is great for building ETLs GraphQL is an excellent way to query There's many cool async libraries Our tools can grow as our needs grow DjangoCon AU 2023 58
59. Thank you DjangoCon AU 2023 59
60. Questions? DjangoCon AU 2023 60
61. DjangoCon AU 2023 61