Green Coding in django

An environmentally friendly approach to development in django (Green Coding with django), rarely does anyone consider the impact of application performance on our planet.

Green Coding is a technique of writing code in an environmentally friendly way. The audience will learn how this can be put into practice with Python and django.

Applications can degrade over time, the common / cheapest answer to most performance based issues tends to be to apply more resources which is against the principles of Green Coding.

This talk is aimed at django developers of all levels, teaching the audience the principles of Green Coding, how they can be incorporated into django, help ensure that applications are built to run efficiently to be environmentally friendly, and to leave with a changed mindset of being more efficient.

Video: https://youtu.be/r4XEF4UHI-g

Transcript

1. Green Coding in django Andrew Aikman djangoCon Europe (Edinburgh) May 2023
2. @aiky30
3. Contents • What is Green Coding • Why should we care • What can we do • Questions
4. What is Green Coding • Fairly new term • Environmentally sustainable computing practice • Considerations ◦ Behavioral ◦ Structural
5. What is Green Coding • Behavioural ◦ Fueling the code (drink, food) ◦ Heating / Cooling ◦ Lighting ◦ Travel ◦ Personal habits • Structural ◦ Code efficiency ◦ Minimal energy usage ◦ Hardware running the code
6. Why should we care?
8. Why should we care? • 66% chance we will pass the 1.5C global warming threshold by 2027 • Data centre usage is over 1% of global emissions • We have a responsibility • Python is not that efficient, but we have a job to do!
9. What can we do? Behavioural
10. What can we do: Behavioural • Use machines more efficiently ◦ Closing unused Apps ◦ Shutdown
13. What can we do: Behavioural • Equipment that we buy ◦ Manufacturing of a laptop consumes on average 214 kgs of CO2 • https://tcocertified.com/ ◦ world-leading sustainability certification for IT products • Dispose of our waste responsibly (WEEE)
15. What can we do: Behavioural • Personal habits • www.ethicalconsumer.org ◦ Articles: equipment, food, banking, travel advice • Talk
16. What I did: Behavioural Problem Solution Heating the whole house Installed smart heating Many unused applications Awareness of SWAP usage Not shutting down Now every Friday, sometimes every day I drink far too much coffee Only boil the water that I need
17. Behavioural: My analysis 350ml Water Boiled = 40 Wh 750ml Water Boiled = 100 Wh Average kettle = 3 KWh
18. Drinking 4 hot drinks a day for 1 year Quantity of water 350ml water (an average mug) 750ml water (1/2 average kettle) Power per boil (Wh) 40 100 Power per day (Wh) 160 400 Power per year (KWh) 240 working days using 48 weeks of 5 days 38.4 96 Cost per year 50p a unit £19.20 £48 Cost of CO2e per year 0.193 kg per kWh 7.41 kg 18.52 kg
20. Savings over 1 year • 1 year cost saving = £28.80 ◦ £48 - £19.20 @ £0.50 per unit • 1 year energy saving = 57.6 KWh ◦ 750ml 96 KWh - 350ml 38.4 KWh • Tesla Model 7 = 200 mile / 322 km range ◦ Average 3.6 miles per KWh ◦ 3.6 x 57.6 = 200 miles • 2 return trips from Edinburgh to Glasgow ◦ Edinburgh to Glasgow = 50 miles
21. Savings over 5 years • 5 year energy saving = 288 KWh ◦ 750ml 480 KWh - 350ml 192 KWh • Average home uses 8 KWh electricity • Power a home for 36 days of electricity
22. So what can we do? Structural
23. “most performance problems in well-written Django sites aren’t at the Python execution level, but rather in inefficient database querying, caching, and templates.” https://docs.djangoproject.com/en/4.2/topics/performance/
25. How do we normally find problems in our code?
27. How can we find issues before they become bugs?
29. How can we “LOOK”? • Local Profiling / debugging tools ◦ Analyse performance by time and resources (CPU, memory) ▪ Cprofile ▪ django-silk ▪ Python timeit module (just time) ▪ Python 3.12 perf profiler support ◦ Performance and Request / Response / DB profiling, tracing ▪ Django-debug-toolbar ▪ KOLO • Cloud logging and alerting ◦ Sentry, Datadog ….
32. By fixing issues in our code: • We become better developers Let’s see some examples …
33. Problem: Outdated libraries
34. Outdated libraries • Cheapest fix! • Found: ◦ `pip list --outdated` ◦ `poetry show --outdated` • Fix: ◦ Read changelogs for performance improvements ◦ Upgrade!
35. Test method • Created small timed code snippets • Ran with empty db • Created as small batch of data where possible • Average times specified from 3 runs • Code presented is edited for presentation
36. Test method start = time.time() # Do stuff end = time.time() duration = end - start print(f"Time to complete: {duration}")
37. Test models class Question(models.Model): question_text = models.CharField(max_length=200) pub_date = models.DateTimeField("date published") class Choice(models.Model): question = models.ForeignKey(Question) choice_text = models.CharField(max_length=200) votes = models.IntegerField(default=0)
38. Problem: slow queries
39. Select N+1 • Find: ◦ Look for many duplicate queries ◦ Common in: ▪ Django admin inlines ▪ Django REST framework serializers ▪ Custom logic for nested relationships • Fix: ◦ For reverse lookups (the FK is in the other model) ▪ use prefetch_related ◦ For forward lookups (the FK is in the current model) ▪ use select_related
40. Select N+1 questions = Question.objects.all() for question in questions: for choice in question.choice_set.all():
41. Select N+1 questions = Question.objects.all().prefetch_related(“choice_set”) for question in questions: for choice in question.choice_set.all():
42. Select N+1 Results 5,000 Questions each with 5 Choices = 25,000 records • Without prefetch_related: ◦ 51.67 seconds ◦ Over 5k queries • With prefetch_related: ◦ 1.1 seconds ◦ 2 queries
43. http://ses4j.github.io/2015/11/23/optimizing-slow-django-rest-framework-performance/
44. Bulk operations • Find: ◦ Slow or many create, update, delete queries • Fix: ◦ Use bulk operations ▪ Built ins: bulk_create, bulk_update ▪ Custom: bulk_delete (_raw_delete)
45. Bulk operations • create ◦ Question.objects.create() ◦ Choice.objects.create() vs • bulk_create ◦ Question.objects.bulk_create() ◦ Choice.objects.bulk_create()
46. Bulk operations 5,000 Questions each with 5 Choices = 25,000 records • Use cases: tests, data migrations • With create: 470.84s • With bulk_create: 1.97s 3 Questions each with 5 Choices = 15 records • Use cases: form submission • With create: 0.041s on average • With bulk_create: 0.013s on average
47. Indexing • Find: ◦ Slow queries when filtering on fields • Fix: ◦ Index commonly filtered fields
48. Indexing class Choice(models.Model): choice_text = models.CharField(max_length=200)
49. Indexing # Create 10,000 Question objects with random text Question.objects.create( question_text=f"Question {random_text}", )
50. Indexing # Querying 1000 times from the middle for i in range(5000, 6000): Question.objects.get(question_text=f"Question {i}") • Query time no index: 26.56s • Query time with Index: 0.28s
51. Django Admin • Find: ◦ Select * from Question • Fixed: ◦ Add autocomplete
53. Django Admin class QuestionAdmin(admin.ModelAdmin): search_fields = ["question_text"] class ChoiceAdmin(admin.ModelAdmin): autocomplete_fields = ["question"]
54. Problem: Slow views
55. Slow views • Find: ◦ View times high, query times low • Fix: ◦ Avoid unnecessary looping ◦ Code cache ▪ lru_cache ▪ @cached_property ◦ Web cache ▪ Not good for the django admin
56. Problem: Slow tests
57. https://youtu.be/-C-XNHAJF-c
58. factory_boy with django • Factories are slow when creating many models • Only populate required fields • Repeated model creation in nested loops is extremely slow ◦ Use factory_boy build_batch with django bulk_create • Related / Sub factories create unnecessary instances ◦ For optional forward relations, use Traits too (recommended for reverse relations)
59. factory_boy with django class Country(models.Model): … class User(models.Model): country = models.ForeignKey(Country) class Company(models.Model): owner = models.ForeignKey(User, null=True, blank=True) country = models.ForeignKey(Country, null=True, blank=True)
60. factory_boy with django class CompanyFactory(DjangoModelFactory): class Meta: model = Company name = "ACME, Inc." country = factory.SubFactory(CountryFactory) owner = factory.SubFactory(UserFactory)
61. factory_boy with django class CompanyWithTraitsFactory(DjangoModelFactory): class Meta: model = Company name = "ACME, Inc." class Params: create_country = factory.Trait( country=factory.SubFactory(CountryFactory) ) create_owner = factory.Trait( owner=factory.SubFactory(UserFactory) )
62. factory_boy with django CompanyFactory.create_batch(1000) 18.62s CompanyWithTraitsFactory.create_batch(1000) 18.12s CompanyWithTraitsFactory.create_batch(1000, create_country=False, create_owner=False) 4.60s
63. “premature optimization is the root of all evil.” Famous saying credited to Donald Knuth
64. Summary • What is Green Coding ◦ Behavioural ◦ Structural • Why should we care • What can we do
66. Question Time Image credit: https://charliegeorgeteam.com/your-planet-needs-you
67. Useful links - Performance in django - https://docs.djangoproject.com/en/4.2/topics/performance/ - Further reading on Green Coding - https://stlpartners.com/articles/sustainability/green-coding-what-is-it/ - https://geekflare.com/green-coding/ - https://www.ibm.com/cloud/blog/green-coding#:~:text=What%20is%20green%20coding%3F,organizations%2 0reduce%20overall%20energy%20consumption - Purchasing advice - www.ethicalconsumer.org - factory_boy testing like a pro - DjangoCon 2022 - Camila Maia - https://pretalx.evolutio.pt/djangocon-europe-2022/talk/XWUYA8/ - DRF Lazy loading - Scott Stafford - http://ses4j.github.io/2015/11/23/optimizing-slow-django-rest-framework-performance/