Tweet
Tweet

Slide 1

Slide 1 text

Green Coding in django Andrew Aikman djangoCon Europe (Edinburgh) May 2023

Slide 2

Slide 2 text

@aiky30

Slide 3

Slide 3 text

Contents ● What is Green Coding ● Why should we care ● What can we do ● Questions

Slide 4

Slide 4 text

What is Green Coding ● Fairly new term ● Environmentally sustainable computing practice ● Considerations ○ Behavioral ○ Structural

Slide 5

Slide 5 text

What is Green Coding ● Behavioural ○ Fueling the code (drink, food) ○ Heating / Cooling ○ Lighting ○ Travel ○ Personal habits ● Structural ○ Code efficiency ○ Minimal energy usage ○ Hardware running the code

Slide 6

Slide 6 text

Why should we care?

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Why should we care? ● 66% chance we will pass the 1.5C global warming threshold by 2027 ● Data centre usage is over 1% of global emissions ● We have a responsibility ● Python is not that efficient, but we have a job to do!

Slide 9

Slide 9 text

What can we do? Behavioural

Slide 10

Slide 10 text

What can we do: Behavioural ● Use machines more efficiently ○ Closing unused Apps ○ Shutdown

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

What can we do: Behavioural ● Equipment that we buy ○ Manufacturing of a laptop consumes on average 214 kgs of CO2 ● https://tcocertified.com/ ○ world-leading sustainability certification for IT products ● Dispose of our waste responsibly (WEEE)

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

What can we do: Behavioural ● Personal habits ● www.ethicalconsumer.org ○ Articles: equipment, food, banking, travel advice ● Talk

Slide 16

Slide 16 text

What I did: Behavioural Problem Solution Heating the whole house Installed smart heating Many unused applications Awareness of SWAP usage Not shutting down Now every Friday, sometimes every day I drink far too much coffee Only boil the water that I need

Slide 17

Slide 17 text

Behavioural: My analysis 350ml Water Boiled = 40 Wh 750ml Water Boiled = 100 Wh Average kettle = 3 KWh

Slide 18

Slide 18 text

Drinking 4 hot drinks a day for 1 year Quantity of water 350ml water (an average mug) 750ml water (1/2 average kettle) Power per boil (Wh) 40 100 Power per day (Wh) 160 400 Power per year (KWh) 240 working days using 48 weeks of 5 days 38.4 96 Cost per year 50p a unit £19.20 £48 Cost of CO2e per year 0.193 kg per kWh 7.41 kg 18.52 kg

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Savings over 1 year ● 1 year cost saving = £28.80 ○ £48 - £19.20 @ £0.50 per unit ● 1 year energy saving = 57.6 KWh ○ 750ml 96 KWh - 350ml 38.4 KWh ● Tesla Model 7 = 200 mile / 322 km range ○ Average 3.6 miles per KWh ○ 3.6 x 57.6 = 200 miles ● 2 return trips from Edinburgh to Glasgow ○ Edinburgh to Glasgow = 50 miles

Slide 21

Slide 21 text

Savings over 5 years ● 5 year energy saving = 288 KWh ○ 750ml 480 KWh - 350ml 192 KWh ● Average home uses 8 KWh electricity ● Power a home for 36 days of electricity

Slide 22

Slide 22 text

So what can we do? Structural

Slide 23

Slide 23 text

“most performance problems in well-written Django sites aren’t at the Python execution level, but rather in inefficient database querying, caching, and templates.” https://docs.djangoproject.com/en/4.2/topics/performance/

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

How do we normally find problems in our code?

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

How can we find issues before they become bugs?

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

How can we “LOOK”? ● Local Profiling / debugging tools ○ Analyse performance by time and resources (CPU, memory) ■ Cprofile ■ django-silk ■ Python timeit module (just time) ■ Python 3.12 perf profiler support ○ Performance and Request / Response / DB profiling, tracing ■ Django-debug-toolbar ■ KOLO ● Cloud logging and alerting ○ Sentry, Datadog ….

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

By fixing issues in our code: ● We become better developers Let’s see some examples …

Slide 33

Slide 33 text

Problem: Outdated libraries

Slide 34

Slide 34 text

Outdated libraries ● Cheapest fix! ● Found: ○ `pip list --outdated` ○ `poetry show --outdated` ● Fix: ○ Read changelogs for performance improvements ○ Upgrade!

Slide 35

Slide 35 text

Test method ● Created small timed code snippets ● Ran with empty db ● Created as small batch of data where possible ● Average times specified from 3 runs ● Code presented is edited for presentation

Slide 36

Slide 36 text

Test method start = time.time() # Do stuff end = time.time() duration = end - start print(f"Time to complete: {duration}")

Slide 37

Slide 37 text

Test models class Question(models.Model): question_text = models.CharField(max_length=200) pub_date = models.DateTimeField("date published") class Choice(models.Model): question = models.ForeignKey(Question) choice_text = models.CharField(max_length=200) votes = models.IntegerField(default=0)

Slide 38

Slide 38 text

Problem: slow queries

Slide 39

Slide 39 text

Select N+1 ● Find: ○ Look for many duplicate queries ○ Common in: ■ Django admin inlines ■ Django REST framework serializers ■ Custom logic for nested relationships ● Fix: ○ For reverse lookups (the FK is in the other model) ■ use prefetch_related ○ For forward lookups (the FK is in the current model) ■ use select_related

Slide 40

Slide 40 text

Select N+1 questions = Question.objects.all() for question in questions: for choice in question.choice_set.all():

Slide 41

Slide 41 text

Select N+1 questions = Question.objects.all().prefetch_related(“choice_set”) for question in questions: for choice in question.choice_set.all():

Slide 42

Slide 42 text

Select N+1 Results 5,000 Questions each with 5 Choices = 25,000 records ● Without prefetch_related: ○ 51.67 seconds ○ Over 5k queries ● With prefetch_related: ○ 1.1 seconds ○ 2 queries

Slide 43

Slide 43 text

http://ses4j.github.io/2015/11/23/optimizing-slow-django-rest-framework-performance/

Slide 44

Slide 44 text

Bulk operations ● Find: ○ Slow or many create, update, delete queries ● Fix: ○ Use bulk operations ■ Built ins: bulk_create, bulk_update ■ Custom: bulk_delete (_raw_delete)

Slide 45

Slide 45 text

Bulk operations ● create ○ Question.objects.create() ○ Choice.objects.create() vs ● bulk_create ○ Question.objects.bulk_create() ○ Choice.objects.bulk_create()

Slide 46

Slide 46 text

Bulk operations 5,000 Questions each with 5 Choices = 25,000 records ● Use cases: tests, data migrations ● With create: 470.84s ● With bulk_create: 1.97s 3 Questions each with 5 Choices = 15 records ● Use cases: form submission ● With create: 0.041s on average ● With bulk_create: 0.013s on average

Slide 47

Slide 47 text

Indexing ● Find: ○ Slow queries when filtering on fields ● Fix: ○ Index commonly filtered fields

Slide 48

Slide 48 text

Indexing class Choice(models.Model): choice_text = models.CharField(max_length=200)

Slide 49

Slide 49 text

Indexing # Create 10,000 Question objects with random text Question.objects.create( question_text=f"Question {random_text}", )

Slide 50

Slide 50 text

Indexing # Querying 1000 times from the middle for i in range(5000, 6000): Question.objects.get(question_text=f"Question {i}") ● Query time no index: 26.56s ● Query time with Index: 0.28s

Slide 51

Slide 51 text

Django Admin ● Find: ○ Select * from Question ● Fixed: ○ Add autocomplete

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

Django Admin class QuestionAdmin(admin.ModelAdmin): search_fields = ["question_text"] class ChoiceAdmin(admin.ModelAdmin): autocomplete_fields = ["question"]

Slide 54

Slide 54 text

Problem: Slow views

Slide 55

Slide 55 text

Slow views ● Find: ○ View times high, query times low ● Fix: ○ Avoid unnecessary looping ○ Code cache ■ lru_cache ■ @cached_property ○ Web cache ■ Not good for the django admin

Slide 56

Slide 56 text

Problem: Slow tests

Slide 57

Slide 57 text

https://youtu.be/-C-XNHAJF-c

Slide 58

Slide 58 text

factory_boy with django ● Factories are slow when creating many models ● Only populate required fields ● Repeated model creation in nested loops is extremely slow ○ Use factory_boy build_batch with django bulk_create ● Related / Sub factories create unnecessary instances ○ For optional forward relations, use Traits too (recommended for reverse relations)

Slide 59

Slide 59 text

factory_boy with django class Country(models.Model): … class User(models.Model): country = models.ForeignKey(Country) class Company(models.Model): owner = models.ForeignKey(User, null=True, blank=True) country = models.ForeignKey(Country, null=True, blank=True)

Slide 60

Slide 60 text

factory_boy with django class CompanyFactory(DjangoModelFactory): class Meta: model = Company name = "ACME, Inc." country = factory.SubFactory(CountryFactory) owner = factory.SubFactory(UserFactory)

Slide 61

Slide 61 text

factory_boy with django class CompanyWithTraitsFactory(DjangoModelFactory): class Meta: model = Company name = "ACME, Inc." class Params: create_country = factory.Trait( country=factory.SubFactory(CountryFactory) ) create_owner = factory.Trait( owner=factory.SubFactory(UserFactory) )

Slide 62

Slide 62 text

factory_boy with django CompanyFactory.create_batch(1000) 18.62s CompanyWithTraitsFactory.create_batch(1000) 18.12s CompanyWithTraitsFactory.create_batch(1000, create_country=False, create_owner=False) 4.60s

Slide 63

Slide 63 text

“premature optimization is the root of all evil.” Famous saying credited to Donald Knuth

Slide 64

Slide 64 text

Summary ● What is Green Coding ○ Behavioural ○ Structural ● Why should we care ● What can we do

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

Question Time Image credit: https://charliegeorgeteam.com/your-planet-needs-you

Slide 67

Slide 67 text

Useful links - Performance in django - https://docs.djangoproject.com/en/4.2/topics/performance/ - Further reading on Green Coding - https://stlpartners.com/articles/sustainability/green-coding-what-is-it/ - https://geekflare.com/green-coding/ - https://www.ibm.com/cloud/blog/green-coding#:~:text=What%20is%20green%20coding%3F,organizations%2 0reduce%20overall%20energy%20consumption - Purchasing advice - www.ethicalconsumer.org - factory_boy testing like a pro - DjangoCon 2022 - Camila Maia - https://pretalx.evolutio.pt/djangocon-europe-2022/talk/XWUYA8/ - DRF Lazy loading - Scott Stafford - http://ses4j.github.io/2015/11/23/optimizing-slow-django-rest-framework-performance/