django CMS @ DUTH - Exploiting Django for a good cause

Given at Django: Under the Hood 2015 05 November 2015

Video: https://opbeat.com/community/posts/django-cms-and-orm-by-iacopo-spalletti/

Transcript

1. DJANGO CMS@DUTH EXPLOITING DJANGO FOR A GOOD CAUSE
2. HELLO, I AM IACOPO And I keep spamming people about django CMS Founder and CTO @NephilaIt django CMS core developer django CMS installer author
3. DJANGO CMS WHAT'S THAT? A Good Django citizen: handles unstructured content using "pages" integrates with other django apps provides features to other applications glues applications together
4. DJANGO CMS A BIT OF BACKGROUND
5. A BIT OF BACKGROUND Started in 2008 by Divio Derived from django-cms-pages it still possible to port contents from cms-pages (see europython website)
6. A BIT OF BACKGROUND 8 feature releases in 7 years 2.0 ➔ 3.2 Extremely upgradable Django 1.1 / django CMS 2.1 ➔ Django 1.8 / django CMS 3.2
7. DJANGO CMS CONCEPTS
8. DJANGO CMS CONCEPTS Page Title Placeholder Plugin Apphook
9. CONCEPTS PAGE Basic CMS item Tree hierarchy information Associated to a Django template
10. CONCEPTS TITLE Language dependent data: Titles Slug ... Publishing state
11. CONCEPTS PLACEHOLDER Container for plugins Defined in the template
12. CONCEPTS PLUGIN Content Any content managed by the CMS Composed by: Plugin Class Plugin Model
13. CONCEPTS APPHOOK Wrapper for Django applications Allows to integrate Django apps in the CMS tree
15. SOME RAW NUMBERS 14400 LOC 3300 admin 2500 models 1000 templatetag 100 view 20000 loc tests 57K downloads/month on PyPi
16. TEMPLATETAGS
17. TEMPLATETAGS django CMS relies on templatetags a lot Frontend Editor Plugin rendering Placeholder discovery ... Two brief examples
18. TEMPLATETAGS Discovering placeholders <div class="content">{% placeholder "my-placeholder" %}</div> for node in nodelist: # check if this is a placeholder first if isinstance(node, Placeholder): placeholders.append(node.get_name()) elif isinstance(node, IncludeNode): ... placeholders += _scan_placeholders(...) # handle {% extends ... %} tags elif isinstance(node, ExtendsNode): placeholders += _extend_nodelist(node) # in block nodes we have to scan for super blocks elif isinstance(node, VariableNode) and current_block: ... placeholders += _scan_placeholders(...)
19. TEMPLATETAGS Hijacking the template rendering {% cms_toolbar %} ... def render_tag(self, context, name, nodelist): ... toolbar.populate() ... # render everything below the tag # i.e.: *all* the template rendered_contents = nodelist.render(context) ... toolbar.post_template_populate() ... content = render_to_string( 'cms/toolbar/toolbar.html', context ) return '%s\n%s' % (content, rendered_contents)
21. IT'S ALL ABOUT THE ADMIN ~25% of the code is admin related
22. IT'S ALL ABOUT THE ADMIN Frontend editor is admin
23. IT'S ALL ABOUT THE ADMIN Frontend editor is a frontend wrapper around the Django admin No code changes to use it for own applications
24. IT'S ALL ABOUT THE ADMIN Plugins are admin
26. IT'S ALL ABOUT THE ADMIN PLUGINS basically a ModelAdmin with a "public" view
27. PLUGINS LIFECYLE Starts at PlaceholderAdmin.add_plugin: def add_plugin(self, request): ... plugin = CMSPlugin(language=language, plugin_type=plugin_type, position=position, placeholder=placeholder) if parent: plugin.position = CMSPlugin.objects.filter(parent=parent).count() plugin.parent_id = parent.pk plugin.save() self.post_add_plugin(request, placeholder, plugin) ... Called via AJAX
28. PLUGINS LIFECYLE The standard change_view is called def edit_plugin(self, request, plugin_id): ... if not instance: # instance doesn't exist, call add view response = plugin_admin.add_view(request) else: # already saved before, call change view # we actually have the instance here, but since i won't override # change_view method, is better if it will be loaded again, so # just pass id to plugin_admin response = plugin_admin.change_view(request, str(plugin_id)) ... return response Now the plugin data are complete
29. LIFECYLE RENDERING How plugin rendering works? Simplified edition
30. LIFECYLE RENDERING Starts in a templatetag <div class="content">{% placeholder "my-placeholder" %}</div>
31. LIFECYLE RENDERING Plugin list is retrieved qs = get_cmsplugin_queryset(request) qs = qs.filter(placeholder__in=placeholders, language=lang) plugins = list(qs.order_by('placeholder', 'path')) ... plugins = downcast_plugins(plugins, non_fallback_phs) plugin_groups = dict((key, list(plugins)) for key, plugins in groupby(plugins, attrgetter('placeholder_id'))) for group in plugin_groups: plugin_groups[group] = build_plugin_tree( plugin_groups[group])
32. LIFECYLE RENDERING Template and context are provided by the plugin class context = plugin.render(context, instance, placeholder_slot) ... if plugin.render_plugin: template = plugin._get_render_template(context, instance, placeholder)
33. LIFECYLE RENDERING the plugin is rendered if isinstance(template, six.string_types): content = render_to_string(template, context) elif (isinstance(template, Template) or (hasattr(template, 'template') and hasattr(template, 'render') and isinstance(template.template, Template))): content = template.render(context)
34. LIFECYLE RENDERING If it resembles views, it's not by chance It also uses PLUGIN_CONTEXT_PROCESSORS and PLUGIN_PROCESSORS (i.e. process_response in MIDDLEWARE ) Each step is heavily cached caching attributes on objects using Django cache
35. QUESTIONS?
37. ORM Simple models structure
38. ORM MODELS Page and CMSPlugin are organised in tree structures handled via mptt or treebeard
39. MODELS PAGE Website tree structure
40. MODELS PLUGIN
41. The content tree structure
42. MODELS MAIN TOPICS Page publication Plugin polymorphism
43. MODELS PAGE/TITLE PUBLICATION draft and live copy of each page for each language (i.e.: 2 Page and 2xN Title instances exists for any page) each live/draft contains copy of placeholders which contain copy of plugins
44. PAGE / TITLE PUBLICATION
45. PAGE / TITLE PUBLICATION On language (title) publishing: old live title (and plugins) is discarded live title is created as copy of draft draft plugins are copied as live
46. PAGE / TITLE PUBLICATION
47. PAGE / TITLE PUBLICATION
48. PAGE / TITLE PUBLICATION 1. CREATE / GET THE "PUBLIC PAGE" if self.publisher_public_id: # Ensure we have up to date mptt properties public_page = Page.objects.get(pk=self.publisher_public_id) else: public_page = Page(created_by=self.created_by) ... self._copy_attributes(public_page) # we need to set relate this new public copy to its draft page (self public_page.publisher_public = self public_page.publisher_is_draft = False # Ensure that the page is in the right position and save it self._publisher_save_public(public_page) public_page = public_page.reload()
49. PAGE / TITLE PUBLICATION 2. COPY THE PUBLISHED TITLES TO THE PUBLIC PAGE old_titles = dict(target.title_set.filter(language=language).values_list( for title in self.title_set.filter(language=language): old_pk = title.pk # this creates a new instance or recycle the old live title.pk = old_titles.pop(title.language, None) title.page = target title.publisher_is_draft = target.publisher_is_draft title.publisher_public_id = old_pk if published: title.publisher_state = PUBLISHER_STATE_DEFAULT else: title.publisher_state = PUBLISHER_STATE_PENDING title.published = published title._publisher_keep_state = True title.save()
50. PAGE / TITLE PUBLICATION 3. COPY THE CONTENT FOR THE PUBLISHED LANGUAGE for ph in self.get_placeholders(): plugins = ph.get_plugins_list(language) found = False for target_ph in target.placeholders.all(): if target_ph.slot == ph.slot: ph = target_ph found = True break if not found: ph.pk = None # make a new instance ph.save() new_phs.append(ph) if plugins: copy_plugins_to(plugins, ph, no_signals=True)
51. MODELS PLUGIN POLYMORPHISM a.k.a. "Poor man's polymorphism"
52. PLUGIN POLYMORPHISM CMSPlugin is the only "extendable" CMS model Actually you have to class MyCMSPlugin(CMSPlugin): title = models.CharField(max_length=255) ... ORM operations on plugins would be very expensive (e.g.: building the list of plugins assigned to a placeholder) Abstract Base class vs. Multi-table inheritance
53. PLUGIN POLYMORPHISM Polymorphism kicks in Only CMSPlugin model used in complex operations: build list of plugins copy plugins reorder plugins ...
54. PLUGIN POLYMORPHISM DOWNCAST_PLUGINS Before rendering cast from generic to real plugins # make a map of plugin types, needed later for downcasting for plugin in queryset: plugin_types_map[plugin.plugin_type].append(plugin.pk) for plugin_type, pks in plugin_types_map.items(): ... # get all the plugins of type cls.model plugin_qs = cls.get_render_queryset().filter(pk__in=pks) # put them in a map to replace base CMSPlugins with downcasted for instance in plugin_qs: plugin_lookup[instance.pk] = instance ... # make the equivalent list of qs, but with downcasted instances return [plugin_lookup.get(plugin.pk, plugin) for plugin in queryset]
55. PLUGIN POLYMORPHISM COPY_PLUGIN Copying a plugin it's a catch 22 issue tree information created on save saving a plugin blocked by required fields in concrete model
56. COPY_PLUGIN
57. COPY_PLUGIN QUESTION How to clone the model-specific data of a model without knowing its structure *and* let mptt/treebeard create tree data?
58. COPY_PLUGIN ANSWER Use the ORM :)
59. COPY_PLUGIN 1. create a generic instance 2. save it (tree data are set) # get the "real" plugin plugin_instance, cls = self.get_plugin_instance() # set up some basic attributes on the new_plugin new_plugin = CMSPlugin() new_plugin.placeholder = target_placeholder parent_cache[self.pk] = new_plugin if self.parent: parent = parent_cache[self.parent_id] parent = CMSPlugin.objects.get(pk=parent.pk) new_plugin.parent_id = parent.pk new_plugin.parent = parent new_plugin.language = target_language new_plugin.plugin_type = self.plugin_type # This triggers the tree setup for the "generic" plugin new_plugin.save()
60. COPY_PLUGIN Swap the plugin model to the concrete type after that if plugin_instance: # get a new instance so references do not get mixed up plugin_instance = plugin_instance.__class__.objects.get( pk=plugin_instance.pk ) plugin_instance.placeholder = target_placeholder # assign the pk of the generic plugin plugin_instance.pk = new_plugin.pk plugin_instance.id = new_plugin.pk # this is the OneToOne field plugin_instance.cmsplugin_ptr = new_plugin plugin_instance.language = target_language # tree information saved before plugin_instance.parent = new_plugin.parent plugin_instance.depth = new_plugin.depth plugin_instance.path = new_plugin.path
61. COPY_PLUGIN Looks weird? A bit Proves the flexibility of the ORM
62. QUESTIONS?
64. URLCONF Exploiting the modular nature of Django
65. EXPLOITING THE MODULAR NATURE OF DJANGO URLCONF Enter Apphooks The way to integrate Django apps in the CMS tree
66. APPHOOKS Tiny wrapper around urls.py class MyApp(CMSApp): name = _('My app') urls = ['myapp.urls']
67. APPHOOKS 1. Add an apphook to the page 2. The Apphook urls is injected in the global urlconf 3. magic happens
68. APPHOOKS heavylift in cms/appresolver.py#L215-L245 for page_id in hooked_applications.keys(): resolver = None for lang in hooked_applications[page_id].keys(): (app_ns, inst_ns), current_patterns, app = hooked_applications[pag if not resolver: resolver = AppRegexURLResolver(r'', 'app_resolver', app_name=a resolver.page_id = page_id if app.permissions: _set_permissions(current_patterns, app.exclude_permissions) resolver.url_patterns_dict[lang] = current_patterns app_patterns.append(resolver) APP_RESOLVERS.append(resolver) return app_patterns
69. APPHOOKS 1. get the apphooks for lang in hooked_applications[page_id].keys(): (app_ns, inst_ns), current_patterns, app =\ hooked_applications[page_id][lang]
70. APPHOOKS 2. creates a custom resolver w/ apphook urls 3. attach resolver to the page url (app_ns, inst_ns), current_patterns, app = hooked_applications[page_id][la if not resolver: resolver = AppRegexURLResolver( r'', 'app_resolver', app_name=app_ns, namespace=inst_ns ) resolver.page_id = page_id if app.permissions: _set_permissions(current_patterns, app.exclude_permissions) resolver.url_patterns_dict[lang] = current_patterns
71. APPHOOKS 4. the apphooks resolvers are loaded into urlconf if apphook_pool.get_apphooks(): urlpatterns = get_app_patterns() else: urlpatterns = [] urlpatterns.extend([ url(regexp, details, name='pages-details-by-slug'), url(r'^$', details, {'slug': ''}, name='pages-root'), ]
72. IMPROVEMENT IN 3.1 NAMESPACES CONFIGURATION (via aldryn-apphooks-config) Detects current namespace Loads a model related to the namespace Attach the model to the current view
73. NAMESPACE CONFIGURATION def get_app_instance(request): ... with override(get_language_from_request(request, check_path=True namespace = resolve(request.path_info).namespace config = app.get_config(namespace) ... def dispatch(self, request, *args, **kwargs): self.namespace, self.config = get_app_instance(request)
74. NAMESPACE CONFIGURATION Example usage class PostListView(AppConfigMixin, ListView): model = Post def get_template_names(self): template_path = self.config.template_prefix \ or 'djangocms_blog' return os.path.join(template_path, self.base_template_name) def get_paginate_by(self, queryset): return self.config.paginate_by or\ get_setting('PAGINATION')
75. IMPROVEMENT IN 3.2 URLCONF RELOAD Into core Uses a middleware 1 DB hit per request Clears the urlconf modules and reload apphook patterns
76. URLCONF RELOAD if 'cms.urls' in sys.modules: reload(sys.modules['cms.urls']) if urlconf is None: urlconf = settings.ROOT_URLCONF if urlconf in sys.modules: reload(sys.modules[urlconf]) clear_app_resolvers() clear_url_caches() get_app_patterns() if new_revision is not None: set_local_revision(new_revision)
77. BONUS INSTANCE AUTO SETUP (via djangocms-apphook-setup) Auto creation of a page and apphook during projects startup 1 hit per project start
78. AUTO SETUP Sample Apphook with namespace configuration auto setup configuration class BlogApp(AutoCMSAppMixin, CMSConfigApp): name = _('Blog') urls = ['djangocms_blog.urls'] app_name = 'djangocms_blog' app_config = BlogConfig auto_setup = { ... 'page title': get_setting('AUTO_BLOG_TITLE'), 'namespace': get_setting('AUTO_NAMESPACE'), 'config_fields': {...}, 'config_translated_fields': {...}, } apphook_pool.register(BlogApp) BlogApp.setup()
79. QUESTIONS?
80. EXERCISE LEFT FOR THE READER Trace the call stack for plugin rendering :)
81. GRAZIE!