Django Admin for Data Analytics @ DjangoVillage

One of the most distinguishing and outstanding features provided by **Django**
surely concerns the **django admin** application.

The django admin app (`django.contrib.admin`) provides a built-in mechanism to
automate the generation of a back-end management web site in a very customary
and flexible fashion.

The resulting web site can be easily extended with additional views,
`admin actions`, `ajax hooks`, new `templates` and `forms`
(`django.contrib.forms.ModelForms`), as well organized into multiple
`AdminSites` with custom permissions.

To this end, this application has been sometimes referred as the *killer*
application of Django, thanks to this unique set of features provided by the
framework out-of-the-box.

However, even if the original intent of the django admin app was to provide
support to some of the most common data management (a.k.a. `CRUD`) operations,
its highly dynamic and extensible nature allows developers to use it as the
basis to build new web applications.

This talk follows exactly in this direction, aimed at presenting a new
Django Admin-based application I recently implemented as part of my job as a
researcher.

This application has been originally designed to provide a
distributed and remotely accessible tool for (source code) data evaluation
in a **pooling** evaluation process [[Manning, 2009]][0].

In particular, the application leverages on the integration of the
`django.contrib.admin` app with **celery** (`djcelery`), mainly necessary to
automate the execution of asynchronous tasks to fetch the data to analyze and
to store them into the database.

Afterwards, the application has been further extended by additional
functionalities, aimed at plotting charts and statistics on evaluated
data (such as the *agreement rate* among judges).

All these features have been integrated into additional admin `views`,
exploiting in the back-end
[`pandas`](http://pandas.pydata.org) (for the analysis part) in
combination with [`matplotlib`](http://matplotlib.org) and
[`d3js`](http://d3js.org) (for graphs and charts plotting).

In this talk, I would like to share my experience in integrating and
combining all these different Python packages and technologies under the
big shell of the Django Admin app.
To this end, I'm going to present some of the main design choices I made, along
with some technical hints and (methodological) conclusions I derived in
implementing this kind of applications.

Very basic knowledge of Django and Django admin are required for this talk.

[0]: http://nlp.stanford.edu/IR-book/pdf/08eval.pdf "Introduction to Information Retrieval, Chap. 8 Evaluation in Information Retrieval"

Transcript

1. Django Admin for Data Analytics Valerio Maggio - valerio.maggio@gmail.com
2. Prologue Using Django Admin to develop a 
 “front end” web application experience and suggestions
3. Why Django Admin ? Built-in support to Data Management Operations CRUD + Permissions Highly Extensible Fast Development DRY principle always in mind
4. Pooling Process Select Judges to manually evaluate data Measure their Agreement Rate
5. Pooling Process: How it works We need (at least) two judges Each Judge (aka User) views only his/her own subset of data Perform Evaluation Post Processing:
 Measure their Agreement Rate
6. Pooling Process:
 In other words Allow each user to access the App with his/her own credentials Show each user only his/her own subset of data to evaluate judges do not see others evaluations (no bias) Allow users to evaluate the data They cannot change data. Only judgements allowed!
7. What Judgements? & What Data?
8. Source Code Clones (Evaluate) Clone Types
9. The Original one @EP2012
10. Type 1: Exact Copy
11. Type 2: Parameter Substituted Clones
12. Type 3: Structure Substituted Clones
13. Type 4: “Functional” Clones
14. The Whole Picture Use Machine Learning, Luke
15. “Cleric: And the people did feast upon the lambs and sloths, and carp and anchovies, and orangutans and breakfast cereals, and fruit-bats and… 
 Brother Maynard: Skip a bit brother…” Let’s talk about Django
16. The Model
 models.py name: CharField version: CharField src_folder_path: FilePathField Project function_name: CharField filename: CharField start_line: IntegerField end_line: IntegerField code_fragment: TextField project: ForeignKey CodeFragment left_fragment: ForeignKey right_fragment: ForeignKey similarity_value: FloatField CandidateClone evaluator: ForeignKey is_clone: BooleanField clone_type: CharField(choices) evaluated: BooleanField candidate: ForeignKey CloneEvaluation project: ForeignKey filepath: FilePathField code_text: TextField SourceCodeFile django.contrib.auth.models.User
17. The Views
 views.py no views at least in views.py Django Admin views + Custom Views
18. Django Admin
19. Django Admin out-of-the-box Users Management Groups and Permissions Restricted access to data
20. Django Admin Evaluation UI editable fields built-in filtering Syntax Highlighted Code
21. Syntax Highlighting: Pygments
22. Pygments Widgets in Forms: CodeFragmentAdmin.change_view
23. Pygments Widgets in Forms: SourceCodeFileAdmin.change_view
24. Pygments Widgets in Forms: SourceCodeFileAdmin.change_view evaluation/ sourcecodefile/588/#65
25. Pygments Widgets in Forms: SourceCodeFileAdmin
26. List Editable & Ajax Views
27. List Editable & Ajax Views
28. Software Projects Data Stats & Charts Admin Actions
29. analyse code file Admin Actions via Celery Tasks
30. Celery Settings
31. generate_code_base Admin Actions via Celery Tasks • Group Data:
 Split into chunks • DB is the bottleneck
32. generate_code_base Admin Actions via Celery Tasks
33. Software Projects Data Stats & Charts
34. Charts
 (excerpt)
35. Chart Example
36. Data Analysis (Post Evaluation Processing)
37. Cohen’s Statistics
38. Limitations and Future Work No messages given to the User on Task Completion post-processing should be better integrated django-runscripts
39. Thanks a lot for your kind attention Valerio Maggio - valerio.maggio@gmail.com