Slide 1
Slide 1 text
Django-proxy-storage
Chibisov Gennady
Slide 2
Slide 2 text
bit.ly/dps-doc
2
Slide 3
Slide 3 text
Plan
1. What is django storage?
2. What is django-proxy-storage?
3. Use cases of django-proxy-storage
3
Slide 4
Slide 4 text
Simple usage with models
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = models.FileField(!
storage=FileSystemStorage(!
location="/media/"!
)!
)!
4
Slide 5
Slide 5 text
Simple usage with models
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = models.FileField(!
storage=FileSystemStorage(!
location="/media/"!
)!
)!
5
Slide 6
Slide 6 text
Simple usage with models
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = models.FileField(!
storage=FileSystemStorage(!
location="/media/"!
)!
)!
6
Slide 7
Slide 7 text
Simple usage with models
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = models.FileField(!
storage=FileSystemStorage(!
location="/media/"!
)!
)!
7
Slide 8
Slide 8 text
Simple usage with models
>>> m = Message.objects.create(!
>>> user=messi,!
>>> user_to=ronaldo,!
>>> )!
>>> m.attach.save(open("om.png"))!
"om.png"
8
Slide 9
Slide 9 text
/media/om.png
9
Slide 10
Slide 10 text
Django-storages
- S3BotoStorage
- LibCloudStorage
- AzureStorage
- GridFSStorage
- DatabaseStorage
10
Slide 11
Slide 11 text
Storage interface
11
path(name)
Slide 12
Slide 12 text
Storage interface
12
open(name,mode)
Slide 13
Slide 13 text
Storage interface
13
save(name,content)
Slide 14
Slide 14 text
Storage interface
14
delete(name)
Slide 15
Slide 15 text
Storage interface
15
exists(name)
Slide 16
Slide 16 text
That’s it (almost)
storage.path(…)!
storage.open(…)!
storage.save(…)!
storage.delete(…)!
storage.exists(…)
16
Slide 17
Slide 17 text
Save example
>>> om = open("om.png")!
>>> st = FileSystemStorage(!
>>> location="/media/"!
>>> )
17
Slide 18
Slide 18 text
Save example
>>> om = open("om.png")!
>>> st = FileSystemStorage(!
>>> location="/media/"!
>>> )
18
Slide 19
Slide 19 text
Save example
>>> om = open("om.png")!
>>> st = FileSystemStorage(!
>>> location="/media/"!
>>> )!
19
Slide 20
Slide 20 text
Save example
>>> om = open("om.png")!
>>> st = FileSystemStorage(!
>>> location="/media/"!
>>> )!
>>> st.save(!
>>> name='om.png'!
>>> content=om!
>>> )!
"om.png"!
20
Slide 21
Slide 21 text
Other methods examples
>>> st.path("om.png")!
"/media/om.png"!
!
>>> st.exists("om.png")!
True!
!
>>> st.open("om.png")!
!
!
>>> st.delete("om.png")!
21
Slide 22
Slide 22 text
Other methods examples
>>> st.path("om.png")!
"/media/om.png"!
!
>>> st.exists("om.png")!
True!
!
>>> st.open("om.png")!
!
!
>>> st.delete("om.png")!
22
Slide 23
Slide 23 text
Other methods examples
>>> st.path("om.png")!
"/media/om.png"!
!
>>> st.exists("om.png")!
True!
!
>>> st.open("om.png")!
!
!
>>> st.delete("om.png")!
23
Slide 24
Slide 24 text
Other methods examples
>>> st.path("om.png")!
"/media/om.png"!
!
>>> st.exists("om.png")!
True!
!
>>> st.open("om.png")!
!
!
>>> st.delete("om.png")!
24
Slide 25
Slide 25 text
django-proxy-storage
25
Slide 26
Slide 26 text
3 building blocks
26
Slide 27
Slide 27 text
1. It’s django storage
27
Slide 28
Slide 28 text
2. Original storage
28
Slide 29
Slide 29 text
2. Original storage
29
FileSystemStorage
Slide 30
Slide 30 text
2. Original storage
30
FileSystemStorage
S3BotoStorage
Slide 31
Slide 31 text
2. Original storage
31
FileSystemStorage
S3BotoStorage
GridFSStorage
Slide 32
Slide 32 text
2. Original storage
32
FileSystemStorage
S3BotoStorage
GridFSStorage
Any storage!
Slide 33
Slide 33 text
3. Meta-backend
33
Slide 34
Slide 34 text
3. Meta-backend
34
MongoDB, ORM
Slide 35
Slide 35 text
3. Meta-backend
35
MongoDB, ORM
Slide 36
Slide 36 text
Proxy-storage example
class MyStorage(ProxyStorageBase):!
original_storage = \!
FileSystemStorage(!
location="/media/"!
)!
meta_backend = MongoMetaBackend(!
database=get_mongo_db(),!
collection="files_meta"!
)
36
Slide 37
Slide 37 text
Proxy-storage base class
class MyStorage(ProxyStorageBase):!
original_storage = \!
FileSystemStorage(!
location="/media/"!
)!
meta_backend = MongoMetaBackend(!
database=get_mongo_db(),!
collection="files_meta"!
)
37
Slide 38
Slide 38 text
Original storage
class MyStorage(ProxyStorageBase):!
original_storage = \!
FileSystemStorage(!
location="/media/"!
)!
meta_backend = MongoMetaBackend(!
database=get_mongo_db(),!
collection="files_meta"!
)
38
Slide 39
Slide 39 text
Original storage
class MyStorage(ProxyStorageBase):!
original_storage = \!
FileSystemStorage(!
location="/media/"!
)!
meta_backend = MongoMetaBackend(!
database=get_mongo_db(),!
collection="files_meta"!
)
39
Slide 40
Slide 40 text
Meta-backend
class MyStorage(ProxyStorageBase):!
original_storage = \!
FileSystemStorage(!
location="/media/"!
)!
meta_backend = MongoMetaBackend(!
database=get_mongo_db(),!
collection="files_meta"!
)
40
Slide 41
Slide 41 text
Meta-backend
class MyStorage(ProxyStorageBase):!
original_storage = \!
FileSystemStorage(!
location="/media/"!
)!
meta_backend = MongoMetaBackend(!
database=get_mongo_db(),!
collection="files_meta"!
)
41
Slide 42
Slide 42 text
nom.png
42
Slide 43
Slide 43 text
Proxy-storage save
>>> nom = open("nom.png")!
>>> st = MyStorage()
43
Slide 44
Slide 44 text
Proxy-storage save
>>> nom = open("nom.png")!
>>> st = MyStorage()
44
Slide 45
Slide 45 text
Proxy-storage save
>>> nom = open("nom.png")!
>>> st = MyStorage()!
!
>>> st.save(!
>>> name="nom.png"!
>>> content=nom!
>>> )
45
Slide 46
Slide 46 text
Proxy-storage save
>>> nom = open("nom.png")!
>>> st = MyStorage()!
!
>>> st.save(!
>>> name="nom.png"!
>>> content=nom!
>>> )!
"/media/nom.png"
46
Slide 47
Slide 47 text
How save works
1. Save file to original storage (file system)
47
Slide 48
Slide 48 text
How save works
1. Save file to original storage (file system)
2. Save file info to meta-backend (mongoDB)
48
Slide 49
Slide 49 text
File info in meta-backend
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}
49
Slide 50
Slide 50 text
File info in meta-backend
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}
50
Slide 51
Slide 51 text
File info in meta-backend
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}
51
Slide 52
Slide 52 text
File info in meta-backend
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}
52
Slide 53
Slide 53 text
Proxy-storage open
>>> st.open("/media/nom.png")!
53
Slide 54
Slide 54 text
Find meta-info by path
>>> st.open("/media/nom.png")!
!
!
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}
54
Slide 55
Slide 55 text
Return original’s storage open
>>> st.open("/media/nom.png")!
!
!
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}
55
Slide 56
Slide 56 text
Proxy-storage exists
>>> st.exists("/media/nom.png")!
True
56
Slide 57
Slide 57 text
Find meta-info by path
>>> st.exists("/media/nom.png")!
True!
!
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}
57
Slide 58
Slide 58 text
Return found or not
>>> st.exists("/media/nom.png")!
True!
!
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}
58
Slide 59
Slide 59 text
Proxy-storage delete
>>> st.delete("/media/nom.png")!
59
Slide 60
Slide 60 text
Find meta-info by path
>>> st.delete("/media/nom.png")!
!
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}!
60
Slide 61
Slide 61 text
Call original’s storage delete
>>> st.delete("/media/nom.png")!
!
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}!
61
Slide 62
Slide 62 text
Remove meta-info
>>> st.delete("/media/nom.png")!
!
{!
"_id": ObjectId("..."),!
"path": "/media/nom.png",!
"proxy_storage_name":"my_storage",!
"original_storage_path":"nom.png"!
}!
62
Slide 63
Slide 63 text
Use cases
63
Slide 64
Slide 64 text
Custom data for meta-backend
class MyStorage(ProxyStorageBase):!
...!
def get_data_for_meta_backend_save(!
self, content, **kwargs!
):!
data = super...!
data["size"] = get_size(content)!
data["mime"] = get_mime(content)!
return data
64
Slide 65
Slide 65 text
Custom data for meta-backend
class MyStorage(ProxyStorageBase):!
...!
def get_data_for_meta_backend_save(!
self, content, **kwargs!
):!
data = super...!
data["size"] = get_size(content)!
data["mime"] = get_mime(content)!
return data
65
Slide 66
Slide 66 text
Custom data for meta-backend
class MyStorage(ProxyStorageBase):!
...!
def get_data_for_meta_backend_save(!
self, content, **kwargs!
):!
data = super...!
data["size"] = get_size(content)!
data["mime"] = get_mime(content)!
return data
66
Slide 67
Slide 67 text
Custom data for meta-backend
{!
"_id": ObjectId("..."),!
"proxy_storage_name":"my_storage",!
"path": "/media/nom.png",!
"original_storage_path":"nom.png",!
"size": 70903,!
"mime": "image/png",!
}
67
Slide 68
Slide 68 text
Multiple original storages
68
Slide 69
Slide 69 text
Multiple original storages
class MyStorage(!
MultipleOriginalStoragesMixin, !
ProxyStorageBase!
):!
...!
original_storages = (!
("fs", FileSystemStorage(!
location="/media/"),!
("amazon", AmazonS3Storage()),!
)
69
Slide 70
Slide 70 text
Multiple original storages
class MyStorage(!
MultipleOriginalStoragesMixin, !
ProxyStorageBase!
):!
...!
original_storages = (!
("fs", FileSystemStorage(!
location="/media/"),!
("amazon", AmazonS3Storage()),!
)
70
Slide 71
Slide 71 text
Multiple original storages
class MyStorage(!
MultipleOriginalStoragesMixin, !
ProxyStorageBase!
):!
...!
original_storages = (!
("fs", FileSystemStorage(!
location="/media/"),!
("amazon", AmazonS3Storage()),!
)
71
Slide 72
Slide 72 text
Multiple original storages
class MyStorage(!
MultipleOriginalStoragesMixin, !
ProxyStorageBase!
):!
...!
original_storages = (!
("fs", FileSystemStorage(!
location="/media/"),!
("amazon", AmazonS3Storage()),!
)
72
Slide 73
Slide 73 text
Multiple original storages
>>> st.save(!
>>> name="om.png"!
>>> content=om,!
>>> using="fs"!
>>> )
73
Slide 74
Slide 74 text
Multiple original storages
>>> st.save(!
>>> name="om.png"!
>>> content=om,!
>>> using="fs"!
>>> )
74
Slide 75
Slide 75 text
Multiple original storages
{!
"_id": ObjectId("..."),!
"proxy_storage_name":"my_storage",!
"path": "/media/om.png",!
"original_storage_path":"om.png",!
"original_storage_name":"fs"!
}
75
Slide 76
Slide 76 text
Multiple original storages
>>> st.save(!
>>> name="nom.png"!
>>> content=nom,!
>>> using="amazon"!
>>> )
76
Slide 77
Slide 77 text
Multiple original storages
{!
"_id": ObjectId("..."),!
"proxy_storage_name":"my_storage",!
"path":"http://am.com/nom.png”,!
"original_storage_path":"nom.png",!
"original_storage_name":"amazon"!
}
77
Slide 78
Slide 78 text
Multiple original storages
>>> st.open("/media/om.png")
From FS
78
Slide 79
Slide 79 text
Multiple original storages
>>> st.open(!
>>> "http://am.com/nom.png")
From
amazon
79
Slide 80
Slide 80 text
Save text files to FS by default
class MyStorage(...):!
...!
def save(self, **kw):!
if not using:!
if is_txt(kw["name"]):!
using = "fs"!
else:!
using = "amazon"!
return super(...).save(**kw)
80
Slide 81
Slide 81 text
Save text files to FS by default
class MyStorage(...):!
...!
def save(self, **kw):!
if not using:!
if is_txt(kw["name"]):!
using = "fs"!
else:!
using = "amazon"!
return super(...).save(**kw)
81
Slide 82
Slide 82 text
Save text files to FS by default
class MyStorage(...):!
...!
def save(self, **kw):!
if not using:!
if is_txt(kw["name"]):!
using = "fs"!
else:!
using = "amazon"!
return super(...).save(**kw)
82
Slide 83
Slide 83 text
Save text files to FS by default
class MyStorage(...):!
...!
def save(self, **kw):!
if not using:!
if is_txt(kw["name"]):!
using = "fs"!
else:!
using = "amazon"!
return super(...).save(**kw)
83
Slide 84
Slide 84 text
Save text files to FS by default
class MyStorage(...):!
...!
def save(self, **kw):!
if not using:!
if is_txt(kw["name"]):!
using = "fs"!
else:!
using = "amazon"!
return super(...).save(**kw)
84
Slide 85
Slide 85 text
Fallback
85
Slide 86
Slide 86 text
Fallback
class MyStorage(!
FallbackProxyStorageMixin, !
ProxyStorageBase!
):!
...!
original_storages = (!
("fs", OrigFSStorage(!
location="/media/"),!
("amazon", AmazonS3Storage()),!
)
86
Slide 87
Slide 87 text
Fallback
class MyStorage(!
FallbackProxyStorageMixin, !
ProxyStorageBase!
):!
...!
original_storages = (!
("fs", OrigFSStorage(!
location="/media/"),!
("amazon", AmazonS3Storage()),!
)
87
Slide 88
Slide 88 text
Fallback
class MyStorage(!
FallbackProxyStorageMixin, !
ProxyStorageBase!
):!
...!
original_storages = (!
("fs", OrigFSStorage(!
location="/media/"),!
("amazon", AmazonS3Storage()),!
)
88
Slide 89
Slide 89 text
Fallback
class MyStorage(!
FallbackProxyStorageMixin, !
ProxyStorageBase!
):!
...!
original_storages = (!
("fs", OrigFSStorage(!
location="/media/"),!
("amazon", AmazonS3Storage()),!
)
89
Slide 90
Slide 90 text
Original storage for fallback
class OrigFSStorage(!
OriginalStorageFallbackMixin,!
FileSystemStorage!
):!
fallback_exceptions = (!
IOError, !
OSError!
)
90
Slide 91
Slide 91 text
Original storage for fallback
class OrigFSStorage(!
OriginalStorageFallbackMixin,!
FileSystemStorage!
):!
fallback_exceptions = (!
IOError, !
OSError!
)
91
Slide 92
Slide 92 text
Original storage for fallback
class OrigFSStorage(!
OriginalStorageFallbackMixin,!
FileSystemStorage!
):!
fallback_exceptions = (!
IOError, !
OSError!
)
92
Slide 93
Slide 93 text
Original storage for fallback
class OrigFSStorage(!
OriginalStorageFallbackMixin,!
FileSystemStorage!
):!
fallback_exceptions = (!
IOError, !
OSError!
)
93
Slide 94
Slide 94 text
Original storage for fallback
class OrigFSStorage(!
OriginalStorageFallbackMixin,!
FileSystemStorage!
):!
fallback_exceptions = (!
IOError, !
OSError!
)
94
Slide 95
Slide 95 text
Save without errors
>>> st.save(!
>>> name="om.png"!
>>> content=om!
>>> )
95
Slide 96
Slide 96 text
Save without errors
{!
"_id": ObjectId("..."),!
"proxy_storage_name":"my_storage",!
"path": "/media/om.png",!
"original_storage_path":"om.png",!
"original_storage_name":"fs"!
}
96
Slide 97
Slide 97 text
Let’s remove dir
$ rm -rf /media/
97
Slide 98
Slide 98 text
Fallback
>>> st.save(!
>>> name="nom.png"!
>>> content=nom,!
>>> )
98
Slide 99
Slide 99 text
Fallback
>>> st.save(!
>>> name="nom.png"!
>>> content=nom,!
>>> )!
IOError
99
Slide 100
Slide 100 text
Fallback
In your face,
FS!
100
Slide 101
Slide 101 text
Fallback
{!
"_id": ObjectId("..."),!
"proxy_storage_name":"my_storage",!
"path":"http://am.com/nom.png”,!
"original_storage_path":"nom.png",!
"original_storage_name":"amazon"!
}
101
Slide 102
Slide 102 text
Authorization
102
Slide 103
Slide 103 text
Simple model
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = models.FileField(!
storage=FileSystemStorage(!
location="/media/"!
)!
)!
103
Slide 104
Slide 104 text
Nginx config without auth
server {!
server_name site.ru;!
!
location / {!
proxy_pass http://app.socket;!
}!
!
location /files/ {!
alias /media/;!
}!
}
104
Slide 105
Slide 105 text
Nginx config without auth
server {!
server_name site.ru;!
!
location / {!
proxy_pass http://app.socket;!
}!
!
location /files/ {!
alias /media/;!
}!
}
105
Slide 106
Slide 106 text
Nginx config without auth
server {!
server_name site.ru;!
!
location / {!
proxy_pass http://app.socket;!
}!
!
location /files/ {!
alias /media/;!
}!
}
106
Slide 107
Slide 107 text
Serve without auth
107
Client
Nginx
/files/
Slide 108
Slide 108 text
Serve without auth
108
Client
Nginx
/files/
Slide 109
Slide 109 text
Serve without auth
109
Client
Nginx
/files/
Slide 110
Slide 110 text
Serve without auth
110
Client
Nginx
/files/
Slide 111
Slide 111 text
Serve without auth
111
Client
Nginx
/files/
Slide 112
Slide 112 text
Serve without auth
112
Client
Nginx
/files/
Slide 113
Slide 113 text
http://site.ru/files/om.png
113
Slide 114
Slide 114 text
Add auth logic
Serve files only for author or receiver of message
114
Slide 115
Slide 115 text
Nginx config without auth
server {!
server_name site.ru;!
!
location / {!
proxy_pass http://app.socket;!
}!
!
location /files/ {!
alias /media/;!
}!
}
115
Slide 116
Slide 116 text
Nginx config with auth
server {!
server_name site.ru;!
!
location / {!
proxy_pass http://app.socket;!
}!
!
location /files/ {!
internal;!
alias /media/;!
}!
}
116
Slide 117
Slide 117 text
Client can’t directly get files
117
Client
Nginx
/files/
Slide 118
Slide 118 text
Rename location
server {!
server_name site.ru;!
!
location / {!
proxy_pass http://app.socket;!
}!
!
location /x-files/ {!
internal;!
alias /media/;!
}!
}
118
Slide 119
Slide 119 text
Auth logic in view
urlpatterns = patterns('',!
url(r'^files/$', file_view)!
)
119
Slide 120
Slide 120 text
Auth logic in view
def file_view(request):!
path = request.GET["path"]!
res = HttpResponse()!
if is_has_access(request,path):!
res["X-Accel-Redirect"] = \!
"/x-files/" + path!
else:!
res.status_code = 404!
return res
120
Slide 121
Slide 121 text
Auth logic in view
def file_view(request):!
path = request.GET["path"]!
res = HttpResponse()!
if is_has_access(request,path):!
res["X-Accel-Redirect"] = \!
"/x-files/" + path!
else:!
res.status_code = 404!
return res
121
Slide 122
Slide 122 text
Auth logic in view
def file_view(request):!
path = request.GET["path"]!
res = HttpResponse()!
if is_has_access(request,path):!
res["X-Accel-Redirect"] = \!
"/x-files/" + path!
else:!
res.status_code = 404!
return res
122
Slide 123
Slide 123 text
Auth logic in view
def file_view(request):!
path = request.GET["path"]!
res = HttpResponse()!
if is_has_access(request,path):!
res["X-Accel-Redirect"] = \!
"/x-files/" + path!
else:!
res.status_code = 404!
return res
123
Slide 124
Slide 124 text
Auth logic in view
def file_view(request):!
path = request.GET["path"]!
res = HttpResponse()!
if is_has_access(request,path):!
res["X-Accel-Redirect"] = \!
"/x-files/" + path!
else:!
res.status_code = 404!
return res
124
Slide 125
Slide 125 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
125
Slide 126
Slide 126 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
126
Slide 127
Slide 127 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
127
Slide 128
Slide 128 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
128
Slide 129
Slide 129 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
X-Accel-Redirect: /x-files/om.png
129
Slide 130
Slide 130 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
X-Accel-Redirect: /x-files/om.png
130
Slide 131
Slide 131 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
X-Accel-Redirect: /x-files/om.png
131
Slide 132
Slide 132 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
X-Accel-Redirect: /x-files/om.png
132
Slide 133
Slide 133 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
X-Accel-Redirect: /x-files/om.png
133
Slide 134
Slide 134 text
Serve with auth
Client
/ /x-files/
Django app
Nginx
X-Accel-Redirect: /x-files/om.png
134
Slide 135
Slide 135 text
http://site.ru/files/?path=om.png
135
Slide 136
Slide 136 text
Auth logic in view
def file_view(request):!
path = request.GET["path"]!
res = HttpResponse()!
if is_has_access(request,path):!
res["X-Accel-Redirect"] = \!
"/x-files/" + path!
else:!
res.status_code = 404!
return res
136
Slide 137
Slide 137 text
Find message with path
def is_has_access(request, path):!
m = Message.objects.get(!
attach=path!
)!
...
137
Slide 138
Slide 138 text
What if user has resume?
class Resume(models.Model):!
user = models.ForeignKey(User)!
file = models.FileField(!
storage=FileSystemStorage(!
location="/resumes/"!
)!
)!
!
# Show resume only for owners!
138
Slide 139
Slide 139 text
Find message or resume by path
def is_has_access(request, path):!
m = Message.objects.get(!
attach=path!
)!
res = Resume.objects.get(!
file=path!
)!
...
139
Slide 140
Slide 140 text
3 Problems
1. Collision of paths.
/media/om.png and /resumes/om.png
!
140
Slide 141
Slide 141 text
3 Problems
1. Collision of paths.
/media/om.png and /resumes/om.png
2. N requests for N models
141
Slide 142
Slide 142 text
3 Problems
1. Collision of paths.
/media/om.png and /resumes/om.png
2. N requests for N models
3. Context of field
142
Slide 143
Slide 143 text
Refactor to proxy-storage
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = models.FileField(!
storage=FileSystemStorage(!
location="/media/"!
)!
)!
143
Slide 144
Slide 144 text
Refactor to proxy-storage
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = models.FileField(!
storage=AttStorage()!
)!
144
Slide 145
Slide 145 text
Refactor to proxy-storage
class Resume(models.Model):!
user = models.ForeignKey(User)!
file = models.FileField(!
storage=FileSystemStorage(!
location="/resumes/"!
)!
)!
145
Slide 146
Slide 146 text
Refactor to proxy-storage
class Resume(models.Model):!
user = models.ForeignKey(User)!
file = models.FileField(!
storage=ResStorage()!
)!
146
Slide 147
Slide 147 text
Proxy-storages
mongo_meta = MongoMetaBackend(…)!
!
class AttStorage(ProxyStorageBase):
original_storage = \!
FSStorage(location="/media/")!
meta_backend = mongo_meta!
!
class ResStorage(ProxyStorageBase):
original_storage = \!
FSStorage(location="/resumes/")!
meta_backend = mongo_meta!
147
Slide 148
Slide 148 text
Proxy-storages
mongo_meta = MongoMetaBackend(…)!
!
class AttStorage(ProxyStorageBase):
original_storage = \!
FSStorage(location="/media/")!
meta_backend = mongo_meta!
!
class ResStorage(ProxyStorageBase):
original_storage = \!
FSStorage(location="/resumes/")!
meta_backend = mongo_meta!
148
Slide 149
Slide 149 text
Path is unique per meta-backend
{!
"_id": ObjectId("..."),!
"path": "/media/om.png",!
"proxy_storage_name":"attachment",!
"original_storage_path":"om.png"!
},!
{!
"_id": ObjectId("..."),!
"path": "/resumes/om.png",!
"proxy_storage_name":"resumes",!
"original_storage_path":"om.png"!
}!
149
Slide 150
Slide 150 text
Has the same original path
{!
"_id": ObjectId("..."),!
"path": "/media/om.png",!
"proxy_storage_name":"attachment",!
"original_storage_path":"om.png"!
},!
{!
"_id": ObjectId("..."),!
"path": "/resumes/om.png",!
"proxy_storage_name":"resumes",!
"original_storage_path":"om.png"!
}!
150
Slide 151
Slide 151 text
Path is unique per meta-backend
{!
"_id": ObjectId("..."),!
"path": "/media/om.png",!
"proxy_storage_name":"attachment",!
"original_storage_path":"om.png"!
},!
{!
"_id": ObjectId("..."),!
"path": "/resumes/om.png",!
"proxy_storage_name":"resumes",!
"original_storage_path":"om.png"!
}!
151
Slide 152
Slide 152 text
2 Problems
1. Collision of paths.
/media/om.png and /resumes/om.png
2. N requests for N models
3. Context of field
152
Slide 153
Slide 153 text
Refactor to ProxyStorageFileField
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = models.FileField(!
storage=AttStorage()!
)!
153
Slide 154
Slide 154 text
Refactor to ProxyStorageFileField
class Message(models.Model):!
user = models.ForeignKey(User)!
to_user = models.ForeignKey(User)!
attach = ProxyStorageFileField(!
storage=AttStorage()!
)!
154
Slide 155
Slide 155 text
Refactor to ProxyStorageFileField
class Resume(models.Model):!
user = models.ForeignKey(User)!
file = models.FileField(!
storage=ResStorage()!
)!
155
Slide 156
Slide 156 text
Refactor to ProxyStorageFileField
class Resume(models.Model):!
user = models.ForeignKey(User)!
file = ProxyStorageFileField(!
storage=ResStorage()!
)!
156
Slide 157
Slide 157 text
ProxyStorageFileField new data
{!
"_id": ObjectId("..."),!
"path": "/media/om.png",!
"proxy_storage_name":"attachment",!
"original_storage_path":"om.png",!
"content_type_id": 1,!
"object_id": 100,!
"field": "attach"!
}
157
Slide 158
Slide 158 text
ProxyStorageFileField new data
{!
"_id": ObjectId("..."),!
"path": "/media/om.png",!
"proxy_storage_name":"attachment",!
"original_storage_path":"om.png",!
"content_type_id": 1,!
"object_id": 100,!
"field": "attach"!
}
158
Slide 159
Slide 159 text
ProxyStorageFileField new data
{!
"_id": ObjectId("..."),!
"path": "/media/om.png",!
"proxy_storage_name":"attachment",!
"original_storage_path":"om.png",!
"content_type_id": 1,!
"object_id": 100,!
"field": "attach"!
}
159
Slide 160
Slide 160 text
No problems
1. Collision of paths.
/media/om.png and /resumes/om.png
2. N requests for N models
3. Context of field
160
Slide 161
Slide 161 text
Find model instance from data
def is_has_access(request, path):!
data = mongo_meta.get(path)!
model = ContentType.objects.get(!
id=data["content_type_id"]!
)!
instance = model.objects.get(!
id=data["object_id"]!
)!
return instance.check_access(!
request, data["field"])
161
Slide 162
Slide 162 text
Find model instance from data
def is_has_access(request, path):!
data = mongo_meta.get(path)!
model = ContentType.objects.get(!
id=data["content_type_id"]!
)!
instance = model.objects.get(!
id=data["object_id"]!
)!
return instance.check_access(!
request, data["field"])
162
Slide 163
Slide 163 text
Find model instance from data
def is_has_access(request, path):!
data = mongo_meta.get(path)!
model = ContentType.objects.get(!
id=data["content_type_id"]!
)!
instance = model.objects.get(!
id=data["object_id"]!
)!
return instance.check_access(!
request, data["field"])
163
Slide 164
Slide 164 text
Find model instance from data
def is_has_access(request, path):!
data = mongo_meta.get(path)!
model = ContentType.objects.get(!
id=data["content_type_id"]!
)!
instance = model.objects.get(!
id=data["object_id"]!
)!
return instance.check_access(!
request, data["field"])
164
Slide 165
Slide 165 text
Find model instance from data
def is_has_access(request, path):!
data = mongo_meta.get(path)!
model = ContentType.objects.get(!
id=data["content_type_id"]!
)!
instance = model.objects.get(!
id=data["object_id"]!
)!
return instance.check_access(!
request, data["field"])
165
Slide 166
Slide 166 text
Message check_access
class Message(models.Model):!
...!
def check_access(self,req,field):!
if field == "attach":!
return req.user in (!
self.user, !
self.to_user!
)
166
Slide 167
Slide 167 text
Message check_access
class Message(models.Model):!
...!
def check_access(self,req,field):!
if field == "attach":!
return req.user in (!
self.user, !
self.to_user!
)
167
Slide 168
Slide 168 text
Resume check_access
class Resume(models.Model):!
...!
def check_access(self,req,field):!
if field == "file":!
return req.user == self.user
168
Slide 169
Slide 169 text
Resume check_access
class Resume(models.Model):!
...!
def check_access(self,req,field):!
if field == "file":!
return req.user == self.user
169
Slide 170
Slide 170 text
Make alias to root
server {!
server_name site.ru;!
!
location / {!
proxy_pass http://app.socket;!
}!
!
location /x-files/ {!
internal;!
alias /;!
}!
}
170
Slide 171
Slide 171 text
Not resume owner
Client
/ /x-files/
Django app
Nginx
404
171
Slide 172
Slide 172 text
/files/?path=/resumes/om.png
172
404
Slide 173
Slide 173 text
Resume owner
Client
/ /x-files/
Django app
Nginx
X-Accel-Redirect: !
/x-files/resumes/om.png
173
Slide 174
Slide 174 text
/files/?path=/resumes/om.png
174
Slide 175
Slide 175 text
True story
class MainStorage(!
FallbackProxyStorageMixin,
ProxyStorageBase):!
original_storages = [!
('elliptics', OrigElliptics()),!
('gridfs', GridFSStorage()),!
]!
meta_backend = ORMMetaBackend(!
model=ProxyStorageModel!
)
175
Slide 176
Slide 176 text
True story
class MainStorage(!
FallbackProxyStorageMixin,
ProxyStorageBase):!
original_storages = [!
('elliptics', OrigElliptics()),!
('gridfs', GridFSStorage()),!
]!
meta_backend = ORMMetaBackend(!
model=ProxyStorageModel!
)
176
Slide 177
Slide 177 text
True story
class MainStorage(!
FallbackProxyStorageMixin,
ProxyStorageBase):!
original_storages = [!
('elliptics', OrigElliptics()),!
('gridfs', GridFSStorage()),!
]!
meta_backend = ORMMetaBackend(!
model=ProxyStorageModel!
)
177
Slide 178
Slide 178 text
True story
from requests.exceptions import \
RequestException!
!
class OrigElliptics(!
OriginalStorageFallbackMixin,
EllipticsStorage!
):!
fallback_exceptions = (!
RequestException,!
)
178
Slide 179
Slide 179 text
True story
from requests.exceptions import \
RequestException!
!
class OrigElliptics(!
OriginalStorageFallbackMixin,
EllipticsStorage!
):!
fallback_exceptions = (!
RequestException,!
)
179
Slide 180
Slide 180 text
180
Slide 181
Slide 181 text
181
Slide 182
Slide 182 text
X-Accel-Redirect: /Спасибо/
Repo - https://github.com/chibisov/django-proxy-storage
Docs - http://chibisov.github.io/django-proxy-storage/docs/
https://github.com/chibisov
@MyNameIss
This fellow is from !
http://www.cuttherope.net/