Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Работа с файлами с помощью django-proxy-storage

Работа с файлами с помощью django-proxy-storage

Геннадий Чибисов (Яндекс)

В этом докладе я хочу рассказать каким образом с помощью нашего инструмента django-proxy-storage можно организовать:

Авторизацию раздачи файлов в веб-приложениях;
Динамическое использование нескольких стораджей;
Автоматический фолбэк до работающего стораджа на уровне приложения.

Moscow Python Meetup

October 01, 2014
Tweet

More Decks by Moscow Python Meetup

Other Decks in Programming

Transcript

  1. Simple usage with models class Message(models.Model):! user = models.ForeignKey(User)! to_user

    = models.ForeignKey(User)! attach = models.FileField(! storage=FileSystemStorage(! location="/media/"! )! )! 4
  2. Simple usage with models class Message(models.Model):! user = models.ForeignKey(User)! to_user

    = models.ForeignKey(User)! attach = models.FileField(! storage=FileSystemStorage(! location="/media/"! )! )! 5
  3. Simple usage with models class Message(models.Model):! user = models.ForeignKey(User)! to_user

    = models.ForeignKey(User)! attach = models.FileField(! storage=FileSystemStorage(! location="/media/"! )! )! 6
  4. Simple usage with models class Message(models.Model):! user = models.ForeignKey(User)! to_user

    = models.ForeignKey(User)! attach = models.FileField(! storage=FileSystemStorage(! location="/media/"! )! )! 7
  5. Simple usage with models >>> m = Message.objects.create(! >>> user=messi,!

    >>> user_to=ronaldo,! >>> )! >>> m.attach.save(open("om.png"))! "om.png" 8
  6. Save example >>> om = open("om.png")! >>> st = FileSystemStorage(!

    >>> location="/media/"! >>> )! >>> st.save(! >>> name='om.png'! >>> content=om! >>> )! "om.png"! 20
  7. Other methods examples >>> st.path("om.png")! "/media/om.png"! ! >>> st.exists("om.png")! True!

    ! >>> st.open("om.png")! <File: /media/om.png>! ! >>> st.delete("om.png")! 21
  8. Other methods examples >>> st.path("om.png")! "/media/om.png"! ! >>> st.exists("om.png")! True!

    ! >>> st.open("om.png")! <File: /media/om.png>! ! >>> st.delete("om.png")! 22
  9. Other methods examples >>> st.path("om.png")! "/media/om.png"! ! >>> st.exists("om.png")! True!

    ! >>> st.open("om.png")! <File: /media/om.png>! ! >>> st.delete("om.png")! 23
  10. Other methods examples >>> st.path("om.png")! "/media/om.png"! ! >>> st.exists("om.png")! True!

    ! >>> st.open("om.png")! <File: /media/om.png>! ! >>> st.delete("om.png")! 24
  11. Proxy-storage example class MyStorage(ProxyStorageBase):! original_storage = \! FileSystemStorage(! location="/media/"! )!

    meta_backend = MongoMetaBackend(! database=get_mongo_db(),! collection="files_meta"! ) 36
  12. Proxy-storage base class class MyStorage(ProxyStorageBase):! original_storage = \! FileSystemStorage(! location="/media/"!

    )! meta_backend = MongoMetaBackend(! database=get_mongo_db(),! collection="files_meta"! ) 37
  13. Original storage class MyStorage(ProxyStorageBase):! original_storage = \! FileSystemStorage(! location="/media/"! )!

    meta_backend = MongoMetaBackend(! database=get_mongo_db(),! collection="files_meta"! ) 38
  14. Original storage class MyStorage(ProxyStorageBase):! original_storage = \! FileSystemStorage(! location="/media/"! )!

    meta_backend = MongoMetaBackend(! database=get_mongo_db(),! collection="files_meta"! ) 39
  15. Proxy-storage save >>> nom = open("nom.png")! >>> st = MyStorage()!

    ! >>> st.save(! >>> name="nom.png"! >>> content=nom! >>> ) 45
  16. Proxy-storage save >>> nom = open("nom.png")! >>> st = MyStorage()!

    ! >>> st.save(! >>> name="nom.png"! >>> content=nom! >>> )! "/media/nom.png" 46
  17. How save works 1. Save file to original storage (file

    system) 2. Save file info to meta-backend (mongoDB) 48
  18. Find meta-info by path >>> st.open("/media/nom.png")! <File: /media/nom.png>! ! {!

    "_id": ObjectId("..."),! "path": "/media/nom.png",! "proxy_storage_name":"my_storage",! "original_storage_path":"nom.png"! } 54
  19. Return original’s storage open >>> st.open("/media/nom.png")! <File: /media/nom.png>! ! {!

    "_id": ObjectId("..."),! "path": "/media/nom.png",! "proxy_storage_name":"my_storage",! "original_storage_path":"nom.png"! } 55
  20. Find meta-info by path >>> st.exists("/media/nom.png")! True! ! {! "_id":

    ObjectId("..."),! "path": "/media/nom.png",! "proxy_storage_name":"my_storage",! "original_storage_path":"nom.png"! } 57
  21. Return found or not >>> st.exists("/media/nom.png")! True! ! {! "_id":

    ObjectId("..."),! "path": "/media/nom.png",! "proxy_storage_name":"my_storage",! "original_storage_path":"nom.png"! } 58
  22. Find meta-info by path >>> st.delete("/media/nom.png")! ! {! "_id": ObjectId("..."),!

    "path": "/media/nom.png",! "proxy_storage_name":"my_storage",! "original_storage_path":"nom.png"! }! 60
  23. Call original’s storage delete >>> st.delete("/media/nom.png")! ! {! "_id": ObjectId("..."),!

    "path": "/media/nom.png",! "proxy_storage_name":"my_storage",! "original_storage_path":"nom.png"! }! 61
  24. Remove meta-info >>> st.delete("/media/nom.png")! ! {! "_id": ObjectId("..."),! "path": "/media/nom.png",!

    "proxy_storage_name":"my_storage",! "original_storage_path":"nom.png"! }! 62
  25. Custom data for meta-backend class MyStorage(ProxyStorageBase):! ...! def get_data_for_meta_backend_save(! self,

    content, **kwargs! ):! data = super...! data["size"] = get_size(content)! data["mime"] = get_mime(content)! return data 64
  26. Custom data for meta-backend class MyStorage(ProxyStorageBase):! ...! def get_data_for_meta_backend_save(! self,

    content, **kwargs! ):! data = super...! data["size"] = get_size(content)! data["mime"] = get_mime(content)! return data 65
  27. Custom data for meta-backend class MyStorage(ProxyStorageBase):! ...! def get_data_for_meta_backend_save(! self,

    content, **kwargs! ):! data = super...! data["size"] = get_size(content)! data["mime"] = get_mime(content)! return data 66
  28. Custom data for meta-backend {! "_id": ObjectId("..."),! "proxy_storage_name":"my_storage",! "path": "/media/nom.png",!

    "original_storage_path":"nom.png",! "size": 70903,! "mime": "image/png",! } 67
  29. Multiple original storages class MyStorage(! MultipleOriginalStoragesMixin, ! ProxyStorageBase! ):! ...!

    original_storages = (! ("fs", FileSystemStorage(! location="/media/"),! ("amazon", AmazonS3Storage()),! ) 69
  30. Multiple original storages class MyStorage(! MultipleOriginalStoragesMixin, ! ProxyStorageBase! ):! ...!

    original_storages = (! ("fs", FileSystemStorage(! location="/media/"),! ("amazon", AmazonS3Storage()),! ) 70
  31. Multiple original storages class MyStorage(! MultipleOriginalStoragesMixin, ! ProxyStorageBase! ):! ...!

    original_storages = (! ("fs", FileSystemStorage(! location="/media/"),! ("amazon", AmazonS3Storage()),! ) 71
  32. Multiple original storages class MyStorage(! MultipleOriginalStoragesMixin, ! ProxyStorageBase! ):! ...!

    original_storages = (! ("fs", FileSystemStorage(! location="/media/"),! ("amazon", AmazonS3Storage()),! ) 72
  33. Save text files to FS by default class MyStorage(...):! ...!

    def save(self, **kw):! if not using:! if is_txt(kw["name"]):! using = "fs"! else:! using = "amazon"! return super(...).save(**kw) 80
  34. Save text files to FS by default class MyStorage(...):! ...!

    def save(self, **kw):! if not using:! if is_txt(kw["name"]):! using = "fs"! else:! using = "amazon"! return super(...).save(**kw) 81
  35. Save text files to FS by default class MyStorage(...):! ...!

    def save(self, **kw):! if not using:! if is_txt(kw["name"]):! using = "fs"! else:! using = "amazon"! return super(...).save(**kw) 82
  36. Save text files to FS by default class MyStorage(...):! ...!

    def save(self, **kw):! if not using:! if is_txt(kw["name"]):! using = "fs"! else:! using = "amazon"! return super(...).save(**kw) 83
  37. Save text files to FS by default class MyStorage(...):! ...!

    def save(self, **kw):! if not using:! if is_txt(kw["name"]):! using = "fs"! else:! using = "amazon"! return super(...).save(**kw) 84
  38. Fallback class MyStorage(! FallbackProxyStorageMixin, ! ProxyStorageBase! ):! ...! original_storages =

    (! ("fs", OrigFSStorage(! location="/media/"),! ("amazon", AmazonS3Storage()),! ) 86
  39. Fallback class MyStorage(! FallbackProxyStorageMixin, ! ProxyStorageBase! ):! ...! original_storages =

    (! ("fs", OrigFSStorage(! location="/media/"),! ("amazon", AmazonS3Storage()),! ) 87
  40. Fallback class MyStorage(! FallbackProxyStorageMixin, ! ProxyStorageBase! ):! ...! original_storages =

    (! ("fs", OrigFSStorage(! location="/media/"),! ("amazon", AmazonS3Storage()),! ) 88
  41. Fallback class MyStorage(! FallbackProxyStorageMixin, ! ProxyStorageBase! ):! ...! original_storages =

    (! ("fs", OrigFSStorage(! location="/media/"),! ("amazon", AmazonS3Storage()),! ) 89
  42. Simple model class Message(models.Model):! user = models.ForeignKey(User)! to_user = models.ForeignKey(User)!

    attach = models.FileField(! storage=FileSystemStorage(! location="/media/"! )! )! 103
  43. Nginx config without auth server {! server_name site.ru;! ! location

    / {! proxy_pass http://app.socket;! }! ! location /files/ {! alias /media/;! }! } 104
  44. Nginx config without auth server {! server_name site.ru;! ! location

    / {! proxy_pass http://app.socket;! }! ! location /files/ {! alias /media/;! }! } 105
  45. Nginx config without auth server {! server_name site.ru;! ! location

    / {! proxy_pass http://app.socket;! }! ! location /files/ {! alias /media/;! }! } 106
  46. Nginx config without auth server {! server_name site.ru;! ! location

    / {! proxy_pass http://app.socket;! }! ! location /files/ {! alias /media/;! }! } 115
  47. Nginx config with auth server {! server_name site.ru;! ! location

    / {! proxy_pass http://app.socket;! }! ! location /files/ {! internal;! alias /media/;! }! } 116
  48. Rename location server {! server_name site.ru;! ! location / {!

    proxy_pass http://app.socket;! }! ! location /x-files/ {! internal;! alias /media/;! }! } 118
  49. Auth logic in view def file_view(request):! path = request.GET["path"]! res

    = HttpResponse()! if is_has_access(request,path):! res["X-Accel-Redirect"] = \! "/x-files/" + path! else:! res.status_code = 404! return res 120
  50. Auth logic in view def file_view(request):! path = request.GET["path"]! res

    = HttpResponse()! if is_has_access(request,path):! res["X-Accel-Redirect"] = \! "/x-files/" + path! else:! res.status_code = 404! return res 121
  51. Auth logic in view def file_view(request):! path = request.GET["path"]! res

    = HttpResponse()! if is_has_access(request,path):! res["X-Accel-Redirect"] = \! "/x-files/" + path! else:! res.status_code = 404! return res 122
  52. Auth logic in view def file_view(request):! path = request.GET["path"]! res

    = HttpResponse()! if is_has_access(request,path):! res["X-Accel-Redirect"] = \! "/x-files/" + path! else:! res.status_code = 404! return res 123
  53. Auth logic in view def file_view(request):! path = request.GET["path"]! res

    = HttpResponse()! if is_has_access(request,path):! res["X-Accel-Redirect"] = \! "/x-files/" + path! else:! res.status_code = 404! return res 124
  54. Auth logic in view def file_view(request):! path = request.GET["path"]! res

    = HttpResponse()! if is_has_access(request,path):! res["X-Accel-Redirect"] = \! "/x-files/" + path! else:! res.status_code = 404! return res 136
  55. What if user has resume? class Resume(models.Model):! user = models.ForeignKey(User)!

    file = models.FileField(! storage=FileSystemStorage(! location="/resumes/"! )! )! ! # Show resume only for owners! 138
  56. Find message or resume by path def is_has_access(request, path):! m

    = Message.objects.get(! attach=path! )! res = Resume.objects.get(! file=path! )! ... 139
  57. 3 Problems 1. Collision of paths. /media/om.png and /resumes/om.png 2.

    N requests for N models 3. Context of field 142
  58. Refactor to proxy-storage class Message(models.Model):! user = models.ForeignKey(User)! to_user =

    models.ForeignKey(User)! attach = models.FileField(! storage=FileSystemStorage(! location="/media/"! )! )! 143
  59. Refactor to proxy-storage class Message(models.Model):! user = models.ForeignKey(User)! to_user =

    models.ForeignKey(User)! attach = models.FileField(! storage=AttStorage()! )! 144
  60. Refactor to proxy-storage class Resume(models.Model):! user = models.ForeignKey(User)! file =

    models.FileField(! storage=FileSystemStorage(! location="/resumes/"! )! )! 145
  61. Proxy-storages mongo_meta = MongoMetaBackend(…)! ! class AttStorage(ProxyStorageBase): original_storage = \!

    FSStorage(location="/media/")! meta_backend = mongo_meta! ! class ResStorage(ProxyStorageBase): original_storage = \! FSStorage(location="/resumes/")! meta_backend = mongo_meta! 147
  62. Proxy-storages mongo_meta = MongoMetaBackend(…)! ! class AttStorage(ProxyStorageBase): original_storage = \!

    FSStorage(location="/media/")! meta_backend = mongo_meta! ! class ResStorage(ProxyStorageBase): original_storage = \! FSStorage(location="/resumes/")! meta_backend = mongo_meta! 148
  63. Path is unique per meta-backend {! "_id": ObjectId("..."),! "path": "/media/om.png",!

    "proxy_storage_name":"attachment",! "original_storage_path":"om.png"! },! {! "_id": ObjectId("..."),! "path": "/resumes/om.png",! "proxy_storage_name":"resumes",! "original_storage_path":"om.png"! }! 149
  64. Has the same original path {! "_id": ObjectId("..."),! "path": "/media/om.png",!

    "proxy_storage_name":"attachment",! "original_storage_path":"om.png"! },! {! "_id": ObjectId("..."),! "path": "/resumes/om.png",! "proxy_storage_name":"resumes",! "original_storage_path":"om.png"! }! 150
  65. Path is unique per meta-backend {! "_id": ObjectId("..."),! "path": "/media/om.png",!

    "proxy_storage_name":"attachment",! "original_storage_path":"om.png"! },! {! "_id": ObjectId("..."),! "path": "/resumes/om.png",! "proxy_storage_name":"resumes",! "original_storage_path":"om.png"! }! 151
  66. 2 Problems 1. Collision of paths. /media/om.png and /resumes/om.png 2.

    N requests for N models 3. Context of field 152
  67. Refactor to ProxyStorageFileField class Message(models.Model):! user = models.ForeignKey(User)! to_user =

    models.ForeignKey(User)! attach = models.FileField(! storage=AttStorage()! )! 153
  68. Refactor to ProxyStorageFileField class Message(models.Model):! user = models.ForeignKey(User)! to_user =

    models.ForeignKey(User)! attach = ProxyStorageFileField(! storage=AttStorage()! )! 154
  69. No problems 1. Collision of paths. /media/om.png and /resumes/om.png 2.

    N requests for N models 3. Context of field 160
  70. Find model instance from data def is_has_access(request, path):! data =

    mongo_meta.get(path)! model = ContentType.objects.get(! id=data["content_type_id"]! )! instance = model.objects.get(! id=data["object_id"]! )! return instance.check_access(! request, data["field"]) 161
  71. Find model instance from data def is_has_access(request, path):! data =

    mongo_meta.get(path)! model = ContentType.objects.get(! id=data["content_type_id"]! )! instance = model.objects.get(! id=data["object_id"]! )! return instance.check_access(! request, data["field"]) 162
  72. Find model instance from data def is_has_access(request, path):! data =

    mongo_meta.get(path)! model = ContentType.objects.get(! id=data["content_type_id"]! )! instance = model.objects.get(! id=data["object_id"]! )! return instance.check_access(! request, data["field"]) 163
  73. Find model instance from data def is_has_access(request, path):! data =

    mongo_meta.get(path)! model = ContentType.objects.get(! id=data["content_type_id"]! )! instance = model.objects.get(! id=data["object_id"]! )! return instance.check_access(! request, data["field"]) 164
  74. Find model instance from data def is_has_access(request, path):! data =

    mongo_meta.get(path)! model = ContentType.objects.get(! id=data["content_type_id"]! )! instance = model.objects.get(! id=data["object_id"]! )! return instance.check_access(! request, data["field"]) 165
  75. Message check_access class Message(models.Model):! ...! def check_access(self,req,field):! if field ==

    "attach":! return req.user in (! self.user, ! self.to_user! ) 166
  76. Message check_access class Message(models.Model):! ...! def check_access(self,req,field):! if field ==

    "attach":! return req.user in (! self.user, ! self.to_user! ) 167
  77. Make alias to root server {! server_name site.ru;! ! location

    / {! proxy_pass http://app.socket;! }! ! location /x-files/ {! internal;! alias /;! }! } 170
  78. True story class MainStorage(! FallbackProxyStorageMixin, ProxyStorageBase):! original_storages = [! ('elliptics',

    OrigElliptics()),! ('gridfs', GridFSStorage()),! ]! meta_backend = ORMMetaBackend(! model=ProxyStorageModel! ) 175
  79. True story class MainStorage(! FallbackProxyStorageMixin, ProxyStorageBase):! original_storages = [! ('elliptics',

    OrigElliptics()),! ('gridfs', GridFSStorage()),! ]! meta_backend = ORMMetaBackend(! model=ProxyStorageModel! ) 176
  80. True story class MainStorage(! FallbackProxyStorageMixin, ProxyStorageBase):! original_storages = [! ('elliptics',

    OrigElliptics()),! ('gridfs', GridFSStorage()),! ]! meta_backend = ORMMetaBackend(! model=ProxyStorageModel! ) 177
  81. True story from requests.exceptions import \ RequestException! ! class OrigElliptics(!

    OriginalStorageFallbackMixin, EllipticsStorage! ):! fallback_exceptions = (! RequestException,! ) 178
  82. True story from requests.exceptions import \ RequestException! ! class OrigElliptics(!

    OriginalStorageFallbackMixin, EllipticsStorage! ):! fallback_exceptions = (! RequestException,! ) 179
  83. 180

  84. 181