Use open source software to develop ideas at work

Transcript

1. Use open source software to develop ideas at work Sammy Fung BarCamp Macau 2014
2. Sammy Fung • Python Developer – Perl → PHP → Python • President at OpenSource.HK • Community Volunteer in HK & Asia – GNOME.Asia, Mozilla......
3. 將來市場領導者必會是 使用及貢獻開放源碼的 企業 Future market leader must be the one which use and contribute open source.
4. 自由 FREEDOM
5. “ Free software means software that respects users' freedom and community. ” http://www.gnu.org/philosophy/free-sw.html
6. 4 Essential Freedoms • Freedom 0: Run the program. • Freedom 1: Open Source to study and change it. • Freedom 2: Re-distribute the program. • Freedom 3: Distribute your modified versions as free software.
7. Ecosystem in Technology Innovation
8. Ecosystem in Technology Innovation • Operating Systems • Servers with Applications • Clients
9. Operating System • Linux – Server, Desktop, Embedded System. – Mobile (Android). • 1983: GNU projects developed many free software programs replacing proprietary programs on UNIX platforms. • 1991: Linus Torvalds released first Linux kernel on internet, then developers integrate with GNU softwares (GNU/Linux) and distribute a new OS. • Documentary Movie: Revolution OS (2001).
10. Web Server • Apache Software Foundation. • Apache HTTP Server (Web Server). • Most popular HTTP server since April 1996. • 1995: First release • 2009: 1st web server serving 100 million websites. • August 2014: 47.92% market share (1st) – http://news.netcraft.com/archives/2014/08/27/august-2014- web-server-survey.html • Documentary Movie: Revolution OS (2001).
11. Applications • Developing Web Applications and Mobile Applications running on OS platforms. • Huge amount of free software (open source software) are esstential and used to develop, manage and operate web & mobile apps and their platforms. • Programming Languages: – PHP, Python, Ruby...... • Web Frameworks, CMS, Blog......: – Wordpress, Drupal, Django, Joomla......
12. Web Browser • Mozilla Foundation (Found in 2003). • Firefox Web Browser. • Market Share: 15-20% (2014), 25-30% (2011). • 1998: Netscape Communications Corp (creator of Netscape Navigator web browser) created Mozilla project to coordinate open source version of internet suite. – Internet Suite: Web + Mail / Newsgroup + Web Composer + IRC. • 2002: 1st release Firefox web browser (experimental branch of Mozilla project). • Sept 2014: Firefox 32. • Documentary Movie: Code Rush
13. Example 1 Open Source Job Board
14. 很多年前 Many years ago...
15. 找工作 Looking for a job
16. 一個香港徵才網站 At a popular job site
17. 傳回錯誤 ! Got error!
18. 原因 ? Why?
19. 你會猜到的 ! You should get it!
21. 高 ! High!
22. 市場佔有率 …...in Market Share
23. Also......
24. 高 ! High!
25. ASP CFM Nginx+PHP
26. ASP
27. 市場佔有率 …...in Market Share
28. 所以 ...... So......
29. ... 我就決定 ... ...I decided...
30. 做開放源碼的 徵才網站軟件 ! Develop open source job board software
31. 從 Drupal 6 模組開始 Development starting from Drupal 6 module
32. 但沒有發表 Never launched
33. 在今年 In this year
34. 轉移到 Drupal 7 模組 Porting from Drupal 6 to Drupal 7
35. API 改變 API Changed
36. 修改了不少 codes Changed some codes
37. GitHub: sammyfung / drupal7-job
38. 張貼徵才 Job Posting
39. 分類 Job Category
40. 公司資料 Company Information
41. 每個分類也有 獨立 RSS RSS Feeds to each job category.
42. 開放源碼 Open Source
43. 你就可以有擁有你 自己的徵才網站 You can own your job board now
44. 如何 ? How ?
45. 安裝 Linux 網頁伺服器 Install a Linux web server.
46. 安裝 Drupal 內容管理系統 Install Drupal Content Management System
47. 安裝這徵才網站 的 Drupal 模組 Install and enable this job board module for Drupal
48. 完成 ! Done!
49. job.opensource.hk
50. 如果你有任何職位需要懂 任何一種開源軟件技術 ... If you got any job positions which requires any one of open source software skill sets....
51. 就貼去 Post it to job.opensource.hk
52. 免費 ! It's Free of Charge!
53. 將來發展 Future Roadmap
54. 推出 Django 模組 Release a Django module
55. 推出 Wordpres 模組 ? Release Wordpress module ?
56. 推出 Javascript 模組 ? Release Javascript module ?
57. 商業化 Commercialize
58. Example 2 Open Source Weather Data Parser
59. http://www.twitter.com/weatherhk
60. WeatherHK • Reports – Hourly Current Weather Information – Weather Forecast • Warning – Tropical Signal issuing, updates, pre-warning.
61. WeatherHK • Interview by MetroPop in 2009.
62. http://sammy.hk/projects/tctrack/tctrack.php
63. TCTrack • Plot paths of any active Tropical Cyclones – On Google Map • Probably the 1st Tropical Cyclones map in Hong Kong using Google Map API. – sammy.hk → Weather Undergroud Hong Kong → HKO – Current Paths. – Forecast Paths. – Tweets JTWC updates to Twitter. – Data Sources: • JTWC • HKO
64. Open Data
66. Five Star Open Data 1.make your stuff available on the Web (whatever format) under an open license. 2.make it available as structured data (e.g., Excel instead of image scan of a table) 3.use non-proprietary formats (e.g., CSV instead of Excel) 4.use URIs to denote things, so that people can point at your stuff. 5.link your data to other data to provide context. 5stardata.info by Tim Berners-Lee, the inventor of the Web.
67. Web Scraping a computer software technique of extracting information from websites. (Wikipedia)
68. Web Scraping with Scrapy
69. Scrapy • Python web scraping framework. • Scrap websites and extract structured data. • From data mining to monitoring and automated testing.
70. Scrapy Features • Define your own data structures. • Write spiders to extract data. • Built-in XPath selectors to extracting data. • Built-in JSON, CSV, XML output. • Interactive shell console, telnet console, logging...... • Scrapyd – a daemon providing Web API and simple Web UI. – http://localhost:6800
71. Scrapy Installation $ apt-get install python python-virtualenv python-pip $ virtualenv env $ source env/bin/activate $ pip install scrapy
72. Creating a Scrapy Project • Creating a new scrapy project. – $ scrapy startproject <project name> • Define your data structure • Write your first spider – Test with scrapy shell console • Output / Store collected data – Output with built-in supported formats – Store to database / object store.
73. Scrapy + Django • Django – Python web framework • Installing Django via pip – $ pip install django • Scrap with Scrapy and store data with Django
74. Creating django project • Creating a Django project. – $ django-admin.py startproject <project name> • Define django settings. – <project name>/settings.py – Create database, tables and first django user. • $ python manage.py syncdb – Define installed django apps with INSTALLED_APPS array.
75. Creating a Django App • Create your own django app. – $ python manage.py startapp <new app name> • Define data model at models.py – $ python manage.py syncdb • Activate django admin UI at admin.py – Add URL router to access admin UI.
76. Define django data model class WeatherData(models.Model): reporttime = models.DateTimeField() station = models.CharField(max_length=3) temperture = models.FloatField(null=True, blank=True) humidity = models.IntegerField(null=True, blank=True)
77. Enable django admin ui • Adding to INSTALLED_APPS at settings.py – django.contrib.admin • Adding URL router at urls.py – $ python manage.py runserver • Access admin UI – http://127.0.0.1:8000/admin
78. Scrapy + Django • Define django environment at scrapy settings. – Load django configuration. • Use Scrapy DjangoItem class – Insteads of Item and Field class – Define which django data model should be linked with. • Query and insert data at scrapy pipelines.
79. hk0weather
80. hk0weather • Weather Data Project. – https://github.com/sammyfung/hk0weather – convert weather information to JSON data from HKO webpages. – python + scrapy + django
81. hk0weather • Hong Kong Weather Data. – 20+ HKO weather stations in Hong Kong. – Regional weather data. – Rainfall data. – Weather forecast report.
82. Hk0weather installation • Setup and activate a python virtual enviornment, and install scrapy and django with pip. • Clone hk0weather from GitHub – $ git clone https://github.com/sammyfung/hk0weather.git • Setup database connection at Django and create database, tables and first django user. • Scrap regional weather data – $ scrapy crawl regionalwx -t json -o regional.json
84. How can you use & participate open source software ?
85. 社群 Community
87. Hong Kong Linux User Group • http://www.linux.org.hk/ • Re-established in 1997. • Linux Talks, Seminiars and Workshops. • Linux Cafe meetups. • Linux Exhibitions and Demostrations.
89. Open Source Hong Kong • http://www.opensource.hk • Established in 2008. • Events: Workshops, Hackathons, Conferences.
90. Monthly Events - Open Source Workshops
91. OpenSource.HK Hackathons
92. Hong Kong Open Source Conferences
93. Hong Kong Creative Open Technology Association • http://www.cota.hk • Established in 2014. • HK registered limited company in guarantee. • Non-profit organization. (in final process) • Promoting Open Standards, Free and Open Source Software, Open Hardware, Free Culture Works, Open Content and Creative Commons. • Work on projects and events. – eg. Hong Kong Open Source Conference 2014.
94. 自由 FREEDOM
95. “ Free software means software that respects users' freedom and community. ” http://www.gnu.org/philosophy/free-sw.html
96. 將來市場領導者必會是 使用及貢獻開放源碼的 企業 Future market leader must be the one which use and contribute open source.
97. Welcome to invite me to...... • Meetings, Talks, Workshops and Events – at your company and schools. • Projects – Development – Consultation • Let's get a name card.
98. Thank You! sammy@sammy.hk