Learn to SourceJuicer

Transcript

1. Learn to SourceJuicer Thomas Wagner Sun Microsystems GmbH Zettachring 10A 77567 Stuttgart Germany Thomas.Wagner@Sun.COM 1 What you get today Today most of the Operatins Systems are delivered in more or less fixed distributions stored on CDROM, DVD, preinstalled images or by a bootstrapping program and “the Net”. If you need more then a minimal install set, you'll most times rely on the manufacturers set of software addons and download them through “the Net”. This is also true for releases of OpenSolaris based on the IPS packaging system and except the software contained in the minimal install image, additional software for OpenSolaris is available through “the Net” only. The SourceJuicer Project now adds even more value to the package repositores on “the Net” by making it very easy to the net-citizen to contribute to the set of available free software addons. This tutorial learns you how to use the SourceJuicer online and to setup a development environment on your local machine. The format of the descriptive “spec files” in RPM-style format is reviewed in more detail and we learn how to setup a local build enviroment to develop “spec files”. Finally we learn how the process for feeding “spec files” into SourceJuicer workflow os done and how other repositories like SFE/spec-files-extra are used to get inspired and to make you a valuable contributor for “spec files”. 1.1 The story about “spec files” and Solaris Sun Microsystems had a product named “Java Desktop System” (JDS) which was essentially a stable release of GNOME Desktop combined with the company's Office Suite and an underlaying Linux or Solaris. The best thing was, you could buy JDS-Support from Sun. Many parts of the GNOME based software in JDS has been build with RPM-style spec files. This is still the case for today's Sun Solaris 10, Solaris Express Community Edition (SXCE) and OpenSolaris. In the past, the software contained in these big bundled products was well- kept (stable, static, minimize changes) and the number of additions grew slowly. Some day the team around the Sun Desktop started splitting theyr repository of spec files into a separate one, named spec-files-extra, to keep the official repository clean from other software not primarily for use in the official products. That time the journey begun. This spec-files-extra reposiory and the opened toolset pkgbuild/pkgtool started to become popular as the geeks around the world (you know them) always wanted more 'geeky' software (“I do mange to run the most complicated F/OSS on Solaris, I definitly do”) or they wanted other releases of programs of the regular distributons and replace them for whatever reason. The spec-files-extra repository got more and more contributors living in “the Net” and grew quickly, containing today over 1000 spec files to build all types of F/OSS software, even very pupular multimedia players. With this backround it started to make sense, to initiate a project like SourceJuicer and let the whole user base of OpenSolaris benefit from the good work of such nice individuals by adding a trusted repository named “/contrib”. SourceJuicer is adding more and more software to OpenSolaris by a growing number of volunteers. The pkgbuild/pkgtool Toolset which is core to SourceJuicer startet years ago as a SVR4-package-only version. The initial code for the IPS packaging enhancement again came from the community. This was long before SourceJuicer. But with IPS Support in pkgbuild/pkgtool, it became easy to build a “SourceJuicer” workflow and share the development of spec files with many people. With the approval process it got possible to make the resulting packages “trusted“, since trusted approvers keep a close look on what the build instructions are.
2. 2 Why you shouldn't do “./configure && make install” any more If you do that, you reinvent the wheel, at least sometimes. There is no automation, it is sometime difficult to repeat and get the same results, you get enlaced by patches, the community has no access to your valuable work and so on. With spec files the knowledge how to build a specific software on the current platform is perfectly documented. Everyone can repeat the same steps with great success. SourceJuicer rely upon repeatable, reliable instructions to build software and to paketize it. 3 Quick tour through the SourceJuicer workflow (Picture source: http://jucr.opensolaris.org/help/sjhowto#PathToContrib) To submit a spec file and supporting files (License, patches) you login to OpenSolaris.org and enter your submission onto a webform. An approver checks if the license is acceptable, propperly referenced and the spec file itself is well done. Once the automatic build process is error free, the package is put into the “/pending” repository and may be installed by the package manager from this repository. Any OpenSolaris user may add comments to your submission and if you ask for voting on your spec file and you get at least +2 and no -1 vote, the package is then promoted to the “/contrib” repository. Read for more details: http://jucr.opensolaris.org/home/ . 4 Behind the SourceJuicer scenes SourceJuicer supports the work flow for submitting, commenting, accepting, building, voting and promoting spec files to the “/pending” and “/contrib” repositories. Behind the scenes works a well-proven toolset. The core of this toolset is pkgbuild/pkgtool (http://pkgbuild.sf.net). This toolset is assisted by a collection of build tools named Common Build Environment (CBE). The CBE is probably going away some day since most parts of it are contained in the more recent releases of the regular OpenSolaris distributions. The spec files you submit are assisted by a predefined set of “include”-spec-files which are referenced by your spec files if needed (e.g. %include Solaris.inc) The Build Grid then shedules the setup of a clean build zone, copy necessary files into that zone, install messing dependencies like other pakcages of the OpenSolaris distribtion and then fires of the build process. The results are transferred to the logs on the SourceJuicer web.
3. 5 The SourceJuicer web How to submit a spec file is documented in depth on the SourceJuicer Web: http://jucr.opensolaris.org/help/sjhowto The following screenshots show the steps in the SourceJuicer workflow: 6 Submit the spec files Fresh spec files are initially submitted with one the “submit” section of the SourceJuicer web: (Picture source: http://jucr.opensolaris.org/images/submit.png) 7 Review the submission Once submitted, you find the spec file on the “review” section of the SourceJuicer web: (Picture source: screenshot of http://jucr.opensolaris.org) 8 The SourceJuicer build view You should email the jucr email list to have your submission validated / approved and if you get the approval, your package is sheduled for a build on the build grid. The state of your build is viewed on the “build” section of the SourceJuicer web:
4. (Picture source: screenshot of http://jucr.opensolaris.org) Note: Once build successfully, installing the packages is a single click with the mouse on the “install” link. There is a file format “p5i” defined, which fires up the package manager and provides every information needed, to install the package on your local machine. 9 Spec files rule the build The RPM-style is for describing build recipes is in use for many years. The format and syntax used for SourceJuicer is not 100% compatible but almost. Some features need extentions to the pkgbuild/pkgtool and volunteers and discussions on extending the tools are always welcome. The spec files describe what is to be done specific to a single software, the include-spec-files contained in the repositores add Solaris specific paths and predefine e.g. architectural differences and compiler flags. 10 How does a spec file look like A (modified) example from the spec-files-jucr repository: bvi.spec ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ %include Solaris.inc Name: bvi Summary: bvi ­ display­oriented editor for binary files (vi like) Version: 1.3.2 License: GPLV2 Source: http://nchc.dl.sourceforge.net/sourceforge/bvi/%{name}­% {version}.src.tar.gz URL: http://bvi.sourceforge.net/ Group: Text Tools Distribution: OpenSolaris Vendor: OpenSolaris Community %include default­depend.inc BuildRoot: %{_tmppath}/%{name}­%{version}­build SUNW_Basedir: %{_basedir}
5. SUNW_Copyright: %{name}.copyright # OpenSolaris IPS Manifest Fields Meta(info.upstream): http://bvi.sourceforge.net/ Meta(info.repository_url): anonymous@bvi.cvs.sourceforge.net:/cvsroot/bvi Meta(info.maintainer): First Last < name@domain.tld > %description The bvi is a display­oriented editor for binary files. %prep %setup ­q ­n %{name}­%{version} %build export LDFLAGS="%{_ldflags}" export CFLAGS="%{optflags}” export CXXFLAGS="%{cxx_optflags}" ./configure ­­prefix=%{_prefix} \ ­­mandir=%{_mandir} \ ­­disable­static make %install rm ­rf $RPM_BUILD_ROOT make install DESTDIR=$RPM_BUILD_ROOT %clean rm ­rf $RPM_BUILD_ROOT %files %defattr(­, root, bin) %doc CHANGES COPYING CREDITS README %dir %attr (0755, root, bin) /usr/local/bin /usr/local/bin/* %dir %attr (0755, root, bin) /usr/local/lib /usr/local/lib/* %dir %attr (0755, root, sys) %{_datadir} %dir %attr (0755, root, other) %{_docdir} %dir %attr(0755, root, bin) /usr/local/man %dir %attr(0755, root, bin) /usr/local/man/* /usr/local/man/*/*
6. %changelog * day month 01 2009 – first.last@domain.tld ­ say what you've done to it See more examples including machine architecture specific builds in the spec-files-extra repository. You may browse these spec files here: http://pkgbuild.svn.sourceforge.net/viewvc/pkgbuild/spec-files-extra/trunk/ (e.g. look for SFElibmcrypt.spec for a dual 32-/64-bit package) 11 I need help starting with my first spec file You should first read some spec files and look how those solve the common task. Then you might choose an anlready existing spec file which build a software similar to the one you want to build. For instance if you build a GNOME desktop addon, you'll like to choose a spec file which already exists for another desktop addon. Another approch is to compile a software manually and write down the steps used. Later on, you put these steps into a template spec file and start over debugging the build process. – choose use a similar software's spec file to start with and edit. In many cases you just have to adjust %files section – write the spec file from scratch by using an empty template and adding Soure, test the download and %prep, then add the %build section with ./configure and make, and so on until the %files section remains to be completed – you may build the package manually outside of the toolset and migrate the steps into a spec file at once (same iterations as above). 12 Build a package by spec files In case of SourceJuicer you have no extra step to build a package. Once it is approved, the build process starts automaticly for the “/pending” repository. If your package got the correct votes, it is automaticly build for the “/contrib” repository. Watch out the SourceJuicer build log for any errors. Even in case the build went through. If you use SourceJuicer web, then you may find the log file of your build and see what went wrong. If you need access to the “config.log” which is produced by a ./configure step, then you my use this Trick: ./configure ­ ­ options || (cat config.log) With this trick, you get the config.log content into the SourceJuicer build log on the web. If you have a local build and development environment set up, you may use this command: . /opt/dtbld/bin/env.sh pkgtool ­–download ­–interactive ­–autodeps build yourspecfile.spec If a step with pkgtool on your spec file fails, then you see the errors right on your terminal. If a %build step fails, then you will be presented with a shell. You may adjust your configuration directly and redo the configure or make step. The changes can be edited in the spec file shortly after the build process. If your want to stop the process sitting at the shell prompt, use “exit 1” or if your want to continue with the next instruction, use “exit 0”. You can read what pkgtool/pkgbuild prepared for compilation or installation, if you look into the file “.pkgbuild.build.sh” with the configure and make steps and “.pkgbuild.install.sh” with the make install steps. Lukcy you, all directories used to store sources, the build directory, the pacakging scripts all reside in your local paths, see the directory structure below ~/packages:
7. ~/packages ls ­l drwxr­xr­x 181 tom other 181 Sep 28 00:49 BUILD drwxr­xr­x 8 tom other 8 Feb 24 2009 PKGMAPS drwxr­xr­x 234 tom other 262 Sep 28 00:49 PKGS drwxr­xr­x 2 tom other 741 Sep 28 00:44 SOURCES drwxr­xr­x 2 tom other 40 Sep 24 23:20 SPECS drwxr­xr­x 139 tom other 139 Sep 28 00:49 SPKGS 13 How is a package build from the compiled binaries The packaging step is included in the last pkgbuild/pkgtool stage. So no need to worry about the syntax of the tools, we just have to have a complete “%files” section in the spec files. To have an idea what the packaging step is, find a quick look at the SVR4 and IPS packages below. SVR4 and IPS packages are best build with scripted use of the package tools. Don't do that manually. pkgtool/pkgbuild wraps all steps for SVR4 with “pkgproto”/”pkgmake” and for IPS with “pkg” and options. On “the Net” there are lots of step guides how to assemble packages, but in our case, we have spec files and the toolset to control the packaging commands. For the concept of RPM-style spec files, the SVR4 and IPS world share the same spec files. The main differenes are in case of SVR4 package that it may include scripting for install/removal of the packages and in case of an IPS package, you have no scripting and store the package on a networked repository. 14 The old SVR4 package prototype Example: prototype map for SFEbvi: ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ pkginfo=/localhomes/tom/packages/PKGMAPS/pkginfo/SFEbvi.pkginfo i depend=/localhomes/tom/packages/PKGMAPS/depend/SFEbvi.depend i copyright=/localhomes/tom/packages/PKGMAPS/copyright/SFEbvi.copyright f none share/doc/SFEbvi/CHANGES=usr/share/doc/SFEbvi/CHANGES 0444 root bin f none share/doc/SFEbvi/COPYING=usr/share/doc/SFEbvi/COPYING 0444 root bin f none share/doc/SFEbvi/CREDITS=usr/share/doc/SFEbvi/CREDITS 0444 root bin f none share/doc/SFEbvi/README=usr/share/doc/SFEbvi/README 0444 root bin d none bin 0755 root bin f none bin/bmore=usr/bin/bmore 0755 root bin f none bin/bvedit=usr/bin/bvedit 0755 root bin f none bin/bvi=usr/bin/bvi 0755 root bin f none bin/bview=usr/bin/bview 0755 root bin d none lib 0755 root bin f none lib/bmore.help=usr/lib/bmore.help 0644 root bin d none share 0755 root sys d none share/doc 0755 root other d none share/doc/SFEbvi 0755 root bin d none share/man 0755 root bin d none share/man/man1 0755 root bin f none share/man/man1/bmore.1=usr/share/man/man1/bmore.1 0644 root bin f none share/man/man1/bvi.1=usr/share/man/man1/bvi.1 0644 root bin
8. ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ The above is used to form the real SVR4 package by pkgmk. We skip the steps with “pkgmk”. 15 The new IPS packages assembly Example: manifest for bvi.spec (never do that manually - thankfully created by pkgtool/pkgbuild) ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ #depend type=require fmri=SUNWcsr dir mode=0755 owner=root group=bin path="usr/lib" file /localhomes/tom/packages/PKGS/bvi/reloc/bin/bvedit mode=0755 owner=root group=bin path="usr/bin/bvedit" dir mode=0755 owner=root group=bin path="/usr/share/doc/bvi" file /localhomes/tom/packages/PKGS/bvi/reloc/share/doc/bvi/COPYING mode=0444 owner=root group=bin path="usr/share/doc/bvi/COPYING" doc=true file /localhomes/tom/packages/PKGS/bvi/reloc/share/man/man1/bvi.1 mode=0644 owner=root group=bin path="usr/share/man/man1/bvi.1" file /localhomes/tom/packages/PKGS/bvi/reloc/bin/bview mode=0755 owner=root group=bin path="usr/bin/bview" file /localhomes/tom/packages/PKGS/bvi/reloc/lib/bmore.help mode=0644 owner=root group=bin path="usr/lib/bmore.help" file /localhomes/tom/packages/PKGS/bvi/reloc/share/man/man1/bmore.1 mode=0644 owner=root group=bin path="usr/share/man/man1/bmore.1" file /localhomes/tom/packages/PKGS/bvi/reloc/share/doc/bvi/CHANGES mode=0444 owner=root group=bin path="usr/share/doc/bvi/CHANGES" doc=true file /localhomes/tom/packages/PKGS/bvi/reloc/bin/bvi mode=0755 owner=root group=bin path="usr/ bin/bvi" file /localhomes/tom/packages/PKGS/bvi/reloc/share/doc/bvi/CREDITS mode=0444 owner=root group=bin path="usr/share/doc/bvi/CREDITS" doc=true dir mode=0755 owner=root group=sys path="usr/share" dir mode=0755 owner=root group=other path="usr/share/doc" dir mode=0755 owner=root group=bin path="usr/share/man/man1" file /localhomes/tom/packages/PKGS/bvi/reloc/bin/bmore mode=0755 owner=root group=bin path="usr/bin/bmore" dir mode=0755 owner=root group=bin path="usr/bin" dir mode=0755 owner=root group=bin path="usr/share/man" file /localhomes/tom/packages/PKGS/bvi/reloc/share/doc/bvi/README mode=0444 owner=root group=bin path="usr/share/doc/bvi/README" doc=true set name=description value="bvi ­ display­oriented editor for binary files (vi like)" set name=pkg.summary value="bvi ­ display­oriented editor for binary files (vi like)" set name=info.upstream_url value="http://bvi.sourceforge.net/" set name=info.maintainer_url value="first.last@domain.tld" license /localhomes/tom/packages/PKGS/bvi/install/copyright license=copyright ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ The above manifest is used together with the following shell script to feed the contents into the IPS package repository: ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ #/usr/bin/bash export PKG_REPO=http://localhost:80 eval `pkgsend open SFEbvi@1.3.2` pkgsend include /localhomes/tom/packages/PKGMAPS/manifests/SFEbvi.manifest pkgsend close ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­
9. 16 My own build environment If you want to setup your own build enviroment instead of using the SourceJuicer web or the Opensolaris testfarm, then there are several ways to get such a setup. You might use the instructions on the sw-porters web http://www.opensolaris.org/os/community/sw-porters/contributing/jucrzones/ or use a bootrapping script (tending towards SFE/spec-files-extra but usable with jucr also): Note: run the script inside a fresh zone and as root user – for the beginning) http://tinyurl.com/bootstrap-sfe-latest-os20nn There are a few other resources which describe duplicating the SourceJuicer enwironment (some more to come). 17 To check your dependencies with check-deps.pl The script check-deps.pl from the Sun Microsystems Xorg Team (Alan Coopersmith) goes through your packages and checks the references in your binaries e.g. linking to a ssl library and if that fact is referenced in your package definition as a “Requires: packagename”. If such a reference is missing, you might break automatic dependency resolution in the packaging system or at least end up with unusable software. Example SVR4: cd packages/PKGS; check­deps.pl ­c SFEbvi Example IPS: pfexec check­deps.pl ­I Read more: http://www.mail-archive.com/pkg-discuss@opensolaris.org/msg10811.html 18 Repositores for local use or for developing SourceJuicer spec files You may use these repositories for learing the spec files: 1. spec-files-jucr (read-only, check-in only via SourceJuicer workflow on the website). Checkot with svn co svn+ssh://anon@svn.opensolaris.org/svn/sourcejuicer/spec­ files­jucr/trunk spec­files­jucr 2. spec-files gnome on all (Open-)Solaris releases. See http://opensolaris.org/os/project/jds/documents/indiana_build/ 3. spec-files-extra aka SFE, see the pkgbuil wiki for help and config/usage tipps for the contained software. See http://pkgbuild.sf.net 4. spec-files-xfce from http://www.opensolaris.org/os/project/xfce/building_xfce/ … more to be added … 19 Use the Testfarm at http://test.opensolaris.org Sun provides unser specific usage terms access to a well equipped testfarm at http://test.opensolaris.org . If you want to work on spec files then book a zone four your tests. It's done in minutes: Log in with your OpenSolaris.org username and password. The Zones are availabe on X86 and Sparc machines and already contain a build environment similar to the one the SourceJuicer Build Grid uses. You may use the provided build enviroment as is or use the bootstrap-sfe-latest-os20nn script from above to add the spec-files-
10. extra repository and have added automaticly a number of commonly required OS packages for standalone builds of spec files from the spec-files-jucr and spec-files-extra repository. 20 Literature http://opensolaris.org/os/project/pkg/ OpenSolaris Project: Image Packaging System http://pkgbuild.sourceforge.net the home of the pkgbuild/pkgtool for build recipes http://wiki.pkgbuild.sourceforge.net the wiki for pkgbuild and the SFE / spec-files-extra repository http://tinyurl.com/bootstrap-sfe-latest-os20nn boostrap a build enviroment (best in a zone and run script as root) http://test.opensolaris.org the Testfarm with bookable zones http://jucr.opensolaris.org/home/ the home of SourceJuicer http://pkgbuild.sourceforge.net/spec-files-extra/ Description of the SFE repository (some specs are SXCE only)