Using OBS for automated & dependable builds of Windows binaries & creation of installers

Transcript

1. Using OBS for automated & dependable builds of Windows binaries & creation of installers Fridrich Štrba Software Engineer & TrainedMonkey™ LibreOffice Team
2. © Novell, Inc. All rights reserved. 2 Who Am I? • Software Engineer in LibreOffice Team • Diverse background • FLOSS enthusiast • Enjoying Win32 porting of FLOSS applications • Working in free time on various projects
3. © Novell, Inc. All rights reserved. 3 Agenda • Differences Between Windows and Linux and Impact on Porting of FLOSS to Windows • Description of Our Windows Cross-Compiling Projects and How To Use Them • Future Directions
4. Differences Between Windows and Linux and Impact on Porting of FLOSS to Windows
5. © Novell, Inc. All rights reserved. 5 Linux • Shared libraries • Package management – Different package management tools (rpm, dpkg,...) • Dependency tracking – RPM tool keeps database of packages and their dependencies • Automatic upgrading possible – In openSUSE / SLE – satsolver tools a powerful dependency resolution – zypper dup
6. © Novell, Inc. All rights reserved. 6 Windows • Very little core shared (C runtime and Win32 API) • No centralized package management • Application in charge of installing all dependencies • Application installer in charge of upgrading
7. © Novell, Inc. All rights reserved. 7 Porting of FLOSS to Windows • Linux way of doing – Majority of FLOSS originally developed for Linux • Where to find the dependencies ? – Tor Lillqvist's web-page of binaries – GnuWin32 packages with their secret sauce • Toolchain for the build – http://www.mingw.org • A lot of insider information needed just to be able to build – Different tutorials focus on the setting of the build environment and location of the dependencies
8. Our approach
9. © Novell, Inc. All rights reserved. 9 Build from sources the Linux way • Why ? – Maintainability – Possibility to provide customer support – Clean stack of libraries • OpenSUSE Build Service – We have the best tool !!! > Build Service makes the building for us > All dependent libraries rebuilt and clean stack available at every point of time
10. © Novell, Inc. All rights reserved. 10 Elements of Our Solution • Cross-compiling framework • Infrastructure package • Toolchain • Porting strategy • Installer creation
11. © Novell, Inc. All rights reserved. 11 Cross-compiling framework • Long tradition of cross-compiling for Windows – Availability of the tools – Speed of the tools – OBS has cross-mingw packages since long time – Still rather manual work • Different existing cross-compiling frameworks – http://mingw-cross.sourceforge.net – http://fedoraproject.org/wiki/SIGs/MinGW – Inspired by the Fedora framework > “There should be no reinvented wheels; no wheel should be reinvented; the number of reinvented wheels should be equal to zero.”
12. © Novell, Inc. All rights reserved. 12 Infrastructure package mingw32-filesystem • Use of cross-binutils – Capacity to read information about the binaries • Macros for builds – Importance to be isolated from native environment • Scripts – Script for requires – Script for provides – Script for debuginfo extraction • Postinstall tweaks
13. © Novell, Inc. All rights reserved. 13 Example objdump information: The Import Tables (interpreted .idata section contents) vma: Hint Time Forward DLL First Table Stamp Chain Name Thunk 00016000 00016050 00000000 00000000 00016530 00016124 DLL Name: KERNEL32.dll vma: Hint/Ord Member­Name Bound­To 161f8 109 DeleteCriticalSection . . . . 163aa 803 VirtualQuery 00016014 000160ac 00000000 00000000 00016554 00016180 DLL Name: msvcrt.dll vma: Hint/Ord Member­Name Bound­To 163ba 5 _close 163c4 47 _lseek 163ce 54 _open 163d6 60 _read 163de 104 _write 00016028 000160c4 00000000 00000000 000165bc 00016198 DLL Name: msvcrt.dll vma: Hint/Ord Member­Name Bound­To 163e8 56 __dllonexit . . . . 164cc 760 vfprintf 0001603c 00000000 00000000 00000000 00000000 00000000
14. © Novell, Inc. All rights reserved. 14 mingw32-find-requires.sh: #!/bin/bash if [ "$1" ] then package_name="$1" fi [ ­z "$OBJDUMP" ] && OBJDUMP=i686­pc­mingw32­objdump # Get the list of files. filelist=`sed "s/['\"]/\\\&/g"` dlls_to_exclude="advapi32.dll [...] wsock32.dll" exclude_pattern="" for i in $dlls_to_exclude; do if test "$exclude_pattern" == ""; then exclude_pattern=$i; else exclude_pattern=$exclude_pattern"|"$i; fi done dlls=$(echo $filelist | tr [:blank:] '\n' | grep ­Ei '\.(dll|exe)$') for f in $dlls; do $OBJDUMP ­p $f | grep 'DLL Name' | tr [:upper:] [:lower:] | grep ­Eo '[­._\+[:alnum:]]+\.dll' | grep ­Ev "$exclude_pattern" | sed 's/\(.*\)/mingw32(\1)/' done | sort ­u
15. © Novell, Inc. All rights reserved. 15 mingw32-find-provides.sh: #!/bin/bash if [ "$1" ] then package_name="$1" fi [ ­z "$OBJDUMP" ] && OBJDUMP=i686­pc­mingw32­objdump filelist=`sed "s/['\"]/\\\&/g"` dlls=$(echo $filelist | tr [:blank:] '\n' | grep '\.dll$') for f in $dlls; do basename=`basename $f | tr [:upper:] [:lower:]` echo "mingw32($basename)" done
16. © Novell, Inc. All rights reserved. 16 mingw32-find-debuginfo.sh: #!/bin/sh if [ ­z "$1" ] ; then BUILDDIR="." else BUILDDIR=$1 fi for f in `find $RPM_BUILD_ROOT ­type f ­name "*.exe" ­or ­name "*.dll"` do case $(i686­pc­mingw32­objdump ­h $f 2>/dev/null | egrep ­o '(debug[\.a­z_]*|gnu.version)') in *debuglink*) continue ;; *debug*) ;; *gnu.version*) echo "WARNING: "`echo $f | sed ­e "s,^$RPM_BUILD_ROOT/*,/,"`" is already stripped!" continue ;; *) continue ;; esac echo extracting debug info from $f i686­pc­mingw32­objcopy ­­only­keep­debug $f $f.debug || : pushd `dirname $f` i686­pc­mingw32­objcopy ­­add­gnu­debuglink=`basename $f.debug` ­­strip­unneeded `basename $f` || : popd done find $RPM_BUILD_ROOT ­type f ­name "*.exe.debug" ­or ­name "*.dll.debug" | sed ­n ­e "s#^$RPM_BUILD_ROOT##p" > $BUILDDIR/debugfiles.list
17. © Novell, Inc. All rights reserved. 17 Toolchain • Self built – Cross-binutils – Cross-compiler – Win32 API and C runtime headers and import libraries • Win32 API and C runtime – First approach > http://www.mingw.org mingw-rt and w32api sources – Choice of http://mingw-w64.sourceforge.net » Better responsiveness of developers » Better API coverage » More complete COM+ headers
18. © Novell, Inc. All rights reserved. 18 Porting strategy • Difference with Fedora – Don't link to a particular openSUSE / SLE release • Work upstream – As little as possible non-upstreamed patches • No “secret sauce” – Windows is not POSIX
19. © Novell, Inc. All rights reserved. 19 Installer creation – First approach • MSI with Wix – http://wix.sourceforge.net – Framework written in C# > Some tools can be run using Mono – Windows only in fact > A critical tool P/Invokes MSI.DLL > Possible to use emulator (wine) – not as clean as one would like > Looked into implementing a replacement of msi.dll for Linux (gave up) > OBS home:hiberis project
20. © Novell, Inc. All rights reserved. 20 Installer creation – Second approach • Nullsoft Scriptable Installer System – http://nsis.sourceforge.net/Main_Page – Written in C and C++ – Command-line tools compilable and usable on Linux > Can be called during the RPM build process > NSI files > Installers wrapped in an RPM package > 7-zip is able to unpack RPM packages just fine » http://www.7-zip.org – Experimental Evolution installer » http://fridrich.blogspot.com/2010/05/experimental-evolution-installer-for.html
21. Future Directions
22. © Novell, Inc. All rights reserved. 22 So, what is this good for? • Old way of developing Evolution on Windows – A long list of dependencies and hackish tweaks > Zero attraction power for new developers » http://www.go-evolution.org/Building_Evolution_on_Windows » http://www.go-evolution.org/Windows • Our way – The development environment is easy to set » zypper si ­d mingw32­evolution­data­server » Using the tried and true tools present on Linux box – Debugging made easy » Installer of debugging symbols that contains a debugger » No need to compile yourself to debug a problem » Limitations because there is nothing like debugedit for windows binaries
23. © Novell, Inc. All rights reserved. 23 MS-DOS cross-compiler toolchain • Needed urgently for a product • Inspired by the Windows cross-compiling framework • Cloned in about a day – Reusing of the experience with Windows cross-compiling – Using DJGPP libraries > http://www.delorie.com/djgpp – Lives in openSUSE Build Service > home:fstrba:msdos project
24. © Novell, Inc. All rights reserved. 24 Status of the project • About 190 active packages in the repositories – 190 source packages targeting win32 – 186 source packages targeting win64 – An authoritative source for GTK+ port of Webkit on Windows • Work done in free time as a hobby • Enthusiastic community of package maintainers born – Taking over package by package according to their interest • Positive community feedback > “By far openSUSE Build Service contains most comprehensive collection of mingw software/libraries. No real competitors that I know of.” » http://old.nabble.com/Re%3A-win32---win64-Gtk%2B-apt-repo--p28979375.html
25. Q&A
27. Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.