f2py: Two Decades Later

f2py is the gold standard for interfacing Fortran and Python. The most famous downstream consumer of the generated wrappers is undoubtedly the scipy ecosystem. Crucially, f2py is not a compiler, and generates a best effort set of wrappers, aided by special comment lines. One of the key inter-operable approaches to compilation of the subsequent wrappers relies on numpy.distutils. Given the planned obsolescence of this module; we discuss the methology by which the build phase can be split into a two phase process and document better the pyf enhancements made to facilitate callbacks. A brief outline of test suites and documentation within the numpy ecosystem and proposed enhancement proposals will be discussed as well. We lay out the roadmap towards f2py remaining relevant beyond f77 by implementing derived types and newer features like parallelism with co-arrays. Playing to its strengths as a code-enhancer and leveraging the flexibility of not being constrained by the actual compilation process allows for the reimagining of f2py as a tool and library.

Site Post --> https://rgoswami.me/posts/fortrancon-2021-meta
FortranCon 2021 Link --> https://tcevents.chem.uzh.ch/event/14/contributions/82/

Transcript

1. f2py: TWO DECADES LATER ROHIT GOSWAMI .and. RALF GOMMERS .and. MELISSA MENDONCA .and. PEARU PETERSON Created: 2021-09-24 Fri 01:07
2. BRIEF INTRODUCTION 3
3. HELLO! Find me here: Who? Rohit Goswami MInstP Doctoral Researcher, University of Iceland, Faculty of Physical Sciences https://rgoswami.me 4
4. LOGISTICS All contents are Slides are in presentations/fortranCon2021/quansightF2PY hosted on GitHub Questions are welcome anytime 5
5. PROGRAMMING LANGUAGES 7
6. MOTIVATION –> Through the magic of automated coding and standards “If a program or package (the words are used interchangeably) is to have a long life and to be of wide application in its field, it is essential for it to be easily moved from one machine to another. It used to be common to dismiss such movement with the statement, ‘There is no such thing as a machine-independent program.’ Nonetheless, a great many packages do now move from one machine to another”[lyonUsingAnsFortran1980] 8
7. LANGUAGE STANDARDS “The standard is the contract between the compiler writer and the application developer.”[clermanModernFortranStyle2012] 9
8. CHANGING STANDARDS character(10) BLAH*8 character*8 :: BLAH_ONE(10) character(8) :: BLAH_ONE(10) #!/usr/bin/env python print("Hello World") print "Hello World" 10
9. F77 ∉ F90 ALWAYS
10. FORTRAN, C, PYTHON F2003 Introduced the ISO_C_BINDING F2008 C_PTR for void * and more F2018 Brought interop for exotic Fortran features which via C descriptors Interop Described in great detail on : fortran90.org 13
11. F2PY 15
12. HISTORY Developed by Pearu Peterson July 9, 1999 f2py.py –> Fortran to Python Interface Generator (FPIG) January 22, 2000 f2py2e –> Fortran to Python Interface Generator, 2nd edition. July 19, 2007 numpy.f2py –> f2py2e moved to NumPy project. This is current stable code of f2py. Used extensively for F77 , :) [petersonF2PYToolConnecting2009] NumPy [waltNumPyArrayStructure2011] SciPy [virtanenSciPyFundamentalAlgorithms2020] MsSpec [sebilleauMsSpec1MultipleScattering2011] 16
13. DESIGN A best effort wrapper Specifications via .pyf or inline comments Not a compiler Can rewrite code :) 17
14. EXPLORATIONS IN F77 19
15. FIBONACCI C FILE: FIB1.F SUBROUTINE FIB(A,N) C CALCULATE FIRST N FIBONACCI NUMBERS INTEGER N REAL*8 A(N) DO I=1,N IF (I.EQ.1) THEN A(I) = 0.0D0 ELSEIF (I.EQ.2) THEN A(I) = 1.0D0 ELSE A(I) = A(I-1) + A(I-2) ENDIF ENDDO END C END FILE FIB1.F f2py -m fib -c fib1.f python -c "import fib; import numpy as np; a=np.zeros(7); fib.fib(a); print(a); exit();" 20
16. UP THE MAGICIAN’S SLEEVE Generated files mkdir blah f2py -m fib -c fib1.f --build-dir blah tree blah blah ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── blah │ │ └── src.macosx-10.9-x86_64-3.9 │ │ ├── fortranobject.o │ │ └── fortranobject.o.d │ ├── fibmodule.o │ └── fibmodule.o.d ├── fib1.o └── src.macosx-10.9-x86_64-3.9 ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── fortranobject.c │ └── fortranobject.h └── fibmodule.c 7 directories, 8 files 21
17. COMPLEXITY NumPy Distutils Which can then be built simply with: Not fun for non python projects wc -l fortranobject.c fortranobject.h fibmodule.c 1107 fortranobject.c 132 fortranobject.h 372 fibmodule.c 1611 total from numpy.distutils.core import Extension, setup fibby = Extension(name = 'fib', sources = ['fib1.f']) if __name__ == "__main__": setup(name = 'fib', ext_modules = [ fibby ]) python setup.py build ag -g .so # build/lib.macosx-10.9-x86_64-3.9/fib.cpython-39-darwin.so 22
18. MODERNIZATION 24
19. MESON AND f2py project('test_builds', 'c', version : '0.1') add_languages('fortran') py_mod = import('python') py3 = py_mod.find_installation() py3_dep = py3.dependency() incnp = run_command(py3, ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'], check : true ).stdout().strip() inc_np = include_directories(incnp) py3.extension_module('fib1', 'fib1.f', 'fib1module.c', 'fortranobject.c', include_directories: inc_np dependencies : py3_dep, install : true) 25
20. FORTRAN-C-NUMPY 26
21. MODERN FORTRAN module fib1 use iso_c_binding implicit none contains subroutine fib(a,n) bind(c,name='c_fib') integer(c_int), intent(in), value :: n integer(c_int) :: i real(c_double) :: a(n) do i=1, n if (i==1) then a(i) = 0.0d0 else if (i==2) then a(i) = 1.0d0 else a(i) = a(i-1) + a(i-2) end if end do end subroutine end module fib1 27
22. NumPy-C I: BOILERPLATE #ifndef PY_SSIZE_T_CLEAN #define PY_SSIZE_T_CLEAN #endif /* PY_SSIZE_T_CLEAN */ #include "Python.h" #include "numpy/ndarrayobject.h" #include "numpy/ufuncobject.h" static PyMethodDef FibbyMethods[] = { {NULL, NULL, 0, NULL} }; // Declare void c_fib(double *a, int n); 28
23. NumPy-C II: FUNCTIONALITY static void double_fib(char **args, npy_intp *dimensions, npy_intp* steps, void* data) { int i; // Standard integer is fine here npy_intp n = dimensions[0]; char *in = args[0], *out = args[1]; npy_intp in_step = steps[0], out_step = steps[1]; double apointer[n]; for (i = 0; i < n; i++) { apointer[i]=(double)in[i]; } // Call the Fortran function c_fib(apointer, n); for (i = 0; i < n; i++) { /*BEGIN main ufunc computation*/ *((double *)out) = apointer[i]; /*END main ufunc computation*/ in += in_step; out += out_step; } } 29
24. NumPy-C III: MODULE DEFINITIONS /*This a pointer to the above function*/ PyUFuncGenericFunction funcs[1] = {&double_fib}; /* These are the input and return dtypes of fib.*/ static char types[2] = {NPY_DOUBLE, NPY_DOUBLE}; static void *data[1] = {NULL}; static struct PyModuleDef moduledef = { PyModuleDef_HEAD_INIT, "fibby", NULL, -1, FibbyMethods, NULL, NULL, NULL, NULL }; 30
25. NumPy-C IV: MODULE INITIALIZATION PyMODINIT_FUNC PyInit_fibby(void) { PyObject *m, *fib, *d; m = PyModule_Create(&moduledef); if (!m) { return NULL; } import_array(); import_umath(); fib = PyUFunc_FromFuncAndData(funcs, data, types, 1, 1, 1, PyUFunc_None, "fib", "Calls fib.f90", 0); d = PyModule_GetDict(m); PyDict_SetItemString(d, "fib", fib); Py_DECREF(fib); return m; } 31
26. COMPILATION py3.extension_module('fibby', 'fib1.f90', 'fibbyhand.c', include_directories:incnp, dependencies : py3_dep ) meson setup bdircythonhand meson compile -C bdircythonhand cd bdircythonhand import fibby import numpy as np a=np.empty(7) b=fibby.fib(a) print(b) exit() 32
27. POINTLESS BENCHMARK Table 1: The results are for 25 runs with 5 warmup for np.empty(7) as the input Command Mean [ms] Min [ms] Max [ms] Handwritten NumPy-C- Fortran 126.0 ± 3.9 119.8 136.8 F2PY (F77) 129.1 ± 4.0 125.1 140.4 Cython 129.5 ± 6.8 121.4 149.1 F2PY (F90) 129.9 ± 5.1 123.9 145.8 ctypes 128.3 ± 7.8 122.7 159.8 33
28. CONCLUSIONS 35
29. ROADMAP Updating the test suite Rewriting the C wrappers for newer standards Build tool support np.distutils is going the way of the dodo Implementing newer standards (90, 95, 2003, 2008, 2018, 2020Y) Automating guarantees Documentation and interop with NumPy-C crackfortran works via dictionaries and strings.. Perhaps a more abstract semantic representation… 36
30. RELEVANCE Writing efficient wrappers without being a language lawyer 37
31. THE END 39
32. ACKNOWLEDGEMENTS as my supervisor, as my co-supervisor, and my committee member at Los Alamos National Laboratory ( , and ) Family, pets, Groupmembers, audience Prof. Hannes Jónsson Prof. Birgir Hrafnkelsson Dr. Elvar Jonsson Dr. Ondřej Čertík Quansight Labs Dr. Ralf Gommers Dr. Melissa Weber Mendonça Dr. Pearu Peterson 40
33. BIBLIOGRAPHY Clerman & Spector, Modern Fortran: Style and Usage, Cambridge University Press . Lyon, Using Ans Fortran, National Bureau of Standards . Peterson, F2PY: A Tool for Connecting Fortran and Python Programs, International Journal of Computational Science and Engineering, 4(4), 296 . . . Sébilleau, Natoli, Gavaza, Zhao, Da Pieve & Hatada, MsSpec-1.0: A Multiple Scattering Package for Electron Spectroscopies in Material Science, Computer Physics Communications, 182(12), 2567-2579 . . . Virtanen, Gommers, Oliphant, Haberland, Reddy, Cournapeau, Burovski, Peterson, Weckesser, Bright, van der Walt, Brett, Wilson, Millman, Mayorov, Nelson, Jones, Kern, Larson, Carey, Polat, Feng, Moore, VanderPlas, Laxalde, Perktold, Cimrman, Henriksen, Quintero, Harris, Archibald, Ribeiro, Pedregosa & van Mulbregt, SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods, 17(3), 261-272 . . . van der Walt, Colbert & Varoquaux, The NumPy Array: A Structure for Efficient Numerical Computation, Computing in Science Engineering, 13(2), 22-30 . . [clermanModernFortranStyle2012] [lyonUsingAnsFortran1980] [petersonF2PYToolConnecting2009] link doi [sebilleauMsSpec1MultipleScattering2011] link doi [virtanenSciPyFundamentalAlgorithms2020] link doi [waltNumPyArrayStructure2011] doi 41
34. THANKS! 42