Upgrade to Pro — share decks privately, control downloads, hide ads and more …

f2py: Two Decades Later

Rohit Goswami
September 24, 2021

f2py: Two Decades Later

f2py is the gold standard for interfacing Fortran and Python. The most famous downstream consumer of the generated wrappers is undoubtedly the scipy ecosystem. Crucially, f2py is not a compiler, and generates a best effort set of wrappers, aided by special comment lines. One of the key inter-operable approaches to compilation of the subsequent wrappers relies on numpy.distutils. Given the planned obsolescence of this module; we discuss the methology by which the build phase can be split into a two phase process and document better the pyf enhancements made to facilitate callbacks. A brief outline of test suites and documentation within the numpy ecosystem and proposed enhancement proposals will be discussed as well. We lay out the roadmap towards f2py remaining relevant beyond f77 by implementing derived types and newer features like parallelism with co-arrays. Playing to its strengths as a code-enhancer and leveraging the flexibility of not being constrained by the actual compilation process allows for the reimagining of f2py as a tool and library.

Site Post --> https://rgoswami.me/posts/fortrancon-2021-meta
FortranCon 2021 Link --> https://tcevents.chem.uzh.ch/event/14/contributions/82/

Rohit Goswami

September 24, 2021
Tweet

More Decks by Rohit Goswami

Other Decks in Programming

Transcript

  1. f2py: TWO DECADES LATER ROHIT GOSWAMI .and. RALF GOMMERS .and.

    MELISSA MENDONCA .and. PEARU PETERSON Created: 2021-09-24 Fri 01:07
  2. HELLO! Find me here: Who? Rohit Goswami MInstP Doctoral Researcher,

    University of Iceland, Faculty of Physical Sciences https://rgoswami.me 4
  3. MOTIVATION –> Through the magic of automated coding and standards

    “If a program or package (the words are used interchangeably) is to have a long life and to be of wide application in its field, it is essential for it to be easily moved from one machine to another. It used to be common to dismiss such movement with the statement, ‘There is no such thing as a machine-independent program.’ Nonetheless, a great many packages do now move from one machine to another”[lyonUsingAnsFortran1980] 8
  4. LANGUAGE STANDARDS “The standard is the contract between the compiler

    writer and the application developer.”[clermanModernFortranStyle2012] 9
  5. CHANGING STANDARDS character(10) BLAH*8 character*8 :: BLAH_ONE(10) character(8) :: BLAH_ONE(10)

    #!/usr/bin/env python print("Hello World") print "Hello World" 10
  6. FORTRAN, C, PYTHON F2003 Introduced the ISO_C_BINDING F2008 C_PTR for

    void * and more F2018 Brought interop for exotic Fortran features which via C descriptors Interop Described in great detail on : fortran90.org 13
  7. HISTORY Developed by Pearu Peterson July 9, 1999 f2py.py –>

    Fortran to Python Interface Generator (FPIG) January 22, 2000 f2py2e –> Fortran to Python Interface Generator, 2nd edition. July 19, 2007 numpy.f2py –> f2py2e moved to NumPy project. This is current stable code of f2py. Used extensively for F77 , :) [petersonF2PYToolConnecting2009] NumPy [waltNumPyArrayStructure2011] SciPy [virtanenSciPyFundamentalAlgorithms2020] MsSpec [sebilleauMsSpec1MultipleScattering2011] 16
  8. DESIGN A best effort wrapper Specifications via .pyf or inline

    comments Not a compiler Can rewrite code :) 17
  9. FIBONACCI C FILE: FIB1.F SUBROUTINE FIB(A,N) C CALCULATE FIRST N

    FIBONACCI NUMBERS INTEGER N REAL*8 A(N) DO I=1,N IF (I.EQ.1) THEN A(I) = 0.0D0 ELSEIF (I.EQ.2) THEN A(I) = 1.0D0 ELSE A(I) = A(I-1) + A(I-2) ENDIF ENDDO END C END FILE FIB1.F f2py -m fib -c fib1.f python -c "import fib; import numpy as np; a=np.zeros(7); fib.fib(a); print(a); exit();" 20
  10. UP THE MAGICIAN’S SLEEVE Generated files mkdir blah f2py -m

    fib -c fib1.f --build-dir blah tree blah blah ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── blah │ │ └── src.macosx-10.9-x86_64-3.9 │ │ ├── fortranobject.o │ │ └── fortranobject.o.d │ ├── fibmodule.o │ └── fibmodule.o.d ├── fib1.o └── src.macosx-10.9-x86_64-3.9 ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── fortranobject.c │ └── fortranobject.h └── fibmodule.c 7 directories, 8 files 21
  11. COMPLEXITY NumPy Distutils Which can then be built simply with:

    Not fun for non python projects wc -l fortranobject.c fortranobject.h fibmodule.c 1107 fortranobject.c 132 fortranobject.h 372 fibmodule.c 1611 total from numpy.distutils.core import Extension, setup fibby = Extension(name = 'fib', sources = ['fib1.f']) if __name__ == "__main__": setup(name = 'fib', ext_modules = [ fibby ]) python setup.py build ag -g .so # build/lib.macosx-10.9-x86_64-3.9/fib.cpython-39-darwin.so 22
  12. MESON AND f2py project('test_builds', 'c', version : '0.1') add_languages('fortran') py_mod

    = import('python') py3 = py_mod.find_installation() py3_dep = py3.dependency() incnp = run_command(py3, ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'], check : true ).stdout().strip() inc_np = include_directories(incnp) py3.extension_module('fib1', 'fib1.f', 'fib1module.c', 'fortranobject.c', include_directories: inc_np dependencies : py3_dep, install : true) 25
  13. MODERN FORTRAN module fib1 use iso_c_binding implicit none contains subroutine

    fib(a,n) bind(c,name='c_fib') integer(c_int), intent(in), value :: n integer(c_int) :: i real(c_double) :: a(n) do i=1, n if (i==1) then a(i) = 0.0d0 else if (i==2) then a(i) = 1.0d0 else a(i) = a(i-1) + a(i-2) end if end do end subroutine end module fib1 27
  14. NumPy-C I: BOILERPLATE #ifndef PY_SSIZE_T_CLEAN #define PY_SSIZE_T_CLEAN #endif /* PY_SSIZE_T_CLEAN

    */ #include "Python.h" #include "numpy/ndarrayobject.h" #include "numpy/ufuncobject.h" static PyMethodDef FibbyMethods[] = { {NULL, NULL, 0, NULL} }; // Declare void c_fib(double *a, int n); 28
  15. NumPy-C II: FUNCTIONALITY static void double_fib(char **args, npy_intp *dimensions, npy_intp*

    steps, void* data) { int i; // Standard integer is fine here npy_intp n = dimensions[0]; char *in = args[0], *out = args[1]; npy_intp in_step = steps[0], out_step = steps[1]; double apointer[n]; for (i = 0; i < n; i++) { apointer[i]=(double)in[i]; } // Call the Fortran function c_fib(apointer, n); for (i = 0; i < n; i++) { /*BEGIN main ufunc computation*/ *((double *)out) = apointer[i]; /*END main ufunc computation*/ in += in_step; out += out_step; } } 29
  16. NumPy-C III: MODULE DEFINITIONS /*This a pointer to the above

    function*/ PyUFuncGenericFunction funcs[1] = {&double_fib}; /* These are the input and return dtypes of fib.*/ static char types[2] = {NPY_DOUBLE, NPY_DOUBLE}; static void *data[1] = {NULL}; static struct PyModuleDef moduledef = { PyModuleDef_HEAD_INIT, "fibby", NULL, -1, FibbyMethods, NULL, NULL, NULL, NULL }; 30
  17. NumPy-C IV: MODULE INITIALIZATION PyMODINIT_FUNC PyInit_fibby(void) { PyObject *m, *fib,

    *d; m = PyModule_Create(&moduledef); if (!m) { return NULL; } import_array(); import_umath(); fib = PyUFunc_FromFuncAndData(funcs, data, types, 1, 1, 1, PyUFunc_None, "fib", "Calls fib.f90", 0); d = PyModule_GetDict(m); PyDict_SetItemString(d, "fib", fib); Py_DECREF(fib); return m; } 31
  18. COMPILATION py3.extension_module('fibby', 'fib1.f90', 'fibbyhand.c', include_directories:incnp, dependencies : py3_dep ) meson

    setup bdircythonhand meson compile -C bdircythonhand cd bdircythonhand import fibby import numpy as np a=np.empty(7) b=fibby.fib(a) print(b) exit() 32
  19. POINTLESS BENCHMARK Table 1: The results are for 25 runs

    with 5 warmup for np.empty(7) as the input Command Mean [ms] Min [ms] Max [ms] Handwritten NumPy-C- Fortran 126.0 ± 3.9 119.8 136.8 F2PY (F77) 129.1 ± 4.0 125.1 140.4 Cython 129.5 ± 6.8 121.4 149.1 F2PY (F90) 129.9 ± 5.1 123.9 145.8 ctypes 128.3 ± 7.8 122.7 159.8 33
  20. ROADMAP Updating the test suite Rewriting the C wrappers for

    newer standards Build tool support np.distutils is going the way of the dodo Implementing newer standards (90, 95, 2003, 2008, 2018, 2020Y) Automating guarantees Documentation and interop with NumPy-C crackfortran works via dictionaries and strings.. Perhaps a more abstract semantic representation… 36
  21. ACKNOWLEDGEMENTS as my supervisor, as my co-supervisor, and my committee

    member at Los Alamos National Laboratory ( , and ) Family, pets, Groupmembers, audience Prof. Hannes Jónsson Prof. Birgir Hrafnkelsson Dr. Elvar Jonsson Dr. Ondřej Čertík Quansight Labs Dr. Ralf Gommers Dr. Melissa Weber Mendonça Dr. Pearu Peterson 40
  22. BIBLIOGRAPHY Clerman & Spector, Modern Fortran: Style and Usage, Cambridge

    University Press . Lyon, Using Ans Fortran, National Bureau of Standards . Peterson, F2PY: A Tool for Connecting Fortran and Python Programs, International Journal of Computational Science and Engineering, 4(4), 296 . . . Sébilleau, Natoli, Gavaza, Zhao, Da Pieve & Hatada, MsSpec-1.0: A Multiple Scattering Package for Electron Spectroscopies in Material Science, Computer Physics Communications, 182(12), 2567-2579 . . . Virtanen, Gommers, Oliphant, Haberland, Reddy, Cournapeau, Burovski, Peterson, Weckesser, Bright, van der Walt, Brett, Wilson, Millman, Mayorov, Nelson, Jones, Kern, Larson, Carey, Polat, Feng, Moore, VanderPlas, Laxalde, Perktold, Cimrman, Henriksen, Quintero, Harris, Archibald, Ribeiro, Pedregosa & van Mulbregt, SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods, 17(3), 261-272 . . . van der Walt, Colbert & Varoquaux, The NumPy Array: A Structure for Efficient Numerical Computation, Computing in Science Engineering, 13(2), 22-30 . . [clermanModernFortranStyle2012] [lyonUsingAnsFortran1980] [petersonF2PYToolConnecting2009] link doi [sebilleauMsSpec1MultipleScattering2011] link doi [virtanenSciPyFundamentalAlgorithms2020] link doi [waltNumPyArrayStructure2011] doi 41