f2py: TWO DECADES LATER ROHIT GOSWAMI .and. RALF GOMMERS .and. MELISSA MENDONCA .and. PEARU PETERSON Created: 2021-09-24 Fri 01:07

BRIEF INTRODUCTION 3

HELLO! Find me here: Who? Rohit Goswami MInstP Doctoral Researcher, University of Iceland, Faculty of Physical Sciences https://rgoswami.me 4

LOGISTICS All contents are Slides are in presentations/fortranCon2021/quansightF2PY hosted on GitHub Questions are welcome anytime 5

PROGRAMMING LANGUAGES 7

MOTIVATION –> Through the magic of automated coding and standards “If a program or package (the words are used interchangeably) is to have a long life and to be of wide application in its field, it is essential for it to be easily moved from one machine to another. It used to be common to dismiss such movement with the statement, ‘There is no such thing as a machine-independent program.’ Nonetheless, a great many packages do now move from one machine to another”[lyonUsingAnsFortran1980] 8

LANGUAGE STANDARDS “The standard is the contract between the compiler writer and the application developer.”[clermanModernFortranStyle2012] 9

CHANGING STANDARDS character(10) BLAH*8 character*8 :: BLAH_ONE(10) character(8) :: BLAH_ONE(10) #!/usr/bin/env python print("Hello World") print "Hello World" 10

F77 ∉ F90 ALWAYS

FORTRAN, C, PYTHON F2003 Introduced the ISO_C_BINDING F2008 C_PTR for void * and more F2018 Brought interop for exotic Fortran features which via C descriptors Interop Described in great detail on : fortran90.org 13

F2PY 15

HISTORY Developed by Pearu Peterson July 9, 1999 f2py.py –> Fortran to Python Interface Generator (FPIG) January 22, 2000 f2py2e –> Fortran to Python Interface Generator, 2nd edition. July 19, 2007 numpy.f2py –> f2py2e moved to NumPy project. This is current stable code of f2py. Used extensively for F77 , :) [petersonF2PYToolConnecting2009] NumPy [waltNumPyArrayStructure2011] SciPy [virtanenSciPyFundamentalAlgorithms2020] MsSpec [sebilleauMsSpec1MultipleScattering2011] 16

DESIGN A best effort wrapper Specifications via .pyf or inline comments Not a compiler Can rewrite code :) 17

EXPLORATIONS IN F77 19

FIBONACCI C FILE: FIB1.F SUBROUTINE FIB(A,N) C CALCULATE FIRST N FIBONACCI NUMBERS INTEGER N REAL*8 A(N) DO I=1,N IF (I.EQ.1) THEN A(I) = 0.0D0 ELSEIF (I.EQ.2) THEN A(I) = 1.0D0 ELSE A(I) = A(I-1) + A(I-2) ENDIF ENDDO END C END FILE FIB1.F f2py -m fib -c fib1.f python -c "import fib; import numpy as np; a=np.zeros(7); fib.fib(a); print(a); exit();" 20

UP THE MAGICIAN’S SLEEVE Generated files mkdir blah f2py -m fib -c fib1.f --build-dir blah tree blah blah ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── blah │ │ └── src.macosx-10.9-x86_64-3.9 │ │ ├── fortranobject.o │ │ └── fortranobject.o.d │ ├── fibmodule.o │ └── fibmodule.o.d ├── fib1.o └── src.macosx-10.9-x86_64-3.9 ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── fortranobject.c │ └── fortranobject.h └── fibmodule.c 7 directories, 8 files 21

COMPLEXITY NumPy Distutils Which can then be built simply with: Not fun for non python projects wc -l fortranobject.c fortranobject.h fibmodule.c 1107 fortranobject.c 132 fortranobject.h 372 fibmodule.c 1611 total from numpy.distutils.core import Extension, setup fibby = Extension(name = 'fib', sources = ['fib1.f']) if __name__ == "__main__": setup(name = 'fib', ext_modules = [ fibby ]) python setup.py build ag -g .so # build/lib.macosx-10.9-x86_64-3.9/fib.cpython-39-darwin.so 22

MODERNIZATION 24

MESON AND f2py project('test_builds', 'c', version : '0.1') add_languages('fortran') py_mod = import('python') py3 = py_mod.find_installation() py3_dep = py3.dependency() incnp = run_command(py3, ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'], check : true ).stdout().strip() inc_np = include_directories(incnp) py3.extension_module('fib1', 'fib1.f', 'fib1module.c', 'fortranobject.c', include_directories: inc_np dependencies : py3_dep, install : true) 25

FORTRAN-C-NUMPY 26

MODERN FORTRAN module fib1 use iso_c_binding implicit none contains subroutine fib(a,n) bind(c,name='c_fib') integer(c_int), intent(in), value :: n integer(c_int) :: i real(c_double) :: a(n) do i=1, n if (i==1) then a(i) = 0.0d0 else if (i==2) then a(i) = 1.0d0 else a(i) = a(i-1) + a(i-2) end if end do end subroutine end module fib1 27

NumPy-C I: BOILERPLATE #ifndef PY_SSIZE_T_CLEAN #define PY_SSIZE_T_CLEAN #endif /* PY_SSIZE_T_CLEAN */ #include "Python.h" #include "numpy/ndarrayobject.h" #include "numpy/ufuncobject.h" static PyMethodDef FibbyMethods[] = { {NULL, NULL, 0, NULL} }; // Declare void c_fib(double *a, int n); 28

NumPy-C II: FUNCTIONALITY static void double_fib(char **args, npy_intp *dimensions, npy_intp* steps, void* data) { int i; // Standard integer is fine here npy_intp n = dimensions[0]; char *in = args[0], *out = args[1]; npy_intp in_step = steps[0], out_step = steps[1]; double apointer[n]; for (i = 0; i < n; i++) { apointer[i]=(double)in[i]; } // Call the Fortran function c_fib(apointer, n); for (i = 0; i < n; i++) { /*BEGIN main ufunc computation*/ *((double *)out) = apointer[i]; /*END main ufunc computation*/ in += in_step; out += out_step; } } 29

NumPy-C III: MODULE DEFINITIONS /*This a pointer to the above function*/ PyUFuncGenericFunction funcs[1] = {&double_fib}; /* These are the input and return dtypes of fib.*/ static char types[2] = {NPY_DOUBLE, NPY_DOUBLE}; static void *data[1] = {NULL}; static struct PyModuleDef moduledef = { PyModuleDef_HEAD_INIT, "fibby", NULL, -1, FibbyMethods, NULL, NULL, NULL, NULL }; 30

NumPy-C IV: MODULE INITIALIZATION PyMODINIT_FUNC PyInit_fibby(void) { PyObject *m, *fib, *d; m = PyModule_Create(&moduledef); if (!m) { return NULL; } import_array(); import_umath(); fib = PyUFunc_FromFuncAndData(funcs, data, types, 1, 1, 1, PyUFunc_None, "fib", "Calls fib.f90", 0); d = PyModule_GetDict(m); PyDict_SetItemString(d, "fib", fib); Py_DECREF(fib); return m; } 31

COMPILATION py3.extension_module('fibby', 'fib1.f90', 'fibbyhand.c', include_directories:incnp, dependencies : py3_dep ) meson setup bdircythonhand meson compile -C bdircythonhand cd bdircythonhand import fibby import numpy as np a=np.empty(7) b=fibby.fib(a) print(b) exit() 32

POINTLESS BENCHMARK Table 1: The results are for 25 runs with 5 warmup for np.empty(7) as the input Command Mean [ms] Min [ms] Max [ms] Handwritten NumPy-C- Fortran 126.0 ± 3.9 119.8 136.8 F2PY (F77) 129.1 ± 4.0 125.1 140.4 Cython 129.5 ± 6.8 121.4 149.1 F2PY (F90) 129.9 ± 5.1 123.9 145.8 ctypes 128.3 ± 7.8 122.7 159.8 33

CONCLUSIONS 35

ROADMAP Updating the test suite Rewriting the C wrappers for newer standards Build tool support np.distutils is going the way of the dodo Implementing newer standards (90, 95, 2003, 2008, 2018, 2020Y) Automating guarantees Documentation and interop with NumPy-C crackfortran works via dictionaries and strings.. Perhaps a more abstract semantic representation… 36

RELEVANCE Writing efficient wrappers without being a language lawyer 37

THE END 39

ACKNOWLEDGEMENTS as my supervisor, as my co-supervisor, and my committee member at Los Alamos National Laboratory ( , and ) Family, pets, Groupmembers, audience Prof. Hannes Jónsson Prof. Birgir Hrafnkelsson Dr. Elvar Jonsson Dr. Ondřej Čertík Quansight Labs Dr. Ralf Gommers Dr. Melissa Weber Mendonça Dr. Pearu Peterson 40

BIBLIOGRAPHY Clerman & Spector, Modern Fortran: Style and Usage, Cambridge University Press . Lyon, Using Ans Fortran, National Bureau of Standards . Peterson, F2PY: A Tool for Connecting Fortran and Python Programs, International Journal of Computational Science and Engineering, 4(4), 296 . . . Sébilleau, Natoli, Gavaza, Zhao, Da Pieve & Hatada, MsSpec-1.0: A Multiple Scattering Package for Electron Spectroscopies in Material Science, Computer Physics Communications, 182(12), 2567-2579 . . . Virtanen, Gommers, Oliphant, Haberland, Reddy, Cournapeau, Burovski, Peterson, Weckesser, Bright, van der Walt, Brett, Wilson, Millman, Mayorov, Nelson, Jones, Kern, Larson, Carey, Polat, Feng, Moore, VanderPlas, Laxalde, Perktold, Cimrman, Henriksen, Quintero, Harris, Archibald, Ribeiro, Pedregosa & van Mulbregt, SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods, 17(3), 261-272 . . . van der Walt, Colbert & Varoquaux, The NumPy Array: A Structure for Efficient Numerical Computation, Computing in Science Engineering, 13(2), 22-30 . . [clermanModernFortranStyle2012] [lyonUsingAnsFortran1980] [petersonF2PYToolConnecting2009] link doi [sebilleauMsSpec1MultipleScattering2011] link doi [virtanenSciPyFundamentalAlgorithms2020] link doi [waltNumPyArrayStructure2011] doi 41

THANKS! 42