Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Maintaining Fortran in Python in Perpetuity

Maintaining Fortran in Python in Perpetuity

Presented at SciPy 2022.

Details here: https://rgoswami.me/posts/scipycon-2022-meta

This talk covers recent advances to the f2py project, including preliminary derived type support, better documentation, and restructured design to enhance new contributor experiences. Additionally, we will discuss the structure and role of interoperability libraries and how modern Fortran intends to harmoniously and symbiotically evolve within the Python ecosystem in the context of Fortran-Lang, the np.distutils deprecation, newer compute devices and the ever changing landscape of high performance programming.

Rohit Goswami

November 15, 2022
Tweet

More Decks by Rohit Goswami

Other Decks in Programming

Transcript

  1. MAINTAINING FORTRAN IN PYTHON IN PERPETUITY ROHIT GOSWAMI .and. MELISSA

    MENDONCA .and. RALF GOMMERS .and. THIRUMALAI SHAKTIVEL .and. PEARU PETERSON
  2. HELLO! Find me here: Who? Rohit Goswami MInstP Doctoral Researcher,

    University of Iceland, Faculty of Physical Sciences Software Engineer, Quansight Labs, Austin TX https://rgoswami.me 4
  3. LOGISTICS All contents are Slides are in presentations/scipy22/F2PYmaint hosted on

    GitHub Questions are welcome at the end, or interrupt me 5
  4. MOTIVATION –> Through the magic of automated coding and standards

    “If a program or package (the words are used interchangeably) is to have a long life and to be of wide application in its field, it is essential for it to be easily moved from one machine to another. It used to be common to dismiss such movement with the statement, ‘There is no such thing as a machine- independent program.’ Nonetheless, a great many packages do now move from one machine to another”lyonUsingAnsFortran1980 8
  5. LANGUAGE STANDARDS “The standard is the contract between the compiler

    writer and the application developer.”clermanModernFortranStyle2012 9
  6. CHANGING STANDARDS character(10) BLAH*8 character*8 :: BLAH_ONE(10) character(8) :: BLAH_ONE(10)

    #!/usr/bin/env python print("Hello World") print "Hello World" 10
  7. FORTRAN, C, PYTHON F2003 Introduced the ISO_C_BINDING F2008 C_PTR for

    void * and more F2018 Brought interop for exotic Fortran features which via C descriptors Interop Described in great detail on : fortran90.org 12
  8. HISTORY Developed by Pearu Peterson July 9, 1999 f2py.py –>

    Fortran to Python Interface Generator (FPIG) January 22, 2000 f2py2e –> Fortran to Python Interface Generator, 2nd edition. July 19, 2007 numpy.f2py –> f2py2e moved to NumPy project. This is current stable code of f2py. Used extensively for F77 , :) petersonF2PYToolConnecting2009 NumPy waltNumPyArrayStructure2011 SciPy virtanenSciPyFundamentalAlgorithms2020 MsSpec sebilleauMsSpec1MultipleScattering2011 15
  9. DESIGN A best effort wrapper Specifications via .pyf or inline

    comments Not a compiler Can rewrite code :) 16
  10. FIBONACCI C FILE: FIB1.F SUBROUTINE FIB(A,N) C CALCULATE FIRST N

    FIBONACCI NUMBERS INTEGER N REAL*8 A(N) DO I=1,N IF (I.EQ.1) THEN A(I) = 0.0D0 ELSEIF (I.EQ.2) THEN A(I) = 1.0D0 ELSE A(I) = A(I-1) + A(I-2) ENDIF ENDDO END C END FILE FIB1.F f2py -m fib -c fib1.f python -c "import fib; import numpy as np; a=np.zeros(7); fib.fib(a); print(a); exit();" 19
  11. UP THE MAGICIAN’S SLEEVE Generated files mkdir blah f2py -m

    fib -c fib1.f --build-dir blah tree blah blah ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── blah │ │ └── src.macosx-10.9-x86_64-3.9 │ │ ├── fortranobject.o │ │ └── fortranobject.o.d │ ├── fibmodule.o │ └── fibmodule.o.d ├── fib1.o └── src.macosx-10.9-x86_64-3.9 ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── fortranobject.c │ └── fortranobject.h └── fibmodule.c 7 directories, 8 files 20
  12. COMPLEXITY NumPy Distutils Which can then be built simply with:

    Not fun for non python projects wc -l fortranobject.c fortranobject.h fibmodule.c 1107 fortranobject.c 132 fortranobject.h 372 fibmodule.c 1611 total from numpy.distutils.core import Extension, setup fibby = Extension(name = 'fib', sources = ['fib1.f']) if __name__ == "__main__": setup(name = 'fib', ext_modules = [ fibby ]) python setup.py build ag -g .so # build/lib.macosx-10.9-x86_64-3.9/fib.cpython-39-darwin.so 21
  13. MESON AND f2py project('test_builds', 'c', version : '0.1') add_languages('fortran') py_mod

    = import('python') py3 = py_mod.find_installation() py3_dep = py3.dependency() incnp = run_command(py3, ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'], check : true ).stdout().strip() inc_np = include_directories(incnp) py3.extension_module('fib1', 'fib1.f', 'fib1module.c', 'fortranobject.c', include_directories: inc_np dependencies : py3_dep, install : true) 24
  14. COMMAND LINE INTERFACE The F2PY UX is rather distinctive -c

    flag acts as a compiler / build-tool Otherwise works to generate C wrappers and Signature files Rewrite ongoing [GSoC] Supports Namami Shankar argparse based design 25
  15. DERIVED TYPES Soon to be a NEP : Tested by

    cmocka e.g. bind(c) types https://github.com/HaoZeke/f2py_derived_nep map to classes meson setup bbdir meson compile -C bbdir python -c "import bbdir.pycart as pycart; aak = pycart.pycart(1,10,2); print(aak); aak.unitstep(); print(aak)" pycart(x: 1.000000, y: 10.000000, z: 2.000000) Modifying derived type pycart(x: 2.000000, y: 11.000000, z: 3.000000) 26
  16. MOVING BEYOND bind(c) Generate Fortran wrappers to manipulate types All

    allocations handled in Fortran Python only calls wrappers to aforementioned types 27
  17. LOGICAL RESTRUCTURE Current flat hierarchy –> intimidating Don’t need much

    Fortran for front-end work codegen Creates wrappers csrcs fortranobject.{c,h} for builds frontend Parser, type inference stds Includes f77, f90, and pyf specs utils For testing and more tests For sanity (no term) WIP : https://github.com/numpy/numpy/pull/20481 28
  18. CONCEPTS Fortran 202X standard 668 pages Python-C API description 320

    pages NumPy-C API description 103 pages C++ 2020 (N4849) standard 1815 pages C11 (N1570) standard 701 pages 31
  19. TEST ISOLATION Write a test for each bug Typically already

    in the issue! class TestNegativeBounds(util.F2PyTest): # Check that negative bounds work correctly sources = [util.getpath("tests", "src", "negative_bounds", "issue_20853.f90")] @pytest.mark.slow def test_negbound(self): xvec = np.arange(12) xlow = -6 xhigh = 4 # Calculate the upper bound, # Keeping the 1 index in mind def ubound(xl, xh): return xh - xl + 1 rval = self.module.foo(is_=xlow, ie_=xhigh, arr=xvec[:ubound(xlow, xhigh)]) expval = np.arange(11, dtype = np.float32) assert np.allclose(rval, expval) 32
  20. FRONT-END Covers (parser, code-gen) pdb and editable installs are good

    choices Tough otherwise (global variables) micromamba create -f environment.yml micromamba activate numpy-dev pip install -e . # EDITABLE MODE # Add a breakpoint anywhere breakpoint() # Profit f2py -m blah buggy.f90 e.g. –> Negative bounds https://github.com/numpy/numpy/pull/21256/files 33
  21. BACK-END Covers (C wrappers, code-gen) gdb works well Sometimes with

    CPython extensions (see the sprint) Manipulate the generated C code Then move back to code-gen in Python e.g. –> Respect value https://github.com/numpy/numpy/pull/21807/files 34
  22. SUPPORTING value ATTRIBUTES Fortran 2003 supports call-by-value f2py wrappers pass

    by reference Details Very well reported issue here 35
  23. VERIFY BUG # Make wrappers f2py -m foo blah.f90 meson

    bbdir meson compile -C bbdir cd bbdir python -c "import foo; print( foo.fortfuncs.square(3) );" 170676880 static PyObject *f2py_rout_foo_fortfun const PyObject *capi_self, PyObject *capi_args, PyObject *capi_keywds, void (*f2py_func)(int*,int*)) { (*f2py_func)(&x,&y); // Passed by refe 36
  24. FIXUP C WRAPPER # Make wrappers meson compile -C bbdir

    cd bbdir python -c "import foo; print( foo.fortfuncs.square(3) );" 9 static PyObject *f2py_rout_foo_fortfun const PyObject *capi_self, PyObject *capi_args, PyObject *capi_keywds, void (*f2py_func)(int,int*)) { (*f2py_func)(x,&y); 37
  25. MODIFY PARSER value should be recognized by crackfortran Add a

    breakpoint and stare So it is already recognized.. name_match = re.compile(r'[A-Za-z][\w$]*').match breakpoint() for v in list(vars.keys()): >>> c # till fortfuncs is processed >>> pp vars {'x': {'attrspec': ['intent(in)', 'value'], 'typespec': 'integer'}, 'y': {'attrspec': ['intent(out)'], 'typespec': 'integer'}} 38
  26. AUGMENT AUX FUNCTIONS Handle the attribute within aux.py Also conditionally

    modify signatures + call conventions def isattr_value(var): return 'value' in var.get('attrspec', []) def getcallprotoargument(rout, cb_map={}): ... if not isattr_value(var): ctype = ctype + '*' 39
  27. GENERATE WRAPPERS This is similar to the handling of the

    existing intent(c) In rules.py 'callfortran': {l_or(isintent_c, isattr_value): '#varname#,', l_not(l_or(isintent_c, isattr_value)): '&#varname#,'}, 40
  28. ROAD-MAP Updating the test suite, using cmocka Rewriting the C

    wrappers for newer standards Build tool support [Namami Shankar] np.distutils is going the way of the dodo Implementing newer standards (90, 95, 2003, 2008, 2018, 2020Y) Automating guarantees Documentation and more interop with NumPy-C crackfortran works via dictionaries and strings.. Perhaps a more abstract semantic representation… Or concepts from G3 F2PY [Pearu] Handle parallelism (proposed) do concurrent and ufuncs 44
  29. WHY DOES FORTRAN STAY IN PYTHON ANYWAY? To write efficient

    wrappers without being a language lawyer 45
  30. HOW DOES FORTRAN STAY IN PYTHON ANYWAY? With all your

    help (Issues, users, developers, non-code contribs) 46
  31. ACKNOWLEDGMENTS as my supervisor, as my co-supervisor at GSI Tech.

    ( , and ) NumPy Team (Matti, Ross, Sebastian, Chuck, Inessa, etc.) Family, pets, Groupmembers, audience Prof. Hannes Jónsson Prof. Birgir Hrafnkelsson Dr. Ondřej Čertík Quansight Labs Dr. Ralf Gommers Dr. Melissa Weber Mendonça Dr. Pearu Peterson 49