How to Bootstrap a Python Project So we can have the least problems in production

About me kahnwong Karnsiree Wong karnwong.me Platform Engineer @Data Cafe Thailand Often known as DevSecMLFinDataOps Enable developers to reach a flow state

Project Goals Release features Reach production stage What Actually Happened It works on my machine! But it doesn’t work on other people’s machines And it definitely does not work on cloud deployment Team members are fighting about which lib version is the correct one Also python version (there’s so many!)

Python Setup Python version Use pyenv ❯ pyenv versions system 3.10.13 3.11.2 3.11.3 * 3.11.8 (set by /Users/kahnwong/.pyenv/version) miniforge3-22.9.0-0 miniforge3-23.3.1-1

Python Setup Dependency management Use poetry [tool.poetry] name = "python_api_template" version = "0.1.0" description = "" authors = ["Karn Wong "] readme = "README.md" packages = [ {include = "python_api_template"} ] [tool.poetry.dependencies] python = "^3.11" fastapi = "^0.109.2" uvicorn = "^0.27.0.post1" pydantic = "^2.5.2"

Define Data Models Always be explicit on Request and Response object, or other data objects your project use ./model/request.py ./model/response.py ./main.py from pydantic import BaseModel class RequestItem(BaseModel): id: str message: str from datetime import datetime from pydantic import BaseModel class ResponseItem(BaseModel): time: datetime message: str from datetime import datetime from fastapi import FastAPI from python_api_template.model.request import RequestItem from python_api_template.model.response import ResponseItem from python_api_template.utils.log import logger app = FastAPI(title="python_api_template") ####################### # routes ####################### @app.get("/version") async def root(): return {"version": "0.1.0"} @app.post("/", response_model=ResponseItem) async def main(request: RequestItem) -> ResponseItem:

Logging stdout is not the same as logging… stdout No log level is specified!!! print(f"Input: {request}") Input: id='string' message='string' INFO: 127.0.0.1:55244 - "POST / HTTP/1.1" 200 OK

Logging logger Can trace filename & function logging.Formatter( "%(asctime)s - [%(levelname)s] - %(name)s - %(funcName)s - %(message)s" ) logger.info(f"Input: {request}") 2024-05-05 15:58:22,041 - [INFO] - python_api_template.main - main - Input: id='string' message='string' INFO: 127.0.0.1:58501 - "POST / HTTP/1.1" 200 OK

pre-commit So you don’t have to format & validate your code manually :) - repo: https://github.com/psf/black rev: 23.12.1 hooks: - id: black - repo: https://github.com/charliermarsh/ruff-pre-commit rev: v0.1.8 hooks: - id: ruff args: ["--fix", "--ignore=E402,E501"] exclude: ^migrations/ - repo: https://github.com/asottile/reorder_python_imports rev: v3.12.0 hooks: - id: reorder-python-imports - repo: https://github.com/python-poetry/poetry rev: 1.7.1 hooks: - id: poetry-check - id: poetry-lock

Tests Because testing everything your project does manually means you have to do it every time you change the code 😱 tests/single.py from fastapi.testclient import TestClient from python_api_template.main import app client = TestClient(app) def test_single(): response = client.post("/", json={"id": "foo", "message": "hello world"}) assert response.status_code == 200 pytest tests/single.py ============================================= test session starts ============================================== platform darwin -- Python 3.11.8, pytest-8.0.0, pluggy-1.4.0 rootdir: /Users/kahnwong/Git/kahnwong/platform/python-api-template plugins: cov-4.1.0, xdist-3.5.0, anyio-4.2.0 collected 1 item

Makefile A.K.A. how to use this project start: uvicorn python_api_template.main:app --port 8080 --reload test: pytest tests/single.py test-full: pytest -n 8 tests/full.py codecov: pytest --cov=python_api_template tests/single.py

Dockerfile Gotta make it reproducible 🐳 Dockerfile compose.yaml FROM python:3.11-slim # app deps COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # app WORKDIR /opt/app COPY . . EXPOSE 8080 CMD uvicorn python_api_template.main:app --port 8080 --host 0.0.0.0 --- services: app: build: context: . ports: - 8080:8080

Conclusion Utilize pyenv and poetry to manage project dependencies Be explicit on data objects Use logger for more visibility into your code Utilize pre-commit to check and validate your code Add tests so you can iterate quickly Declare your project entrypoints in Makefile Use Dockerfile to test a deployment

Thank You! Slides Blog GitHub LinkedIn