• Software Engineer, BBC News Labs • Former Community Manager at Raspberry Pi • PyPI critical project maintainer • Based in Cambridgeshire • bennuttall.com • twitter.com/ben_nuttall • github.com/bennuttall Ben Nuttall
• Multi-disciplinary innovation team within BBC News & BBC R&D • Prototypes of new audience experiences • Solutions to help journalists • Research and trying out ideas • bbcnewslabs.co.uk • twitter.com/bbc_news_labs BBC News Labs
IDX (Identify the X) Automated clipping of content in live radio for social media mosromgr Processing TV/radio running orders to extract structured metadata BBC Images Image metadata enrichment pipeline Projects
• Identify stakeholders • Set up calls with journalists • Learn about existing systems and workflows • Get access to systems & data and get to know them • Set up shadowing Research week
• Sit with a journalist or producer • Watch them do their job using existing tools • Work out what their workflows are • Look for pain points, inefficiencies, slowness, manual work that could be automated Shadowing
• Run code without managing server infrastructure • Pay for compute time instead of provisioning for peak capacity • Python/JavaScript/Go/etc • Python 3.9 AWS Lambda
• Workflow design • Sequence of Lambdas • Lambdas can be implemented in different languages • Failures, retries, parallelisation Step functions / state machines
• Execute with initial data • Pass new data on • Parallel paths and decisional logic • Specify retry logic • Whole state machine succeeds or fails • Easy access to data, exception info and lambda logs Step functions / state machines
• Execute with initial data • Pass new data on • Parallel paths and decisional logic • Specify retry logic • Whole state machine succeeds or fails • Easy access to data, exception info and lambda logs Step functions / state machines
• Amazon Aurora PostgreSQL-Compatible Edition • DB instance class: serverless v1 • Specify capacity range and scaling configuration • Web service data API Serverless PostgreSQL
• EC2 web server hosting static files in S3 • Access via BBC Login or BBC certificate • Every project can re-use the infrastructure • Great for SPAs and static sites News Labs Apps Portal
• Devise a website content structure with a layout template • Create Chameleon templates for each page type • Create logic layer for retrieving data required for each page write • Create lambda for writing/rewriting relevant pages • e.g. new episode processed: • write new episode page ///index.htm • update brand index page //index.htm • update homepage /index.htm • Create CLI for manual rewrites Static HTML websites with Chameleon
• Structured logging • Looks great when running locally • easy to see relevant information • JSON logging support ideal for running in AWS • can access and search structured logs in CloudWatch • Encourages good logging practice! Structlog import structlog if os.environ.get('MOS_LOGGING') == 'JSON': processors = [ structlog.stdlib.add_log_level, structlog.processors.StackInfoRenderer(), structlog.processors.format_exc_info, structlog.processors.JSONRenderer(), ] structlog.configure(processors=processors)