Troubleshooting Node.js applications Debugging in SAP Cloud Platform Cloud Foundry Vadim Klimov SAP Inside Track Belgium May 2019 

Demo application 2 Houston, we have a problem! 

Ideas on diagnostics and troubleshooting 3 Logging library (Winston, Log4js, Bunyan, Pino, {logging library of your choice}) + HTTP logger (Morgan, {HTTP logger of your choice}) console.log() Simple, but may overfill source code with excessive number of logging statements More advanced, but may still overfill source code with excessive number of logging statements No There shall be a right tool for the job. Mindful logging – for continuous monitoring and diagnostics, verbose tracing on demand – for common troubleshooting, debugger – for advanced troubleshooting No Let’s go for a debugger! 

Node.js debugging Ahead of remote application debugging – debug in local environment, if possible 4 ▪ Do not kick off remote debugging straightaway – firstly, assess feasibility of running a Node.js application in a local environment in a representational way, reproducing an issue and debugging it locally. ▪ Reproduce integral components of an application environment (runtime, dependencies, etc.) in a local environment, or ensure a local environment has access to corresponding remote dependencies, or mock them. ▪ Use the same code base of the application when running it in local and remote environments. ▪ Few useful NPM packages to simplify access to application environment variables and unify it between local and Cloud Foundry deployments: ▪ dotenv (https://www.npmjs.com/package/dotenv) – to load environment variables. ▪ cfenv (https://www.npmjs.com/package/cfenv) – to access an application container environment in Cloud Foundry (when application runs in Cloud Foundry) or emulate it in a local environment (when application runs locally). 

Node.js debugger and debuggee Simplified overview 5 Code editor (Visual Studio Code, Atom, Brackets, …) Node.js application Inspector API node --inspect[-brk][=port] {main module} Browser built-in DevTools (Chrome DevTools, …) IDE (WebStorm, Eclipse, Komodo, …) ▪ Arguments --debug and --debug-brk – to establish debugging session between debugger and debuggee using V8 Debugging Protocol (legacy). Where possible, its successor – V8 Inspector Protocol – shall be considered (can be enabled using arguments --inspect and --inspect-brk). ▪ --inspect-brk / --debug-brk – to pause debuggee before user code (invocation of main module). ▪ --inspect / --debug – to pause debuggee at active breakpoints. Inspecto r client Inspecto r client Inspecto r client 

Developer’s machine SCP CF subaccount Organization Space Node.js debugger and debuggee Application deployed to Cloud Foundry 6 Container Node.js application Inspector API node --inspect[-brk][=port] {main module} CF CLI Port forwarding through SSH tunnel for Inspector port of Node.js application Code editor Browser built-in DevTools IDE Inspecto r client Inspecto r client Inspecto r client 

Debugging Node.js application deployed to Cloud Foundry Preparatory steps 7 ▪ Step 1. Define a script to start a Node.js application with an enabled Inspector. ▪ Step 2. Use the defined script as a start up command of the deployed application. ▪ Step 3. Enable SSH access to a container of a deployed Node.js application. ▪ Step 4. Configure port forwarding through SSH tunnel for an Inspector port of the deployed Node.js application. ▪ Step 5. Create debugging configuration to attach to a l o cal No d e. j s p ro ces s (representing local end of the SSH tunnel). Node.js application Cloud Foundry Debugger 

Node.js application Steps 1 and 2. Adjust application manifest files 8 ▪ In Node.js application manifest (package.json), define a script to start the application with an enabled Inspector. ▪ In Cloud Foundry deployment manifest (manifest.xml), use the defined script as a start up command of the deployed application. 

Developer’s machine SCP CF subaccount Organization Space Cloud Foundry Access to Inspector of Node.js application through SSH tunnel 9 Container Check if SSH access is enabled at space level cf space-ssh-allowed {space} Enable SSH access cf allow-space-ssh {space} Disable SSH access cf disallow-space-ssh {space} Check if SSH access is enabled at application level cf ssh-enabled {app} Enable SSH access cf enable-ssh {app} Disable SSH access cf disable-ssh {app} * If SSH access has been disabled, restart application after enabling SSH access cf restart {app} Node.js application Inspector API node --inspect[-brk][=port] {main module} CF CLI cf ssh [-N] –L [bind host:]{bind port}: {host}:{port} {app} Port forwarding through SSH tunnel for Inspector port of Node.js application Code editor Browser built-in DevTools IDE Inspecto r client Inspecto r client Inspecto r client 

Cloud Foundry Step 3. Enable SSH access to application container 10 ▪ By default, SSH access is enabled at SCP Cloud Foundry space level, but it is not enabled at application level. ▪ Command: cf enable-ssh {app}. ▪ Restart the application after enabling SSH access to its application container. ▪ SSH access has been enabled and an application presents itself as SSH enabled, but attempt to connect using SSH fails if the application has not been restarted… ▪ …and attempt to connect using SSH after the application has been restarted. 

Cloud Foundry Step 4. Configure port forwarding through SSH tunnel for Node Inspector port 11 ▪ Command: cf ssh [-N] -L [bind host:]{bind port}:{host}:{port} {app}. ▪ Argument -N – to omit access to an SSH shell of an application container right after setting up port forwarding. If the argument is provided, port forwarding through SSH tunnel is enabled and an SSH shell to the application container can be started at a later time using command cf ssh {app}. ▪ In either case, port forwarding through SSH tunnel for a Node Inspector port is configured and usage of a specified local port can be verified: ▪ When started, port forwarding through SSH tunnel does not verify if the remote is accessible, if the remote port is opened and if the corresponding listener service is ready to accept requests on it. 

SCP CF region – IaaS provider (AWS, GCP, Azure) IP ranges: {check SAP and IaaS provider’s documentation} Developer’s machine Cloud Foundry Access to Inspector of Node.js application through SSH tunnel – communication 12 Container Node.js application Inspector API CF CLI IDE Code editor Browser built-in DevTools Inspector client node --inspect[-brk][=port] {main module} node --inspect ./dist/app.js cf ssh [-N] –L [bind host:]{bind port}:{host}:{port} {app} cf ssh -N -L 9229:127.0.0.1:9229 weather-demo Port of SSH proxy: 2222 Host of application within its container: 127.0.0.1 Can be omitted when CF CLI runs and is accessed on local network interface In sake of simplicity, default Inspector port (9229) is used and is mapped to the same bind port on a local machine. A bind port can be any unused port and does not need to be the same as the forwarded port. SSH daemon of application container 

Side note: Cloud Foundry Access to service instance through SSH tunnel – overview 13 Developer’s machine SCP CF subaccount Organization Space Container Node.js application Inspector API CF CLI MongoDB client (example) Service {…} client IDE Inspecto r client MongoD B driver Container Service {…} Container MongoDB (example) Published port Exposed port Container IP Host IP 

Side note: Cloud Foundry Access to service instance through SSH tunnel – MongoDB example 14 ▪ Option #1. Bind a service instance to an application. Connection and authentication information can be retrieved from an application environment variable VCAP_SERVICES. ▪ Option #2. Create a service key for a service instance. Connection and authentication information can be retrieved from the service key. ▪ Option #3. For some services, connection and authentication information can be retrieved from service instance dashboard. Restart the application before using port forwarding for the service instance through SSH tunnel, if the application was started before the service instance creation. 

Side note: Cloud Foundry Access to service instance through SSH tunnel – why is application restart required? 15 ▪ Concept of application security groups in Cloud Foundry. ▪ A new application security group is automatically created for a newly created service instance at space level. ▪ A new application security group does not affect already started applications until they are restarted. ▪ Accessibility of services exposed by a service instance can be verified from an application container. ▪ A new feature – dynamic egress policies (currently in beta in Cloud Foundry): in contrast to application security groups, does not require application restart and can be set at application level. 

Cloud Foundry Explore environment using SSH shell – application process and application directory 16 ▪ Deployed and started Node.js application: ▪ Application directory – $HOME/app (/home/vcap/app) – contains pushed files of a Node.js application: 

Debugger Step 5. Debugging configuration 17 ▪ request – “attach”, as debugger attaches to an already running Node.js process (no need to launch it). ▪ port – local port which has been used when configuring port forwarding through SSH tunnel. ▪ localRoot – Node.js project workspace on a local machine (where debugger is executed). ▪ remoteRoot – directory that contains a Node.js application on a remote server (in an application container). It is essential to specify both local root and remote root of Node.js project / application root directory to allow correct source code mapping between debugger and debuggee. Create debugging configuration (e.g. in launch.json in Visual Studio Code) to attach debugger to a locally running Node.js process. Attach point is a local end of the SSH tunnel, through which debugger will reach Inspector of a Node.js application deployed to Cloud Foundry: 

Debugging Node.js application deployed to Cloud Foundry Example 18 

Cloud Foundry Explore environment using Node REPL console – application process 19 ▪ process.pid – process ID ▪ process.execPath – absolute path to executable that started Node process. ▪ process.execArgv – Node.js specific command line options passed when starting Node process. ▪ process.mainModule – main module of the application. Enter Node REPL (Read-Eval-Print-Loop) console and explore process attributes – such as: 

Debugging Traditional breakpoint 20 Execution of a program flow is paused: ▪ for breakpoint – whenever the program reaches corresponding statement, ▪ for conditional breakpoint – whenever the program reaches corresponding statement and the specified condition is met (evaluates to true). 

Debugging Logpoint 21 ▪ In certain cases, a developer has no need to pause program flow – instead, it is only required to observe content (specific variables) or evaluate expressions. In such cases, use logpoints instead of traditional breakpoints. ▪ Logpoint is a type of breakpoint that produces the specified log entry and outputs it to debug terminal without suspending program execution. Essentially, logpoint is a statement console.log() that is added at the specific location of the analyzed program at debug time, rather than at development time. 

Debugging Logpoint – under the hood 22 Enable debug adapter trace to collect detailed information: in debugging configuration, set value of an attribute “trace” to: ▪ true – to output only to debug adapter log file, or ▪ “verbose” – to output both to debug adapter log file and debug console. 

Debugging Hints and tips 23 ▪ A lot of variables and properties can be declared when reaching certain processing steps, whereas a developer might be interested in continuously observing only few of them. Use watch expressions to get access to content of a selection of variables and to evaluate expressions: ▪ Use debug console as Node REPL console to execute statements. 

Debugging Hints and tips: transpiled Node.js application – TypeScript example 24 ▪ For Node.js applications developed in programming languages that require transpiling (such as TypeScript), ensure that generation of source map files is enabled (e.g. for TypeScript – in TypeScript Compiler configuration) to make it possible to map originally developed source code and corresponding JavaScript code that has been generated by compiler and is interpreted at runtime by JavaScript engine: ▪ Source map contains Base64 VLQ encoded mapping between original TypeScript and generated JavaScript versions of source code. 

Debugging Node.js application deployed to Cloud Foundry in production 25 Basic reflex of a developer when they are asked to debug an application in a production environment shall be… Reminder: ALWAYS carefully assess possible risks and potential impact on the debugged application and environment where it runs, and evaluate consequences associated with running debugging session in a production environment BEFORE you start debugging. Any debugging exercise in a production environment shall be taken with high caution. …otherwise, imprudent debugging in a production environment introduces a (not so) small chance of a disaster. 

Debugging Node.js application deployed to Cloud Foundry in production 26 Common case for a production environment: ▪ A Node.js application has been deployed to Cloud Foundry. ▪ The application has been started without an enabled Inspector. ▪ SSH access to the application container can be enabled, if needed. ▪ There is no point in configuring port forwarding for an Inspector port and attempting to connect to a running application in debug mode in such a state, because Inspector does not listen on a dedicated port. 

Debugging Node.js application deployed to Cloud Foundry in production Enable Inspector in a running Node.js application 27 ▪ Option #1. Make corresponding amendments to application manifest files and re-push the application to Cloud Foundry. Commonly is not acceptable in a production environment. ▪ Option #2. Connect to an application container using SSH and send signal USR1 to a process of the application – this will enable Inspector for an already running Node.js application. Having this done, configure port forwarding through SSH tunnel and proceed to remote debugging: 

Debugging Node.js application deployed to Cloud Foundry in production Cleanup after debugging – disable Inspector 28 Debugging session has been completed and debugger has been disconnected, but Inspector remains enabled and listening for incoming connections. In contrast to enabling Inspector, it cannot be disabled by sending signal to a process of the application (Inspector cannot be disabled from outside the process). ▪ Option #1. Restart an application – the application will be started with disabled Inspector. Commonly is not acceptable in a production environment. ▪ Option #2. In Node REPL console, use Inspector API and trigger deactivation (closure) of Inspector: require('inspector').close() ▪ Option #3. Enhance the application by implementing and registering signal event listener for a dedicated process signal that is not in use yet. The listener shall implement similar logic as mentioned above and disable Inspector if it is enabled. With this enhancement, Inspector can be disabled by sending corresponding signal to a process of the application. 

Debugging and challenge of the flow of time 29 Traditional debugger allows to step forward an analyzed program flow (forward debugging). What if we need to rewind program flow back from the breakpoint (reverse debugging) to have a look into what has happened within the program several statements ago? ▪ We need a time machine… ▪ … or Node.js runtime and debugger that support Time Travel Debugging (TTD). 

Time travel debugging Node.js application 30 ▪ Time travel debugging (reverse debugging) allows to step back to earlier executed statements of the program. Can be also used for postmortem debugging – debugging of the program that does not run anymore (crashed or has been terminated). ▪ Based on record-replay-snapshot architecture. ▪ Requires usage of an alternative JavaScript engine for Node.js runtime – Node ChakraCore (https:// github.com/nodejs/node-chakracore), and an enhanced JavaScript debugger – Jardis. ▪ As per the documentation, “this project is still work in progress and not an officially supported Node.js branch”. ▪ The feature is not available in Chrome V8 JavaScript engine. Warning: Stunts and tricks displayed hereafter are performed in controlled environments, such as a local development environment. Do not attempt to duplicate, re-create, or perform the same or similar stunts and tricks in your production environment. 

Time travel debugging Node.js application Visual Studio Code extension 31 ▪ An extension for Visual Studio Code – NodeChakra Time Travel Debug (https:// marketplace.visualstudio.com/items?itemName=ttd-trace-tools.node-chakracore-time-travel-debugger): ▪ The extension enriches Visual Studio Code debugger with additional functionality to support time travel debugging – such as reverse steps (reverse step local and reverse step dynamic operations) and replay: 

Time travel debugging Node.js application Example 32 

Time travel debugging Node.js application Constraints 33 If time travel debugging is so effective, why it is not enabled by default in debugging toolkit? ▪ Runtime overhead and costs associated with collection of large volume of non-deterministic data about program state for each discrete snapshot during forward execution of the program. ▪ Delays when stepping through collected and persisted snapshots in reverse debugging and snapshots replay modes. The goal is to provide affordable time travel debugging system that addresses constraints and finds a compromise between ultimate reverse debugging functionality and overhead that is incurred by a system. 

Side note: Node Version Switcher Rapid switch between JavaScript engines 34 ▪ Node Version Switcher (https://github.com/jasongin/nvs) – to quickly switch between JavaScript engines used by Node.js runtime, and Node.js versions. ▪ Debugging configuration to utilize NVS and specific Node.js engine: 

Side note: Node Version Switcher (cont.) Rapid switch between JavaScript engines 35 After switching Node.js engine, details of the used engine can be verified in Node REPL console by checking process attributes – they will indicate which engine and version is currently used: 

➢ Thank you Dr. Vadim Klimov SAP Integration Architect SAP Community https://people.sap.com/Vadim.Klimov LinkedIn https://www.linkedin.com/in/Vadim-Klimov