How to use LLVM frontend library-libtooling

🔳 COSCUP 2020 Taiwan
🔳 HackMD Note (https://bit.ly/libtooling)
🔳 Author Douglas Chen (陳鍵源 )
🔳 Clang frontend libraries:
◾ LibClang
◾ Clang Plugins
◾ LibTooling
◾ $ clang -cc1 -ast-dump min.c

🔳 Problems I got, then what I did:
◾ My side project
◾ Problems & Answers

🔳 Several naming conventions :
◾ UpperCamel,
◾ lowerCamel,
◾ UPPER_SNAKE_CASE,
◾ lower_snake_case,
◾ szHungarainNotion

🔳 Demonstrate to know more details by 10 questions
◾ #1: Parse input commands. (argc & argv) Command Library
◾ #2: Feed file path as a parameter ? loadFromFile()
◾ #3: How to get the name from Decls ?
◾ #4: Is it a(n) array / builtin-type / pointer ?
▪ QualType VarQualType = pVarDecl->getType();
▪ bIsArray = VarQualType->isArrayType();
▪ bIsBuiltinType = VarQualType->isBuiltinType();
▪ bIsPtr = VarQualType->isPointerType();

◾ #5: How to know its location (row & column)?
▪ ASTContext& AstCxt = pParamDecl->getASTContext();
▪ FullSourceLoc FullSrcLoc = AstCxt .getFullLoc(pDecl->getBeginLoc())
▪ size_t nLineNumb = FullSrcLoc.getSpellingLineNumber();
▪ size_t nColNumb = FullSrcLoc.getSpellingColumnNumber();
◾ #6: How to know parsing result ?
▪ pDecl->isInvalidDecl()
◾ #7: How to check a virtual file in memory ? (Unit Test)
▪ ClangTool::mapVirtualFile(VirtFileName, SourceCode);
◾ #8: How to check an header file ?
▪ getInsertArgumentAdjuster() "--language=c++"
▪ appendArgumentsAdjuster()
◾ #9: Specific header directories ?
▪ getInsertArgumentAdjuster() "-I %s"
▪ appendArgumentsAdjuster()
◾ #10: Absolute paths ?
▪ llvm::sys::fs::exists()
▪ llvm::sys::fs::access()
▪ llvm::sys::fs::make_absolute()
▪ llvm::sys::path::append()
▪ llvm::sys::path::is_relative()
▪ llvm::sys::path::is_absolute()
▪ llvm::sys::path::native()
▪ llvm::sys::path::filename()
▪ llvm::sys::path::extension()

🔳 The last
◾ CI/CD (Microsoft Azure DevOps)
◾ 創惟科技 (Genesys Logic)

Transcript

1. Douglas Chen [陳鍵源] <dougpuob@gmail.com> How to use LLVM’s Clang frontend library `libtooling` COSCUP 2020 Taiwan 2020/Aug/01
2. HackMD Note https://bit.ly/coscup2020libtooling 2
3. 3 3. Problems I got, then what I did: a. My side project b. Problems & Answers 4. The last 1. How Clang helps me! a. Motivation 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling
4. 4 1. How Clang helps me!
5. 5 Ref: https://guycookson.com/2015/06/26/design-vs-user-experience/ Coding guideline ! Several naming conventions : 1. UpperCamel, 2. lowerCamel, 3. UPPER_SNAKE_CASE, 4. lower_snake_case, 5. szHungarainNotion
6. 6 $ clang -cc1 -ast-dump min.c // min.c int min(int a, int b) { if (a < b) return a; return b; } Clang is the frontend tool of LLVM
7. TranslationUnitDecl `-FunctionDecl <line:2:1, line:6:1> line:2:5 min 'int (int, int)' |-ParmVarDecl <col:9, col:13> col:13 used a 'int' |-ParmVarDecl <col:16, col:20> col:20 used b 'int' `-CompoundStmt <col:23, line:6:1> |-IfStmt <line:3:3, line:4:12> | |-BinaryOperator <line:3:7, col:11> 'bool' '<' | | |-ImplicitCastExpr <col:7> 'int' <LValueToRValue> | | | `-DeclRefExpr <col:7> 'int' lvalue ParmVar 0x56015970ab88 'a' 'int' | | `-ImplicitCastExpr <col:11> 'int' <LValueToRValue> | | `-DeclRefExpr <col:11> 'int' lvalue ParmVar 0x56015970ac08 'b' 'int' | `-ReturnStmt <line:4:5, col:12> | `-ImplicitCastExpr <col:12> 'int' <LValueToRValue> | `-DeclRefExpr <col:12> 'int' lvalue ParmVar 0x56015970ab88 'a' 'int' `-ReturnStmt <line:5:3, col:10> `-ImplicitCastExpr <col:10> 'int' <LValueToRValue> `-DeclRefExpr <col:10> 'int' lvalue ParmVar 0x56015970ac08 'b' 'int' // min.c int min(int a, int b) { if (a < b) return a; return b; } $ clang -cc1 -ast-dump min.c https://godbolt.org/z/f_WYLM 7 Dump the AST to console
8. 8 Where is the LibTooling
9. Compiler Source Code Executable Binary 9 Frontend IR Optimizer IR Backend Source Code Machine Code Portable IR Transformed IR lib Which part is our topic today ?
10. 10 3. Problems I got, then what I did: a. My side project b. Problems & Answers 4. The last 1. How Clang helps me! a. Motivation 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling
11. 11 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling
12. 12 libclang.dll libclang.so libclang.dylib ClangSharp cindex.py https:/ /github.com/microsoft/ClangSharp https:/ /github.com/go-clang/clang-v3.9 https:/ /github.com/KyleMayes/clang-rs go-clang C# Python Golang clang-rs Rustlang How to use LibClang ... ✔ C language. ✔ Stable interface. ✔ Separate complicated compiler framework. ✖ Full Clang AST.
13. 13 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling
14. Build Environment clang 14 ClangPlugin.so How to use Clang Plugins clang -cc1 -load ClangPlugin.so -plugin -print-fns Hello.cpp ✔ C++. ✔ Loaded at runtime by compiler. ✔ Run additional actions on the AST, as part of a compiliation. ✔ Full Clang AST. ✖ Interface is not stable. Hello.cpp
15. 15 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling
16. 16 Standalone.exe How to use LibTooling ✔ C++. ✔ A single executable file. ✔ Full Clang AST. ✔ Share code with ClangPlugins. ✖ Interface is not stable.
17. 17 3. Problems I got, then what I did: a. My side project b. Problems & Answers 4. The last 1. How Clang helps me! a. Motivation 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling
18. 18 3. Problems I got, then what I did: a. My side project b. Problems & Answers
19. 19 CppNameLint Windows Linux macOS https://github.com/dougpuob/cppnamelint 1. UpperCamel, 2. lowerCamel, 3. UPPER_SNAKE_CASE, 4. lower_snake_case, 5. szHungarainNotion
20. 20 Let’s try to visit the Dummy function (sample code) // Fool.c typedef struct _FoolSt { int Bool; int Cool; } FoolSt; int GetWholePool(const FoolSt* const pFool) { int iWool = pFool->Bool + pFool->Cool + 2020; return Wool; }
21. 21 TOML config file of CppNameLint utility # cppnamelint.toml [General.Options] Version = 0.3 FileExtNameList = ["*.c","*.h","*.cpp"] CheckFileName = true CheckVariableName = true CheckFunctionName = true CheckEnum = true CheckStruct = true [General.Rules] FileName = 0 FunctionName = 0 VariableName = 1.... ClassName = 0 EnumTagName = 0 EnumValueName = 0 StructTagName = 0 StructValueName = 0 # 0: Default (UpperCamel) # 1: UpperCamel # 2: lowerCamel # 3: lower_snake # 4: Hungarian # 5: UPPER_SNAKE [General.IgnoredList] FunctionPrefix = [ "_", "__", "~"] VariablePrefix = [ "m_" ] EnumTagPrefix = [ "_", "e" ] StructTagPrefix = [ "_", "s" ] FunctionName = ["main", "newASTConsumer"]
22. 22 check the Round.cpp file with CppNameLint utility
23. 23 Decl classes (inheritance hierarchy)
24. 24 Decls with source code // Fool.c typedef struct _FoolSt { int Bool; int Cool; } FoolSt; int GetWholePool(const FoolSt* const pFool) { int iWool = pFool->Bool + pFool->Cool + 2020; return Wool; }
25. 25 UML class diagrams (CppNameLint utility) Yours LLVM 2 1 3
26. ↑ cppnamelint.exe!MyASTVisitor::VisitFunctionDecl(clang::FunctionDecl * pDecl) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromFunctionDecl(clang::FunctionDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromCXXMethodDecl(clang::CXXMethodDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromCXXConstructorDecl(clang::CXXConstructorDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::TraverseCXXConstructorDecl(clang::CXXConstructorDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::TraverseDecl(clang::Decl * D) ↑ cppnamelint.exe!MyASTConsumer::HandleTopLevelDecl(clang::DeclGroupRef DeclGrpRef) ↑ cppnamelint.exe!clang::ParseAST(clang::Sema & S, bool PrintStats, bool SkipFunctionBodies) ↑ cppnamelint.exe!clang::ASTFrontendAction::ExecuteAction() ↑ cppnamelint.exe!clang::FrontendAction::Execute() ↑ cppnamelint.exe!clang::CompilerInstance::ExecuteAction(clang::FrontendAction & Act) ↑ cppnamelint.exe!clang::tooling::FrontendActionFactory::runInvocation(...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::runInvocation(const char * BinaryName, ...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::run() ↑ cppnamelint.exe!clang::tooling::ClangTool::run(clang::tooling::ToolAction * Action) ↑ cppnamelint.exe!RunCheck(namelint::MemoBoard & Memo, clang::tooling::ClangTool & Tool) ↑ cppnamelint.exe!RunCheckFormFile(namelint::MemoBoard & Memo) ↑ cppnamelint.exe!main(int Argc, const char * * Argv) 26 Callstack
27. 27 UML class diagrams (CppNameLint utility) [❶ ❷ ❸ ❹ ❺ ❻ ❼ ❽ ❾ ❿] ❸ ❺ ❹ ❻ ❼ ❷ ❽ ❶ ❾ Yours LLVM 2 1 3
28. 28 Callstack (1) ↑ cppnamelint.exe!MyASTVisitor::VisitFunctionDecl(clang::FunctionDecl * pDecl) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromFunctionDecl(clang::FunctionDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromCXXMethodDecl(clang::CXXMethodDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromCXXConstructorDecl(clang::CXXConstructorDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::TraverseCXXConstructorDecl(clang::CXXConstructorDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::TraverseDecl(clang::Decl * D) ↑ cppnamelint.exe!MyASTConsumer::HandleTopLevelDecl(clang::DeclGroupRef DeclGrpRef) ↑ cppnamelint.exe!clang::ParseAST(clang::Sema & S, bool PrintStats, bool SkipFunctionBodies) ↑ cppnamelint.exe!clang::ASTFrontendAction::ExecuteAction() ↑ cppnamelint.exe!clang::FrontendAction::Execute() ↑ cppnamelint.exe!clang::CompilerInstance::ExecuteAction(clang::FrontendAction & Act) ↑ cppnamelint.exe!clang::tooling::FrontendActionFactory::runInvocation(...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::runInvocation(const char * BinaryName, ...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::run() ↑ cppnamelint.exe!clang::tooling::ClangTool::run(clang::tooling::ToolAction * Action) ↑ cppnamelint.exe!RunCheck(namelint::MemoBoard & Memo, clang::tooling::ClangTool & Tool) ↑ cppnamelint.exe!RunCheckFormFile(namelint::MemoBoard & Memo) ↑ cppnamelint.exe!main(int Argc, const char * * Argv) 1 Yours LLVM 2 3 6 7 5 ? ? ? ? 4 3 1
29. 29 Callstack (2) ↑ cppnamelint.exe!MyFactory::newASTConsumer() ↑ cppnamelint.exe!clang::tooling::newFrontendActionFactory<...> ::FrontendActionFactoryAdapter::ConsumerFactoryAdaptor::CreateASTConsumer() ↑ cppnamelint.exe!clang::FrontendAction::CreateWrappedASTConsumer() ↑ cppnamelint.exe!clang::FrontendAction::BeginSourceFile() ↑ cppnamelint.exe!clang::CompilerInstance::ExecuteAction(clang::FrontendAction & Act) ↑ cppnamelint.exe!clang::tooling::FrontendActionFactory::runInvocation() ↑ cppnamelint.exe!clang::tooling::ToolInvocation::runInvocation(const char * BinaryName, ...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::run() ↑ cppnamelint.exe!clang::tooling::ClangTool::run(clang::tooling::ToolAction * Action) ↑ cppnamelint.exe!RunCheck(namelint::MemoBoard & Memo, clang::tooling::ClangTool & Tool) ↑ cppnamelint.exe!RunCheckFormFile(namelint::MemoBoard & Memo) ↑ cppnamelint.exe!main(int Argc, const char * * Argv) 1 2 4 2 1
30. // Main.cpp MyFactory MyFactory; std::unique_ptr<FrontendActionFactory> NewFactory = newFrontendActionFactory(&MyFactory); ToolAction *pAction = NewFactory.get(); if (pAction) { RetCode = Tool.run(pAction); } // ParseAST.cpp!clang::ParseAST() for (bool AtEOF = P.ParseFirstTopLevelDecl(ADecl);...) { if (ADecl && !Consumer->HandleTopLevelDecl(ADecl.get())) 30 // clang/include/clang/Tooling/Tooling.h class ConsumerFactoryAdaptor : public ASTFrontendAction { std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &, StringRef) override { return ConsumerFactory->newASTConsumer(); } } // MyFactory.cpp unique_ptr<MyASTConsumer> MyFactory::newASTConsumer() { return llvm::make_unique<MyASTConsumer>(); } // MyAstConsumer.cpp class MyASTConsumer : public clang::ASTConsumer { private: public: bool HandleTopLevelDecl( DeclGroupRef declGroupRef); }; ❸... ❹... ❺... ❷... ❶... Yours LLVM
31. // MyAstConsumer.cpp bool MyASTConsumer::HandleTopLevelDecl( DeclGroupRef DeclGrpRef) { const SourceManager &SrcMgr = ASTCxt.getSourceManager(); MyASTVisitor myVisitor(&SrcMgr, &ASTCxt, pConfig); myVisitor.TraverseDecl(*Iter); } 31 // ParseAST.cpp!clang::ParseAST() template <typename Derived> bool RecursiveASTVisitor<Derived>::TraverseDecl(Decl *D) { if (!D) return true; // As a syntax visitor, by default we want to ignore declarations for // implicit declarations (ones not typed explicitly by the user). if (!getDerived().shouldVisitImplicitCode() && D->isImplicit()) return true; switch (D->getKind()) { #define ABSTRACT_DECL(DECL) #define DECL(CLASS, BASE) \ case Decl::CLASS: \ if (!getDerived().Traverse##CLASS##Decl(static_cast<CLASS##Decl *>(D))) \ return false; \ break; #include "clang/AST/DeclNodes.inc" } // Visit any attributes attached to this declaration. for (auto *I : D->attrs()) { if (!getDerived().TraverseAttr(I)) return false; } return true; } // MyAstVisitor.cpp bool MyASTVisitor::VisitFunctionDecl( clang::FunctionDecl *pDecl) { return true; } Yours LLVM 2 3
32. 32 Problems & Answers 3 Demonstrate to know more details by 10 questions. 1
33. 33 Problems & Answers #1 #1: Parse input commands. (argc & argv)
34. 34 Problems & Answers #1 // CommandLine.h //==----------------------------------------------------------------------- static cl::SubCommand CheckSubcommand("check", "Check source with naming rules."); static cl::opt<string> CheckInputSrc(cl::Positional, cl::desc("<input file>"), cl::Required, cl::sub(CheckSubcommand)); // Main.cpp // Initialize LLVM's CommandLine library. cl::HideUnrelatedOptions(CppNameLintCategory); cl::ParseCommandLineOptions(Argc, Argv); if (CheckSubcommand) { LogConfig(); iRet = RunCheckFormFile(InputFilePath); LogCheckResult();
35. 35 Problems & Answers #1 // MyCommandLine.h //==--------------------------------------------------------------------------------------------- static cl::SubCommand CheckSubcommand("check", "Check source with naming rules."); static cl::opt<string> CheckInputSrc(cl::Positional, cl::desc("<input file>"), cl::Required, cl::sub(CheckSubcommand)); static cl::opt<string> CheckInputConfig("config", cl::desc("Specific your config file(.toml)."), cl::value_desc("File"), cl::cat(CppNameLintCategory), cl::sub(CheckSubcommand)); static cl::opt<string> CheckOutputJson("jsonout", cl::desc("Generate result to a JSON file."), cl::value_desc("File"), cl::cat(CppNameLintCategory), cl::sub(CheckSubcommand)); static cl::list<string> CheckIncludes("include", cl::desc("Specific header folers."), cl::value_desc("-include Dir1 ..."), cl::ZeroOrMore, cl::cat(CppNameLintCategory), cl::sub(CheckSubcommand));
36. 36 Problems & Answers #2 #2: Feed file path as a parameter ?
37. 37 Problems & Answers #2 int main(int argc, const char **argv) { CommonOptionsParser OptParse(argc, argv, ToolingSampleCategory); ClangTool Tool(OptParse.getCompilations(), OptParse.getSourcePathList()); return Tool.run(newFrontendActionFactory<MyFrontendAction>().get()); } int RunCheckFormFile(std::string FilePath) { std::string Mesg; auto Compilations = FixedCompilationDatabase::loadFromFile(FilePath, Mesg); ClangTool Tool(*Compilations, {FilePath}); return Tool.run(newFrontendActionFactory<MyFrontendAction>().get()); } Origin Now
38. 38 Problems & Answers #3 #3: How to get the name from Decls ?
39. 39 int Dummy ( int iVal1 ) { const int iVal2 = iVal1; ParmVarDecl FunctionDecl pParmDecl->getBeginLoc() pParmDecl->getLocation() substr pParmVarDecl->getName() pFnDecl->getDeclName() VarDecl pVarDecl->getNameAsString() pVarDecl->getBeginLoc() pVarDecl->getLocation() substr Problems & Answers #3
40. 40 Problems & Answers #4 #4: Is it a(n) array / builtin-type / pointer ?
41. 41 int Dummy ( int iVal1 ) { const int iVal2 = iVal1; VarDecl QualType VarQualType = pVarDecl->getType(); bIsArray = VarQualType->isArrayType(); bIsBuiltinType = VarQualType->isBuiltinType(); bIsPtr = VarQualType->isPointerType(); Problems & Answers #4
42. 42 Problems & Answers #5 #5: How to know its location (row & column)?
43. 43 Problems & Answers #5 int Dummy ( int iVal1 ) { pParmDecl->getLocation() substr pParmVarDecl->getName() ASTContext& AstCxt = pParamDecl->getASTContext(); FullSourceLoc FullSrcLoc = AstCxt .getFullLoc(pDecl->getBeginLoc()) size_t nLineNumb = FullSrcLoc.getSpellingLineNumber(); size_t nColNumb = FullSrcLoc.getSpellingColumnNumber(); pParmDecl->getBeginLoc() ParmVarDecl
44. 44 Problems & Answers #6 #6: How to know parsing result ?
45. 45 Problems & Answers #6 // AstVisitor.cpp if (pDecl->isInvalidDecl()) { AppCxt::getInstance().MemoBoard.Assert.nInvalidDecl++; if (true ==m_pConfig->General.Options.bBypassInvalidDecl) { return true; } } bool bErr = diagEngine.hasErrorOccurred(); size_t nWarn = diagEngine.getNumWarnings();
46. 46 Problems & Answers #7 #7: How to check a virtual file in memory ? (Unit Test)
47. 47 Problems & Answers #7 ClangTool::mapVirtualFile(VirtFileName, SourceCode);
48. 48 Problems & Answers #9 #8: How to check an header file ?
49. 49 Problems & Answers #9 int RunCheck(ClangTool &Tool) { int iRet = 0; ArgumentsAdjuster ArgAdj = getInsertArgumentAdjuster( "--language=c++", // C++ ArgumentInsertPosition::BEGIN)); Tool.appendArgumentsAdjuster(ArgAdj); return iRet; }
50. 50 Problems & Answers #8 #9: Specific header directories ?
51. 51 Problems & Answers #8 int RunCheck(ClangTool &Tool) { int iRet = 0; for (auto inc : Memo.Dir.Includes) { llvm::SmallString<128> IncDirPath(inc); llvm::sys::fs::make_absolute(IncDirPath); vector<string> inc_dir = {string("-I"), IncDirPath.str()}; DcLib::Log::Out(INFO_ALL, "-I %s", IncDirPath.c_str()); Tool.appendArgumentsAdjuster(getInsertArgumentAdjuster(inc_dir, ArgumentInsertPosition::BEGIN)); } return iRet; }
52. 52 Problems & Answers #10 #10: Absolute paths ?
53. / / FileSystem llvm::sys::fs::exists() llvm::sys::fs::access() llvm::sys::fs::make_absolute() / / Path llvm::sys::path::append() llvm::sys::path::is_relative() llvm::sys::path::is_absolute() llvm::sys::path::native() llvm::sys::path::filename() llvm::sys::path::extension() / / Json llvm::json::fromJSON() llvm::json::isUTF8() llvm::json::parse() llvm::json::toJSON() / / zlib llvm::zlib::compress() llvm::zlib::crc32() llvm::zlib::uncompress() 53 Problems & Answers #10
54. 54 3. Problems I got, then what I did: a. My side project b. Problems & Answers 4. The last 1. How Clang helps me! a. Motivation 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling
55. 55 CI/CD (Azure DevOps) source repo ❸ binary repo ❷ ❶ ❹
56. 56 創 惟 科 技 CppNameLint ❤ Appreciate someone who Helped me
57. 57 • Diagonostics (Warning) • Rewrite (Fixit) Others intereting classes but ...
58. 58 Thank You
59. END