Tweet
Tweet

Slide 1

Slide 1 text

Douglas Chen [陳鍵源] How to use LLVM’s Clang frontend library `libtooling` COSCUP 2020 Taiwan 2020/Aug/01

Slide 2

Slide 2 text

HackMD Note https://bit.ly/coscup2020libtooling 2

Slide 3

Slide 3 text

3 3. Problems I got, then what I did: a. My side project b. Problems & Answers 4. The last 1. How Clang helps me! a. Motivation 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling

Slide 4

Slide 4 text

4 1. How Clang helps me!

Slide 5

Slide 5 text

5 Ref: https://guycookson.com/2015/06/26/design-vs-user-experience/ Coding guideline ! Several naming conventions : 1. UpperCamel, 2. lowerCamel, 3. UPPER_SNAKE_CASE, 4. lower_snake_case, 5. szHungarainNotion

Slide 6

Slide 6 text

6 $ clang -cc1 -ast-dump min.c // min.c int min(int a, int b) { if (a < b) return a; return b; } Clang is the frontend tool of LLVM

Slide 7

Slide 7 text

TranslationUnitDecl `-FunctionDecl line:2:5 min 'int (int, int)' |-ParmVarDecl col:13 used a 'int' |-ParmVarDecl col:20 used b 'int' `-CompoundStmt |-IfStmt | |-BinaryOperator 'bool' '<' | | |-ImplicitCastExpr 'int' | | | `-DeclRefExpr 'int' lvalue ParmVar 0x56015970ab88 'a' 'int' | | `-ImplicitCastExpr 'int' | | `-DeclRefExpr 'int' lvalue ParmVar 0x56015970ac08 'b' 'int' | `-ReturnStmt | `-ImplicitCastExpr 'int' | `-DeclRefExpr 'int' lvalue ParmVar 0x56015970ab88 'a' 'int' `-ReturnStmt `-ImplicitCastExpr 'int' `-DeclRefExpr 'int' lvalue ParmVar 0x56015970ac08 'b' 'int' // min.c int min(int a, int b) { if (a < b) return a; return b; } $ clang -cc1 -ast-dump min.c https://godbolt.org/z/f_WYLM 7 Dump the AST to console

Slide 8

Slide 8 text

8 Where is the LibTooling

Slide 9

Slide 9 text

Compiler Source Code Executable Binary 9 Frontend IR Optimizer IR Backend Source Code Machine Code Portable IR Transformed IR lib Which part is our topic today ?

Slide 10

Slide 10 text

10 3. Problems I got, then what I did: a. My side project b. Problems & Answers 4. The last 1. How Clang helps me! a. Motivation 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling

Slide 11

Slide 11 text

11 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling

Slide 12

Slide 12 text

12 libclang.dll libclang.so libclang.dylib ClangSharp cindex.py https:/ /github.com/microsoft/ClangSharp https:/ /github.com/go-clang/clang-v3.9 https:/ /github.com/KyleMayes/clang-rs go-clang C# Python Golang clang-rs Rustlang How to use LibClang ... ✔ C language. ✔ Stable interface. ✔ Separate complicated compiler framework. ✖ Full Clang AST.

Slide 13

Slide 13 text

13 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling

Slide 14

Slide 14 text

Build Environment clang 14 ClangPlugin.so How to use Clang Plugins clang -cc1 -load ClangPlugin.so -plugin -print-fns Hello.cpp ✔ C++. ✔ Loaded at runtime by compiler. ✔ Run additional actions on the AST, as part of a compiliation. ✔ Full Clang AST. ✖ Interface is not stable. Hello.cpp

Slide 15

Slide 15 text

15 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling

Slide 16

Slide 16 text

16 Standalone.exe How to use LibTooling ✔ C++. ✔ A single executable file. ✔ Full Clang AST. ✔ Share code with ClangPlugins. ✖ Interface is not stable.

Slide 17

Slide 17 text

17 3. Problems I got, then what I did: a. My side project b. Problems & Answers 4. The last 1. How Clang helps me! a. Motivation 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling

Slide 18

Slide 18 text

18 3. Problems I got, then what I did: a. My side project b. Problems & Answers

Slide 19

Slide 19 text

19 CppNameLint Windows Linux macOS https://github.com/dougpuob/cppnamelint 1. UpperCamel, 2. lowerCamel, 3. UPPER_SNAKE_CASE, 4. lower_snake_case, 5. szHungarainNotion

Slide 20

Slide 20 text

20 Let’s try to visit the Dummy function (sample code) // Fool.c typedef struct _FoolSt { int Bool; int Cool; } FoolSt; int GetWholePool(const FoolSt* const pFool) { int iWool = pFool->Bool + pFool->Cool + 2020; return Wool; }

Slide 21

Slide 21 text

21 TOML config file of CppNameLint utility # cppnamelint.toml [General.Options] Version = 0.3 FileExtNameList = ["*.c","*.h","*.cpp"] CheckFileName = true CheckVariableName = true CheckFunctionName = true CheckEnum = true CheckStruct = true [General.Rules] FileName = 0 FunctionName = 0 VariableName = 1.... ClassName = 0 EnumTagName = 0 EnumValueName = 0 StructTagName = 0 StructValueName = 0 # 0: Default (UpperCamel) # 1: UpperCamel # 2: lowerCamel # 3: lower_snake # 4: Hungarian # 5: UPPER_SNAKE [General.IgnoredList] FunctionPrefix = [ "_", "__", "~"] VariablePrefix = [ "m_" ] EnumTagPrefix = [ "_", "e" ] StructTagPrefix = [ "_", "s" ] FunctionName = ["main", "newASTConsumer"]

Slide 22

Slide 22 text

22 check the Round.cpp file with CppNameLint utility

Slide 23

Slide 23 text

23 Decl classes (inheritance hierarchy)

Slide 24

Slide 24 text

24 Decls with source code // Fool.c typedef struct _FoolSt { int Bool; int Cool; } FoolSt; int GetWholePool(const FoolSt* const pFool) { int iWool = pFool->Bool + pFool->Cool + 2020; return Wool; }

Slide 25

Slide 25 text

25 UML class diagrams (CppNameLint utility) Yours LLVM 2 1 3

Slide 26

Slide 26 text

↑ cppnamelint.exe!MyASTVisitor::VisitFunctionDecl(clang::FunctionDecl * pDecl) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromFunctionDecl(clang::FunctionDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromCXXMethodDecl(clang::CXXMethodDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromCXXConstructorDecl(clang::CXXConstructorDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::TraverseCXXConstructorDecl(clang::CXXConstructorDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::TraverseDecl(clang::Decl * D) ↑ cppnamelint.exe!MyASTConsumer::HandleTopLevelDecl(clang::DeclGroupRef DeclGrpRef) ↑ cppnamelint.exe!clang::ParseAST(clang::Sema & S, bool PrintStats, bool SkipFunctionBodies) ↑ cppnamelint.exe!clang::ASTFrontendAction::ExecuteAction() ↑ cppnamelint.exe!clang::FrontendAction::Execute() ↑ cppnamelint.exe!clang::CompilerInstance::ExecuteAction(clang::FrontendAction & Act) ↑ cppnamelint.exe!clang::tooling::FrontendActionFactory::runInvocation(...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::runInvocation(const char * BinaryName, ...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::run() ↑ cppnamelint.exe!clang::tooling::ClangTool::run(clang::tooling::ToolAction * Action) ↑ cppnamelint.exe!RunCheck(namelint::MemoBoard & Memo, clang::tooling::ClangTool & Tool) ↑ cppnamelint.exe!RunCheckFormFile(namelint::MemoBoard & Memo) ↑ cppnamelint.exe!main(int Argc, const char * * Argv) 26 Callstack

Slide 27

Slide 27 text

27 UML class diagrams (CppNameLint utility) [❶ ❷ ❸ ❹ ❺ ❻ ❼ ❽ ❾ ❿] ❸ ❺ ❹ ❻ ❼ ❷ ❽ ❶ ❾ Yours LLVM 2 1 3

Slide 28

Slide 28 text

28 Callstack (1) ↑ cppnamelint.exe!MyASTVisitor::VisitFunctionDecl(clang::FunctionDecl * pDecl) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromFunctionDecl(clang::FunctionDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromCXXMethodDecl(clang::CXXMethodDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::WalkUpFromCXXConstructorDecl(clang::CXXConstructorDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::TraverseCXXConstructorDecl(clang::CXXConstructorDecl * D) ↑ cppnamelint.exe!clang::RecursiveASTVisitor<...>::TraverseDecl(clang::Decl * D) ↑ cppnamelint.exe!MyASTConsumer::HandleTopLevelDecl(clang::DeclGroupRef DeclGrpRef) ↑ cppnamelint.exe!clang::ParseAST(clang::Sema & S, bool PrintStats, bool SkipFunctionBodies) ↑ cppnamelint.exe!clang::ASTFrontendAction::ExecuteAction() ↑ cppnamelint.exe!clang::FrontendAction::Execute() ↑ cppnamelint.exe!clang::CompilerInstance::ExecuteAction(clang::FrontendAction & Act) ↑ cppnamelint.exe!clang::tooling::FrontendActionFactory::runInvocation(...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::runInvocation(const char * BinaryName, ...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::run() ↑ cppnamelint.exe!clang::tooling::ClangTool::run(clang::tooling::ToolAction * Action) ↑ cppnamelint.exe!RunCheck(namelint::MemoBoard & Memo, clang::tooling::ClangTool & Tool) ↑ cppnamelint.exe!RunCheckFormFile(namelint::MemoBoard & Memo) ↑ cppnamelint.exe!main(int Argc, const char * * Argv) 1 Yours LLVM 2 3 6 7 5 ? ? ? ? 4 3 1

Slide 29

Slide 29 text

29 Callstack (2) ↑ cppnamelint.exe!MyFactory::newASTConsumer() ↑ cppnamelint.exe!clang::tooling::newFrontendActionFactory<...> ::FrontendActionFactoryAdapter::ConsumerFactoryAdaptor::CreateASTConsumer() ↑ cppnamelint.exe!clang::FrontendAction::CreateWrappedASTConsumer() ↑ cppnamelint.exe!clang::FrontendAction::BeginSourceFile() ↑ cppnamelint.exe!clang::CompilerInstance::ExecuteAction(clang::FrontendAction & Act) ↑ cppnamelint.exe!clang::tooling::FrontendActionFactory::runInvocation() ↑ cppnamelint.exe!clang::tooling::ToolInvocation::runInvocation(const char * BinaryName, ...) ↑ cppnamelint.exe!clang::tooling::ToolInvocation::run() ↑ cppnamelint.exe!clang::tooling::ClangTool::run(clang::tooling::ToolAction * Action) ↑ cppnamelint.exe!RunCheck(namelint::MemoBoard & Memo, clang::tooling::ClangTool & Tool) ↑ cppnamelint.exe!RunCheckFormFile(namelint::MemoBoard & Memo) ↑ cppnamelint.exe!main(int Argc, const char * * Argv) 1 2 4 2 1

Slide 30

Slide 30 text

// Main.cpp MyFactory MyFactory; std::unique_ptr NewFactory = newFrontendActionFactory(&MyFactory); ToolAction *pAction = NewFactory.get(); if (pAction) { RetCode = Tool.run(pAction); } // ParseAST.cpp!clang::ParseAST() for (bool AtEOF = P.ParseFirstTopLevelDecl(ADecl);...) { if (ADecl && !Consumer->HandleTopLevelDecl(ADecl.get())) 30 // clang/include/clang/Tooling/Tooling.h class ConsumerFactoryAdaptor : public ASTFrontendAction { std::unique_ptr CreateASTConsumer(CompilerInstance &, StringRef) override { return ConsumerFactory->newASTConsumer(); } } // MyFactory.cpp unique_ptr MyFactory::newASTConsumer() { return llvm::make_unique(); } // MyAstConsumer.cpp class MyASTConsumer : public clang::ASTConsumer { private: public: bool HandleTopLevelDecl( DeclGroupRef declGroupRef); }; ❸... ❹... ❺... ❷... ❶... Yours LLVM

Slide 31

Slide 31 text

// MyAstConsumer.cpp bool MyASTConsumer::HandleTopLevelDecl( DeclGroupRef DeclGrpRef) { const SourceManager &SrcMgr = ASTCxt.getSourceManager(); MyASTVisitor myVisitor(&SrcMgr, &ASTCxt, pConfig); myVisitor.TraverseDecl(*Iter); } 31 // ParseAST.cpp!clang::ParseAST() template bool RecursiveASTVisitor::TraverseDecl(Decl *D) { if (!D) return true; // As a syntax visitor, by default we want to ignore declarations for // implicit declarations (ones not typed explicitly by the user). if (!getDerived().shouldVisitImplicitCode() && D->isImplicit()) return true; switch (D->getKind()) { #define ABSTRACT_DECL(DECL) #define DECL(CLASS, BASE) \ case Decl::CLASS: \ if (!getDerived().Traverse##CLASS##Decl(static_cast(D))) \ return false; \ break; #include "clang/AST/DeclNodes.inc" } // Visit any attributes attached to this declaration. for (auto *I : D->attrs()) { if (!getDerived().TraverseAttr(I)) return false; } return true; } // MyAstVisitor.cpp bool MyASTVisitor::VisitFunctionDecl( clang::FunctionDecl *pDecl) { return true; } Yours LLVM 2 3

Slide 32

Slide 32 text

32 Problems & Answers 3 Demonstrate to know more details by 10 questions. 1

Slide 33

Slide 33 text

33 Problems & Answers #1 #1: Parse input commands. (argc & argv)

Slide 34

Slide 34 text

34 Problems & Answers #1 // CommandLine.h //==----------------------------------------------------------------------- static cl::SubCommand CheckSubcommand("check", "Check source with naming rules."); static cl::opt CheckInputSrc(cl::Positional, cl::desc(""), cl::Required, cl::sub(CheckSubcommand)); // Main.cpp // Initialize LLVM's CommandLine library. cl::HideUnrelatedOptions(CppNameLintCategory); cl::ParseCommandLineOptions(Argc, Argv); if (CheckSubcommand) { LogConfig(); iRet = RunCheckFormFile(InputFilePath); LogCheckResult();

Slide 35

Slide 35 text

35 Problems & Answers #1 // MyCommandLine.h //==--------------------------------------------------------------------------------------------- static cl::SubCommand CheckSubcommand("check", "Check source with naming rules."); static cl::opt CheckInputSrc(cl::Positional, cl::desc(""), cl::Required, cl::sub(CheckSubcommand)); static cl::opt CheckInputConfig("config", cl::desc("Specific your config file(.toml)."), cl::value_desc("File"), cl::cat(CppNameLintCategory), cl::sub(CheckSubcommand)); static cl::opt CheckOutputJson("jsonout", cl::desc("Generate result to a JSON file."), cl::value_desc("File"), cl::cat(CppNameLintCategory), cl::sub(CheckSubcommand)); static cl::list CheckIncludes("include", cl::desc("Specific header folers."), cl::value_desc("-include Dir1 ..."), cl::ZeroOrMore, cl::cat(CppNameLintCategory), cl::sub(CheckSubcommand));

Slide 36

Slide 36 text

36 Problems & Answers #2 #2: Feed file path as a parameter ?

Slide 37

Slide 37 text

37 Problems & Answers #2 int main(int argc, const char **argv) { CommonOptionsParser OptParse(argc, argv, ToolingSampleCategory); ClangTool Tool(OptParse.getCompilations(), OptParse.getSourcePathList()); return Tool.run(newFrontendActionFactory().get()); } int RunCheckFormFile(std::string FilePath) { std::string Mesg; auto Compilations = FixedCompilationDatabase::loadFromFile(FilePath, Mesg); ClangTool Tool(*Compilations, {FilePath}); return Tool.run(newFrontendActionFactory().get()); } Origin Now

Slide 38

Slide 38 text

38 Problems & Answers #3 #3: How to get the name from Decls ?

Slide 39

Slide 39 text

39 int Dummy ( int iVal1 ) { const int iVal2 = iVal1; ParmVarDecl FunctionDecl pParmDecl->getBeginLoc() pParmDecl->getLocation() substr pParmVarDecl->getName() pFnDecl->getDeclName() VarDecl pVarDecl->getNameAsString() pVarDecl->getBeginLoc() pVarDecl->getLocation() substr Problems & Answers #3

Slide 40

Slide 40 text

40 Problems & Answers #4 #4: Is it a(n) array / builtin-type / pointer ?

Slide 41

Slide 41 text

41 int Dummy ( int iVal1 ) { const int iVal2 = iVal1; VarDecl QualType VarQualType = pVarDecl->getType(); bIsArray = VarQualType->isArrayType(); bIsBuiltinType = VarQualType->isBuiltinType(); bIsPtr = VarQualType->isPointerType(); Problems & Answers #4

Slide 42

Slide 42 text

42 Problems & Answers #5 #5: How to know its location (row & column)?

Slide 43

Slide 43 text

43 Problems & Answers #5 int Dummy ( int iVal1 ) { pParmDecl->getLocation() substr pParmVarDecl->getName() ASTContext& AstCxt = pParamDecl->getASTContext(); FullSourceLoc FullSrcLoc = AstCxt .getFullLoc(pDecl->getBeginLoc()) size_t nLineNumb = FullSrcLoc.getSpellingLineNumber(); size_t nColNumb = FullSrcLoc.getSpellingColumnNumber(); pParmDecl->getBeginLoc() ParmVarDecl

Slide 44

Slide 44 text

44 Problems & Answers #6 #6: How to know parsing result ?

Slide 45

Slide 45 text

45 Problems & Answers #6 // AstVisitor.cpp if (pDecl->isInvalidDecl()) { AppCxt::getInstance().MemoBoard.Assert.nInvalidDecl++; if (true ==m_pConfig->General.Options.bBypassInvalidDecl) { return true; } } bool bErr = diagEngine.hasErrorOccurred(); size_t nWarn = diagEngine.getNumWarnings();

Slide 46

Slide 46 text

46 Problems & Answers #7 #7: How to check a virtual file in memory ? (Unit Test)

Slide 47

Slide 47 text

47 Problems & Answers #7 ClangTool::mapVirtualFile(VirtFileName, SourceCode);

Slide 48

Slide 48 text

48 Problems & Answers #9 #8: How to check an header file ?

Slide 49

Slide 49 text

49 Problems & Answers #9 int RunCheck(ClangTool &Tool) { int iRet = 0; ArgumentsAdjuster ArgAdj = getInsertArgumentAdjuster( "--language=c++", // C++ ArgumentInsertPosition::BEGIN)); Tool.appendArgumentsAdjuster(ArgAdj); return iRet; }

Slide 50

Slide 50 text

50 Problems & Answers #8 #9: Specific header directories ?

Slide 51

Slide 51 text

51 Problems & Answers #8 int RunCheck(ClangTool &Tool) { int iRet = 0; for (auto inc : Memo.Dir.Includes) { llvm::SmallString<128> IncDirPath(inc); llvm::sys::fs::make_absolute(IncDirPath); vector inc_dir = {string("-I"), IncDirPath.str()}; DcLib::Log::Out(INFO_ALL, "-I %s", IncDirPath.c_str()); Tool.appendArgumentsAdjuster(getInsertArgumentAdjuster(inc_dir, ArgumentInsertPosition::BEGIN)); } return iRet; }

Slide 52

Slide 52 text

52 Problems & Answers #10 #10: Absolute paths ?

Slide 53

Slide 53 text

/ / FileSystem llvm::sys::fs::exists() llvm::sys::fs::access() llvm::sys::fs::make_absolute() / / Path llvm::sys::path::append() llvm::sys::path::is_relative() llvm::sys::path::is_absolute() llvm::sys::path::native() llvm::sys::path::filename() llvm::sys::path::extension() / / Json llvm::json::fromJSON() llvm::json::isUTF8() llvm::json::parse() llvm::json::toJSON() / / zlib llvm::zlib::compress() llvm::zlib::crc32() llvm::zlib::uncompress() 53 Problems & Answers #10

Slide 54

Slide 54 text

54 3. Problems I got, then what I did: a. My side project b. Problems & Answers 4. The last 1. How Clang helps me! a. Motivation 2. Clang frontend libraries: a. LibClang b. Clang Plugins c. LibTooling

Slide 55

Slide 55 text

55 CI/CD (Azure DevOps) source repo ❸ binary repo ❷ ❶ ❹

Slide 56

Slide 56 text

56 創 惟 科 技 CppNameLint ❤ Appreciate someone who Helped me

Slide 57

Slide 57 text

57 ● Diagonostics (Warning) ● Rewrite (Fixit) Others intereting classes but ...

Slide 58

Slide 58 text

58 Thank You

Slide 59

Slide 59 text

END