1 *WARNING:* Kythe is alpha. Use at your own risk. 2 3 Kythe is a pluggable, (mostly) language-agnostic ecosystem for building tools 4 that work with code. This release contains the core set of indexers, 5 extractors, and tools directly supported by the Kythe team. 6 7 *License:* Apache Licence, Version 2.0 8 9 # Contents 10 - indexers 11 - cxx_indexer :: C++ indexer 12 - java_indexer.jar :: Java indexer 13 - extractors 14 - bazel_cxx_extractor :: C++ extractor for Bazel extra_actions 15 - bazel_java_extractor.jar :: Java extractor for Bazel extra_actions 16 - cxx_extractor :: C++ extractor 17 - javac_extractor.jar :: Java extractor 18 - javac-wrapper.sh :: javac wrapper script for extractor 19 - proto :: Protocol buffer definitions of public APIs 20 - tools 21 - dedup_stream :: Removes duplicates entries from a delimited stream 22 - directory_indexer :: Emits Kythe file nodes for some local paths 23 - entrystream :: Generic Kythe entry stream processor 24 - http_server :: HTTP/GRPC server for Kythe service APIs (xrefs, filetree, search) 25 - indexpack :: Converts between .kindex archives and indexpacks 26 - kwazthis :: Determine what Kythe node(s) are at a particular point in a file 27 - kythe :: CLI for the service APIs exposed by http_server 28 - read_entries :: Dumps a GraphStore's contents as an entry stream 29 - triples :: Converts an entry stream (or GraphStore) to N-Triples 30 - verifier :: Verifies indexer outputs with source-inlined goals 31 - viewindex :: Dumps a .kindex file as JSON 32 - write_entries :: Writes an entry stream to a GraphStore 33 - write_tables :: Processes a GraphStore into efficient serving tables for http_server 34 35 # Dependencies 36 - Java JDK >=8 37 - libncurses 38 - libuuid 39 40 ## Debian Jessie Install 41 42 echo "deb http://http.debian.net/debian jessie-backports main" >> /etc/apt/sources.list 43 apt-get install openjdk-8-jdk libncurses5 libssl1.0.0 44 45 # End-to-end Java Example 46 47 ``` 48 # Install Kythe 49 tar xzf kythe-v*.tar.gz 50 rm -rf /opt/kythe 51 mv kythe-v*/ /opt/kythe 52 53 git clone https://github.com/GoogleCloudPlatform/DataflowJavaSDK.git 54 cd DataflowJavaSDK 55 56 # Setup the Java extractor's environment 57 export REAL_JAVAC="$(which javac)" 58 export JAVAC_EXTRACTOR_JAR=/opt/kythe/extractors/javac_extractor.jar 59 export KYTHE_ROOT_DIRECTORY="$PWD" 60 export KYTHE_OUTPUT_DIRECTORY="/tmp/kythe" 61 export KYTHE_CORPUS=DataflowJavaSDK 62 mkdir -p "$KYTHE_OUTPUT_DIRECTORY" 63 64 # Force Maven to compile with extractors/javac-wrapper.sh 65 mvn clean compile \ 66 -Dmaven.compiler.forceJavacCompilerUse=true \ 67 -Dmaven.compiler.fork=true \ 68 -Dmaven.compiler.executable=/opt/kythe/extractors/javac-wrapper.sh 69 70 cd .. 71 72 # Index the resulting .kindex files, deduplicate the entries, and finally store 73 # them in a GraphStore 74 GRAPHSTORE=/tmp/kythe_graphstore 75 rm -rf "$GRAPHSTORE" 76 find "$KYTHE_OUTPUT_DIRECTORY" -name '*.kindex' | \ 77 xargs -L1 java -jar /opt/kythe/indexers/java_indexer.jar | \ 78 /opt/kythe/tools/dedup_stream | \ 79 /opt/kythe/tools/write_entries --graphstore $GRAPHSTORE 80 81 # Process the GraphStore into serving tables 82 SERVING=/tmp/kythe_serving 83 rm -rf "$SERVING" 84 /opt/kythe/tools/write_tables --graphstore $GRAPHSTORE --out "$SERVING" 85 86 # Launch Kythe's service APIs as an HTTP server listening to only local 87 # connections on port 9898. Using `--listen :9898` instead will allow 88 # connections from other networked machines. 89 /opt/kythe/tools/http_server --serving_table "$SERVING" \ 90 --public_resources /opt/kythe/web/ui --listen localhost:9898 91 92 # The sample web UI will be serving at http://localhost:9898 93 94 # /opt/kythe/tools/kythe and /opt/kythe/tools/kwazthis can be used with the 95 # local running server by passing the '--api=http://localhost:9898' flag. 96 ``` 97 98 # Usage 99 100 ## Extractors 101 102 `extractors/cxx_extractor` and `extractors/javac_extractor.jar` are 103 flag-compatible replacements for clang and javac, respectively. Instead of 104 performing a full compilation, they extract the compilation's full context 105 including all file inputs and archive them in a .kindex file. 106 107 Since each extractor is meant to replace a compiler, all configuration is passed 108 by the following environment variables: 109 110 KYTHE_ROOT_DIRECTORY (required): root directory of the repository being compiled. 111 This helps the extractor correctly infer file input paths. 112 KYTHE_OUTPUT_DIRECTORY (required): output directory for resulting .kindex 113 files 114 KYTHE_VNAMES: path to a JSON-encoded VNames configuration file. See 115 https://godoc.org/github.com/google/kythe/kythe/go/storage/vnameutil for 116 more details on the file's format and 117 https://kythe.io/repo/kythe/data/vnames.json for an example. 118 KYTHE_CORPUS: the name of the corpus for all constructed VNames (only used if 119 KYTHE_VNAMES is unset; defaults to "kythe") 120 121 `extractors/javac-wrapper.sh` is provided as one way to inject an extractor into 122 a build system -- by wrapping each compiler command. For instance, Maven 123 compilations can easily be extracted by replacing `$JAVA_HOME/bin/javac` with 124 this script and setting the following Maven options: 125 126 -Dmaven.compiler.forceJavacCompilerUse=true 127 -Dmaven.compiler.fork=true 128 -Dmaven.compiler.executable=$JAVA_HOME/bin/javac 129 130 Read `extractors/javac-wrapper.sh` for more details on its usage and see 131 examples usages in the Docker images defined at 132 https://kythe.io/repo/kythe/java/com/google/devtools/kythe/extractors/java/standalone/Dockerfile 133 and 134 https://kythe.io/repo/kythe/java/com/google/devtools/kythe/extractors/java/maven/Dockerfile. 135 136 ### Examples: 137 138 export KYTHE_ROOT_DIRECTORY="$PWD" 139 export KYTHE_OUTPUT_DIRECTORY=/tmp/kythe 140 141 mkdir -p "$KYTHE_OUTPUT_DIRECTORY" 142 143 java -Xbootclasspath/p:extractors/javac_extractor.jar \ 144 -jar extractors/javac_extractor.jar \ 145 -cp "${$(find third_party -name '*.jar' -printf '%p:')%:}" \ 146 $(find src/main -name '*.java') 147 148 extractors/cxx_extractor -Ithird_party/include some.cc 149 150 ## Indexers 151 152 `indexers/cxx_indexer` and `indexers/java_indexer.jar` analyze the .kindex files 153 produced by the extractors and emit a stream of protobuf wire-encoded facts 154 (entries) that conform to https://kythe.io/schema. The output stream can be 155 processed by many of the accompanying binaries in the tools/ directory. 156 157 ### Examples 158 159 java -jar indexers/java_indexer.jar \ 160 /tmp/kythe/065c7a9a0789d09b73d74b0ca17cfcec6643d69a6218d095d19a770b10dffdf9.kindex \ 161 > java.entries 162 163 indexers/cxx_indexers \ 164 /tmp/kythe/579d266e5914257a9bd4458eb9b218690280ae15123d642025f224d10f64e6f3.kindex \ 165 > cxx.entries 166
