1 # Project overview 2 3 ## Title 4 5 Enable Building of gRPC Python with Bazel 6 7 ## Overview 8 9 gRPC Python currently has a constellation of scripts written to build the 10 project, but it has a lot of limitations in terms of speed and maintainability. 11 [Bazel](https://bazel.build/) is the open-sourced variant of Google's internal 12 system, Blaze, which is an ideal replacement for building such projects in a 13 fast and declarative fashion. But Bazel in itself is still in active 14 development, especially in terms of Python (amongst a few other languages). 15 16 The project aimed to fill this gap and build gRPC Python with Bazel. 17 18 [Project page](https://summerofcode.withgoogle.com/projects/#6482576244473856) 19 20 [Link to proposal](https://storage.googleapis.com/summerofcode-prod.appspot.com/gsoc/core_project/doc/5316764725411840_1522049732_Naresh_Ramesh_-_GSoC_proposal.pdf) 21 22 ## Thoughts and challenges 23 24 ### State of Bazel for Python 25 26 Although previously speculated, the project didn't require any contributions 27 directly to [bazelbuild/bazel](https://github.com/bazelbuild/bazel). The Bazel 28 rules for Python are currently being separated out into their own repo at 29 [bazelbuild/rules_python](https://github.com/bazelbuild/rules_python/). 30 31 Bazel is [still very much in active development for 32 Python](https://groups.google.com/forum/#!topic/bazel-sig-python/iQjV9sfSufw) 33 though. There's still challenges when it comes to building for Python 2 vs 3. 34 Using pip packages is still in experimental. Bazel Python support is currently 35 distributed across these two repositories and is yet to begin migration to one 36 place (which will be 37 [bazelbuild/rules_python](https://github.com/bazelbuild/rules_python/)). 38 39 Bazel's roadmap for Python is publicly available [here as a Google 40 doc](https://docs.google.com/document/d/1A6J3j3y1SQ0HliS86_mZBnB5UeBe7vExWL2Ryd_EONI/edit). 41 42 ### Cross collaboration between projects 43 44 Cross contribution surprisingly came up because of building protobuf sources 45 for Python, which is still not natively supported by Bazel. An existing 46 repository, [pubref/rules_protobuf](https://github.com/pubref/rules_protobuf), 47 which was maintained by an independent maintainer (i.e. not a part of Bazel) 48 helped solve this problem, but had [one major blocking 49 issue](https://github.com/pubref/rules_protobuf/issues/233) and could not be 50 resolved at the source. But [a solution to the 51 issue](https://github.com/pubref/rules_protobuf/pull/196) was proposed by user 52 dududko, which was not merged because of failing golang tests but worked well 53 for Python. Hence, a fork of this repo was made and is to be used with gRPC 54 until the solution can be merged back at the source. 55 56 ### Building Cython code 57 58 Building Cython code is still not supported by Bazel, but the team at 59 [cython/cython](https://github.com/cython/cython) have added support for Bazel 60 on their side. The way it works is by including Cython as a third-party Bazel 61 dependency and using custom Bazel rules for building our Cython code using the 62 binary within the dependency. 63 64 ### Packaging Python code using Bazel 65 66 pip and PyPI still remain the de-facto standard for distributing Python 67 packages. Although Bazel is pretty versatile and is amazing for it's 68 reproducible and incremental build capabilities, these can only be still used 69 by the contributors and developers for building and testing the gRPC code. But 70 there's no way yet to build Python packages for distribution. 71 72 ### Building gRPC Python with Bazel on Kokoro (internal CI) 73 74 Integration with the internal CI was one of the areas that highlighted how 75 simple Bazel can be to use. gRPC was already using a dockerized Bazel setup to 76 build some of it's core code (but not as the primary build setup). Adding a new 77 job on the internal CI ended up being as simple as creating a new shell script 78 to install the required dependencies (which were python-dev and Bazel) and a 79 new configuration file which pointed to the subdirectiory (src/python) under 80 which to look for targets and run the tests accordingly. 81 82 ### Handling imports in Python code 83 84 When writing Python packages, imports in nested modules are typically made 85 relative to the package root. But because of the way Bazel works, these paths 86 wouldn't make sense from the Workspace root. So, the folks at Bazel have added 87 a nifty `imports` parameter to all the Python rules which lets us specify for 88 each target, which path to consider as the root. This parameter allows for 89 relative paths like `imports = ["../",]`. 90 91 ### Fetching Python headers for Cython code to use 92 93 Cython code makes use of `Python.h`, which pulls in the Python API for C 94 extension modules to use, but it's location depending on the Python version and 95 operating system the code is building on. To make this easier, the folks at 96 Tensorflow wrote [repository rules for Python 97 autoconfiguration](https://github.com/tensorflow/tensorflow/tree/e447ae4759317156d31a9421290716f0ffbffcd8/third_party/py). 98 This has been [adapted with some some 99 modifications](https://github.com/grpc/grpc/pull/15992) for use in gRPC Python 100 as well. 101 102 ## How to use 103 104 All the Bazel tests for gRPC Python can be run using a single command: 105 106 ```bash 107 bazel test --spawn_strategy=standalone --genrule_strategy=standalone //src/python/... 108 ``` 109 110 If any specific test is to be run, like say `LoggingPoolTest` (which is present 111 in 112 `src/python/grpcio_tests/tests/unit/framework/foundation/_logging_pool_test.py`), 113 the command to run would be: 114 115 ```bash 116 bazel test --spawn_strategy=standalone --genrule_strategy=standalone //src/python/grpcio_tests/tests/unit/framework/foundation:logging_pool_test 117 ``` 118 119 where, `logging_pool_test` is the name of the Bazel target for this test. 120 121 Similarly, to run a particular method, use: 122 123 ```bash 124 bazel test --spawn_strategy=standalone --genrule_strategy=standalone //src/python/grpcio_tests/tests/unit/_rpc_test --test_arg=RPCTest.testUnrecognizedMethod 125 ``` 126 127 ## Useful Bazel flags 128 129 - Use `bazel build` with a `-s` flag to see the logs being printed out to 130 standard output while building. 131 - Similarly, use `bazel test` with a `--test_output=streamed` to see the the 132 test logs while testing. Something to know while using this flag is that all 133 tests will be run locally, without sharding, one at a time. 134 135 ## Contributions 136 137 ### Related to the project 138 139 - [435c6f8](https://github.com/grpc/grpc/commit/435c6f8d1e53783ec049b3482445813afd8bc514) 140 Update grpc_gevent cython files to include .pxi 141 - [74426fd](https://github.com/grpc/grpc/commit/74426fd2164c51d6754732ebe372133c19ba718c) 142 Add gevent_util.h to grpc_base_c Bazel target 143 - [b6518af](https://github.com/grpc/grpc/commit/b6518afdd610f0115b42aee1ffc71520c6b0d6b1) 144 Upgrade Bazel to 0.15.0 145 - [ebcf04d](https://github.com/grpc/grpc/commit/ebcf04d075333c42979536c5dd2091d363f67e5a) 146 Kokoro setup for building gRPC Python with Bazel 147 - [3af1aaa](https://github.com/grpc/grpc/commit/3af1aaadabf49bc6274711a11f81627c0f351a9a) 148 Basic setup to build gRPC Python with Bazel 149 - [11f199e](https://github.com/grpc/grpc/commit/11f199e34dc416a2bd8b56391b242a867bedade4) 150 Workspace changes to build gRPC Python with Bazel 151 - [848fd9d](https://github.com/grpc/grpc/commit/848fd9d75f6df10f00e8328ff052c0237b3002ab) 152 Minimal Bazel BUILD files for grpcio Python 153 154 ### Other contibutions 155 156 - [89ce16b](https://github.com/grpc/grpc/commit/89ce16b6daaad4caeb1c9ba670c6c4b62ea1a93c) 157 Update Dockerfiles for python artifacts to use latest git version 158 - [32f7c48](https://github.com/grpc/grpc/commit/32f7c48dad71cac7af652bf994ab1dde3ddb0607) 159 Revert removals from python artifact dockerfiles 160 - [712eb9f](https://github.com/grpc/grpc/commit/712eb9ff91cde66af94e8381ec01ad512ed6d03c) 161 Make logging after success in jobset more apparent 162 - [c6e4372](https://github.com/grpc/grpc/commit/c6e4372f8a93bb0eb996b5f202465785422290f2) 163 Create README for gRPC Python reflection package 164 - [2e113ca](https://github.com/grpc/grpc/commit/2e113ca6b2cc31aa8a9687d40ee1bd759381654f) 165 Update logging in Python to use module-level logger 166 167 ### Pending PRs 168 169 - BUILD files for all tests in 170 [tests.json](https://github.com/ghostwriternr/grpc/blob/70c8a58b2918a5369905e5a203d7ce7897b6207e/src/python/grpcio_tests/tests/tests.json). 171 - BUILD files for gRPC testing, gRPC health checking, gRPC reflection. 172 - (Yet to complete) BUILD files for grpcio_tools. One test depends on this. 173 174 ## Known issues 175 176 - [grpc/grpc #16336](https://github.com/grpc/grpc/issues/16336) RuntimeError 177 for `_reconnect_test` Python unit test with Bazel 178 - Some tests in Bazel pass despite throwing an exception. Example: 179 `testAbortedStreamStream` in 180 `src/python/grpcio_tests/tests/unit/_metadata_code_details_test.py`. 181 - [#14557](https://github.com/grpc/grpc/pull/14557) introduced a minor bug 182 where the module level loggers don't initialize a default logging handler. 183 - Sanity test doesn't make sense in the context of Bazel, and thus fails. 184 - There are some issues with Python2 vs Python3. Specifically, 185 - On some machines, cygrpc.so: undefined symbol: _Py_FalseStruct error 186 shows up. This is because of incorrect Python version being used to build 187 Cython. 188 - Some external packages like enum34 throw errors when used with Python 3 and 189 some extra packages are currently installed as Python version in current 190 build scripts. For now, the extra packages are added to a 191 `requirements.bazel.txt` file in the repository root. 192
