1GRPC Server Reflection Protocol 2=============================== 3 4This document describes server reflection as an optional extension for servers 5to assist clients in runtime construction of requests without having stub 6information precompiled into the client. 7 8The primary usecase for server reflection is to write (typically) command line 9debugging tools for talking to a grpc server. In particular, such a tool will 10take in a method and a payload (in human readable text format) send it to the 11server (typically in binary proto wire format), and then take the response and 12decode it to text to present to the user. 13 14This broadly involves two problems: determining what formats (which protobuf 15messages) a server’s method uses, and determining how to convert messages 16between human readable format and the (likely binary) wire format. 17 18## Method reflection 19 20We want to be able to answer the following queries: 21 1. What methods does a server export? 22 2. For a particular method, how do we call it? 23Specifically, what are the names of the methods, are those methods unary or 24streaming, and what are the types of the argument and result? 25 26The first version of the protocol is here: 27https://github.com/grpc/grpc/blob/master/src/proto/grpc/reflection/v1/reflection.proto 28 29Note that a server is under no obligation to return a complete list of all 30methods it supports. For example, a reverse proxy may support server reflection 31for methods implemented directly on the proxy but not enumerate all methods 32supported by its backends. 33 34 35### Open questions on method reflection 36 * Consider how to extend this protocol to support non-protobuf methods. 37 38## Argument reflection 39The second half of the problem is converting between the human readable 40input/output of a debugging tool and the binary format understood by the 41method. 42 43This is obviously dependent on protocol type. At one extreme, if both the 44server and the debugging tool accept JSON, there may be no need for such a 45conversion in the first place. At the opposite extreme, a server using a custom 46binary format has no hope of being supported by a generic system. The 47intermediate interesting common case is a server which speaks binary-proto and 48a debugging client which speaks either ascii-proto or json-proto. 49 50One approach would be to require servers directly support human readable input. 51In the future method reflection may be extended to document such support, 52should it become widespread or standardized. 53 54## Protobuf descriptors 55 56A second would be for the server to export its 57google::protobuf::DescriptorDatabase over the wire. This is very easy to 58implement in C++, and Google implementations of a similar protocol already 59exist in C++, Go, and Java. 60 61This protocol mostly returns FileDescriptorProtos, which are a proto encoding 62of a parsed .proto file. It supports four queries: 63 1. The FileDescriptorProto for a given file name 64 2. The FileDescriptorProto for the file with a given symbol 65 3. The FileDescriptorProto for the file with a given extension 66 4. The list of known extension tag numbers of a given type 67 68These directly correspond to the methods of 69google::protobuf::DescriptorDatabase. Note that this protocol includes support 70for extensions, which have been removed from proto3 but are still in widespread 71use in Google’s codebase. 72 73Because most usecases will require also requesting the transitive dependencies 74of requested files, the queries will also return all transitive dependencies of 75the returned file. Should interesting usecases for non-transitive queries turn 76up later, we can easily extend the protocol to support them. 77 78### Reverse proxy traversal 79 80One potential issue with naive reverse proxies is that, while any individual 81server will have a consistent and valid picture of the proto DB which is 82sufficient to handle incoming requests, incompatibilities will arise if the 83backend servers have a mix of builds. For example, if a given message is moved 84from foo.proto to bar.proto, and the client requests foo.proto from an old 85server and bar.proto from a new server, the resulting database will have a 86double definition. 87 88To solve this problem, the protocol is structured as a bidirectional stream, 89ensuring all related requests go to a single server. This has the additional 90benefit that overlapping recursive requests don’t require sending a lot of 91redundant information, because there is a single stream to maintain context 92between queries. 93 94``` 95package grpc.reflection.v1alpha; 96message DescriptorDatabaseRequest { 97 string host = 1; 98 oneof message_request { 99 string files_for_file_name = 3; 100 string files_for_symbol_name = 4; 101 FileContainingExtensionRequest file_containing_extension = 5; 102 string list_all_extensions_of_type = 6; 103 } 104} 105 106message FileContainingExtensionRequest { 107 string base_message = 1; 108 int64 extension_id = 2; 109} 110 111message DescriptorDatabaseResponse { 112 string valid_host = 1; 113 DescriptorDatabaseRequest original_request = 2; 114 oneof message_response { 115 // These are proto2 type google.protobuf.FileDescriptorProto, but 116 // we avoid taking a dependency on descriptor.proto, which uses 117 // proto2 only features, by making them opaque 118 // bytes instead 119 repeated bytes fd_proto = 4; 120 ListAllExtensionsResponse extensions_response = 5; 121 // Notably includes error code 5, NOT FOUND 122 int32 error_code = 6; 123 } 124} 125 126message ListAllExtensionsResponse { 127 string base_type_name; 128 repeated int64 extension_number; 129} 130 131service ProtoDescriptorDatabase { 132 rpc DescriptorDatabaseInfo(stream DescriptorDatabaseRequest) returns (stream DescriptorDatabaseResponse); 133} 134``` 135 136Any given request must either result in an error code or an answer, usually in 137the form of a series of FileDescriptorProtos with the requested file itself 138and all previously unsent transitive imports of that file. Servers may track 139which FileDescriptorProtos have been sent on a given stream, for a given value 140of valid_host, and avoid sending them repeatedly for overlapping requests. 141 142| message_request message | Result | 143| --------------------------- | ----------------------------------------------- | 144| files_for_file_name | transitive closure of file name | 145| files_for_symbol_name | transitive closure file containing symbol | 146| file_containing_extension | transitive closure of file containing a given extension number of a given symbol | 147| list_all_extensions_of_type | ListAllExtensionsResponse containing all known extension numbers of a given type | 148 149At some point it would make sense to additionally also support any.proto’s 150format. Note that known any.proto messages can be queried by symbol using this 151protocol even without any such support, by parsing the url and extracting the 152symbol name from it. 153 154## Language specific implementation thoughts 155All of the information needed to implement Proto reflection is available to the 156code generator, but I’m not certain we actually generate this in every 157language. If the proto implementation in the language doesn’t have something 158like google::protobuf::DescriptorPool the grpc implementation for that language 159will need to index those FileDescriptorProtos by file and symbol and imports. 160 161One issue is that some grpc implementations are very loosely coupled with 162protobufs; in such implementations it probably makes sense to split apart these 163reflection APIs so as not to take an additional proto dependency. 164 165## Known Implementations 166 167Enabling server reflection differs language-to-language. Here are links to docs relevant to 168each language: 169 170- [Java](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md#enable-server-reflection) 171- [Go](https://github.com/grpc/grpc-go/blob/master/Documentation/server-reflection-tutorial.md#enable-server-reflection) 172- [C++](https://grpc.io/grpc/cpp/md_doc_server_reflection_tutorial.html) 173- [Python](https://github.com/grpc/grpc/blob/master/doc/python/server_reflection.md) 174- Ruby: not yet implemented [#2567](https://github.com/grpc/grpc/issues/2567) 175- Node: not yet implemented [#2568](https://github.com/grpc/grpc/issues/2568) 176