Projects/Windows CCAPI
Contents
RPC design for Windows CCAPI clients and server
The CCAPI is a local service provided to local clients. The implementation is operating system independent, but the transport between the clients and server is not. The transport is what's described here.
The clients are threads of processes belonging to a login session.
The proposal is for a single user; the solution is replicated for each user logged onto the PC.
-Kevin 13:44, 8 January 2008 (EST) This is a cut and paste of the original html document. Responses to Danny Mayer's email exchange have not been incorporated yet.
Conventions & clarifications
The CCAPI client acts as both an RPC client and RPC server and the CCAPI server acts as both an RPC client and RPC server.
- The RPC call from the CCAPI client to the CCAPI server is called the "request." In this mode, the CCAPI client is the RPC client and the CCAPI server is the RPC server. The CCAPI client calls the "request routine" and the RPC mechanism invokes the identically named "request handler" on the CCAPI server.
- The RPC call from the CCAPI server to the CCAPI client is called the "reply." In this mode, the CCAPI client is the RPC server and the CCAPI server is the RPC client. The CCAPI server calls the "reply routine" and the RPC mechanism invokes the identically named "reply handler" on the CCAPI client.
<UUID> is a UUID for a client thread.
<SST> means server start time, a time_t.
<LSID> is a logon-session-specific identifier.
Design Requirements
- The server's OS-independent code is single threaded, because it must operate on platforms that do not allow multiple threads.
- The client and server must be able to maintain connections, where state is maintained between individual messages.
- Individual messages must be handled in a single threaded server.
- The server must be able to detect when a client dies, so that any connection state can be cleaned up.
Design
The design of server endpoint names, server authentication and RPC thread safety is carried over from the previous CCAPI implementation. References to terminology from that implementation are annotated with '[CC-]' and assume some familiarity with the old code.
References to terminology from the new os-independent CCAPI implementation are annotated '[CC+]' and assume some familiarity with the new code.
The server and each client create an RPC endpoint. The server's endpoint is krbcc.<LSID>.ep [cc-] and the client's endpoint is CCAPI_<UUID>.
The server's ccs_pipe_t type [CC+] is a struct containing the UUID and a handle to the client's TSP. <TODO: Move this to implementation details.>
How is the request handled in the server and the reply sent to the client?
One straightforward way is for the reply to be the returned data in the request RPC call (an [out] parameter). That is, data passed from the RPC server to the RPC client. The request handler calls ccs_server_handle_request. Eventually, the server code calls ccs_os_server_send_reply, which saves the reply somewhere. When the server eventually returns to the request handler, the handler returns the saved reply to the client.
But this doesn't work. Consider client A taking a lock and then client B asking for the same lock. A single threaded server waiting for the lock on behalf of B will never be able to process A's unlock request.
Instead, the server asynchronously sends a reply for each request. Each client must wait for a reply to its request.
This will resolve the deadlock described above. The server will complete B's request. B will wait for a separate reply from the server that indicates that the lock has been acquired. The server will be free to process A's unlock request.
The client's cci_os_ipc function waits for ccs_reply. The client sends the request, including it's UUID, from which the server can construct the endpoint on which to call ccs_reply.
The server's listener thread listens for RPC requests. The request handler puts each request/reply endpoint in a queue and returns to the client.
The server's worker thread removes items from the queue, calls ccs_server_handle_request. ccs_server_handle_request takes both the request data and the client UUID . Eventually ccs_os_server_send_reply is called, with the reply data and client UUID in the reply_pipe. ccs_os_server_send_reply calls ccs_reply on the client's endpoint, which sends the reply to the client.
Connections
If the client wants state to be maintained on the server, the client creates a connection. When the connection is closed, the server cleans up any state associated with the connection.
Any given thread in an application process could want to create a connection. When cci_ipc_thread_init is called, the connection thread-local variables are initialized. New connections are created when cci_os_ipc() (via _cci_ipc_send) is called and no connection was previously established. Basically we lazily establish connections so the client doesn't talk to the server until it has to.
Detecting client exit
The server must be able to detect when clients disappear, so the server can free any resources that had been held for the client.
The Windows RPC API does not appear to provide a notification for an endpoint disappearing. It does provide a way to ask if an endpoint is listening. This is useful for polling, but we want a better performing solution than that.
The client has an isAlive function on its endpoint.
To detect the client disappearing without using polling, the server makes an asynchronous call to the isAlive function on the client's endpoint. The isAlive function never returns. When the client exits for any reason, it's endpoint will be closed and the server's function call will return an error. The asynchronous call on the server means no additional threads are used.
Windows provides a number of notification methods to signal I/O completion. Among them are I/O completion ports and callback functions. I chose callback functions because they appear to consume fewer resources.
RPC Endpoint / Function summary
- The server creates one CCS_<LSID> endpoint to listen for connection requests and client requests. It has the functions
- ccs_rpc_connect(msgtype, UUIDlen, <UUID>, status)
- ccs_rpc_request(msgtype, UUIDlen, <UUID>, msglen, msg, SST, status) called by client. NB: The windows server sets the in_client_pipe to the in_reply_pipe.
- Each client thread creates a CCAPI_<UUID> endpoint. It has the functions
- isAlive [function never returns.]
- ccs_rpc_request_reply(msgtype, SST, replylen, reply, status)
- ccs_rpc_connect_reply(msgtype, SST, status
Windows-specific implementation details
Client CCAPI library initialization:
This code runs when the CCAPI DLL is loaded.
- <placeholder>
Client initialization:
This code runs when cci_os_ipc_thread_init is called:
- Generate <UUID> and save in thread-specific storage. This serves as the client ID / ccs_pipe_t.
- Create client endpoint.
- Listen on client endpoint.
- Create canonical server connection endpoint from the <LSID>, which the client and server should have in common.
- Test if server is listening to the CCS_<LSID> endpoint.
- If not, quit. (! Start it?)
- Call ccs_connect(<UUID>) on the CCS_<LSID> endpoint.
- Save SST in thread-specific storage.
Server initialization:
[old]
Server is initialized by client starting a new process. There should be only one server process per Windows username. [new]
- Server is started by kfwlogon (as is done currently).
- Capture server start time (SST).
- Start listener thread, create listener endpoint, listen on CCS_<LSID> endpoint.
Establishing a connection:
- Client calls ccs_connect(<UUID>) on server's CCS_<LSID> endpoint.
- Client gets back and stores SST in thread-specific storage.
- If new connection, server ...
- adds connection to connection table
- calls isAlive on CCAPI_<UUID>.
- NB: isAlive never returns.
Client request:
The server's reply to the client's request is not synchronous.
- Client calls ccs_rpc_request(msglen, msg, msgtype, UUIDlen, <UUID>, SST, status) on server's endpoint.
- Server listen thread receives message, queues request.
- Server worker thread dequeues request, processes, calls ccs_rpc_reply(replylen, reply, msgtype, status) on CCAPI_<UUID>.
- Server checks SST. If server's SST is different, it means server has restarted since client created connection.
- Client receives reply.
Detecting client exit
- When connection created, client created an endpoint.
- Server calls isAlive on client's endpoint.
- When isAlive returns, the server's notification callback will be called. Call back routine queues a DISCONNECT pseudo-message. When the server's worker thread handles the DISCONNECT, it will release connection resources.
Detecting server exit
- Client's call to ccs_rpc_request will return an error if the server has gone away.
To do
Code / implementation
Notes to myself:
- Recover from weird error in RpcMgmtIsServerListening
- Convert makefiles to work with existing build system
Administration
Explorations
- Verify: If a user has tickets and then starts a process with elevated privileges, the elevated process will not be able to communicate with the pre-existing credential cache.
Review
This section documents the review of the project according to Project policy. It is divided into multiple sections. First, approvals should be listed. To list an approval type
- #~~~~
on its own line. The next section is for discussion. Use standard talk pageconventions. In particular, sign comments with
- --~~~~
and indent replies.
Members of Krbcore raising Blocking objections should preface their comment with {{project-block}}. The member who raised the objection should remove this markup when their objection is handled.
