Projects/Windows CCAPI

An announcement has been sent to krbdev@mit.edu starting a review of this project. That review will conclude on 2008-03-18.

Comments can be sent to krbdev@mit.edu.

RPC Design for Windows CCAPI Clients and Server

Client/Server Transport

The CCAPI is a local service provided to local clients. The implementation is operating system independent, but the transport between the clients and server is not. The transport is what's described here.

External APIs

There are two external APIs -- CCAPI V2 and V3. Both use the transport described here.

Client Definition

The clients are threads of processes belonging to a single user's login session.

Server Definition

The server is for a single user logon session. If there are multiple user logon sessions on the same PC, a separate server runs for each user session.

Supported Platforms

The CCAPI is part of MIT Kerberos for Windows [KfW]. KfW runs on Windows XP, Windows Vista and Windows Terminal Server.

Conventions & clarifications

The CCAPI client acts as both an RPC client and RPC server and the CCAPI server acts as both an RPC client and RPC server.

• The RPC call from the CCAPI client to the CCAPI server is called the "request." In this mode, the CCAPI client is the RPC client and the CCAPI server is the RPC server. The CCAPI client calls the "request routine" and the RPC mechanism invokes the identically named "request handler" on the CCAPI server.
• The RPC call from the CCAPI server to the CCAPI client is called the "reply." In this mode, the CCAPI client is the RPC server and the CCAPI server is the RPC client. The CCAPI server calls the "reply routine" and the RPC mechanism invokes the identically named "reply handler" on the CCAPI client.
• When 'client' or 'server' appears without a qualifier, it means the CCAPI client or CCAPI server.

<UUID> is a UUID for a client thread.

<SST> means server start time, a time_t.

<LSID> is a logon-session-specific identifier.

The build environment is Visual Studio 2005.

Design Requirements

• The server's OS-independent code is single threaded, because it must operate on platforms that do not allow multiple threads. The server's RPC interface and the client are multithreaded.
• The client and server must be able to maintain connections, where state is maintained between individual messages.
• Individual messages must be handled in a single threaded server.
• The server must be able to detect when a client dies, so that any connection state can be cleaned up.

Design

The design of server endpoint names, server authentication and RPC thread safety is carried over from the previous CCAPI implementation. References to terminology from that implementation are annotated with '[CC-]' and assume some familiarity with the old code.

References to terminology from the new os-independent CCAPI implementation are annotated '[CC+]' and assume some familiarity with the new code.

The server and each client create an RPC endpoint. The server's endpoint is krbcc.<LSID>.ep [CC-] and the client's endpoint is CCAPI_<UUID>.

The server's ccs_pipe_t type [CC+] is a struct containing the UUID and a handle to the client's TSP. <TODO: Move this to implementation details.>

How is the request handled in the server and the reply sent to the client?

One straightforward way is for the reply to be the returned data in the request RPC call (an [out] parameter). That is, data passed from the RPC server to the RPC client. The request handler calls ccs_server_handle_request. Eventually, the server code calls ccs_os_server_send_reply, which saves the reply somewhere. When the server eventually returns to the request handler, the handler returns the saved reply to the client.

But this doesn't work. Consider client A taking a lock and then client B asking for the same lock. A single threaded server waiting for the lock on behalf of B will never be able to process A's unlock request.

Instead, the server asynchronously sends a reply for each request. Each client must wait for a reply to its request.

This will resolve the deadlock described above. The server will complete B's request. B will wait for a separate reply from the server that indicates that the lock has been acquired. The server will be free to process A's unlock request.

More details:

The Windows RPC mechanism runs each RPC procedure in its own thread.

Client threads use Microsoft Thread Local Storage to store a pointer to the client's thread-specific data [TSP].

Request and reply RPC calls include the client's UUID and TSP*.

The request handler in the CCAPI server builds a struct describing the request and adds it to an interlocked queue. This enables the server to handle simultaneous requests and requests that overlap with operations.

The server's main thread removes requests from the queue and processes them sequentially. The call into ccs_server_handle_request is the interface to the OS-independent CCAPI code [OSIC]. This call includes the request data and a ccs_pipe_t [CC+]. The Windows ccs_pipe_t is a ccs_win_pipe_t*. ccs_win_pipe_t is a struct containing the UUID and TSP*.

Eventually, the processing initiated in ccs_server_handle_request produces a call to ccs_os_server_send_reply. The OSIC passed the ccs_pipe_t along and ccs_os_server_send_reply makes the client endpoint from the included UUID and calls ccs_rpc_reply.

The client's cci_os_ipc function calls ccs_rpc_request and then waits for ccs_rpc_reply to set an event flag. How does the reply data go from the reply procedure thread to the client thread? One field in the TSP is a cci_stream_t*. The reply procedure builds the stream from the reply data, stores the cci_stream_t* in the TSP and sets a UUID-based event flag. Then the client can retrieve the stream* from the TSP.

CCAPI 'Connections'

Client state exists on the server. The server needs to know when the client is finished so it can clean up the client's state.

The Windows CCAPI client code manages a 'connection' to the server. This happens automatically, without any participation from the client application. The connection is established in cci_os_ipc() [CC+]when no connection was previously established. This 'lazy' technique avoids creating connections for client threads that do not use the CCAPI.

The connection is discarded when the client thread exits.

Detecting client exit

The server must be able to detect when clients disappear, so the server can free any resources that had been held for the client.

The Windows RPC API does not appear to provide a notification for an endpoint disappearing. It does provide a way to ask if an endpoint is listening. This is useful for polling, but we want a better performing solution than polling.

The CCAPI client's RPC interface includes a ccapi_listen function.

To detect the CCAPI client disappearing without using polling, the CCAPI server makes an asynchronous call to ccapi_listen, requesting a callback when the function completes. When the CCAPI client exits for any reason, it's endpoint will be closed and the RPC mechanism will call the CCAPI server's callback function. The server stored the client's UUID in the RPC asynchronous data structure, which the callback function can see. The callback function queues a DISCONNECT message which will be handled in the CCAPI server's main thread.

Windows provides a number of notification methods to signal I/O completion. Among them are I/O completion ports and callback functions. I chose callback functions because they appear to consume fewer resources.

Detecting server exit

When the server starts, it captures the current time in Server Start Time (SST).

ccs_rpc_connect_reply sends the SST to the client, which remembers it.

All subsequent requests from the client include the SST. If the server had exitted and restarted, it would have a new SST. The server's new SST wouldn't match the previous server's SST, which the client stored. The server would return an error indicating to the client that the previous connection was lost. <TODO: implement this!>

Thread safety

Server

The server spawns one thread to listen for RPC calls. This might not be necessary if the dontwait parameter is set.

The Windows RPC mechanism runs each RPC procedure in a separate thread. Each of these threads adds a message to the work queue. Access to the queue is protected with a critical section.

The Server's main thread removes items from the work queue and calls into the single threaded OSIC.

The only interaction between the multi-threaded RPC handlers and the single threaded OSIC is the work queue.

Client

A process could have multiple threads, each of which can be a CCAPI client. The CCAPI DLL is shared among threads and must be thread safe.

Threads use Microsoft Thread Local Storage to store and access necessary thread-specific data.

RPC calls are protected with the same code used in the previous CCAPI implementation.

RPC Endpoint / Function summary

The LSID is the same for the server and client(s).

The client funnels all CCAPI calls through cci_os_ipc_msg, which checks that the server is listening. If the server isn't listening, the client spawns it. Then the client uses CCAPI V2 code to authenticate the server.

Server endpoint functions

When started, the server creates one krbcc.<LSID>.ep endpoint to listen for connection requests and client requests. It has the functions

CC_UINT32 ccs_authenticate("<LSID>.auth")

This code is copied from the previous CCAPI implementation. It authenticates that the server is running as the same user as the client. The client caller puts a [random] value in a file <LSID>.auth and passes the name of the file to ccs_authenciate. If the server can read the file, it increments the value and returns it.

ccs_rpc_connect(msgtype, TSP*, UUID, status)

Each client checks a thread-specific bCCAPI_Connected flag before attempting to communicate with the server. If the flags is clear, the client creates a 'connection' by calling ccs_rpc_connect. The connection allocates server resources for the client.
msgtype CCMSG_CONNECT.
TSP* Address of the client's thread specific storage. The server doesn't do anything with this, it sends it back in the ccs_rpc_connect_reply message.
UUID The client's UUID. Used by the server to construct the client's endpoint so it know's which client to send the reply to.
status Return code indicating successful receipt of the call, not successful execution of the connect function on the server.

ccs_rpc_request(msgtype, TSP*, UUID, msglen, msg, SST, status)

Sends all the data associated with a CCAPI operation to the server.
msgtype CCMSG_REQUEST.
msg[msglen] The CCAPI message - a counted binary array, which will be passed into the os-independent server code.
TSP* Address of the client's thread specific storage. The server doesn't do anything with this, it sends it back in the ccs_rpc_connect_reply message.
UUID The client's UUID. Used by the server to construct the client's endpoint so it know's which client to send the reply to.
SST The time when the server started, as supplied by ccapi_rpc_connect_reply. If it is different from the SST the server has, it means the server has restarted since the connection was established.
status return code indicating successful receipt of the call, not successful execution of the connect function on the server.

Client endpoint functions

Each client thread creates a CCAPI_<UUID> endpoint. It has the functions

ccapi_rpc_connect_reply(msgtype, TSP*, UUID, SST, status)

Sends result of the connect request to the client. msgtype CCMSG_CONNECT_REPLY.
TSP* Address of the client's thread specific storage. The reply procedure uses TSP to know which thread's storage to put the reply.
UUID The client's UUID. Used by the reply procedure to construct an event name used to signal the client thread that a reply has been received.
SST The time when the server started. If it is different from the SST the client saved when connecting, it means the server has restarted since the connection was established.
status return code indicating successful receipt of the call, not successful execution of the reply function on the client.

ccapi_rpc_request_reply(msgtype, TSP*, UUID, SST, replylen, reply, status)

Operating system independent routine ccs_server_send_reply calls ccs_rpc_request_reply to send any data to be returned as a result of a CCAPI function call. The client will copy the reply data to a cci_stream that the client CCAPI code can interpret. There is exactly one ccapi_rpc_request_reply per ccs_rpc_request.
msgtype CCMSG_REQUEST_REPLY.
TSP* Address of the client's thread specific storage. The reply procedure uses TSP to know which thread's storage to put the reply.
UUID The client's UUID. Used by the reply procedure to construct an event name used to signal the client thread that a reply has been received.
SST The time when the server started. If it is different from the SST the client saved when connecting, it means the server has restarted since the connection was established.
reply[replylen] CCAPI data - a counted binary array. Any data the CCAPI function needs to return to the caller.
status return code indicating successful receipt of the call, not successful execution of the reply function on the client.

ccapi_listen(hBinding, msgtype, status)

Called asynchronously from Windows-specific code when a connection is being established. When the client exits, a completion routine will run in the server. This is how the server detects a client exiting.
hBinding Required RPC mechanism parameter.
msgtype CCMSG_
status return code indicating successful receipt of the call, not successful execution of the reply function on the client.

Test Plan

There are four dimensions to the test plan:

• User identity(ies) -- must test with
  • One user identity logon session
  • Different user identity logon sessions
  • Multiple logon sessions for one user identity (Scheduled tasks don't run in the logon session).
  • Multiple logon sessions for multiple user identities
• One or more threads per logon session
• Failure scenarios
  • User client thread is killed
  • User logon session is killed
  • Server is killed
    Server should restart on next client operation. Client should see that SST is different.

If client is waiting for a reply, should the wait time out?

• Operating systems
  • XP
  • Vista SP1
  • Terminal Server

Windows-specific implementation details

Client initialization

There are three parts to this:

• when the CCAPI DLL is loaded
• whenever a thread it inited
• when a thread wants to use the CCAPI

Client CCAPI library initialization

This code runs when the CCAPI DLL is loaded.

• Create process-wide mutex for making RPC calls thread safe. [Carried over from previous implementation.]
• Init Thread Local Storage.

Client CCAPI initialization

This code runs when cci_os_ipc_thread_init is called:

• Generate <UUID> and save in thread-specific storage. This serves as the client ID.
• Create client endpoint.
• Listen on client endpoint.
• Create canonical server connection endpoint from the <LSID>, which the client and server should have in common.
• Test if server is listening to the krbcc.<LSID>.ep endpoint.
  • If not, start the server.
• Call ccs_connect(<UUID>) on the krbcc.<LSID>.ep endpoint.
• Save SST in thread-specific storage.

Server initialization:

The server is initialized by client starting a new process. There should be only one server process per Windows username.

• As described above, when a client thread wants to use the CCAPI, it checks if the server is listening. If not, the client starts the server. This creates a server with the same LSID and privileges as the client.
• Capture server start time (SST).
• Start listener thread, create listener endpoint, listen on krbcc.<LSID>.ep endpoint.

Establishing a connection:

• Client calls ccs_connect(<UUID>) on server's krbcc.<LSID>.ep endpoint.
• Client gets back and stores SST in thread-specific storage.
• If new connection, server ...
  • adds connection to connection table (? Is it sufficient to call ccs_server_add_client?)
  • calls ccapi_listen on CCAPI_<UUID>.

Client request:

The server's reply to the client's request is not synchronous.

• Client calls ccs_rpc_request(msglen, msg, msgtype, UUIDlen, <UUID>, SST, status) on server's endpoint.
• Server listen thread receives message, queues request.
• Server worker thread dequeues request, processes, calls ccs_rpc_reply(msgtype, TSP*, UUID, SST, replylen, reply) on CCAPI_<UUID>.
• Server checks SST. If server's SST is different, it means server has restarted since client created connection.
• Client receives reply.

Detecting client exit

• When connection created, client created an endpoint.
• Server calls ccapi_listen on client's endpoint, specifying a notification callback when the call is marked completed (via RpcAsyncCompleteCall or client exit).
• The server's notification callback queues a DISCONNECT pseudo-message. When the server's worker thread handles the DISCONNECT, it will release connection resources.

Detecting server exit

Client's call to ccs_rpc_request will return an error if the server has gone away.

Data structures / RPC calls

Thread-specific data (TSP)

The TSP is per client thread.

struct tspdata {
   BOOL                _CCAPI_Connected;
   RPC_ASYNC_STATE*    _rpcState;
   HANDLE              _replyEvent;
   time_t              _sst;
   cci_stream_t        _stream;
   char                _uuid[UUID_SIZE];
   };

WIN_PIPE

The ccs_pipe_t is passed all the way through the server OSIC to ccs_os_ipc.

struct ccs_win_pipe_t {
   char*   uuid;
   HANDLE  clientHandle;
   };
typedef struct ccs_win_pipe_t  WIN_PIPE;
typedef struct ccs_win_pipe_t* ccs_pipe_t;

Request call

void ccs_rpc_request(
   [in]                        const long  rpcmsg,
   [in, size_is(HSIZE)]        const char  tsphandle[],
   [in, string]                const char* pszUUID,
   [in]                        const long  lenRequest,
   [in,  size_is(lenRequest)]  const char* pszRequest,
   [in]                        const long  serverStartTime,
   [out]                       long*       status );

Request reply call

void ccs_rpc_request_reply(
   [in]                    const long              rpcmsg,
   [in, size_is(HSIZE)]    const char              tsphandle[],
   [in, string]            const char*             uuid,
   [in]                    const long              srvStartTime,
   [in]                    const long              cbIn,        
   [in,  size_is(cbIn)]    const unsigned char     chIn[],      
   [out]                   long*                   status );    

Connect call

void ccs_rpc_connect(
   [in]                        const long  rpcmsg,    
   [in, size_is(HSIZE)]        const char  tsphandle[],
   [in,  string]               const char* pszUUID,    
   [out]                       long*       status );

Connect reply call

void ccs_rpc_connect_reply(
   [in]                    const long      rpcmsg,      
   [in, size_is(HSIZE)]    const char      tsphandle[],
   [in, string]            const char*     uuid,
   [in]                    const long      srvStartTime,
   [out]                   long*           status );

ccapi_listen call

void ccapi_listen(
   handle_t                hBinding,   
   [in]                    const long rpcmsg,
   [out]                   long* status );

Disconnect pseudomessage

The server's ccapi_listen async notification routine queues a Disconnect. The parameters to worklist_add are in the same order as in the WorkItem definition, below.

worklist_add(   CCMSG_DISCONNECT, 
                pipe,
                NULL,               /* No payload with connect request */
                (const time_t)0 );  /* No server session number with connect request */

Server WorkItem

Request, Connect and Disconnect messages are queued as WorkItems for the server's main thread to process.

class WorkItem {
private:
   const long            _rpcmsg;
         WIN_PIPE*       _pipe;
         cci_stream_t    _buf;
   const long            _sst;
   ...

To do

Overview

• Add CCAPI V2 compatibility description.

Code / implementation

• Be explicit about sizeof(SST) [64 bits].

Explorations

• Confirm or deny: If a user has tickets and then starts a process with elevated privileges, will the elevated process be able to communicate with the pre-existing credential cache?

Review

This section documents the review of the project according to Project policy. It is divided into multiple sections. First, approvals should be listed. To list an approval type

#~~~~

on its own line. The next section is for discussion. Use standard talk pageconventions. In particular, sign comments with

--~~~~

and indent replies.

Members of Krbcore raising Blocking objections should preface their comment with {{project-block}}. The member who raised the objection should remove this markup when their objection is handled.

Discussion

blocking There are significant problems with this project proposal.

• Actually act on all the todo items that are things that need to happen to this proposal before submitting for approval.
• For todo items that are things that need to happen to the code, open bugs targeted at the 1.7 release; the project page is mostly not supposed to change once approved unless design changes are needed.
• API descriptions need to actually describe what the API does. It's not enough to describe where it is called from; describe what its job is as well.
• What is ccs_authenticate?
• You need to follow API conventions I.E. ccs_* needs to be a server side routine.
• Why do you believe that having elevated process be unable to access tickets is the correct behavior for the product?
• The CCAPI server is not started by kfwlogon. IT may be started by a side effect but if you're not using kfwlogon it will be started by something else. Also, you should not have stuff marked [old]

--SamHartman 14:07, 4 March 2008 (EST)