Discovery Service

Question: A discovery service is one of the most powerful applications a team can have at its disposal. A discovery service maintains information about every service in the application estate, and this information can be queried via a discovery service client API. This enables services to easily obtain location details of other services, and communicate directly with them. No more excessive hardcoded config with connection details is necessary. We will build a discovery service using an asynchronous heartbeat model. Every client will send an asynchronous heartbeat to the discovery service on an interval, which the discovery service will receive and store in a ‘status’ table and ‘latestStatus’ table. Status will maintain the history of heartbeats, whereas latestStatus will maintain the latest heartbeat. A client which disconnects is considered inactive, whereas a client that has sent a heartbeat in the past and has not disconnected is considered active. The discovery service can even heartbeat to itself and maintain its own information, which we will do as well in this implementation. The discovery service should have functions defined to return host and port information of active services, and the client API should call these functions remotely synchronously. The below solution demonstrates a basic and portable discovery service/client API implementation that can be applied to any kdb+ estate.

Example

                                
                                // Start up discovery service and have it load client API so it can heartbeat to itself 
$ q discoServer.q -p 7001 -discoPort 7001 -name discovery_1 -serviceType discovery -t 5000 -region amrs -sendHB 1 
q)\l discoClient.q 
 
// Start up client ‘services’ (empty q sessions with client API loaded in) 
$ q discoClient.q -p 8000 -discoHost localhost -discoPort 7001 -name rdb_amrs_1 -serviceType rdb -t 5000 -region amrs -sendHB 1 > /dev/null 2>&1 & 
$ q discoClient.q -p 8001 -discoHost localhost -discoPort 7001 -name rdb_amrs_2 -serviceType rdb -t 5000 -region amrs -sendHB 1 > /dev/null 2>&1 & 
$ q discoClient.q -p 8002 -discoHost localhost -discoPort 7001 -name hdb_amrs_1 -serviceType hdb -t 5000 -region amrs -sendHB 1 > /dev/null 2>&1 & 
$ q discoClient.q -p 8003 -discoHost localhost -discoPort 7001 -name hdb_amrs_2 -serviceType hdb -t 5000 -region amrs -sendHB 1 > /dev/null 2>&1 & 
$ q discoClient.q -p 8004 -discoHost localhost -discoPort 7001 -name hdb_amrs_3 -serviceType hdb -t 5000 -region amrs -sendHB 1 > /dev/null 2>&1 & 
$ q discoClient.q -p 9000 -discoHost localhost -discoPort 7001 -name rdb_emea_1 -serviceType rdb -t 5000 -region emea -sendHB 1 > /dev/null 2>&1 & 
 
// Back in discovery service 
q)latestStatus 
name       | time                          pid  serviceType region hostname        port qVersion qVersionDate ipcHandle active 
-----------| ----------------------------------------------------------------------------------------------------------------- 
discovery_1| 2020.10.08D19:26:02.423369000 1559 discovery   amrs   desktop-71jn2qg 7001 4        2020.06.01   0         1 
hdb_amrs_2 | 2020.10.08D19:26:05.274433000 1574 hdb         amrs   desktop-71jn2qg 8003 4        2020.06.01   6         1 
rdb_amrs_1 | 2020.10.08D19:26:05.274455000 1566 rdb         amrs   desktop-71jn2qg 8000 4        2020.06.01   7         1 
hdb_amrs_3 | 2020.10.08D19:26:05.282092000 1577 hdb         amrs   desktop-71jn2qg 8004 4        2020.06.01   8         1 
hdb_amrs_1 | 2020.10.08D19:26:05.282112000 1570 hdb         amrs   desktop-71jn2qg 8002 4        2020.06.01   9         1 
rdb_emea_1 | 2020.10.08D19:26:05.374412000 1578 rdb         emea   desktop-71jn2qg 9000 4        2020.06.01   10        1 
rdb_amrs_2 | 2020.10.08D19:26:05.380388000 1568 rdb         amrs   desktop-71jn2qg 8001 4        2020.06.01   11        1 
q)status 
name        time                          pid  serviceType region hostname        port qVersion qVersionDate ipcHandle active 
----------------------------------------------------------------------------------------------------------------------------- 
discovery_1 2020.10.08D19:25:07.423366000 1559 discovery   amrs   desktop-71jn2qg 7001 4        2020.06.01   0         1 
discovery_1 2020.10.08D19:25:12.422955000 1559 discovery   amrs   desktop-71jn2qg 7001 4        2020.06.01   0         1 
discovery_1 2020.10.08D19:25:17.423374000 1559 discovery   amrs   desktop-71jn2qg 7001 4        2020.06.01   0         1 
discovery_1 2020.10.08D19:25:22.423375000 1559 discovery   amrs   desktop-71jn2qg 7001 4        2020.06.01   0         1 
discovery_1 2020.10.08D19:25:27.423375000 1559 discovery   amrs   desktop-71jn2qg 7001 4        2020.06.01   0         1 
hdb_amrs_2  2020.10.08D19:25:30.274206000 1574 hdb         amrs   desktop-71jn2qg 8003 4        2020.06.01   6         1 
rdb_amrs_1  2020.10.08D19:25:30.274225000 1566 rdb         amrs   desktop-71jn2qg 8000 4        2020.06.01   7         1 
hdb_amrs_3  2020.10.08D19:25:30.281500000 1577 hdb         amrs   desktop-71jn2qg 8004 4        2020.06.01   8         1 
hdb_amrs_1  2020.10.08D19:25:30.281659000 1570 hdb         amrs   desktop-71jn2qg 8002 4        2020.06.01   9         1 
rdb_emea_1  2020.10.08D19:25:30.373906000 1578 rdb         emea   desktop-71jn2qg 9000 4        2020.06.01   10        1 
rdb_amrs_2  2020.10.08D19:25:30.379487000 1568 rdb         amrs   desktop-71jn2qg 8001 4        2020.06.01   11        1 
discovery_1 2020.10.08D19:25:32.423191000 1559 discovery   amrs   desktop-71jn2qg 7001 4        2020.06.01   0         1 
hdb_amrs_2  2020.10.08D19:25:35.274438000 1574 hdb         amrs   desktop-71jn2qg 8003 4        2020.06.01   6         1 
rdb_amrs_1  2020.10.08D19:25:35.274460000 1566 rdb         amrs   desktop-71jn2qg 8000 4        2020.06.01   7         1 
hdb_amrs_1  2020.10.08D19:25:35.282184000 1570 hdb         amrs   desktop-71jn2qg 8002 4        2020.06.01   9         1 
.. 
 
// Open up new q session with client API loaded in and test API functions 
$ q discoClient.q -discoHost localhost -discoPort 7001 
q).discoClient.getServiceAll (1#`region)!1#`amrs 
`:desktop-71jn2qg:7001`:desktop-71jn2qg:8003`:desktop-71jn2qg:8000`:desktop-71jn2qg:8004`:desktop-71jn2qg:8002`:desktop-71jn2qg:8001 
q).discoClient.getServiceAll `region`serviceType!`amrs`rdb 
`:desktop-71jn2qg:8000`:desktop-71jn2qg:8001 
q).discoClient.getServiceAll `region`serviceType!`amrs`hdb 
`:desktop-71jn2qg:8003`:desktop-71jn2qg:8004`:desktop-71jn2qg:8002 
q).discoClient.getServiceAll `region`serviceType!`emea`hdb 
`symbol$() 
q).discoClient.getService `region`serviceType!`amrs`hdb 
`:desktop-71jn2qg:8003 
q).discoClient.getService `region`serviceType!`amrs`hdb 
`:desktop-71jn2qg:8002 
q).discoClient.getService `region`serviceType!`emea`hdb 
` 
q).discoClient.getServiceAll (1#`name)!enlist"rdb_amrs*" 
`:desktop-71jn2qg:8000`:desktop-71jn2qg:8001 
q).discoClient.getServiceAll `serviceType`qVersion!(`rdb;4) 
`:desktop-71jn2qg:8000`:desktop-71jn2qg:9000`:desktop-71jn2qg:8001
                                
                            

Solution

Tags:
api architecture asynchronous ipc
Searchable Tags
algorithms api architecture asynchronous c csv data structures dictionaries disk feedhandler finance functions ingestion ipc iterators machine learning math multithreading optimizations realtime shared library sql statistics streaming strings tables temporal utility websockets