Crypto Feed Handler

Question: There are data sources today that disseminate data real-time through the use of WebSockets. WebSockets are a modern day advancement which allow for real-time bidirectional communication between a server and a client via TCP/IP. WebSockets are fully asynchronous and allow for event driven responses, which is perfect for creating a feed handler. Coinbase Pro disseminates real-time cryptocurrency data via their WebSockets API.

Given the below trade table, subscribe to real-time ticker data for the following product ids: "BTC-USD,ETH-USD,XRP-USD,LTC-USD,BCH-USD,OXT-USD,XLM-USD,ATOM-USD,ETC-USD,LINK-USD,ALGO-USD,DAI-USD".

If you are receiving a 'con error, then you have SSL verification turned on. You need to either have the SSL_CA_CERT_FILE environment variable set to your CA cert file, or turn off SSL verification. Do either of the following below:

Download CA cert file and export environment variable
$ curl https://curl.haxx.se/ca/cacert.pem > $HOME/certs/cabundle.pem && export SSL_CA_CERT_FILE=$HOME/certs/cabundle.pem

Turn off SSL verification of the target server SSL certificate:
$ export SSL_VERIFY_SERVER=NO

More Information:

https://docs.pro.coinbase.com/#subscribe

Example

                                
                                q)trade:([]time:();product_id:();sequence:();price:();open_24h:();volume_24h:();low_24h:();high_24h:();volume_30d:();best_bid:();best_ask:();side:();trade_id:();last_size:())

// After running your code

q)trade
time                          product_id sequence     price    open_24h volume_24h   low_24h high_24h volume_30d   best_bid best_ask side trade_id     last_size
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
2020.05.17D16:22:55.144618000 BTC-USD    1.414503e+10 9791.45  9389.91  12892.92     9261.23 9896     645411.3     9790.16  9791.45  buy  9.223133e+07 0.00387904
2020.05.17D16:22:56.816019000 XRP-USD    9.193879e+08 0.2029   0.1993   2.665403e+07 0.1975  0.205    1.643819e+09 0.2029   0.203    sell 9147536      6898
2020.05.17D16:22:55.683882000 ETH-USD    9.140631e+09 208.31   201.01   109032.8     197.77  210      5854496      208.31   208.35   sell 5.834095e+07 0.07329224
2020.05.17D16:22:57.044887000 XLM-USD    1.072446e+09 0.070938 0.067764 2.978889e+07 0.06707 0.071784 2.099961e+09 0.070922 0.070938 buy  4288973      11161
2020.05.17D16:22:20.463744000 ETC-USD    9.94501e+08  6.721    6.711    248230.7     6.501   6.786    1.390851e+07 6.721    6.728    sell 4136261      11.61295
..

Solution

                                
                                ######## solution.q ########

trade:([]time:();product_id:();sequence:();price:();open_24h:();volume_24h:();low_24h:();high_24h:();volume_30d:();best_bid:();best_ask:();side:();trade_id:();last_size:());
.z.wc:{-1"Disconnected, hdl:",string x;};
.z.ws:{
    data:update `$product_id, "F"$price, "F"$open_24h, "F"$volume_24h, "F"$low_24h, "F"$high_24h, "F"$volume_30d, "F"$best_bid, "F"$best_ask, `$side, "P"$-1_time, "F"$last_size from .j.k x;
    .[insert;(`trade;value cols[trade]#data);()];
 };
con:{@[;0](`$":wss://ws-feed.pro.coinbase.com")"GET / HTTP/1.1\r\nHost: ws-feed.pro.coinbase.com\r\n\r\n"};
sub:{[h;products]
    neg[h] .j.j `type`channels!(`subscribe;enlist (`name`product_ids!(`ticker;(),products)));
 };
unsub:{[h;products]
    neg[h] .j.j `type`product_ids`channels!(`unsubscribe;(),products;(),`ticker);
 };

// connect and subscribe to products
h:con[];
products:`$("BTC";"ETH";"XRP";"LTC";"BCH";"OXT";"XLM";"ATOM";"ETC";"LINK";"ALGO";"DAI"),\:"-USD";
sub[h;products];

Explanation: To establish a websocket connection, we need to send a HTTP request to the WebSocket endpoint. From this point forward, a WebSocket connection replaces the HTTP connection. To subscribe to the products of interest, we send a JSON string in the desired format by Coinbase. After sending this we start receiving incoming marketdata. The data is received in the .z.ws callback as a JSON string. Appropriately cast the fields and insert into the trade table. The first JSON will fail the insert because it is metadata regarding the subscription request. To unsubscribe, call unsub with the WebSocket handle and the product list.

Code:

https://bitbucket.org/alvikabir/dailyq/src/master/Advanced/crypto-feed-handler

Tags:

api dictionaries feedhandler realtime tables websockets

Searchable Tags

algorithms api architecture asynchronous c csv data structures dictionaries disk feedhandler finance functions ingestion ipc iterators machine learning math multithreading optimizations realtime shared library sql statistics streaming strings tables temporal utility websockets

daily q

Crypto Feed Handler

Example

Solution

Tags:

Searchable Tags