We would like to alter our LOB data for our machine learning purposes.
We import the trade book of January 2017
import pandas as pd
import string
from prepare_data import *
project_dir = os.path.abspath(os.path.join(os.getcwd(), os.pardir))
filedates, filenames = read('GARAN')
with open(project_dir + '/DATA/GARAN_TRADE' + '/GARAN_2017-01.csv', 'rb') as input:
trade_book = pd.read_csv(input,index_col =False,names =list(string.ascii_uppercase)[:-5])
trade_book.head(5)
We are interested in the 'L' column, where we get information on the type of session at a given time. The times are given in the column 'J'. We first look at the types of sessions. We have matching, continuous and closing sessions:
trade_book = trade_book[['A','J','L']]
trade_book['L'].unique()
P_SUREKLI_ISLEM stands for 'continuous session'. We would like to include times in the order book which fall into continuous session times. To do that we determine the continuous session intervals and take the times inside these intervals only.
Let's take the day Jan 2nd as an example: First, we take the corresponding entries in the trade book for Jan 2nd and make sure that the times are ordered. We sort the entries by the time column 'J' to do that.
date = '2017-01-02'
if trade_book['J'].apply(lambda x: True if len(x)==15 else False).sum() == len(trade_book): #for November
trade_book['J'] = trade_book['J'].apply(make_clock)
trade_book_df = trade_book[trade_book['A']==date].sort_values(by=['J'])
print('Checking if trade book times are in ascending order...')
for i in range(len(trade_book_df)-1):
if get_time_inbetween(make_clock(trade_book_df.iloc[i]['J']),make_clock(trade_book_df.iloc[i+1]['J']),'ms') < 0:
raise Exception('Given trade book times are not correctly ordered.')
trade_book_df
The trade books of GARAN stock for 2017 have seconds precision except for November. In November we have miliseconds precision.
As it can be seen in the above data frame, the trade book has multiple entries with the same time and session type. To simplify things, we look at each unique time entry in the trade book whether the session type changes for that time point. Because if not, we could simply pass on one entry for each unique time entry with its corresponding session type:
print('Looking if trade book has one session type per second/ms...')
trade_book_prec = get_prec_of_clock(trade_book_df.iloc[0]['J'])
unique_seconds = trade_book_df['J'].unique()
for time in unique_seconds:
if len(trade_book_df[trade_book_df['J']==time]['L'].unique())!=1:
raise Exception('Session type changes inside one unit of precision: ', time)
trade_book_simplified = trade_book_df['L'].to_frame()
trade_book_simplified.index=pd.Index(trade_book_df['J'].apply(make_clock).values)
trade_book_simplified = get_only_last_entry(trade_book_simplified,trade_book_prec)
trade_book_simplified
Now, for instance, we have one time entry 09:55:08 with its session type instead of multiple identical ones.
We use the simplified trade book to get continuous session time blocks. We take the time entries from our LOB, which fall into these time blocks. Then, from the LOB with continuous session times, we only take the last entry of each minute. Note that while taking the last entry of each minute from the LOB with continuous session times, the missing minutes that fall into a continuous session time block are forward filled.
filename = filenames[filedates.index(date)]
df = get_df(filename).astype(float) #Importing LOB Data
df = df[df.columns[df.columns.isin([i for i in df.columns if not i.count('ord')])]] #eliminating order column
df.index = pd.Index(get_clocks(df)[-1], name='time')
times = get_continuous_trading_times(trade_book_simplified)
LOB_times = [] ; LOB_conti_ffilled = pd.DataFrame([])
print('Continuous session blocks: ',times)
for time_pair in times:
t1 = make_clock(time_pair[0]); t2 = make_clock(time_pair[1])
LOB_times_to_add = df.index[[True if get_time_inbetween(t1,i,'ms')>=0 and get_time_inbetween(i,t2,'ms')>=0 else False for i in df.index]].to_list()
LOB_times += LOB_times_to_add
LOB_conti_ffilled = pd.concat([LOB_conti_ffilled,get_only_last_entry(df.loc[LOB_times_to_add],'m',ffill=True)[0]],axis=0)
LOB_conti = df.loc[LOB_times]
Now we have the LOB with only the continuous session times:
LOB_conti
And the LOB with the continuous session minutes, which is also forward filled:
LOB_conti_ffilled
At this point we can also calculate the Liquidity Measures. We use here the LOB with continuous session entries to calculate them. We then forward fill the missing minutes.
df_liq = get_all(LOB_conti,LOB_conti.index[0])
data = []
for t in LOB_conti_ffilled.index:
if df_liq.index.to_list().count(t):
data.append(df_liq.loc[t].to_list())
else:
data.append([None]*len(df_liq.columns))
df_liq_ffilled = pd.DataFrame(data,index = LOB_conti_ffilled.index,columns=df_liq.columns).ffill()
df_liq_ffilled
As the last step of data preprocessing, we apply normalization on prices, sizes and times. Prices are normalized by the mid price of the first continuous session entry, in this case the entry at 10:00:00.123 in the above data frame. Bid and ask side volumes are normalized by the total volume on their respective side at each row. The time index is replaced by equidistant points between 0 and 1. The data frame below represents how each day's preprocessed data looks like:
midprice = (LOB_conti.iloc[0]['ask1'] + LOB_conti.iloc[0]['bid1'])*0.5
result_df = normalize(pd.concat([LOB_conti_ffilled,df_liq_ffilled], axis=1), midprice)
result_df = result_df[[i for i in result_df.columns if i!='mid']]
result_df
As a side note, after processing each day's data in this way results in entries of 420 minutes for each day, indicating 7 hours of continuous session time per day, except for the days shown below:
filenames = os.listdir(project_dir + '/CODES/dataset/LOB_LIQ_VARS/GARAN')
[filenames.pop(i) for i,k in enumerate(filenames) if k.split('.')[-1]!='npy']
filenames.sort()
for filename in filenames:
with open(project_dir + '/CODES/dataset/LOB_LIQ_VARS/GARAN/' + f'{filename}', 'rb') as input:
data = np.load(input,allow_pickle='TRUE').item()
assert data['X'].shape[0] == data['y'].shape[0]
no_of_entries = data['y'].shape[0] + 60
if no_of_entries != 420:
print(f'{filename[:-4]}: ',no_of_entries)
From the preprocessed data of each day, we would like to create 60 minute rolling windows and use it as input to the neural network. Each day's set of rolling windows is used isolated from another, i.e. forecasting takes place within one day's set and does not involve another day's rolling windows.
The quantities we would like to forecast are the midprice and expectation and variance of prices of the following minute of each rolling window. This would result in vector with a length of 5, since expectation and variance are calculated for both ask and bid side seperately.
Note that a day with 420 minute entries would result in a set of 360 rolling windows and not 361, since the last entry of the day is to be forecast.
Our model accepts a rolling window of 60 minutes as input, which corresponds to a tensor with shape (60, no. of features), where the features could be normalized time of entry, prices, volumes and the liquidity measures.
We can decide how many days one batch will consist of, for both training and validation batches. Our batch size would be then the total number of rolling windows of all days in the batch. This is demonstrated below, where we have chosen the number of training and validation days to be 10 and 2 respectively:
from torchsummary import summary
from train import *
cfg.TRAIN.BATCH_SIZE = 10
cfg.TRAIN.VAL_BATCH_SIZE = 2
cfg.LOB = True
cfg.LIQ_VARS = False
train_dataloader , val_dataloader = load_data()
for i in train_dataloader:
train = i
break
for i in val_dataloader:
val = i
break
print(f'No of days in training batch: {train_dataloader.batch_size}\n', \
f'No of days in validation batch: {val_dataloader.batch_size}\n')
print('Shapes for one batch: \n\n',
'TRAIN:\n',
f'training data input shape: {train.input.numpy().shape}', \
f'training data output shape: {train.target.numpy().shape}', \
'\n VALIDATION:\n',
f'validation data input shape: {val.input.numpy().shape}', \
f'validation data output shape: {val.target.numpy().shape}')
print('\nLoading model with one hidden layer.\n')
model = load_model()
print(f'Model output shape for input shape {train.input.numpy().shape}: \
\n {model(train.input).detach().numpy().shape}')
summary(model.float(), input_size=(60,21))
As mentioned in the previous section, the target data consists of the mid price, expectation and variance of bid and ask prices for each minute entry in the LOB, starting from the 61st trading minute for each day.
We will now explain how expectation and variance are calculated. We take the 61st minute of Jan 2nd, 11:00 as example.
We will concentrate only on the bid side, since the calculations are analogous for the ask side. At 11:00 we have the following bid prices and volumes:
_61st_entry = LOB_conti_ffilled.iloc[60]
bid_prices = _61st_entry[['bid'+ str(i) for i in range(1,cfg.LOB_LVL+1)]].to_list()
bid_vols = _61st_entry[['bsize'+ str(i) for i in range(1,cfg.LOB_LVL+1)]].apply(int).to_list()
print('Prices :', bid_prices)
print('Volumes :', bid_vols)
For each minute we would like to create a probability measure by normalizing the volumes with the total volume at the bid side and calculate the expectation and variance of bid price with it. We show it for 11:00:
bid_vols_norm = [i/sum(bid_vols) for i in bid_vols]; print('Normalized Volumes :', bid_vols_norm)
bid_expectation = sum([i*k for i,k in zip(bid_prices,bid_vols_norm)]) ; print('Expected price: ',bid_expectation)
bid_var = sum([i*k for i,k in zip([(l-bid_expectation)**2 for l in bid_prices],bid_vols_norm)])
print('Variance of price: ',bid_var)
Due to forward filling it will happen that information at the last minute of a rollling window is the same as the next minute's information, which would mean that the target data's information is present in that window. This is the case for the forward fills that take place after the first hour, i.e. the first 60 entries of a day.
Another reason for the window to possess its target could simply be that some entries in the LOB stay unchanged or reoccur at certain points.
We now check every rolling window whether it includes its target. While doing that, we also make sure that the next window's last entry has the information of the current window's target.
Here we also made sure that no error occurs due to floating point precision while changing between lists, numpy arrays and pytorch tensors.
from train import load_data
from tqdm.notebook import tqdm
cfg.TRAIN.SPLIT_RATIO=1
cfg.TRAIN.BATCH_SIZE = 1
cfg.TRAIN.VAL_BATCH_SIZE = 1
cfg.LOB = True
cfg.LIQ_VARS = False
filenames = os.listdir(cfg.STOCKS_DIR+f'/GARAN/')
[filenames.pop(i) for i,k in enumerate(filenames) if k.split('.')[-1]!='npy']
filenames.pop(filenames.index('midprices.npy'))
filenames.sort()
filedates = [*map(lambda x: x[14:-4],filenames)]
train_dataloader , val_dataloader = load_data()
train = [] ; target = []
for i in train_dataloader:
train.append(i.input)
target.append(i.target)
due_to_ffill = {i:[] for i in filedates}
due_to_LOB = {i:[] for i in filedates}
def get_vector(line,midprice_factor):
line = line.numpy()
bid_prices = line[1:10:2].tolist()
ask_prices = line[11:20:2].tolist()
mid_price = (line[1:10:2][0] + line[11:20:2][0])*midprice_factor*0.5
bid_prices = [i*midprice_factor for i in bid_prices]
ask_prices = [i*midprice_factor for i in ask_prices]
bid_vols_norm = line[2:11:2].tolist()
bid_expectation = sum([i*k for i,k in zip(bid_prices,bid_vols_norm)])
bid_var = sum([i*k for i,k in zip([(l-bid_expectation)**2 for l in bid_prices],bid_vols_norm)])
ask_vols_norm = line[12:21:2].tolist()
ask_expectation = sum([i*k for i,k in zip(ask_prices,ask_vols_norm)])
ask_var = sum([i*k for i,k in zip([(l-ask_expectation)**2 for l in ask_prices],ask_vols_norm)])
return [mid_price,bid_expectation,ask_expectation,bid_var,ask_var]
midprices = np.load(cfg.STOCKS_DIR+'/GARAN/midprices.npy',allow_pickle='TRUE').item()
for a,filename in tqdm(enumerate(filenames),total=253):
ffilled_times = np.load(cfg.STOCKS_DIR+f'/GARAN/{filename}',allow_pickle='TRUE').item()
data = train[a] #shape: (-1,60,21)
target_data = target[a]
midprice_factor = midprices['midprice'][a]
for b,(window,window_target) in enumerate(zip(data,target_data)):
if b<data.shape[0]-1:
#Here we make sure that no indexing error has taken place
assert get_vector(data[b+1][-1],midprice_factor) == window_target.numpy().tolist()
# Here we count the number of times when the target is inside its window, later we will only talk about
# the number of windows with its target data.
# The two are different since there are consecutive forward fills and there will be windows including
# its target multiple times in multiple entries because of that.
for c,line in enumerate(window):
if get_vector(line,midprice_factor) == window_target.numpy().tolist():
if ffilled_times['index'].count(b + c + 1):
due_to_ffill[filedates[a]].append([b,c])
else:
due_to_LOB[filedates[a]].append([b,c])
def sort(liste):
liste.sort()
return liste
for a,filename in tqdm(enumerate(filenames),total=253):
# ffills = np.load(cfg.STOCKS_DIR+f'/GARAN/{filename}',allow_pickle='TRUE').item()['index']
# assert [sort(list(set(sum(k)+1 for k in due_to_ffill[i]))) for i in due_to_ffill][a]==ffills or [59,74,166].count(a),a
ffills = []
for i in np.load(cfg.STOCKS_DIR+f'/GARAN/{filename}',allow_pickle='TRUE').item()['index']:
if i > 60:
ffills.append(i)
assert [sort(list(set(sum(k)+1 for k in due_to_ffill[i]))) for i in due_to_ffill][a]==ffills
windows_due_to_ffill = [[*set(k[0] for k in due_to_ffill[i])] for i in due_to_ffill]
windows_due_to_LOB = [[*set(k[0] for k in due_to_LOB[i])] for i in due_to_LOB]
no_of_windows_due_to_ffill = sum(map(len,windows_due_to_ffill))
no_of_windows_due_to_LOB = sum(map(len,windows_due_to_LOB))
total_data = torch.cat(target).shape[0]
print(f'Total count of data points: {total_data}')
print(f'Incidence of target inclusion due to forward filling: {(no_of_windows_due_to_ffill/total_data)*100}%')
print(f'Incidence of target inclusion due to recurrence of LOB entries: {(no_of_windows_due_to_LOB/total_data)*100}%')
print(f'Total incidence of target inclusion: {((no_of_windows_due_to_ffill+no_of_windows_due_to_LOB)/total_data)*100}%')
In total we have 90797 rolling windows and their target vectors consisting of midprice, bid and ask expectations and variances. The number of windows including its target's information is 146 (~ 0.16%), out of which 116 are due to forward filling (~ 0.13%).
Here we do not count the number of times when the target is inside its window but only the number of windows including its target. The two are different since there are consecutive forward fills and therefore some windows will be including its target multiple times in multiple entries.
Below we have the dates and times of forward filled entries, where the consecutive forward fills are highlighted:
ffill_dict = {}
def highlight(x):
indices = []
x_ = [make_clock(i) for i in x if i.count(':')]
for i in range(len(x_)-1):
if get_time_inbetween(x_[i],x_[i+1],'m')==1:
indices.extend([i,i+1])
return ['background-color: lightgreen' if indices.count(v) else '' for v in range(len(x))]
for filename in filenames:
data = np.load(cfg.STOCKS_DIR+f'/GARAN/{filename}',allow_pickle='TRUE').item()['time']
if data != []:
ffill_dict.update({filename[14:-4]: data})
pd.DataFrame.from_dict(ffill_dict,'index',columns=range(1,16)).fillna('').style.apply(highlight,axis=1)