Jane Street Market EDA 📈
Jane Street Market Prediction Kaggle Competition
Project Definition 🏆
The project is based on Kaggle competition by Jane Street - Jane Street Market Prediction
"Buy low, sell high" sounds easy. In reality, we know trading is difficult to solve and even more so in today's fast financial markets. Developing strategy with machine learning model can help us maximize returns using market data from a major global stock exchange. Then the competition will take our predictiveness to model against future market returns and give feedback on the leaderboard. My goal is to explore financial area of data science and explore Kaggle community as much as possible.
In general, if one is able to generate a highly predictive model which selects the right trades to execute, they would also be playing an important role in sending the market signals that push prices closer to “fair” values. That is, a better model will mean the market will be more efficient going forward. However, developing good models will be challenging for many reasons, including a very low signal-to-noise ratio, potential redundancy, strong feature correlation, and difficulty of coming up with a proper mathematical formulation.
(src: https://www.kaggle.com/c/jane-street-market-prediction/overview/description)
Dataset: 6GB of Real world financial markets.
- anonymized set of features, feature_{0...129}, representing real stock market data.
- each row in the dataset represents a trading opportunity, for which you will be predicting an action value: 1 to make the trade and 0 to pass on it.
- each trade has an associated weight and resp, which together represents a return on the trade.
- date column is an integer which represents the day of the trade, while ts_id represents a time ordering.
- in addition to anonymized feature values, you are provided with metadata about the features in features.csv.
- more info: https://www.kaggle.com/c/jane-street-market-prediction/data
Problem: Predict 'Action' columns
Where 1 to make the trade and 0 to pass on it. The goal is minimize choosing bad trades and maximizing good trades.
Proposing a solution
Even the world's finest financial expert wouldn't be able to gain anything from this anonymized data. Hence we are forced to use machine learning or data science approach to solve this problem.
Since the goal is clear (maximize profit) we want to see which features seem important or strange through EDA and visualizations. We will also have to grasp understanding of the data so it is not just a big 6GB of numbers. This will help us decide what to do with missing values and do feature engineering to help the computer learn.
Then we are going to build classifier model to let the computer do the hard work of learning to make good predictions. Of course, these models will be terrible at first so we will have to supervise it closely with some hyperparameters and strict evaluations.
Finally, the Jane Street will evaluate our submission and tell us if it is any good : )
Metrics
The competition is evaluated on a utility score. Each row in the test set represents a trading opportunity for which you will be predicting an action value. Each trade j has an associated weight and resp, which represents a return. Weight and resp are not defined on test data set.
For each date i, we define:
$pi=∑_{j}(weight_{ij}∗resp_{ij}∗action{i_j})$
$t=\dfrac{∑p_i}{∑p_{i}^2}\sqrt{\dfrac{250}{|i|}}$
where |i| is the number of unique dates in the test set. The utility is then defined as: $u=min(max(t,0),6)∑pi.$
df = dt.fread('../../../Kaggle/Jane-Street-Market-Prediction/input/train.csv')
df = df.to_pandas()
feat = pd.read_csv("../../../Kaggle/Jane-Street-Market-Prediction/input/features.csv")
df.describe()
feat.describe()
As told, all the featues and even tags are anonymized. There's not much human interpretability just from describe tables. Except feature_0 is unique by being binary.
A lot of the histogram of above features has extreme outliers. For the full enlarged version of the histograms, check out here It would be safe to fill the null values with medians. Other imputation method considered were mean and KNN-Imputation. Check out my other notebook where KNN-Imputation was used to train MLP.
If we just remove all nans, we would be removing more than 16.54% of the dataset.
Interesting points so far:
- feature_0 is binary.
- A lot of features seems to be normally distributed.
- A lot of missing values.
We can see that resp is closely related to resp_4 (blue and purple). Resp_1 and resp_2 also seem to be closely related but much much linear. Resp_3 seem to be in the middle, where the shape is closer to upper group but position is slightly closer to green and orange.
Note: weight and resp multiplied together represents a return on the trade.
We can see that most weights are around 0.2 and we can see two 'peaks' which is around 0.2 and 0.3. Note that maximum weight was 167.29 represented by 1.0 on x-axis. Thus 0.2 represents around 33.458 and 0.3 represents around 50.187.
Note that the graph plots all the positive gains. (Our 1's for our action column). So we can see that there were 'bigger' gains in the beginning and as time approach 500, the gain becomes smaller. In conclusion, the earlier trades are much bigger but we don't know what it's going to be like in our competition test set.
We know that we probability want to invest more 'weight' if there are bigger 'resp'(return). We learn here that higher weights are only when resp is close to 0. In other words, it is dumb to trade if resp is away from 0 but it is safe to invest even a lot if it is near 0.
In the Kaggle community, there's been lots of discussion on how the trends changed significantly since day ~85. We can see much more trades happening before day 100. Rest of the days are still very active but not as noisy. We can suggest that there has been a change of trading model from Jane Street as discussed here by Carl.
Let us look at the most important feature, 'feature_0'
df['feature_0'].value_counts()
Interestingly, when feature_0 is 1, plot shows negative slope while in contrast, when feature_0 is -1, plot shows positive slope. My guess is that feature_0 corresponds to Buy(1) and Sell(-1) or vice versa. So if we set action to 1 with feature_0 = 1 then we are selling and when we set action to 0 with feature_0 = -1, then we are buying. This makes sense since whether we are buying or selling we can still lose or gain profit.
Remember that we have another file called features.csv. Which can help us understand 100+ features and maybe cluster into groups. Let's take a look.
Let us see what tag_0 groups tells us.
Correlation between features of tag_0. It looks like there certainly are correlation between elements of the group except a few.
Interesting points:
- feature_0 has no tags
- feature 79 to 119 all has 4 tags
- feature 7 to 36 have 3 and 4 tags periodically
- Similar trend between 2 to 7, 37 to 40, 120 to 129
- tag_n doesn't tell too much about the features
In another notebook.
Implementation Planning
Thoughts going into predicting phase.
- Days before ~100 can be dropped as suspicion of model shift.
- Feature_0 seem very important to find slope of cummulative resp.
- Resp near 0 is prefered over other values.
- A lot of features are normally distributed.
- We have over 2 million datas, it would be safe to add lot more features(feature enginerring)
- There are a lot of missing values too. Can try mean, median or KNN imputation methods.
- Note that although this is kind of a time series data, we can only predict with features 0 to 129