Stock Prediction Kafka
An end-to-end ML system that streams live prices and news through Kafka, scores headlines with DistilBERT and predicts price direction with XGBoost, all on free APIs.
What it does
A production-style pipeline that takes in live stock prices and news headlines, scores the sentiment of every headline, predicts whether each stock will go up or down, and shows it all on a dashboard that refreshes every five seconds. It runs entirely on a laptop with free APIs and no cloud account.
The pipeline
- Producers pull price bars from yfinance and headlines from NewsAPI and publish them to Kafka topics.
- Three consumer groups read the stream: DistilBERT scores every headline, XGBoost predicts UP or DOWN with a confidence score, and a monitor watches for drift, collapsing confidence and sentiment spikes.
- Results land in SQLite, are served by a FastAPI backend with eight REST endpoints, and are charted in a React and Recharts dashboard.
The model
XGBoost on 16 features: returns over 1, 5 and 10 bars, moving averages, MACD, RSI, Bollinger band width, volume ratio and rolling news sentiment, trained on two years of price history. The rest of the system talks to the model only through one predict module, so it can be swapped for LightGBM or a neural network without touching anything else.