Posts

Causal Modeling

Image
                                                               generated by meta ai Yes. In fact, causal  modelling  is one of the most advanced topics in quantitative finance and is becoming increasingly important because traditional ML models (LSTM, XGBoost, Transformers) often learn  correlations , whereas causal models aim  to discover why prices move. Since I am  pursuing an  MSc in Financial Engineering  at  WQU , I'll teach it at that level, covering  theory, mathematics, and Python implementations. Learning Roadmap We'll build from scratch. Correlation vs Causation Structural Causal Models (SCM) Directed Acyclic Graphs (DAGs) Causal Discovery Algorithms Do-Calculus (Pearl) Counterfactual Prediction Causal Forecasting Applying causal models to stock prediction Building an end-t...

Scaling a Python application to serve millions of users

Scaling a Python application to serve millions of users requires moving past single-server setups and bypassing Python’s Global Interpreter Lock (GIL) through modern architecture . True scale is achieved by making the application stateless, caching aggressively, offloading heavy lifting, and ensuring the database never becomes a bottleneck. 1. Master Concurrency and Framework Selection Python handles traffic differently depending on the chosen framework and runtime strategy: Use Async Frameworks : Transition from synchronous frameworks (like standard Flask or Django) to asynchronous frameworks like FastAPI or Sanic. Async frameworks handle thousands of concurrent I/O-bound connections on a single process using an event loop. Optimize WSGI/ASGI Servers : Run applications behind multi-process workers. For Django/Flask, use Gunicorn with or workers. For FastAPI, utilize Uvicorn with a defined number of worker processes to fully utilize multi-core CPU architectures. 2. Design for Horiz...

Beyond Spot-Checking: Why LLM Applications Require Specialized Evaluation

Image
  images generated by meta ai Building applications with Large Language Models (LLMs) feels deceptively fast at first. A single engineer can write a prompt or connect a database using Retrieval-Augmented Generation (RAG) and get a working prototype in a afternoon. However, moving that prototype into production is where the real challenge begins. Unlike traditional software that fails loudly with a stack trace when a bug occurs, LLMs fail silently and plausibly . A system can return confident answers that are completely hallucinated, subtly outdated, or entirely off-topic without throwing a single runtime error. Manual "vibes-based" spot-checking—asking 5 to 10 questions and assuming the app works—does not scale. Modern AI evaluation (often called LLM Eval ) replaces guesswork with structured measurement. Why Is LLM Evaluation Necessary? 1. Catching Silent Regressions When you tweak a prompt to fix one edge case, change your vector database's top- $k$ search paramete...