Week 25-26: Natural Language Processing (NLP)
NLP Intelligence Platform: From Fragmented NLP Scripts to a Unified Production Stack
Introduction
Most NLP tutorials stop at isolated notebooks: one file for tokenization, another for sentiment, a third for intent, none wired to HTTP, persistence, or observability.
week_25_26_aiml_integrated_projectis a self-contained capstone that consolidates classical text processing, transformer tokenization, embeddings, sequence modeling, sentiment analysis, intent routing, and response selection into one runnable system:
An installable Python package (
week2526_python) with a core engine, learning lab wrappers, and a product NLP pipelineA single FastAPI backend exposing a lab path for component exploration and a product path for analyze, predict, and experiment training
A React dashboard with live telemetry, a support-assistant workflow, a pipeline studio, and an operations console
Postgres-backed experiment metadata, PyTorch artifacts on disk, and production hardening (timeouts, structured logging, graceful ML degradation)
The engineer payoff is operational clarity: one
/analyzecall runs preprocess → intent → sentiment → response; training jobs becomeRunrows you poll from the dashboard; missing models fall back to lexicon and keyword rules instead of crashing the API.
Core Components
Each block below is a distinct capability inside this repository—not an external dependency.
Classical text pipeline
week2526_python.core.text_pipeline provides NLTK-backed normalization, tokenization, stemming, lemmatization, POS tagging, and TF-IDF feature extraction. nltk_bootstrap.py lazily downloads punkt and stopwords only when classical processing runs, keeping cold starts fast.
Subword tokenization
core.tokenization.distilbert and core.tokenization.subword wrap Hugging Face tokenizers (DistilBERT, BERT, GPT-2). Lab endpoints expose side-by-side token comparisons without loading full models on every health check.
Word embeddings
core.embeddings.glove loads GloVe vectors or falls back to a deterministic synthetic vocabulary for offline demos. glove_bootstrap.py caches vector files under GLOVE_CACHE_DIR on startup when WEEK2526_BOOTSTRAP_MISSING=true.
Character sequence model
core.sequence_models.char_lstm implements a character-level LSTM for text generation. Checkpoints live under artifacts/sequence/char_lstm.pt after a lab train call.
Sentiment stack
The sentiment module spans vocabulary building (vocabulary.py), Bi-LSTM architecture (model.py), training (training.py), lexicon scoring (lexicon.py), and inference (inference.py). Artifacts: artifacts/sentiment/best_model.pt, vocab.json, and metrics.json.
Intent classification
core.intent.classifier trains a DistilBERT head over six support intents. core.intent.lightweight provides keyword-based routing when no checkpoint exists or when WEEK2526_LIGHTWEIGHT_MODE=true. Checkpoint path: artifacts/intent/intent_classifier.pt.
Response selection
core.response_selection.selector maps intent + sentiment score to template replies, optionally reranking with embedding similarity when ML artifacts are present.
Product NLP engine
core.nlp_engine.pipeline implements analyze_text, predict_sentiment_product, system_status, and bootstrap_missing_assets. This is the orchestration layer the product API calls through app.services.model_loader.
Model registry loader
backend/app/services/model_loader.py lazy-loads sentiment and intent models behind a thread-safe registry, exposes readiness(), and never raises on missing artifacts—lexicon and lightweight intent are used instead.
Experiment and training orchestration
SQLAlchemy models (Experiment, Run, Artifact) plus async repositories manage product training. nlp_train_job.py schedules background jobs with asyncio.create_task and supports synchronous training via ?sync=true.
FastAPI surface and middleware
backend/app/main.py defines the only FastAPI() instance. Middleware includes request timeouts (30s default, 600s for long train routes), upload size limits (WEEK2526_MAX_UPLOAD_MB), JSON request logging with request_id and latency_ms, and environment-aware CORS.
React dashboard
The frontend (frontend/src/ui/) ships four views—Overview, Support Assistant, Pipeline Studio, and Operations—with sidebar navigation, live metric polling (4s global, 3s during active training runs), animated pipeline progress on analyze, and module cards grouped by topic (text processing, embeddings, sentiment, intent and response).
Architecture Overview
Surface table
http://localhost:3000
OperatorsMulti-panel NLP console proxied through Nginx
Architecture diagram
The diagram below shows how the dashboard, API layer, Python package tiers, and persistence volumes connect. External caches (NLTK, Hugging Face) feed the core package without blocking the product path in lightweight mode.
