Machine Learning Dictionary

A breakthrough technique that allows neural networks to focus on relevant parts of input data, similar to human attention. By learning which parts of the input are important for each output, it dramatically improves performance in tasks like translation and image captioning. This mechanism is a key component of transformer architectures.

Automatic differentiation (autograd) is a computational technique that automatically calculates derivatives of functions defined by computer programs. Unlike symbolic differentiation (which manipulates mathematical expressions) or numerical differentiation (which approximates derivatives using finite differences), autograd computes exact derivatives efficiently by applying the chain rule systematically during program execution.

In machine learning, autograd is fundamental to training neural networks through gradient-based optimization. It enables frameworks like PyTorch, TensorFlow, and JAX to automatically compute gradients of loss functions with respect to model parameters, eliminating the need for manual derivative calculations. This automation is crucial for deep learning, where models may have millions or billions of parameters.

Autograd works by tracking operations performed on tensors and building a computational graph that records how outputs depend on inputs. During the backward pass, it traverses this graph in reverse order, applying the chain rule to compute gradients efficiently. This process, combined with backpropagation, enables the training of complex neural architectures that would be impractical to differentiate manually.

Modern autograd systems support both forward-mode and reverse-mode automatic differentiation, with reverse-mode (used in backpropagation) being particularly efficient for functions with many inputs and few outputs, which is typical in machine learning scenarios.

Simple Examples:

1. Strong Positive Correlation (r ≈ 0.9): Height and weight in a population. As height increases, weight tends to increase proportionally.

2. Moderate Positive Correlation (r ≈ 0.4): Study hours and test scores. More study time generally leads to better scores, but other factors also influence performance.

3. No Correlation (r ≈ 0): Shoe size and intelligence. These variables have no meaningful linear relationship.

4. Moderate Negative Correlation (r ≈ -0.4): Age of a car and its resale value. Older cars typically have lower resale values, though condition and other factors matter.

5. Strong Negative Correlation (r ≈ -0.8): Outdoor temperature and home heating usage. As temperature drops, heating usage increases substantially.

The core learning algorithm for neural networks that calculates how much each neuron contributed to the error, then adjusts weights backwards through the network. It's like tracing back through a chain of decisions to identify and correct mistakes, enabling efficient network training.

[The Most Important Algorithm in Machine Learning](https://www.youtube.com/watch?v=SmZmBKc7Lrs&ab_channel=ArtemKirsanov)

Also known as CANDECOMP/PARAFAC decomposition, CP breaks down a tensor into a sum of rank-one tensors (outer products of vectors). This decomposition provides a highly interpretable representation where each component represents a distinct pattern or factor in the data.

CP decomposition serves as a powerful unsupervised learning tool for discovering latent factors in multi-way data, with applications in chemometrics (analyzing chemical measurements), neuroscience (identifying functional networks), and recommendation systems (capturing user-item-context interactions).

A fundamental rule in calculus for finding the derivative of composite functions. The chain rule states that if you have a composite function f(g(x)), then its derivative is the derivative of the outer function evaluated at the inner function, multiplied by the derivative of the inner function.

Mathematically expressed as:
$$\frac{d}{dx}[f(g(x))] = f'(g(x)) \cdot g'(x)$$

Or in Leibniz notation:
$$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$$

The chain rule is essential for differentiating complex functions and is widely used in calculus, physics, engineering, and machine learning (particularly in backpropagation algorithms for neural networks). Common applications include finding derivatives of exponential functions, trigonometric functions with inner functions, and nested polynomial expressions.

A specialized neural network architecture inspired by the visual cortex. It uses sliding filters to automatically learn and detect important features in grid-like data (especially images), making it powerful for tasks like facial recognition, object detection, and medical image analysis.

A loss function commonly used in classification problems, particularly for multi-class classification and neural networks. Cross entropy loss measures the difference between the predicted probability distribution and the true distribution (one-hot encoded labels). It penalizes confident wrong predictions more heavily than uncertain predictions.

Mathematically, for a single sample with true class y and predicted probabilities p, the cross entropy loss is:

$$L = -\sum_{i=1}^{C} y_i \log(p_i)$$

where C is the number of classes. For binary classification, this simplifies to:

$$L = -[y \log(p) + (1-y) \log(1-p)]$$

Cross entropy loss is particularly effective because it provides strong gradients when predictions are wrong and approaches zero as predictions become more accurate. It's widely used in deep learning for training neural networks on classification tasks, often combined with softmax activation in the output layer.

• Definition: Computational methods used to separate mixed biological signals from heterogeneous samples into their constituent components.

• Applications:
- Cell type composition estimation from bulk tissue transcriptomics data
- Tumor microenvironment characterization from mixed tumor samples
- Immune cell profiling from complex tissue samples
- Epigenetic signal deconvolution from mixed cell populations

• Key methodologies:
- Reference-based deconvolution: Uses known cell type-specific signatures as reference
- Reference-free deconvolution: Identifies cell types without prior knowledge using statistical approaches
- Semi-supervised approaches: Combines reference data with unsupervised learning
- Spatial deconvolution: Incorporates spatial information to resolve cellular heterogeneity

• Algorithms and tools:
- CIBERSORT: Estimating immune cell fractions from gene expression profiles
- CellMix: R package for linear unmixing of heterogeneous tissue samples
- MuSiC: Multi-subject single cell deconvolution
- DSA (Digital Sorting Algorithm): Marker-free deconvolution for transcriptomics

• Challenges and considerations:
- Reference dataset quality and comprehensiveness
- Assumption of linear mixing in most algorithms
- Handling of technical and biological noise
- Validation of deconvolution results with orthogonal methods

A specialized form of machine learning using multiple layers of neural networks. It's particularly powerful for complex tasks like understanding images, text, and speech. Deep learning has enabled breakthrough applications like ChatGPT and stable diffusion models.

A hybrid regression technique that combines the penalties of both Lasso and Ridge Regression, incorporating both L1 and L2 regularization terms. This balanced approach overcomes limitations of each method alone: it can select variables like Lasso while handling groups of correlated features better, similar to Ridge. The mixing parameter allows data scientists to tune the model between pure Lasso and pure Ridge behavior.

Elastic Net is particularly valuable for complex datasets with many correlated features, such as in genomics (where groups of genes may work together), neuroimaging (where brain regions have correlated activities), and recommendation systems (where user preferences show complex patterns).

A technique that converts discrete data (like words or categories) into dense vectors of continuous numbers. These learned representations capture semantic relationships and similarities, enabling AI models to process categorical data effectively. It's fundamental to modern NLP and recommendation systems.

In machine learning and neural networks, an epoch refers to one complete pass through the entire training dataset during the training process. During each epoch, the model sees every training example once and updates its parameters accordingly. Multiple epochs are typically required to train a model effectively, with the number of epochs being a hyperparameter that affects model performance and training time.

The process of taking a pre-trained model and adapting it to a specific task by training it on a smaller, task-specific dataset. This transfer learning approach saves computational resources and often yields better results than training from scratch.

An extension of the standard CP decomposition that incorporates different loss functions and constraints to handle various data types (binary, count, continuous) and missing values. GCP provides more flexibility for modeling complex real-world data with non-Gaussian characteristics.

In unsupervised learning, GCP enables robust pattern discovery in heterogeneous multi-way data, supporting applications like topic modeling across document collections, community detection in dynamic networks, and analyzing sparse, noisy biological measurements across multiple experimental conditions.

An AI architecture where two networks compete: one creates fake data, while the other tries to distinguish real from fake. This competition drives both to improve, resulting in increasingly realistic synthetic data. GANs have revolutionized AI-generated art, deepfakes, and synthetic data generation.

A fundamental optimization algorithm used to train machine learning models by iteratively adjusting parameters to minimize a loss function. The algorithm computes the gradient (partial derivatives) of the loss function with respect to each parameter and updates parameters in the direction opposite to the gradient, effectively moving downhill toward a minimum.

Mathematically, the update rule is:
$$\theta_{t+1} = \theta_t - \alpha \nabla J(\theta_t)$$

where θ represents parameters, α is the learning rate, and ∇J(θ) is the gradient of the loss function.

Main Variants:

Batch Gradient Descent: Uses the entire dataset to compute gradients at each step. Provides stable convergence but can be computationally expensive for large datasets.

**Stochastic Gradient Descent (SGD)**: Uses a single random sample to compute gradients at each step. Much faster per iteration and can escape local minima due to noise, but convergence is more erratic.

Mini-batch Gradient Descent: Uses small batches of samples (typically 32-256) to compute gradients. Balances the stability of batch gradient descent with the efficiency of SGD, making it the most commonly used variant in practice.

Gradient descent is the foundation of most machine learning optimization and is essential for training neural networks, linear regression, logistic regression, and many other models.

A linear regression technique that performs both variable selection and regularization to enhance prediction accuracy and interpretability. Lasso (Least Absolute Shrinkage and Selection Operator) adds an L1 penalty term to the cost function, which can shrink some coefficients exactly to zero, effectively removing less important features from the model. This feature selection capability makes Lasso particularly valuable for high-dimensional datasets where many features may be irrelevant or redundant.

Lasso Regression is widely used in fields like genomics (selecting relevant genetic markers), finance (identifying key economic indicators), and image processing (extracting important features while discarding noise).

An advanced RNN architecture that solves the "vanishing gradient" problem, allowing it to remember important information for longer sequences. It uses specialized gates to control information flow, making it excellent for tasks requiring long-term memory like language translation and speech recognition.

A mathematical function that measures how far a model's predictions are from the actual target values, providing a quantifiable way to assess model performance during training. The loss function calculates the "cost" or "error" of the model's current state, with lower values indicating better performance. Different types of problems require different loss functions: mean squared error for regression tasks, cross-entropy for classification, and specialized losses for tasks like object detection or generative modeling.

The choice of loss function is crucial as it directly influences how the model learns through gradient descent optimization. During training, the algorithm adjusts model parameters to minimize the loss function, effectively teaching the model to make better predictions. Common examples include mean absolute error (L1 loss), mean squared error (L2 loss), binary cross-entropy, categorical cross-entropy, and Huber loss. Modern deep learning often employs custom loss functions tailored to specific tasks, such as focal loss for handling class imbalance or perceptual loss for image generation tasks.

A subset of AI where systems improve their performance on a task through experience (data), without being explicitly programmed. It's crucial for applications like image recognition, recommendation systems, and natural language processing.

An approach to modeling that incorporates known physical, chemical, or biological processes to explain phenomena, rather than relying solely on statistical patterns in data. Unlike purely data-driven approaches, mechanistic models are built on theoretical understanding of the underlying mechanisms that generate the observed data. These models attempt to represent causal relationships and system dynamics based on first principles.

Mechanistic models are particularly valuable in scientific domains where understanding the "why" and "how" is as important as prediction accuracy. They offer greater interpretability and can often extrapolate beyond the range of observed data more reliably than black-box statistical models. In machine learning, mechanistic approaches are increasingly being combined with data-driven methods to create hybrid models that benefit from both theoretical knowledge and empirical patterns, especially in fields like computational biology, climate science, and physics-informed neural networks.

Flight Simulator:
A flight simulator uses mathematical equations to recreate the experience of flying a plane, demonstrating a real-world application of a mechanistic model.
Chromatography Model:
In chromatography, mechanistic models consider physical and biochemical effects like convection, dispersion, and adsorption, based on natural laws

A machine learning model (e.g., a neural network) can be trained to predict cell phenotypes (e.g., cell growth, cell death) based on the outputs of the mechanistic model, which include changes in protein localization, phosphorylation, and downstream effects

A statistical framework that incorporates both fixed effects (parameters that apply to an entire population) and random effects (parameters specific to individual groups or subjects within the data). These models are particularly valuable for analyzing hierarchical, nested, or longitudinal data where observations are not fully independent. Mixed-effects models account for both within-group and between-group variations, making them powerful tools for fields like medicine, ecology, and social sciences where data often has complex, multi-level structures.

By simultaneously modeling group-level and individual-level effects, these models provide more accurate parameter estimates and standard errors than traditional regression approaches when dealing with clustered data. They're especially useful for repeated measures designs, panel data, and any scenario where measurements are taken from the same subjects over time or under different conditions.

Example:
A mixed-effects model could be used to predict the progression of Alzheimer's disease by analyzing longitudinal MRI data, incorporating patient-specific factors and overall trends.

A fundamental type of feedforward neural network consisting of multiple layers of interconnected nodes (perceptrons). An MLP typically includes an input layer, one or more hidden layers, and an output layer, with each layer fully connected to the next. Unlike single-layer perceptrons, MLPs can learn non-linear relationships through their hidden layers and activation functions, making them capable of solving complex classification and regression problems. MLPs are the building blocks of deep learning and serve as the foundation for more sophisticated architectures like CNNs and RNNs. They're widely used in applications ranging from image recognition to financial modeling.

A computing system inspired by biological neural networks in human brains. It consists of interconnected nodes (neurons) that process and transmit information. Neural networks are fundamental to modern AI, enabling systems to learn patterns from data and make predictions or decisions.

An algorithm used to adjust the parameters of machine learning models during training to minimize the loss function. Optimizers determine how the model's weights and biases are updated based on the computed gradients, directly affecting the speed and quality of learning.

Common optimizers include:

**Gradient Descent (SGD)**: The fundamental optimizer that updates parameters in the direction opposite to the gradient.

Adam (Adaptive Moment Estimation): Combines momentum and adaptive learning rates, widely used for its robustness and efficiency.

RMSprop: Adapts learning rates based on recent gradient magnitudes, effective for non-stationary objectives.

AdaGrad: Adapts learning rates based on historical gradients, useful for sparse data.

Momentum: Accelerates SGD by adding a fraction of the previous update to the current one.

AdamW: A variant of Adam with decoupled weight decay for better regularization.

The choice of optimizer significantly impacts training convergence, stability, and final model performance. Modern deep learning frameworks typically default to Adam or its variants due to their adaptive nature and robust performance across various tasks.

A common problem in machine learning where a model learns the training data too precisely, including its noise and irregularities. This reduces its ability to generalize to new data, like memorizing answers instead of understanding the underlying concepts.

The Pearson Correlation Coefficient (PCC) is a statistical measure that quantifies the linear relationship between two continuous variables. It produces a value ranging from -1 to +1, where +1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship between the variables.

Mathematically, it is calculated as the ratio between the covariance of two variables and the product of their standard deviations, making it a normalized measurement of covariance. The formula is often expressed as:

$$
r = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n} (x_i - \bar{x})^2 \sum_{i=1}^{n} (y_i - \bar{y})^2}}
$$

In machine learning, the Pearson correlation coefficient serves several critical functions:

1. Feature Selection: It helps identify which features have strong relationships with the target variable, allowing data scientists to select the most relevant features for model training.

2. Multicollinearity Detection: It identifies highly correlated input features that might cause instability in models like linear regression.

3. Dimensionality Reduction: Understanding correlation patterns helps in techniques like Principal Component Analysis (PCA) to reduce the number of features while preserving information.

4. Data Exploration: It provides insights into relationships within the data, guiding further analysis and model selection.

The interpretation of correlation strength varies by field, but generally:
- Values between ±0.1 and ±0.3 indicate weak correlation
- Values between ±0.3 and ±0.5 indicate moderate correlation
- Values between ±0.5 and ±1.0 indicate strong correlation

It's important to note that Pearson correlation only captures linear relationships and is sensitive to outliers. For non-linear relationships or when dealing with ordinal data, alternative measures like Spearman's rank correlation coefficient may be more appropriate.

In practical machine learning applications, Pearson correlation is used in genomics to identify relationships between genes, in financial modeling to analyze market dependencies, and in recommendation systems to measure similarities between user preferences or items.

Simple Examples:

1. Strong Positive Correlation (r ≈ 0.9): Height and weight in a population. As height increases, weight tends to increase proportionally.

2. Moderate Positive Correlation (r ≈ 0.4): Study hours and test scores. More study time generally leads to better scores, but other factors also influence performance.

3. No Correlation (r ≈ 0): Shoe size and intelligence. These variables have no meaningful linear relationship.

4. Moderate Negative Correlation (r ≈ -0.4): Age of a car and its resale value. Older cars typically have lower resale values, though condition and other factors matter.

5. Strong Negative Correlation (r ≈ -0.8): Outdoor temperature and home heating usage. As temperature drops, heating usage increases substantially.

A technique that adds location information to data in neural networks, helping them understand the order and structure of sequences. It's crucial for transformer models to process ordered data like text or time series, as it provides context about where each piece of information belongs in the sequence.

The art and science of crafting effective inputs for large language models to achieve desired outputs. It involves understanding model behavior and using specific techniques to guide the model's responses, crucial for getting optimal results from AI systems.

A neural network designed for sequential data that maintains a "memory" of previous inputs. Like having a short-term memory, it processes information in order and uses past context to understand current inputs, making it suitable for tasks like text prediction and time series analysis.

A type of machine learning where an agent learns to make decisions by interacting with an environment. The agent receives rewards or penalties based on its actions, similar to how humans learn through trial and error. It's crucial for robotics, game AI, and autonomous systems.

A widely-used activation function in neural networks that outputs the input directly if it's positive, otherwise it outputs zero. Mathematically defined as f(x) = max(0, x), ReLU is simple yet effective at introducing non-linearity into neural networks while being computationally efficient. It helps solve the vanishing gradient problem that plagued earlier activation functions like sigmoid and tanh, allowing for faster training of deep networks. ReLU has become the default activation function for hidden layers in most modern neural network architectures, though variants like Leaky ReLU and ELU address some of its limitations, such as the "dying ReLU" problem where neurons can become permanently inactive.

A regularization technique that addresses multicollinearity in linear regression by adding an L2 penalty term to the cost function. Unlike Lasso, Ridge Regression shrinks coefficients toward zero but rarely sets them exactly to zero, keeping all features in the model while reducing their impact. This approach is particularly effective when dealing with highly correlated predictors, preventing the model from assigning excessive importance to any single variable.

Ridge Regression excels in scenarios where all features contribute to the outcome but need to be constrained to prevent overfitting, such as in economic forecasting, climate modeling, and biomedical research.

• Definition: A computational method that integrates multiple data types to create a unified patient similarity network, enabling more comprehensive analysis than single-data approaches.

• Algorithm principles:
- Constructs similarity networks for each data type separately
- Iteratively updates each network by fusing information from other networks
- Converges to a single integrated network that captures complementary information across data types
- Uses spectral clustering for patient stratification and subtype identification

• Applications in multi-omics integration:
- Cancer subtyping: Identifying disease subtypes by integrating genomic, transcriptomic, and clinical data
- Biomarker discovery: Finding robust biomarkers across multiple data platforms
- Patient stratification: Grouping patients with similar molecular profiles across different data types
- Drug response prediction: Integrating molecular and pharmacological data to predict treatment outcomes

• Advantages over single-data analysis:
- Increased statistical power through data integration
- Robustness to noise in individual data types
- Ability to capture complementary information across heterogeneous data
- Improved prediction accuracy for clinical outcomes

• Implementation considerations:
- Parameter selection (number of neighbors, fusion iterations)
- Data normalization across different platforms
- Computational efficiency for large datasets
- Visualization of integrated networks

A mathematical function that converts a vector of real numbers into a probability distribution, where each output value is between 0 and 1 and all outputs sum to 1. Softmax is commonly used as the final activation function in multi-class classification problems, transforming raw model outputs (logits) into interpretable probabilities for each class.

Mathematically defined as: $$\text{softmax}(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}$$ for all j, where x is the input vector. The exponential function ensures all outputs are positive, while the normalization by the sum creates a valid probability distribution. Softmax amplifies the differences between values - larger inputs receive disproportionately higher probabilities, making it useful for confident predictions.

Softmax is essential in neural networks for tasks like image classification (determining which of several objects appears in an image), natural language processing (predicting the next word from a vocabulary), and any scenario requiring probabilistic outputs across multiple mutually exclusive categories. It's often paired with cross-entropy loss during training to optimize classification performance.

A learning approach where the AI system is trained on labeled data (examples with known correct answers). It's like learning with a teacher who provides the correct answers. This is the most common form of machine learning, used in applications like spam detection and image classification.

A mathematical approach for breaking down multi-dimensional arrays (tensors) into simpler components, similar to how matrix factorization works for two-dimensional data. Tensor decompositions reveal underlying patterns and structures in complex multi-way data, making them powerful tools for dimensionality reduction, feature extraction, and latent factor discovery in unsupervised learning.

Tensor methods are particularly valuable for analyzing data with multiple aspects or modes (such as users-items-time in recommendation systems or subjects-features-time in neuroimaging), where traditional matrix methods would lose important multi-way relationships.

A specialized hardware accelerator developed by Google specifically for neural network machine learning. TPUs are application-specific integrated circuits (ASICs) designed to accelerate tensor operations, which are fundamental to deep learning models. Unlike CPUs and GPUs, TPUs are optimized for matrix operations with high computational throughput and lower precision, making them significantly faster and more energy-efficient for AI workloads.

TPUs play a crucial role in training and running Google's Gemini models, providing the massive computational power needed to process the enormous datasets required for large language models. Their architecture enables parallel processing at scale, reducing training time from weeks to days or hours, and allowing for more efficient inference when deploying these models in production environments.

The process of breaking text into smaller units (tokens) that can be processed by AI models. These tokens might be words, subwords, or characters. Modern systems often use subword tokenization to handle unknown words and reduce vocabulary size while maintaining meaning.

An influential neural network architecture that revolutionized natural language processing. It uses self-attention mechanisms to process sequential data, enabling better understanding of context in language. Transformers power most modern language models like GPT and BERT.

A higher-order extension of principal component analysis (PCA) that decomposes a tensor into a core tensor multiplied by a matrix along each mode. Tucker decomposition provides a more flexible representation than other tensor methods, allowing different ranks for different dimensions.

In unsupervised learning, Tucker decomposition excels at subspace learning and dimensionality reduction for multi-way data, enabling applications like multi-aspect data mining, anomaly detection in network traffic, and feature extraction from multi-modal signals.

A learning method where the AI system finds patterns in unlabeled data without explicit guidance. It's like discovering categories or relationships naturally. Important for data clustering, anomaly detection, and understanding hidden patterns in data.

A generative model that learns to compress data into a compact representation and then reconstruct it. Unlike regular autoencoders, VAEs learn smooth, continuous representations that allow meaningful data generation and manipulation. They're vital for tasks like image generation and data compression.

A specialized database designed to store and efficiently search through high-dimensional vectors (embeddings). These databases enable rapid similarity search and are crucial for modern AI applications like semantic search, recommendation systems, and image retrieval.

A specific type of embedding that maps words to vectors of real numbers, capturing semantic relationships between words. Similar words cluster together in the embedding space, allowing models to understand word meanings and relationships. Examples include Word2Vec and GloVe.