How We Learn Value While Making Quick Decisions

How does Reinforcement Learning (RL) describe the process of making decisions based on personal experience?

RL describes how the subjective value associated with a decision option is updated over time following the receipt of rewards or punishments. This allows the decision-maker to integrate past experiences with new feedback.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What are the two standard components of most Reinforcement Learning (RL) computational models?

RL models typically consist of a learning component, which updates the subjective values of options using feedback, and a choice model, which maps those values to final choice probabilities. This division separates value updating from decision selection.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

Why is a simple mapping function, such as the soft-max choice rule, insufficient for fully describing decision-making?

Although the soft-max rule fits choice probabilities well, it does not provide a mechanistic description of the cognitive processes that occur during a decision. For instance, it cannot capture how decision conflict or response times vary.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How can analyzing choice data and response times (RTs) simultaneously improve cognitive models?

Inspecting choices and RTs together reveals latent cognitive mechanisms, such as decision conflict, which would be completely overlooked if researchers analyzed choice choices alone. Slower response times, for example, can indicate higher conflict.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How do sequential sampling models (SSMs) conceptualize the decision-making process?

SSMs define decision-making as an integration-to-bound process where noisy evidence in favor of one option is accumulated over time. A response is initiated only when this accumulated evidence reaches a pre-set threshold.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How do cautious decision-makers adjust their decision threshold in Sequential Sampling Models (SSMs)?

Cautious decision-makers increase their threshold, requiring more evidence before responding. This results in more accurate decisions but increases response times.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How does an easy decision affect the evidence accumulation process in Sequential Sampling Models (SSMs)?

For easy decisions, such as choosing between a highly rewarding option and a very poor one, the integration (drift) rate is higher. This leads to faster and more accurate decisions.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

Which sequential sampling model is the dominant paradigm in perceptual and value-based decision research?

The diffusion decision model (DDM) is the dominant model, which has transitioned from perceptual decision-making to value-based decision-making. It has also been used to link behavior to eye-tracking and neural measures.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What is a major limitation of standard Sequential Sampling Models (SSMs) when applied to feedback tasks?

SSMs typically assume that subjective values are stable over time and that repeated choices do not alter preferences. This assumption is violated when feedback is given after choices, demanding a learning component.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What was the primary goal of the modeling framework proposed in this study?

The goal was to propose a combined reinforcement learning diffusion decision model (RLDDM). This model describes the single-decision process using SSM mechanics while incorporating RL to show how subjective values update over time.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How is the 'difficulty effect' defined in reward-based decision-making?

The difficulty effect refers to the finding that when individuals choose between options that have highly similar values, they tend to be both slower and less accurate. This effect is driven by a reduced drift rate in SSMs.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What is the 'magnitude effect' in value-based decision-making?

The magnitude effect is the tendency for participants to make faster decisions when choosing between two highly-valued options compared to two lower-valued options, often without changes in accuracy.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What was the performance exclusion criterion used for participants in this study, and why?

Only participants with an accuracy above 56% across 240 trials were included, because performing below this threshold is highly likely to be the result of random behavior according to a binomial test.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What experimental paradigm was used to test the proposed learning and decision models?

The study used a multi-armed bandit paradigm where participants had to choose between pairs of visual stimuli and received full feedback on rewards after each choice.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

Which four pairs of options were presented in each block of the learning task?

The presented pairs were AB, AC, BD, and CD. Options AD and BC were excluded from presentation in order to properly test the difficulty and magnitude effects.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

In reinforcement learning models, what does the learning rate parameter (η) represent?

The learning rate η dictates the weight assigned to new feedback relative to past information. A rate near 0 preserves old values, whereas a rate near 1 means the subjective value is updated to almost match the latest feedback.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What function does the sensitivity parameter (θ) serve in reinforcement learning models?

The sensitivity parameter (or inverse temperature) controls how deterministic choices are based on value differences. Higher values make the choice rule more sensitive to differences, making higher-value selections highly likely.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How were subjective Q-values initialized in the first learning block of the study?

They were initialized to 27.5, which represented the average payoff value presented in the instruction screens shown to all participants before starting.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What asymmetric learning mechanism is often used to model 'optimism bias'?

Optimism bias is modeled by fitting separate learning rates (η) for positive and negative reward prediction errors, representing how individuals process good versus bad news differently.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What are the four core parameters of the standard, simple Diffusion Decision Model (DDM)?

The four core parameters are the drift rate (v), the decision threshold (a), the starting-point bias, and the non-decision time (Ter), which accounts for motor and sensory processes.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

Why did the researchers assume no starting-point bias in their DDM modeling?

Because the left and right layout positions of the options were completely randomized across trials, preventing participants from having an a priori bias toward choosing a specific side.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How is the drift rate (v) defined on a trial-by-trial basis in the simplest RLDDM?

The drift rate on trial t is defined as the difference between the updated subjective values of the correct and incorrect options, scaled by a parameter (v_mod).

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How does the proposed RLDDM dynamically adjust the decision threshold (a) over trials?

The threshold is modulated trial-by-trial based on the average learned subjective values of the presented options, meaning threshold height is influenced by expected rewards.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What is a major consequence of assuming a strictly linear mapping between value differences and drift rates in RLDDMs?

Models assuming a strictly linear mapping fail to capture sensitivity differences and tend to severely underestimate accuracy in difficult decision trials.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What nonlinear mapping function was proposed to relate value differences to the drift rate in the RLDDM?

An S-shaped (sigmoidal) function centered at zero was used, bounded by a maximum drift parameter (v_max) to regulate sensitivity to value differences.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What statistical models were deployed to empirically test the difficulty and magnitude effects in this study?

The researchers fit two separate Bayesian hierarchical models: a logistic regression model on accuracy and a linear regression model on log-transformed response times.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What convergence diagnostic statistic was used to assess if PyStan models were successfully fit?

The Gelman-Rubin convergence diagnostic R-hat was calculated, with a value of 1.01 or less indicating that the chains had successfully converged.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What specific index was used to compare the predictive accuracy and fit of the various cognitive models?

The widely applicable information criterion (WAIC) was computed, which corrects for model complexity by penalizing for the effective number of parameters.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What empirical behavioral results did the study show regarding the magnitude effect on decision accuracy?

In line with regression results, magnitude did not affect choice accuracy; participants did not select the superior option more often in higher-valued pairs.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

Which of the basic Reinforcement Learning (RL) models provided the most parsimonious fit?

Model 3, which incorporated separate learning rates for positive and negative prediction errors alongside a fixed sensitivity parameter (theta), was the most parsimonious.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

Which DDM variant proved to be the most parsimonious during model comparison?

Model 2, which featured separate drift rates for difficulty levels and separate response thresholds for pairs of options with different mean reward magnitudes.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What features characterized the best-fitting RLDDM (the full model) overall?

The best-fitting RLDDM combined separate positive/negative learning rates, an S-shaped drift rate mapping, and a decision threshold modulated by the learned average subjective value.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

How does a negative threshold modulation parameter (a_mod) in the RLDDM explain the magnitude effect?

A negative threshold modulation parameter reduces the decision threshold when expected values are high, allowing for faster decision speeds when choosing between high-value options.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What advantage does the RLDDM have over standard DDM fitting methods with respect to trial sequences?

The RLDDM preserves trial order by updating the drift rate and decision threshold dynamically based on the subjective values learned on each trial. Standard DDMs collapse trials and ignore this feedback order.

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified

What key limitation of the RLDDM framework do the authors explicitly note?

A key limitation is that RLDDMs are restricted to two-alternative forced-choice (2AFC) paradigms and cannot easily fit tasks with more than two options (like the Iowa Gambling Task).

Source: A reinforcement learning diffusion decision model for value-based decisions · DOI · quote verified