Standard deviation

UML diagrams presenting the flow of the analysis for each module are available here: https://github.com/ANCPLabOldenburg/MEG-QC-code/tree/main/diagrams

meg_qc.calculation.metrics.STD_meg_qc.RMSE(data_m_or_g: array)[source]

RMSE - general root means squared error. Currently NOT USED, as np.std is slightly faster. Was used before as alternative to std calculation, was faster.

Parameters:

data_m_or_g (np.array or list) – data for magnetometer or gradiometer given as np array or list (it can be 1 or several channels data as 2 dimentional array or as list of lists)

Returns:

rmse as numpy array (1-dimentional if 1 channel was given, 2-dim if more channels)

Return type:

np.ndarray

meg_qc.calculation.metrics.STD_meg_qc.STD_meg_qc(std_params: dict, channels: dict, chs_by_lobe: dict, dict_epochs_mg: dict, data: Raw, m_or_g_chosen: list, verbose_plots: bool)[source]

Main STD function. Calculates: - Std of data for each channel over all time series. - Channels with big std (noisy) and small std (flat) over all time series. - Std of data for each channel in each epoch. - Epochs with big std (noisy) and small std (flat).

Parameters:

  • std_params (dict) – dictionary with parameters for std metric, originally from config file

  • channels (dict) – dictionary with channel names for each channel type: channels[‘mag’] or channels[‘grad’]

  • chs_by_lobe (dict) – dictionary with channels grouped first by ch type and then by lobe: chs_by_lobe[‘mag’][‘Left Occipital’] or chs_by_lobe[‘grad’][‘Left Occipital’]

  • dict_epochs_mg (dict) – dictionary with epochs for each channel type: dict_epochs_mg[‘mag’] or dict_epochs_mg[‘grad’]

  • data (mne.io.Raw) – raw data

  • m_or_g_chosen (list) – list of strings with channel types chosen by user: [‘mag’, ‘grad’] or [‘mag’] or [‘grad’]

  • verbose_plots (bool) – True for showing plot in notebook.

Returns:

  • list – list of QC_derivative objects containing data frames and figures for std metric.

  • dict – dictionary with simple metric for std/ptp.

  • std_str (str) – String with notes about STD for report

meg_qc.calculation.metrics.STD_meg_qc.get_big_small_std_ptp_all_data(ptp_or_std_channels_named: dict, channels: list, std_multiplier: float)[source]

Function calculates peak-to-peak amplitude or STDs over the entire data set for each channel. Threshold for noisy = mean + multiplier*std, above it - noisy, Threshold for flat = mean - multiplier*std, below it - flat, where:

  • mean is mean stds/ptp values over all channels

  • std is standard deviation of stds values over all channels

  • multiplier is a parameter set in config, defines how many stds/ptp above or below mean should be taken as threshold.

Parameters:

  • ptp_or_std_channels_named (dict) – peak-to-peak amplitude or std for each channel

  • channels (list) – list of channel names

  • std_multiplier (float) – multipliar for std, used to define thresholds for noisy and flat channels

Returns:

  • dict – dictionary with channel names and their stds/ptp values. Noisy channels.

  • dict – dictionary with channel names and their stds/ptp values. Flat channels.

meg_qc.calculation.metrics.STD_meg_qc.get_big_small_std_ptp_epochs(df_std: DataFrame, ch_type: str, std_lvl: int, std_or_ptp: str)[source]

NOT USED ANY MORE

  • Calculate std for every separate epoch of a given list of channels

  • Find which channels in which epochs have too high/too small stds or PtP amplitudes

  • Create MEG_QC_derivative as dfs

meg_qc.calculation.metrics.STD_meg_qc.get_noisy_flat_std_ptp_epochs(df_std: DataFrame, ch_type: str, std_or_ptp: str, noisy_channel_multiplier: float, flat_multiplier: float, percent_noisy_flat_allowed: float)[source]

1. Define if the channels data inside the epoch is noisy or flat: Compare the std of this channel for this epoch (df_std) to the mean STD of this particular channel or over all epchs.

  • If std of this channel for this epoch is over the mean std of this channel for all epochs together * flat multiplyer, then the data for this channel in this epoch is noisy.

  • If std of this channel for this epoch is under the mean std of this channel for all epochs together * noisy multiplyer, then this the data for this channel in this epoch is flat.

Multiplyer is set by user in the config file.

2. Count how many channels are noisy/flat in each epoch. If more than percent_noisy_flat_allowed of channels are noisy/flat, then this epoch is noisy/flat. Percent is set by user in the config file.

  1. Create MEG_QC_derivative as 3 dfs:

  • df_epoch_vs_mean: ratio of std of this channel for this epoch to the mean std of this channel over all epochs together

  • df_noisy_epoch: df with True/False values for each channel in each epoch, True if this channel is noisy in this epoch

  • df_flat_epoch: df with True/False values for each channel in each epoch, True if this channel is flat in this epoch

Parameters:

  • df_std (pd.DataFrame) – dataframe with std/ptp values for each channel and each epoch

  • ch_type (str) – channel type, ‘mag’, ‘grad’

  • std_or_ptp (str) – ‘std’ or ‘ptp’ - to use std or peak to peak amplitude as a metric

  • noisy_channel_multiplier (float) – multiplier to define noisy channel, if std of this channel for this epoch is over (the mean std of this channel for all epochs together*multipliar), then this channel is noisy set by user in the config file

  • flat_multiplier (float) – multiplier to define flat channel, if std of this channel for this epoch is under (the mean std of this channel for all epochs together*multipliar), then this channel is flat set by user in the config file

  • percent_noisy_flat_allowed (float) – percent of noisy/flat channels allowed in each epoch, if more than this percent, then this epoch is noisy/flat. Example: 70 Means that if more than 70% of channels are noisy/flat in this epoch, then this epoch is noisy/flat.

Returns:

list of 3 MEG_QC_derivative objects: - df_epoch_vs_mean: ratio of std of this channel for this epoch to the mean std of this channel over all epochs together - df_noisy_epoch: df with True/False values for each channel in each epoch, True if this channel is noisy in this epoch - df_flat_epoch: df with True/False values for each channel in each epoch, True if this channel is flat in this epoch

Return type:

list

meg_qc.calculation.metrics.STD_meg_qc.get_std_all_data(data: Raw, channels: list)[source]

Calculate RMSE/std (same mathematically) for each channel - for the entire time duration.

Parameters:

  • data (mne.io.Raw) – raw data

  • channels (list) – list of channel names

Returns:

rmse/std for each channel

Return type:

dict

meg_qc.calculation.metrics.STD_meg_qc.get_std_epochs(channels: list, epochs_mg: Epochs)[source]

Calculate std for multiple epochs for a list of channels. Used as internal function in std_meg_epoch()

Parameters:

  • channels (list) – list of channel names

  • epochs_mg (mne.Epochs) – epochs data as mne.Epochs object

Returns:

dataframe with std values for each channel and each epoch

Return type:

pd.DataFrame

meg_qc.calculation.metrics.STD_meg_qc.make_dict_global_std_ptp(std_ptp_params: dict, big_std_with_value_all_data: List[dict], small_std_with_value_all_data: List[dict], channels: List[str], std_or_ptp: str)[source]

Make a dictionary with global metric content for std or ptp metric. Global means that it is calculated over entire data series, not over epochs.

Parameters:

  • std_ptp_params (dict) – dictionary with parameters for std or ptp metric

  • big_std_with_value_all_data (list) – list of dictionaries (channel_name: value) for channels with big std or ptp

  • small_std_with_value_all_data (list) – list of dictionaries (channel_name: value) for channels with small std or ptp

  • channels (list) – list of channel names

  • std_or_ptp (str) – ‘std’ or ‘ptp’: use STD or Peak-to-peak metric

Returns:

dictionary with global metric content for std or ptp metric

Return type:

dict

meg_qc.calculation.metrics.STD_meg_qc.make_dict_local_std_ptp(std_ptp_params: dict, noisy_epochs_df: DataFrame, flat_epochs_df: DataFrame)[source]

Make a dictionary with local metric content for std or ptp metric. Local means that it is calculated over epochs.

Parameters:

  • std_ptp_params (dict) – dictionary with parameters for std or ptp metric, originally from config file

  • noisy_epochs_df (pd.DataFrame) – dataframe with True/False values for noisy channels in each epoch

  • flat_epochs_df (pd.DataFrame) – dataframe with True/False values for flat channels in each epoch

Returns:

dictionary with local metric content for std or ptp metric

Return type:

dict

meg_qc.calculation.metrics.STD_meg_qc.make_simple_metric_std(std_params: dict, big_std_with_value_all_data: List[dict], small_std_with_value_all_data: List[dict], channels: List[str], deriv_epoch_std: dict, metric_local_present: bool, m_or_g_chosen: List[dict])[source]

Make simple metric for STD.

Parameters:

  • std_params (dict) – dictionary with parameters for std metric, originally from config file

  • big_std_with_value_all_data (list) – list of dictionaries (channel_name: value) for channels with big std

  • small_std_with_value_all_data (list) – list of dictionaries (channel_name: value) for channels with small std

  • channels (list) – list of channel names

  • deriv_epoch_std (dict) – dictionary with QC_derivative objects containing data frames. Used only data frame 1 and 2. 1: contains True/False values for noisy channels in each epoch. 2: contains True/False values for flat channels in each epoch.

  • metric_local_present (bool) – True if local metric was calculated (epochs present). False if not calculated (epochs were not detected).

  • m_or_g_chosen (list) – list of strings with channel types chosen by user: [‘mag’, ‘grad’] or [‘mag’] or [‘grad’]

Returns:

dictionary with simple metric for std/ptp

Return type:

dict