SageMaker AI LSTMネットワークとESA STIXデータを用いた太陽フレア検出システムの構築

The effective monitoring and characterization of solar flares demands sophisticated analysis of X-ray emissions across multiple energy spectrums. Machine learning-based anomaly detection serves as a powerful tool for identifying significant patterns that could indicate notable solar activity. Through the identification of distinct radiation signatures, key solar event characteristics can be detected, analyzed, and comprehensively understood. These detected patterns are essential for various applications, including space weather forecasting, solar physics investigations, and satellite operation planning. In recent years, solar monitoring capabilities have dramatically expanded, generating unprecedented volumes of X-ray measurement data. As this data continues to grow, analytical methods must evolve to efficiently process these massive datasets while capturing even the most subtle variations in solar behavior. Advanced deep learning architectures, particularly Long Short-Term Memory (LSTM) networks, have emerged as highly capable solutions for these challenges.

This post presents an implementation of LSTM neural networks for anomaly detection in multi-channel X-ray data collected by the Spectrometer/Telescope for Imaging X-rays (STIX). Our analysis emphasizes the detection of anomalous patterns across various energy ranges, spanning low (4–10 keV), medium (10–25 keV), and high (25+ keV) energy bands. This multi-channel approach facilitates comprehensive solar activity monitoring and enables robust identification of potential flare events through sophisticated pattern analysis of X-ray emission data. In this post, we show you how to use Amazon SageMaker AI to build and deploy a deep learning model for detecting solar flares using data from the European Space Agency’s STIX instrument. SageMaker AI will use Random Cut Forest (RCF), an unsupervised learning algorithm that detects abnormal data points by assigning anomaly scores based on the density and sparsity of the data points. You will learn how to implement a Long Short-Term Memory (LSTM) neural network that processes multi-channel X-ray data to identify potential solar flare events.

Key concepts

In this section, we discuss some key concepts of solar radiation analysis and machine learning (ML) in this solution.

X-ray energy channels in solar observations

X-ray emissions in our STIX data are measured across multiple energy bands, categorized into low (4–10 keV), medium (10–25 keV), and high (25+ keV) energy channels. This multi-channel approach enables comprehensive monitoring of solar activity across different energy levels. The energy bands provide crucial information about various aspects of solar flares, from their initiation to peak intensity. By analyzing patterns across these channels, we can identify different phases of solar flare evolution and characterize their intensity.

The combination of data from multiple energy channels offers detailed insights into solar flare characteristics. Higher energy channels typically indicate more intense flare activity, while lower energy channels can capture precursor events or post-flare phenomena. This multi-spectral approach allows for early detection of flare onset and accurate assessment of flare magnitude and duration.

Quaternions in spacecraft dynamics

LSTM networks are unlike traditional neural networks, in which they maintain an internal memory state that allows them to capture long-term dependencies in time series data. This unique quality places it as a Recurrent Neural Network (RNN), to where it is a variant. LSTM’s capability is particularly valuable for solar flare detection, where patterns may develop over extended time periods.

An LSTM architecture operates through a sophisticated system of gates and states. At its core, the input gate controls the flow of new information into the network, determining which data points are significant enough to be stored. Working in tandem, the forget gate evaluates existing information and decides which elements should be discarded, preventing the accumulation of irrelevant data. These decisions are reflected in the cell state, which serves as the network’s long-term memory, maintaining crucial information through the sequence processing. The output gate then regulates what information from this cell state should be presented as output at each time step.

This sophisticated architecture enables the model to learn intricate relationships between different energy channels while maintaining temporal coherence in the analysis. The multilayer approach allows the network to build increasingly abstract representations of the data, while the regularization techniques maintain reliable performance on new observations. Through continuous monitoring of reconstruction errors between predicted and actual values, the system effectively identifies anomalous patterns that may indicate solar flare events.

Time series analysis and anomaly detection

The anomaly detection system processes multi-dimensional time series data, where each dimension corresponds to a different energy channel. The LSTM model learns normal patterns in the X-ray emission data and identifies deviations that could indicate solar flare events. These anomalies might represent sudden intensity increases, unusual spectral patterns, or other signatures of solar activity.

For reliable flare detection, the LSTM analyzes both temporal and spectral characteristics of the cross channel X-ray data. Temporal analysis focuses on identifying sudden changes or unusual patterns in emission intensity over time. Spectral analysis examines the relationship between different energy channels, as solar flares often produce characteristic patterns across multiple energy bands.The combination of LSTM-based pattern recognition and multi-channel analysis creates a robust framework for automated solar flare detection. This approach can identify subtle precursor patterns, track flare evolution, and characterize flare intensity across different energy ranges, providing valuable insights for space weather forecasting and solar physics research.

Solution architecture

The solution architecture implements anomaly detection for ESA Solar Orbiter Solar STIX data using the LSTM algorithm, as illustrated in the following diagram. This solution uses Bring Your Own Script (BYOS) with Amazon SageMaker AI, so you can use your custom training scripts while using the managed infrastructure of SageMaker AI. With BYOS, you can:

• Use your preferred ML frameworks (in this case, PyTorch)

• Maintain control over your training logic

• Continue to use the training infrastructure of SageMaker AI and scaling capabilities

• Deploy your custom LSTM model without having to manage containers

To use this approach, upload your Python script to SageMaker AI and specify it as the entry point when creating your training job. The Python used for this blog post is located within the referenced GitHub repository.

The data pipeline initiates with raw STIX (Spectrometer/Telescope for Imaging X-rays) measurements stored in FITS format. These observations undergo initial processing in a JupyterLab environment, which serves as our primary development and analysis platform. The environment hosts custom Python notebooks that handle the conversion of FITS data to CSV format and execute our advanced detection algorithms.At the heart of the system lies our neural network processing pipeline. The workflow begins with data preparation in JupyterLab, followed by the training of our specialized CrossChannelLSTM model implemented in PyTorch. This model processes X-ray emissions across multiple energy ranges (4–10 keV, 10–25 keV, and 25+ keV). Following the training phase, the system analyzes temporal sequences to identify potential solar flare signatures through anomaly detection. The pipeline culminates in the generation of comprehensive visualizations, encompassing temporal analysis plots, energy evolution diagrams, and inter-channel correlation displays.

The system produces extensive analytical outputs, including identified anomalies, detailed channel statistics, and temporal pattern analysis. The visualization suite generates intricate plots highlighting anomalous patterns across energy bands, accompanied by temporal evolution charts and model performance metrics. All findings are documented in structured CSV files, containing anomaly indicators and reconstruction error measurements, facilitating in-depth analysis of detected solar events. Throughout the entire process, strict data handling protocols maintain analytical integrity and reproducibility.

Solution overview

This solution demonstrates a comprehensive approach to solar flare detection using deep learning techniques. First, we perform data preprocessing by converting FITS file format containing STIX quicklook lightcurve data into manageable CSV format. Then, we conduct data normalization and sequence preparation to maintain quality input for our analysis. Using PyTorch, we implement a custom CrossChannelLSTM model specifically designed for detecting anomalies in multi-channel X-ray data. The system processes data across multiple energy bands to identify patterns that may indicate solar flare activity.After the model training phase, which uses multiple LSTM layers with dropout regularization, the solution provides extensive visualization capabilities. These include time series plots with anomaly highlighting, energy-time evolution visualizations, and cross-channel analysis plots for clear interpretation of the findings. The system generates detailed outputs including channel-specific statistics, temporal evolution of anomalies, and comprehensive CSV files containing anomaly flags and reconstruction errors.

This combination of deep learning architecture and visualization tools creates a robust framework for automated solar flare detection. This combination makes it particularly suitable for space weather monitoring applications where rapid detection and precise analysis are crucial. The solution’s ability to process large-scale STIX data while identifying subtle radiation patterns demonstrates its effectiveness for solar physics research and space weather forecasting.

Prerequisites

Before implementing the solar flare detection system, ensure that you have the following tools and dependencies installed.

• AWS Requirements:

An AWS account with appropriate permissions

• IAM role with appropriate SageMaker AI and Amazon Simple Storage Service (Amazon S3) access policies

• Amazon S3 bucket for storing data and model artifacts

• Required AWS Services:

Amazon SageMaker AI (ml.m5.2xlarge JupyterLab instance recommended)

• Amazon S3

• AWS IAM Identity Center

• The development environment requires Python 3.7 or later, along with essential Python packages included in the provided Python script and requirements file:

• For the JupyterLab environment:

Amazon SageMaker AI Studio notebooks

• Additional requirements:

Access to STIX data in FITS format

• Sufficient RAM for processing large datasets (recommended minimum 16GB)

• GPU support recommended for faster model training

• Basic understanding of Python programming and deep learning concepts

• Estimated Costs:

SageMaker AI ml.m5.4xlarge instance: ~$0.922 per hour

• S3 storage: ~$0.023 per GB per month

• Total estimated cost for running this solution: ~$10–15 for a few hours of experimentation

Set up the solution

The setup process involves configuring the Amazon SageMaker AI Python environment, where all the data analysis and model training is executed.

• On the Amazon SageMaker AI console, open the SageMaker AI domain details page.

• Open JupyterLab, then create a new Python notebook instance for this project.

• When the environment is ready, open a terminal in Amazon SageMaker AI JupyterLab to clone the project repository using the following commands:

git clone https://github.com/aws-samples/sample-SageMaker-ai-lstm-anomaly-detection-solar-orbiter.git
cd sample-SageMaker-ai-lstm-anomaly-detection-solar-orbiter

• Install the required Python libraries:

pip install -r requirements.txt

This process will set up the necessary dependencies for running anomaly detection analysis on the Solar Orbiter sensor data.

Execute anomaly detection

Update the bucket_name and file_name variables in the script with your S3 bucket and data file names.

Run the script in JupyterLab as a Jupyter notebook or run as a Python script:

python ESA_SolOrb_AD.py

Upon execution, the notebook or script performs a series of automated tasks to analyze the solar X-ray data. It begins by loading and preprocessing the FITS file, converting it to CSV format and normalizing the data across energy channels. Next, it trains the CrossChannelLSTM model using PyTorch, establishing the foundation for our anomaly detection system. When the model is operational, it processes the multi-channel X-ray data to identify potential solar flare events through pattern analysis across different energy bands (4–10 keV, 10–25 keV, and 25+ keV).

Code structure

The Python implementation centers around a solar flare detection pipeline, structured in the main script. At its core are two main classes: CrossChannelLSTM and CrossChannelDataset, which together orchestrate the workflow from data ingestion to visualization. These classes work in tandem to process STIX X-ray data and identify potential solar flare events.

The explore_ql_lightcurve method handles the initial data ingestion and preprocessing, converting FITS files to CSV format and ensuring X-ray measurements are properly formatted for analysis. The plot_lightcurve method creates initial visualizations of the data across different energy channels. The print_channel_stats method provides statistical analysis for each energy band.

The CrossChannelLSTM class implements the neural network architecture, with multiple LSTM layers and dropout regularization. The CrossChannelDataset class manages data preparation and sequencing for the model. The detect_cross_channel_anomalies method then uses this trained model to identify unusual patterns in the X-ray emissions across different energy bands.

For visualization, the plot_cross_channel_anomalies and plot_flare_anomalies methods create detailed graphs highlighting detected anomalies, temporal evolution patterns, and energy band distributions. These visualizations include time series analysis, energy-time evolution diagrams, and cross-channel correlation plots.

Together, these components create a comprehensive pipeline for processing multi-channel X-ray data and identifying potential solar flare events that warrant further investigation. The system’s modular design allows for the modification of model parameters and visualization options to suit specific analysis needs.

Configuration

Adjust the following parameters in the script as needed for appropriate use such as accuracy, performance, compute time, and other needs: The LSTM model architecture and training parameters significantly influence the detection of solar flare events.

The following parameters can be modified:

Model Architecture Parameters:

• hidden_size: Size of LSTM hidden layers (default: 128–256)

• num_layers: Number of LSTM layers (default: 2–3)

• dropout_rate: Dropout regularization rate (default: 0.2)

• sequence_length: Length of input sequences (default: 30–50)

Training Parameters:

• batch_size: Number of sequences per training batch (default: 32)

• num_epochs: Number of training iterations (default: 15–20)

• learning_rate: Rate of model parameter updates (default: 0.001)

• threshold_multiplier: Anomaly detection sensitivity (default: 1.5)

For improved performance on standard hardware configurations, we recommend:

• hidden_size=256

• num_layers=3

• dropout_rate=0.2

• sequence_length=30

• batch_size=32

• num_epochs=20

Hardware requirements can significantly impact training time and model performance. GPU acceleration is recommended for faster training, though CPU-only execution is supported. Minimum recommended system specifications include 16 GB RAM and 4 CPU cores.

The system supports customization of visualization parameters and output formats. Results can be saved as CSV files containing detailed anomaly flags and reconstruction errors for each energy channel. The visualization suite can be configured to display different aspects of the analysis, from time series plots to energy band distributions.

Data

The script uses public ESA Solar Orbiter STIX (Spectrometer/Telescope for Imaging X-rays) data in FITS file format. The data contains X-ray measurements across multiple energy channels, ranging from 4 keV to over 25 keV. The FITS files include:

• Time series data for each energy channel

• Energy band information (4–10 keV, 10–25 keV, 25+ keV)

• Control indices and timing information

• Measurement error data

<