We are currently experiencing disruptions affecting DSSFS04 and DSSFS06. This is impacting access to parts of the AI Systems storage and can also cause login sessions or applications to hang when affected filesystem paths are accessed.

The DSS team is investigating the issue together with IBM.

We apologize for the disruption and will provide updates as soon as more information is available.

Web Service: DEGRADED (Jupyter & RStudio)
SSH Service: DEGRADED

We are currently facing disruptions in dssfs04 and dssfs06.
We are working together with IBM on a solution.

We deeply regret the interruption.

Your DSS Team

The following changes were introduced:

• Applied the latest Linux kernel security updates across the cluster to improve system security and stability.

A limited subset of the CPU partition is currently unavailable due to an issue requiring an upstream kernel fix. Most CPU resources remain available.

OOD applications requesting CPU resources may experience increased queue times, while small CPU jobs should generally be unaffected.

GPU resources, SSH access, and GPU batch jobs are not affected.

We will restore the affected CPU resources once the required upstream fix becomes available.

Web Service: DEGRADED (Jupyter & RStudio)

The OOD web applications are currently unavailable, including Jupyter Notebooks and RStudio.

The CPU partition is currently unavailable due to maintenance. In addition, several legacy GPU systems (P100 and V100) are temporarily unavailable.

All current-generation AI Systems resources, including the H100 and standard A100 GPU partitions, remain fully operational. SSH access and GPU batch jobs on these systems are not affected.

We are working to restore the affected resources as quickly as possible. Thank you for your patience.

Web Service: DEGRADED (Jupyter & RStudio)
SSH Service: UP

The AI Systems are now operational again and were in unplanned maintenance in response to a security advisory. None of the jobs that were either running or queued during the maintenance had to be terminated and could be kept running.

We sincerely apologize for the disruption.

The AI Systems have entered an unplanned maintenance in response to an urgent advisory. We expect the system to be available again on Tuesday, June 30, at 10:00.

As part of the unscheduled maintenance, all running jobs were terminated and will need to be restarted once the AI Systems are back online. Jobs that were queued but had not yet started will remain in the queue and will be scheduled automatically after the AI Systems resume operation.

We sincerely apologize for the disruption and will keep you informed as new information becomes available. In the meantime, we kindly ask for your patience and understanding.

The following changes were introduced:

• Operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes.
• Upgraded the GPFS parallel file system to version 5.3.2-4 for security and reliability improvements.
• Upgraded the GPU software stack to NVIDIA Driver 580 and CUDA 13 for support of the latest AI software and CUDA features.

Recovery of the AI Systems infrastructure is currently ongoing following the recent power outage.

Services and system components are being brought back online gradually and in a controlled manner. Please expect partial availability during the recovery phase. While many systems may already become reachable again, diagnostics, integrity checks, and validation procedures are still ongoing to ensure stable and reliable operation.

Users may therefore continue to experience temporary interruptions, reduced availability, or delayed service restoration for individual components while recovery work continues.

Further updates will follow as additional systems return to operation and diagnostics are completed.

previous title: “Offline due to Power Outage, 25 May 2026”

Due to a power grid outage, the AI Systems (including BayernKI and MCML) were unavailable between 25 May 04:00 CET and 26 May 2026 15:00 CET.

Action required: Jobs that were running at the time of the outage were abruptly terminated. Please review your job queue and history, and manually resubmit any affected jobs.

The AI Systems login nodes were taken offline between 15.5.2026 09:30 CET and 19.5.2026 11:30 CET.

System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority.

Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation.

Another maintenance window will be scheduled to apply upstream patches as they become available.

The AI Systems login nodes were taken offline between 7.5.2026 23:00 CET and 8.5.2026 15:00 CET.

System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority.

Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation.

Another maintenance window will be scheduled to apply upstream patches as they become available.

The AI Systems login nodes were taken offline between 30.4.2026 09:00 CET and 30.4.2026 17:00 CET.

System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority.

Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation.

Another maintenance window will be scheduled to apply upstream patches as they become available.

The following changes were introduced:

• Firmware, operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes
• Networking interface & switch configurations have been homogenized and switch firmware has been updated
• Virtual infrastructure components have been moved to new backend

The following changes were introduced:

• The AI Systems have been added to status.lrz.de and any future maintenance and service disruption announcements will be made public primarily via this channel
• Firmware, operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes
• The web-based frontend, Open OnDemand, has been updated to version 3.1.16
• Data Science Storage configuration parameters have been updated

We are currently experiencing disruptions affecting DSSFS04 and DSSFS06. This is impacting access to parts of the AI Systems storage and can also cause login sessions or applications to hang when affected filesystem paths are accessed.

The DSS team is investigating the issue together with IBM.

We apologize for the disruption and will provide updates as soon as more information is available.

Web Service: DEGRADED (Jupyter & RStudio)
SSH Service: DEGRADED

We are currently facing disruptions in dssfs04 and dssfs06.
We are working together with IBM on a solution.

We deeply regret the interruption.

Your DSS Team

The following changes were introduced:

• Applied the latest Linux kernel security updates across the cluster to improve system security and stability.

A limited subset of the CPU partition is currently unavailable due to an issue requiring an upstream kernel fix. Most CPU resources remain available.

OOD applications requesting CPU resources may experience increased queue times, while small CPU jobs should generally be unaffected.

GPU resources, SSH access, and GPU batch jobs are not affected.

We will restore the affected CPU resources once the required upstream fix becomes available.

Web Service: DEGRADED (Jupyter & RStudio)

The OOD web applications are currently unavailable, including Jupyter Notebooks and RStudio.

The CPU partition is currently unavailable due to maintenance. In addition, several legacy GPU systems (P100 and V100) are temporarily unavailable.

All current-generation AI Systems resources, including the H100 and standard A100 GPU partitions, remain fully operational. SSH access and GPU batch jobs on these systems are not affected.

We are working to restore the affected resources as quickly as possible. Thank you for your patience.

Web Service: DEGRADED (Jupyter & RStudio)
SSH Service: UP

The AI Systems are now operational again and were in unplanned maintenance in response to a security advisory. None of the jobs that were either running or queued during the maintenance had to be terminated and could be kept running.

We sincerely apologize for the disruption.

The AI Systems have entered an unplanned maintenance in response to an urgent advisory. We expect the system to be available again on Tuesday, June 30, at 10:00.

As part of the unscheduled maintenance, all running jobs were terminated and will need to be restarted once the AI Systems are back online. Jobs that were queued but had not yet started will remain in the queue and will be scheduled automatically after the AI Systems resume operation.

We sincerely apologize for the disruption and will keep you informed as new information becomes available. In the meantime, we kindly ask for your patience and understanding.

The following changes were introduced:

• Operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes.
• Upgraded the GPFS parallel file system to version 5.3.2-4 for security and reliability improvements.
• Upgraded the GPU software stack to NVIDIA Driver 580 and CUDA 13 for support of the latest AI software and CUDA features.

Recovery of the AI Systems infrastructure is currently ongoing following the recent power outage.

Services and system components are being brought back online gradually and in a controlled manner. Please expect partial availability during the recovery phase. While many systems may already become reachable again, diagnostics, integrity checks, and validation procedures are still ongoing to ensure stable and reliable operation.

Users may therefore continue to experience temporary interruptions, reduced availability, or delayed service restoration for individual components while recovery work continues.

Further updates will follow as additional systems return to operation and diagnostics are completed.

previous title: “Offline due to Power Outage, 25 May 2026”

Due to a power grid outage, the AI Systems (including BayernKI and MCML) were unavailable between 25 May 04:00 CET and 26 May 2026 15:00 CET.

Action required: Jobs that were running at the time of the outage were abruptly terminated. Please review your job queue and history, and manually resubmit any affected jobs.

The AI Systems login nodes were taken offline between 15.5.2026 09:30 CET and 19.5.2026 11:30 CET.

System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority.

Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation.

Another maintenance window will be scheduled to apply upstream patches as they become available.

The AI Systems login nodes were taken offline between 7.5.2026 23:00 CET and 8.5.2026 15:00 CET.

System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority.

Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation.

Another maintenance window will be scheduled to apply upstream patches as they become available.

The AI Systems login nodes were taken offline between 30.4.2026 09:00 CET and 30.4.2026 17:00 CET.

System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority.

Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation.

Another maintenance window will be scheduled to apply upstream patches as they become available.

The following changes were introduced:

• Firmware, operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes
• Networking interface & switch configurations have been homogenized and switch firmware has been updated
• Virtual infrastructure components have been moved to new backend

The following changes were introduced:

• The AI Systems have been added to status.lrz.de and any future maintenance and service disruption announcements will be made public primarily via this channel
• Firmware, operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes
• The web-based frontend, Open OnDemand, has been updated to version 3.1.16
• Data Science Storage configuration parameters have been updated