Thesis/chapters/chap04/chap04.tex

% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Author:   Phillip Rothenbeck
% Title:    Investigating the Evolution of the COVID-19 Pandemic in Germany Using Physics-Informed Neural Networks
% File:     chap04/chap04.tex
% Part:     Experiments
% Description:
%         summary of the content in this chapter
% Version:  01.01.2012
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Experiments   10}
\label{chap:evaluation}
In the preceding chapters, we explained the methods (see~\Cref{chap:methods})
based the theoretical background, that we established in~\Cref{chap:background}.
In this chapter present the setups and results from the experiments and
simulations, we ran. First, we discuss the experiments dedicated to identify
the epidemiological parameters of $\beta$ and $\alpha$ in synthetic and
real-world data. Second, we examine the reproduction number in synthetic and
real-world data of Germany. Each section, is divided into a description of the
experimental setup and the results.

% -------------------------------------------------------------------

\section{Identifying the Transition Rates on Real-World and Synthetic Data  5}
\label{sec:sir}
In this section, we aim to identify the transmission rate $\beta$ and the
recovery rate $\alpha$ from either synthetic or preprocessed real-world data.
The methodology that we employ to identify the transition rates is described
in~\Cref{sec:pinn:sir}. Meanwhile, the methods we utilize to preprocess the
real-world data are detailed in~\Cref{sec:preprocessing:rq}.

% -------------------------------------------------------------------

\subsection{Setup   1}
\label{sec:sir:setup}

In this subsection, we present the configurations for the training of our
PINNs, which are designed to identify the transition parameters. This
encompasses the specific parameters for the preprocessing and the configuration
of the PINN themselves.\\

In order to validate our method, we first generate a dataset of synthetic data.
We achieve this by solving~\Cref{eq:modSIR} for a given set of parameters.
The parameters are set to $\alpha = \nicefrac{1}{3}$ and $\beta = \nicefrac{1}{2}$.
The size of the population is $N = \expnumber{7.6}{6}$ and the initial amount of
infectious individuals of is $I_0 = 10$. We conduct the simulation over 150
days, resulting in a dataset of the form of~\Cref{fig:synthetic_SIR}.\\ In
order to process the real-world RKI data, it is necessary to preprocess the raw
data for each state and Germany separately. This is achieved by utilizing a
recovery queue with a recovery period of 14 days. With regard to population
size of each state, we set it to the respective value counted at the end of
2019\footnote{\url{https://de.statista.com/statistik/kategorien/kategorie/8/themen/63/branche/demographie/\#overview}}.
The initial number of infectious individuals is set to the number of infected
people on March 09. 2020 from the dataset. The data we extract spans from
March 09. 2020 to June 22. 2023, encompassing a period of 1200 days and
representing the time span during which the COVID-19 disease was the most
active and severe.

\begin{figure}[h]
    %\centering
    \setlength{\unitlength}{1cm} % Set the unit length for coordinates
    \begin{picture}(12, 9.5) % Specify the size of the picture environment (width, height)
        \put(1.5, 4.5){
            \begin{subfigure}{0.3\textwidth}
                \centering
                \includegraphics[width=\textwidth]{SIR_synth.pdf}
                \label{fig:synthetic_SIR}
            \end{subfigure}
        }
        \put(8, 4.5){
            \begin{subfigure}{0.3\textwidth}
                \centering
                \includegraphics[width=\textwidth]{datasets_states/Germany_SIR_14.pdf}
                \label{fig:germany_sir}
            \end{subfigure}
        }
        \put(0, 0){
            \begin{subfigure}{0.3\textwidth}
                \centering
                \includegraphics[width=\textwidth]{datasets_states/Schleswig_Holstein_SIR_14.pdf}
                \label{fig:schleswig_holstein_sir}
            \end{subfigure}
        }
        \put(4.75, 0){
            \begin{subfigure}{0.3\textwidth}
                \centering
                \includegraphics[width=\textwidth]{datasets_states/Berlin_SIR_14.pdf}
                \label{fig:berlin_sir}
            \end{subfigure}
        }
        \put(9.5, 0){
            \begin{subfigure}{0.3\textwidth}
                \centering
                \includegraphics[width=\textwidth]{datasets_states/Thueringen_SIR_14.pdf}
                \label{fig:thüringen_sir}
            \end{subfigure}
        }

    \end{picture}
    \caption{Synthetic and real-world training data. The synthetic data is
        generated with $\alpha=\nicefrac{1}{3}$ and $\beta=\nicefrac{1}{2}$
        and~\Cref{eq:modSIR}. The Germany data is taken from the death case
        data set. Exemplatory we show illustrations of the datasets of Schleswig
        Holstein, Berlin, and Thuringia. For the other states see~\Cref{chap:appendix} }
    \label{fig:datasets_sir}
\end{figure}

The PINN that we utilize comprises of seven hidden layers with twenty neurons
each, and an activation function of ReLU. We employ the Adam optimizer and the
polynomial scheduler of the PyTorch library, for training, with a base learning rate
of $\expnumber{1}{-3}$. We train the model for 10000 epochs to extract the
parameters. For each set of parameters, we conduct five iterations to
demonstrate stability of the values. The configuration is similar to the
configuration, that Shaier \etal ~\cite{Shaier2021} use for their work aside
from the learning rate and the scheduler choice.\\

The following section presents the results of the simulations conducted with the
setups that we describe in this section.

% -------------------------------------------------------------------

\subsection{Results    4}
\label{sec:sir:results}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.7\textwidth]{reproducability.pdf}
    \caption{Visualization of all 5 predictions for the synthetic dataset,
        compared to the true values of $\alpha = \nicefrac{1}{3}$ and $\beta = \nicefrac{1}{2}$}
    \label{fig:reprod}
\end{figure}

In this section, we present the results, that we obtain from the conducted
experiments, that we describe in the preceding section. We begin by examining
the results for the synthetic dataset, focusing the accuracy and
reproducibility. We then proceed to present and discuss the results for the
German states and Germany.\\

The results of the experiment regarding the synthetic data can be seen
in~\Cref{table:alpha_beta_synth} and in~\Cref{fig:reprod}.~\Cref{fig:reprod}
depicts the values of $\beta$ and $\alpha$ for each iteration in comparison to the true
values of $\beta=\nicefrac{1}{2}$ and $\alpha=\nicefrac{1}{3}$. In~\Cref{table:alpha_beta_synth}
we present the mean $\mu$ and standard deviation $\sigma$ of both values across
all five iterations.\\

\begin{table}[h]
    \begin{center}
        \begin{tabular}{ccc ccc}
            \toprule
            \multicolumn{3}{c}{$\alpha$} & \multicolumn{3}{c}{$\beta$}                                         \\
            \cmidrule{1-3}\cmidrule{4-6}
            true                         & $\mu$                       & $\sigma$ & true   & $\mu$  & $\sigma$ \\
            \midrule
            0.3333                       & 0.3334                      & 0.0011   & 0.5000 & 0.5000 & 0.0017   \\
            \bottomrule
        \end{tabular}
        \caption{The mean $\mu$ and standard deviation $\sigma$ across the 5
            independent iterations of training our PINNs with the synthetic dataset.}
        \label{table:alpha_beta_synth}
    \end{center}
\end{table}

The results demonstrate that the model is capable of approximating the correct
parameters for the small, synthetic dataset in each of the five iterations.
While the predicted value is not precisely accurate, the standard deviation is
sufficiently small, and taking the mean of multiple iterations produces an
almost perfect result.\\

In~\Cref{table:state_mean_std} we present the results of the training for the
real-world data. The results are presented from top to bottom, in the order of
the community identification number, with the last entry being Germany. Both
the mean $\mu$ and the standard deviation $\sigma$ are calculated across all
five iterations of our experiment. We can observe that the values of
\emph{Hamburg} have the highest standard deviation, while \emph{Mecklenburg Vorpommern}
has the lowest $\sigma$.\\

\begin{table}[h]
    \begin{center}
        \begin{tabular}{lccccc}
            \toprule
                                   & \multicolumn{2}{c}{$\alpha$} & \multicolumn{2}{c}{$\beta$} &                                        \\
            \cmidrule{2-3}\cmidrule{4-5}
            state name             & $\mu$                        & $\sigma$                    & $\mu$  & $\sigma$ & $e_{\text{synth}}$ \\
            \midrule
            Schleswig Holstein     & 0.0771                       & 0.0010                      & 0.0966 & 0.0013   & 0.0849             \\
            Hamburg                & 0.0847                       & 0.0035                      & 0.1077 & 0.0037   & 0.0948             \\
            Lower Saxony           & 0.0735                       & 0.0014                      & 0.0962 & 0.0018   & 0.0774             \\
            Bremen                 & 0.0588                       & 0.0018                      & 0.0795 & 0.0025   & 0.0933             \\
            North Rhine-Westphalia & 0.0780                       & 0.0009                      & 0.1001 & 0.0011   & 0.0777             \\
            Hesse                  & 0.0653                       & 0.0016                      & 0.0854 & 0.0020   & 0.1017             \\
            Rhineland-Palatinate   & 0.0808                       & 0.0016                      & 0.1036 & 0.0018   & 0.0895             \\
            Baden-Württemberg      & 0.0862                       & 0.0014                      & 0.1132 & 0.0016   & 0.0796             \\\addlinespace
            Bavaria                & 0.0809                       & 0.0021                      & 0.1106 & 0.0027   & 0.0952             \\
            Saarland               & 0.0746                       & 0.0021                      & 0.0996 & 0.0024   & 0.1080             \\
            Berlin                 & 0.0901                       & 0.0008                      & 0.1125 & 0.0008   & 0.0667             \\
            Brandenburg            & 0.0861                       & 0.0008                      & 0.1091 & 0.0010   & 0.0724             \\
            Mecklenburg-Vorpommern & 0.0910                       & 0.0007                      & 0.1167 & 0.0008   & 0.0540             \\
            Saxony                 & 0.0797                       & 0.0017                      & 0.1073 & 0.0022   & 0.1109             \\
            Saxony-Anhalt          & 0.0932                       & 0.0019                      & 0.1207 & 0.0027   & 0.0785             \\
            Thuringia              & 0.0952                       & 0.0011                      & 0.1248 & 0.0016   & 0.0837             \\\addlinespace
            Germany                & 0.0803                       & 0.0012                      & 0.1044 & 0.0014   & 0.0804             \\
            \bottomrule
        \end{tabular}
        \caption{Mean and standard deviation across the 5 iterations, that we
            conducted for each German state and Germany as the whole country.}
        \label{table:state_mean_std}
    \end{center}
\end{table}

\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{mean_std_alpha_beta_res.pdf}
    \caption{Visualization of the mean $\mu$ and standard deviation $\sigma$ of
        the transition rates $\alpha$ and $\beta$ for each state compared to the
        mean values of $\alpha$ and $\beta$ for Germany.}
    \label{fig:alpha_beta_mean_std}
\end{figure}

In~\Cref{fig:alpha_beta_mean_std}, we present a visual representation of the
means and standard deviations in comparison to the national values. It is
noteworthy that the states of Saxony-Anhalt and Thuringia have the highest
transmission rates of all states, while Bremen and Hessen have the lowest
values for $\beta$. The transmission rates of Hamburg, Baden Württemberg,
Bavaria, and all eastern states lay above the national rate of transmission.
Similarly, the recovery rate yields comparable outcomes. For the recovery rate,
the same states that exhibit a transmission rate exceeding the national value,
have a higher recovery rate than the national standard, with the exception of
Saxony.It is noteworthy that the recovery rates of all states exhibit a
tendency to align with the recovery rate of $\alpha=\nicefrac{1}{14}$, which is
equivalent to a recovery period of 14 days.\\

It is evident that there is a correlation between the values of $\alpha$ and
$\beta$ for each state. States with a high transmission rate tend to have a
high recovery rate, and vice versa. The correlation between $\alpha$ and
$\beta$ can be explained by the implicate definition of $\alpha$ using a
recovery queue with a constant recovery period of 14 days. This might result to
the PINN not learning $\alpha$ as a standalone parameter but rather as a
function of the transmission rate $\beta$. This phenomenon occurs because the
transmission rate determines the number of individuals that get infected per
day, and the recovery queue moves a proportional number of people to the
removed compartment. Consequently, a number of people defined by $\beta$ move
to the $R$ compartment 14 days after they were infected.\\

This issue can be addressed by reducing the SIR model, thereby eliminating the
significance of the $R$ compartment size. In the following section, we present
our experiments for the reduced SIR model with time-independent parameters.

% -------------------------------------------------------------------

\section{Reduced SIR Model   5}
\label{sec:rsir}
In this section we describe the experiments we conduct to identify the
time-dependent reproduction number for both synthetic and real-world data.
Similar to the previous section, we first describe the setup of our experiments
and afterwards present the results. The methods we employ for the preprocessing
are described in~\Cref{sec:preprocessing:rq} and for the PINN, that we use,
are described in~\Cref{sec:pinn:rsir}.

% -------------------------------------------------------------------

\subsection{Setup    1}
\label{sec:rsir:setup}
This section outlines the selection of parameters and configuration for data
generation, preprocessing, and the neural networks. We employ these setups to
train the PINNs to identify the reproduction number on both synthetic and
real-world data.\\

For the purposes of validation, we create a synthetic dataset, by setting the parameter
of $\alpha$ and the reproduction value each to a specific values, and solving~\Cref{eq:reduced_sir_ODE}
for a given time interval. We set $\alpha=\nicefrac{1}{3}$ and $\Rt$ to the
values as can be seen in~\Cref{fig:Rt_dataset} as well as the population
size $N=\expnumber{7.6}{6}$ and the initial amount of infected people to
$I_0=10$. Furthermore, we set our simulated time span to 150 days. We use this
dataset to demonstrate, that our method is working on a simple and minimal
dataset.\\ To obtain a dataset of the infectious group, consisting of the
real-world data, we processed the data of the dataset
\emph{COVID-19-Todesfälle in Deutschland} to extract the number of infections
in Germany as a whole. For the German states, we use the data of \emph{SARS-CoV-2
    Infektionen in Deutschland}. In the preprocessing stage, we employ a constant
rate for $\alpha$ to move individuals into the removed compartment. For each
state we generate two datasets with a different recovery rate. First, we choose
$\alpha = \nicefrac{1}{14}$, which aligns with the time of recovery\footnote{\url{https://github.com/robert-koch-institut/SARS-CoV-2-Infektionen_in_Deutschland.git}}.
Second, we use $\alpha=\nicefrac{1}{5}$, as 5 days into the infection is the
point at which the infectiousness is at its peak\footnote{\url{https://www.infektionsschutz.de/coronavirus/fragen-und-antworten/ansteckung-uebertragung-und-krankheitsverlauf/}}.
As in~\Cref{sec:sir}, we set the population size $N$ of each state and Germany
to the corresponding size at the end of 2019. Furthermore, for the same reason
we restrict the data points to an interval of 1200 days, beginning on March 09.
2020.\\

\begin{figure}[t]
    \centering
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=\textwidth]{I_synth.pdf}
    \end{subfigure}
    \quad
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=\textwidth]{I_synth_r_t.pdf}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}{0.67\textwidth}
        \centering
        \includegraphics[width=\textwidth]{datasets_states/Germany_datasets.pdf}
    \end{subfigure}
    \caption{The upper two graphics show the curve of the size of the
        infectious group (left) and the corresponding true reproduction value
        $\Rt$ (right) for the synthetic data. The lower graphic exemplary
        illustrates the different curves for Germany.}
    \label{fig:Rt_dataset}
\end{figure}

In order to achieve the desired output, the selected neural network
architecture comprises of four hidden layers, each containing 100 neurons. The
activation function is the tangens hyperbolicus function. For the real-world
data, we weight the data loss by a factor of $\expnumber{1}{6}$, to the total
loss. The model is trained using a base learning rate of $\expnumber{1}{-3}$,
with the same scheduler and optimizer as we describe in~\Cref{sec:sir:setup}.
We train the model for 20000 epochs. To reduce the standard deviation, each
experiment is conducted 15 times.\\

% -------------------------------------------------------------------

\subsection{Results   4}
\label{sec:rsir:results}

In this section we provide the results for our experiments considering the
reduced SIR model and the reproduction number $\Rt$. First, we present
our findings for the synthetic dataset. Then, we provide and discuss the
results for the real-world data.\\

\Cref{fig:synth_results} illustrates the results of our experiments conducted on
the synthetic dataset, which can be seen in~\Cref{fig:Rt_dataset}. It is evident
that the model is capable of learning the infection data across all data points.
The error for this is, $e_{\text{synth}} = 0.0016$, which is of a negligible
magnitude.\\

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.45\textwidth}
        \includegraphics[width=\textwidth]{synthetic_I_prediction.pdf}
    \end{subfigure}
    \quad
    \begin{subfigure}{0.45\textwidth}
        \includegraphics[width=\textwidth]{synthetic_R_t_statistics.pdf}
    \end{subfigure}
    \label{fig:synth_results}
    \caption{Results for the reproduction rate $\Rt$ on synthetic data. The
        left graphic show the prediction of the model regarding the $I$ group. The
        right graphic presents the predicted $\Rt$ against the true value, with the
        standard deviation.}
\end{figure}

An examination of the predictions for the representation value $\Rt$ reveals
that here as well, the model is capable of accurately delineating the value at
each time point. However, during the first 30 days, the standard deviation is
exhibits an upward trend, while during the final 120 days, the predictions
demonstrate remarkable precision. The overall prediction of $\Rt$ has an error
of $e_{\Rt} = 0.0521$.\\

In~\Cref{fig:state_results}, we present the graphs of $\Rt$ for the state with
the highest value of $\beta$, namely Thuringia, and for the state with the lowest
transmission rate $\beta$, namely Bremen. Further visualizations of the results
can be found in~\Cref{chap:appendix}. In all datasets, the graphs with $\alpha =
    \nicefrac{1}{5}$ are of a smaller size than those with
$\alpha = \nicefrac{1}{14}$. This is due to the fact that the individuals are
being moved to the removed compartment at a faster rate. Resulting, it can be
observed that the value of $\Rt$ is constantly remaining closer to the threshold
of $\Rt=1$, while the reproduction number for datasets with $\alpha = \nicefrac{1}{14}$
reaches values of up to 1.6. In states with higher values of $\beta$, the period
during which the value of $\Rt$ is above the threshold of one 1 is longer, but
the peak is lower. In states with a lower transmission rate, the period above 1
is shorter, but the peak value is higher.\\

\begin{figure}[t]
    \centering
    \begin{subfigure}{0.45\textwidth}
        \includegraphics[width=\textwidth]{I_prediction/Thueringen_I_prediction.pdf}
    \end{subfigure}
    \quad
    \begin{subfigure}{0.45\textwidth}
        \includegraphics[width=\textwidth]{I_prediction/Bremen_I_prediction.pdf}
    \end{subfigure}
    \begin{subfigure}{0.45\textwidth}
        \includegraphics[width=\textwidth]{R_t/Thueringen_R_t_statistics.pdf}
    \end{subfigure}
    \quad
    \begin{subfigure}{0.45\textwidth}
        \includegraphics[width=\textwidth]{R_t/Bremen_R_t_statistics.pdf}
    \end{subfigure}
    \label{fig:state_results}
    \caption{Visualization of the prediction of the training and  the graphs of
        $\Rt$ for Thuringia (left) and Bremen (right) with both
        $\alpha = \nicefrac{1}{14}$ and $\alpha = \nicefrac{1}{5}$. Events like
        the peak of an influential variant are marked horizontally.}
\end{figure}

\Cref{table:state_error} presents data regarding the discrepancy between the
predicted and actual values from the dataset for compartment $I$. It is evident,
that the error for all experiments falls within a range of values that is not
negligible and will have an influence on the resulting reproduction values that
are learned while fitting the data. A comparison of the results for the various
values of $\alpha$ reveals that the errors associated with $\alpha = \nicefrac{1}{14}$
are consistently smaller, with the exception of Saxony and Germany. This can be
attributed to the differing sizes of infection counts, particularly in relation
to the normalization factor $C$. The model is unable to learn effectively if the
values of the data loss $\mathcal{L}_{\text{data}}$ are too large or too small
at the beginning.\\

As illustrated in~\Cref{fig:state_results}, the training data is overlaid with the
corresponding prediction of the model. We can observe that the prediction, though
an exact reconstruction, accurately captures the general trajectory of the
pandemic. The model's prediction demonstrates an ability to capture larger
peaks, exhibiting a tendency to ignore smaller changes. This suggests that the
prediction of the model is capable show the rough outline of the progression of COVID-19. In the
beginning, the majority of predictions below $\Rt=1$, indicating an outbreak.
As we observed in the synthetic data, the model exhibits a higher standard
deviation at the boundaries. In the graphs, we mark the
peaks of the most severe COVID-19 variants in Germany. While the peaks of the
Alpha and Delta variants are clearly visible in the data, the model does not
learn these, and thus they are not reflected in the results. The peak of the
Omicron variant  represents the culmination of the COVID-19 pandemic in Germany
and can be identified as the most prominent peak in the dataset. Immediately preceding this peak, we observe the highest
value of the reproduction number across all states. This phenomenon can be explained, by
number  of individuals infected by one infectious person reaching its peak. In
some states the peaks of other Omicron variants after the maximum peak are visible (see Thuringia).\\

The experiments demonstrate, that our model encounteres difficulties in learning the data for the
states and Germany and consequently in predicting the reproduction values for each dataset.
Nonetheless, the predictions illustrate the general trends of the most impactful
events of the COVID-19 pandemic.\\

\begin{table}[t]
    \begin{center}
        \begin{tabular}{lcc}
            \toprule
                                   & \multicolumn{2}{c}{$e_I$}                            \\
            \cmidrule{2-3}
            state name             & $\alpha=\nicefrac{1}{14}$ & $\alpha=\nicefrac{1}{5}$ \\
            \midrule
            Schleswig Holstein     & 0.2005                    & 0.2514                   \\
            Hamburg                & 0.3045                    & 0.3357                   \\
            Lower Saxony           & 0.2140                    & 0.3082                   \\
            Bremen                 & 0.2370                    & 0.3838                   \\
            North Rhine-Westphalia & 0.1718                    & 0.2460                   \\
            Hesse                  & 0.2736                    & 0.3172                   \\
            Rhineland-Palatinate   & 0.2442                    & 0.2674                   \\
            Baden-Württemberg      & 0.1984                    & 0.2958                   \\\addlinespace
            Bavaria                & 0.1928                    & 0.2825                   \\
            Saarland               & 0.2554                    & 0.4676                   \\
            Berlin                 & 0.1885                    & 0.2948                   \\
            Brandenburg            & 0.2023                    & 0.2571                   \\
            Mecklenburg-Vorpommern & 0.1518                    & 0.3272                   \\
            Saxony                 & 0.3382                    & 0.2807                   \\
            Saxony-Anhalt          & 0.1959                    & 0.2564                   \\
            Thuringia              & 0.1401                    & 0.2221                   \\\addlinespace
            Germany                & 0.3371                    & 0.2533                   \\
            \bottomrule
        \end{tabular}
        \caption{This table displays all average values of the error $e_{\text{synth}}$
            for all German states and Germany. The average is formed across all
            10 iteration.}
        \label{table:state_error}
    \end{center}
\end{table}




% -------------------------------------------------------------------