\chapter{Complete calculations}
 \label{sec:complcalcs}

\starthistory
 130219 & First version by Patrick Eriksson.\\
\stophistory

\graphicspath{{Figs/rte/}}

This chapter outlines how complete radiative transfer simulations are
performed. ARTS is not only performing atmospheric radiative transfer, also
sensor characteristics can be considered. As a consequence, the topics of this
chapter include the distinction between monochromatic pencil beam and ``full''
calculations, and how the sensor is introduced.



\section{Overview}
%
An attempt to illustrate a ``standard'' ARTS calculation is found in Figure~
\ref{fig:ycalc_flow}. The figure shows that most calculation tasks are handled
by an agenda. For example, \builtindoc{ppath\_agenda} has the task of
determining the propagation path for the given observation geometry. In
principle, the agenda could solve the task by loading data from a file, but
most likely it will use some of the dedicated workspace methods. These
workspace methods are targeted towards different observation types, for
example, radio link calculations require special consideration. Anyhow, the
main message here is that by using the agenda concept, a very high degree of
flexibility can be achieved and new features can be added fairly simply. On the
other hand, the concept require that the user actually apply methods that make
sense for the agenda. The code of ARTS performs some consistency checks of the
agenda output, but this can only catch some types of mistakes.

Complete radiative transfer calculations are normally performed by
\wsmindex{yCalc}. This method incorporates sensor responses and has the
variable \wsvindex{y} as main output. The letter y here refers to the
measurement vector, \MsrVct, found in the formalism of
\citet{rodgers:90,rodgers:00} (see also Sec.~\ref{T-sec:formalism:wfuns} of
\theory). The vector can hold anything from a single value, to a high number of
spectra appended. The spectra in the last case can correspond to a limb
sounding sequence (hence measured for different zenith angles), or even be
measured by different sensors. In any case, the data in \MsrVct\ contain likely
significant impact of different parts of the sensor used, such as the angular
weighting by the antenna pattern. 

\begin{figure*}[p]
 \begin{center}
  \includegraphics*[clip,trim=25 30 25 40,width=0.99\hsize]{ycalc_flow}\\
  {\small (For caption see top of the next page.)}
 \end{center}
\end{figure*}
% This figure was produced by LibreOffice draw: ycalc_flow.odg

\begin{figure}[t]
  \caption{A flowchart of a radiative transfer calculation (on previous page),
    using \wsmindex{yCalc} or \wsmindex{iyCalc}. The figure assumes that
    \wsaindex{iy\_main\_agenda} holds \wsmindex{iyEmissionStandard}, that
    represents the most complex case. Some important input data are shown in
    orange, and main output are shown in green. The two main methods are
    plotted as grey boxes, while agendas are shown as ovals. For cases when
    \builtindoc{iy\_sub\_agenda} not is defined (the most common situation),
    the agendas communicate directly with \builtindoc{iy\_main\_agenda}. For
    connections between methods and agendas the arrows show the direction of
    output (calculated) data. Every call of an agenda involves also some input
    data (settings), but this aspect has been ignored for clarity reasons. Data
    entering an agenda along a common line indicates mutually independent
    options. For example, absorption can be calculated "on-the-fly" from basic
    spectroscopic data or be extracted from a pre-calculated look-up table (see
    bottom-right corner). The dotted lines indicate that some methods and
    agendas can make further calls of \builtindoc{iy\_main\_agenda}.}
  \label{fig:ycalc_flow}
\end{figure}

On the other hand, atmospheric radiative transfer is performed for
monochromatic frequencies, along pencil beam directions. The outcome of one
such calculation is the Stokes vector for each frequency considered, and as
workspace variable this quantity is denoted as \wsvindex{iy}. Please, note the
distinction to \builtindoc{y}, that can contain information from a number of
monochromatic pencil beam calculations, as shown in
Section~\ref{sec:fm_defs:seqsandblocks}.

Further, the term ``iy'' is not restricted to the final outcome of atmospheric
radiative transfer, it is also used to indicate that quantities and operations
are of monochromatic pencil beam character. For example, the agenda returning
the radiation entering the atmosphere from space is named as
\builtindoc{iy\_space\_agenda} to indicate that the agenda output is directly
associated with the calculation of \builtindoc{iy}. If only a single
\builtindoc{iy} is of concern, the radiative transfer can be performed by
\wsmindex{iyCalc}, having a smaller set of input arguments than
\builtindoc{yCalc}.

The output is not restricted to \builtindoc{y} or \builtindoc{iy}, also some
auxiliary data can be extracted as described in Section~\ref{sec:fm_defs:aux}.
In addition, \builtindoc{yCalc} can also provide weighting functions
(Chapter~\ref{sec:wfuns}).








\section{Compulsory sensor and data reduction variables}
%==============================================================================
\label{sec:fm_defs:sensor1}

The instrument real or hypothetical, that detects the simulated radiation is
denoted as the sensor\index{sensor, the}. The forward model is constructed in
such way that a sensor must exist. For cases when only monochromatic pencil
beam radiation is of interest, the positions and directions for which the
radiation shall be calculated are given by specifying an imaginary sensor with
infinite frequency and angular resolution. The workspace variables for the
sensor that always must be specified are \builtindoc{sensor\_response},
\builtindoc{sensor\_pos}, \builtindoc{sensor\_los} and
\builtindoc{mblock\_dlos\_grid}. These variables are presented separately
below.


\subsection{Sensor position\index{sensor position}}
%===================
\label{sec:fm_defs:sensorpos}

The observation positions of the sensor are stored in
\wsvindex{sensor\_pos}. This is a matrix where each row corresponds to
a sensor position. The number of columns in the matrix equals the
atmospheric dimensionality (1 column for 1D etc.). The columns of the
matrix (from first to last) are geometrical altitude, latitude and longitude.
Accordingly, row $i$ of \builtindoc{sensor\_pos} for a 3D case is
$(\aAlt{i},\aLat{i},\aLon{i})$. The sensor position can be set to any
value, but the resulting propagation paths (also dependent on
\builtindoc{sensor\_los}) must be valid with respect to the model
atmosphere (see Section~\ref{sec:fm_defs:ppaths}). An obviously
incorrect choice is to place the senor below the surface altitude. If
the sensor is placed inside the model atmosphere, any sensor
line-of-sight is allowed, this including the cases that the sensor is
placed on the surface looking down, and that the sensor is placed
inside the cloud box. 

One or several spectra can be calculated for each position as described in
Section~\ref{sec:fm_defs:seqsandblocks}. The corresponding workspace variable
for single pencil beam calculations is \wsvindex{rte\_pos}, that is an input
argument to e.g.\ \wsaindex{iy\_main\_agenda} and \wsmindex{iyCalc}.



\subsection{Line-of-sight\index{line-of-sight}}
%===================
\label{sec:fm_defs:los}

The viewing direction of the sensor, the line-of-sight, is described
by two angles, the zenith angle (\ZntAng) and the azimuth angle
(\AzmAng). The zenith angle exists for all atmospheric
dimensionalities, while the azimuth angle is defined only for 3D.
The term line-of-sight is not only used in connection with the sensor,
it is also used to describe the local propagation direction along the
path taken by the observed radiation
(Section~\ref{sec:fm_defs:ppaths}).  The zenith and azimuth angles
are defined in an identical way in both of these contexts (sensor
pointing direction; local propagation direction). This is expected as
the position of the sensor is the end point of the propagation path.
The sensor line-of-sight is the direction the antenna is pointed to
receive the radiation. The line-of-sight for propagation paths is
defined likewise, it is the direction in which a hypothetical sensor
must be placed to receive the radiation along the propagation path at
the point of interest. This means that the line-of-sight and the
photons are going in opposite directions. As a true sensor has a
finite spatial resolution (described by the antenna pattern),
theoretically there is an infinite number of line-of-sights associated
with the sensor, but in the forward model, spectra are only calculated
for a discrete set of directions. If a sensor line-of-sight is
mentioned without any comments, it refers to the direction in which
the centre of the antenna pattern is directed.

\begin{figure}
 \begin{center}
  \begin{minipage}[c]{0.6\textwidth}
   \includegraphics*[width=0.99\hsize]{za_and_aa_angles}
  \end{minipage}%
  \begin{minipage}[c]{0.4\textwidth}
   \caption{Definition of zenith angle, \ZntAng, and azimuth angle, 
       \AzmAng, for a line-of-sight. The figure shows a line-of-sight
       with a negative azimuth angle.}
   \label{fig:fm_defs:los}
  \end{minipage}
 \end{center}
\end{figure}           
 
The \textindex{zenith angle}, \ZntAng, is simply the angle between the
line-of-sight and the zenith direction (Figure~\ref{fig:fm_defs:los}).
The valid range for 1D and 3D cases is $[0,180\degree]$. In the case
of 2D, zenith angles down to -180\degree\ are also allowed, where the
distinction is that positive angles mean a viewing direction towards
higher latitudes, and negative angles mean a viewing direction towards
lower latitudes. It should be mentioned that the zenith and nadir
directions are here defined to be along the line passing the centre of
the coordinate system and the point of concern
(Section~\ref{sec:ppath:geoid}). A nadir observation,
$\ZntAng=180\degree$, is thus a measurement towards the centre of the
coordinate system.

The \textindex{azimuth angle}, \AzmAng, is given with respect to the
\textindex{meridian plane}.  That is, the plane going through the
north and south poles of the coordinate system $(\Lat=\pm90\degree)$
and the sensor. The valid range is $[-180\degree,180\degree]$ where
angles are counted clockwise; 0\degree means that the viewing or
propagation direction is north-wise and +90\degree\ means that the
direction of concern goes eastward. This definition does not work for
position on the poles. To cover these special cases, the definition is
extended to say that for positions on the poles the azimuth angle
equals the longitude along the viewing direction. For example, if
standing on any of the poles and the viewing direction is towards
Greenwich, the azimuth angle is 0\degree.

The sensor line-of-sights are stored in \wsvindex{sensor\_los}. This workspace
variable is a matrix, where the first column holds zenith angles and the second
column is azimuth angles. For 1D and 2D there is only one column in the matrix,
while for 3D a row $i$ of the matrix is $(\aZntAng{i},\aAzmAng{i})$. The number
of rows for \builtindoc{sensor\_los} must be the same as for
\builtindoc{sensor\_pos}. The correspondance to \builtindoc{rte\_pos} is
\wsvindex{rte\_los}.


\subsection{Sensor characteristics and data reduction}
%===================
\label{sec:fm_defs:sensorchar}

The term ``sensor characteristics''\index{sensor characteristics} is
used here as a comprehensive term for the response of all sensor parts
that affect how the field of monochromatic pencil beam intensities are
translated to the recorded spectrum. For example, the antenna pattern,
the side-band filtering and response of the spectrometer channels are
normally the most important characteristics of a microwave heterodyne
radiometer. Any processing of the spectral data that takes place
before the retrieval is denoted as \textindex{data reduction}. The
most common processing is to represent the original spectra with a
smaller set of values, that is, a reduction of the data size. The most
common data reduction techniques is binning and Hotelling
transformation by an eigenvector expansion.

In ARTS, the influence of sensor characteristics and data reduction is
incorporated by transfer matrices\index{sensor transfer matrix}. The
application of these transfer matrices assumes that each step is a
linear operation, which should be the case for the response of the
parts of a well designed instrument. Non-linear data reduction could
be handled by special workspace methods.

The sensor and data reduction are described as a series of units, each having
its own transfer matrix. There is only one compulsory transfer matrix and it is
\wsvindex{sensor\_response}. There are several workspace variables associated
with this transfer matrix where \wsvindex{antenna\_dim} and
\wsvindex{mblock\_dlos\_grid} are the compulsory ones.

The variable \builtindoc{antenna\_dim} gives the dimensionality of the antenna
pattern\index{antenna pattern dimensionality}, where the options are 1 and 2,
standing for 1D and 2D, respectively. A 1D antenna dimensionality means that
the azimuth extension of the antenna pattern is neglected, there is only a
zenith angle variation of the response. A 2D antenna pattern is converted to a
1D pattern by integrating the azimuth response for each zenith angle. 

See further Chapter~\ref{sec:sensor}, where inclusion of sensor characteristics
is described in .



\section{Measurement sequences and blocks}
%===================
\label{sec:fm_defs:seqsandblocks}

The series of observations modelled by the simulations is denoted as the
\textindex{measurement sequence}. That is, a measurement sequence covers all
spectra recorded at all considered sensor positions. A measurement sequence
consists of one or several \textindex{measurement block}s. A measurement block
can be treated as a measurement cycle that is repeated, an integer number of
times, to form the measurement sequence.

A measurement block covers one or several recorded spectra, depending
on the measurement conditions and the atmospheric dimensionality. A
block can consist of several spectra when there is no effective motion
of the sensor with respect to the atmospheric fields. It should be
noted that for 1D cases, a motion along a constant altitude has no
influence on the simulated spectra as the same atmospheric fields are
seen for a given viewing direction. It is favourable, if possible, to
handle all spectra as a single block, instead of using a block for
each sensor position. This is the case as the antenna patterns for the
different line-of-sights are normally overlapping and a pencil beam
spectrum can be used in connection with several measurement spectra to
estimate the intensity field. If a measurement sequence is divided
into several blocks even if a single block would be sufficient, pencil
beam spectra for basically identical propagation paths can be
calculated several times, which of course will increase the
computational time. To summarise, for cases when the sensor is not in
motion, or with a 1D atmosphere and a sensor not moving vertically,
the aim should be to use a single block for the measurement sequence.

If not a single block is used, the standard option should be that the
blocks cover one spectrum each. There could exist reasons to select an
intermediate solution, to let the extent of the blocks be several
spectra (but not the full measurement sequence). This could be the
case when the atmospheric dimensionality is 2D or 3D, and the sensor
is moving but the movement during some subsequent spectra can be
neglected. 

The pencil beam spectra for each line-of-sight are appended vertically to form
a common vector, \aMpiVct{b}. Values are put in following the order in
\builtindoc{f\_grid}. Hence, the frequencies for this vector are
\begin{equation}
  \aMpiVct{b} = 
  \left[ \begin{array}{c} 
     \left[
          \begin{array}{c} \aFrq{1}\\\vdots\\\aFrq{n} \end{array} 
     \right] \\
     \vdots \\
     \left[
          \begin{array}{c} \aFrq{1}\\\vdots\\\aFrq{n} \end{array} 
     \right]
     \end{array} \right]
  \label{eq:fm_defs:freqs_of_ib}
\end{equation}
where \aFrq{i} is element $i$ of \builtindoc{f\_grid} and $n$ the length of the
same vector. The order of the angles inside \builtindoc{mblock\_dlos\_grid} is
followed when looping the pencil beam directions

The workspace variable \builtindoc{sensor\_response} is here denoted as
\aSnsMtr{b}. It is applied on each \aMpiVct{b} and the results are
appended vertically, following the order of the positions in
\builtindoc{sensor\_pos}
\begin{equation}
  \MsrVct = \left[ \begin{array}{c} \aSnsMtr{b}\aMpiVct{{b,1}} \\ 
                                    \aSnsMtr{b}\aMpiVct{{b,2}} \\
                                    \vdots                     \\
                                    \aSnsMtr{b}\aMpiVct{{b,n}} 
            \end{array} \right]
  \label{eq:fm_defs:measseq}
\end{equation}
where $1$ indicates the first sensor position etc. 

It should be noted that the compulsory sensor variables give no
information about the content of the obtained \MsrVct, as it is not
clear which parts and features the block transfer matrix covers. If
\aSnsMtr{b} only incorporates the antenna pattern, the result is a set
of hypothetical spectra corresponding to a point inside the sensor. On
the other hand, if \aSnsMtr{b} includes the whole of the sensor and an
eigenvector data reduction, the result is not even a spectrum in
traditional way, it is just a column of coefficients with a vague
physical meaning.


