\chapter{Introduction}
\label{sec:concept}

\starthistory
110505 & Complete revision by Stefan Buehler. Also integrated text
         from ARTS2 article first submission.\\
2002-10 & Written, mainly by Stefan Buehler, some parts by Patrick Eriksson. 
%100421 & Section \ref{sec:concept:doc} rewritten by Stefan Buehler.\\
%090507 & Section on controlfiles and workspace moved 
%         and rewritten by Stefan Buehler.\\
%090424 & Complete revision and new text by Patrick Eriksson.\\
%080729 & Section on command line parameters updated by Stefan Buehler.\\
%020613 & Updated and extended by Stefan Buehler.\\
%000616 & ARTS concept described by Stefan Buehler. \\
\stophistory

\graphicspath{{Figs/concept/}}


This section introduces and describes the basic ideas underlying the
ARTS program. It also presents some terminology. You should read
it if you want to understand how the program works and how it can be
used efficiently.



\section{What is ARTS?}
%==============================

The Atmospheric Radiative Transfer Simulator, \textindex{ARTS}, is a software
for performing simulations of atmospheric radiative transfer. ARTS is a
relatively general and flexible program, where new calculation features can be
easily added. Originally, the development of ARTS was initiated to deal with
passive mm and sub-mm measurements. The radiation source for such measurements
is emission in the atmosphere or by the Earth's surface. Thermal IR radiation
is governed by the same basic physical principles and therefore this wavelength
region is also well handled in ARTS now. But ARTS contains so far no dedicated
methods for scattering of solar radiation and there is therefore a restriction
to simulations of longwave radiation (microwave to thermal IR). However, ARTS
can be used for basic studies of lonwave radiation fluxes, as for example in
\citet{buehler:recen:06} or \citet{john:under:06}. More lately, some support
for handling radio link calculations have been added.

One main application of ARTS should be to perform \textindex{retrievals} for
remote sensing data. A special feature of ARTS in this context is its high
flexibility when defining observation geometry (including scanning features)
and sensor characteristics. Jacobians (weighting functions) are also provided.

There exist two versions of ARTS. This user guide deals with the later
of the two versions \citep{eriksson:arts2:11}, here denoted as just
ARTS. \textindex{ARTS-1}, the first version of ARTS
\citep{buehler:artst:05}, can only handle 1D atmospheres with
unpolarised radiation and situations where scattering can be
neglected.  These restrictions have been removed in the current
version. A short summary of ARTS's main features is:
\begin{description}
\item[The \textindex{atmosphere}] can be 1D, 2D or 3D. That is, atmospheric
  variables (temperature, gas concentrations etc.) can be assumed to
  only vary in the vertical dimension (1D), to have no longitude
  variation (2D) or vary in all three spatial dimensions (3D).
\item[The \textindex{surface}] is by default assumed to be spherical. For 2D
  and 3D, a complete reference ellipsoid is used and the surface can have
  arbitrary shape.
\item[Polarisation] is fully described by using the Stokes formalism.
\item[Scattering] can be considered in several manners. Extinction from
  scatterers can be included in transmission type calculations.
  For radiance calculations of thermal emission (in contrast to solar radiation)
  there are two modules at hand to take the scattering into account: DOIT
  (Chapter~\ref{sec:scattering:doit}) and MC (Chapter~\ref{sec:scattering:mc}).
  The \textindex{scattering} particles are
  for efficiency reasons confined to a region of the atmosphere denoted as the
  cloudbox.
\item[Observation geometry] is free. That is, the forward model can be
  used to simulate ground-based, down-looking, limb sounding and
  balloon/aircraft measurements.
\item[Sensor characteristics] can be incorporated in a flexible and
  efficient manner.
\item[Jacobians,] the partial derivatives of simulated measurement
  with respect to forward model variables, can be provided for a
  number of variables, where analytical expressions are used as far as
  possible.
\end{description}
Details are found in later parts of the user guide. Use the table of
contents and the index for navigating through the user guide.



\section{Documentation}
%========================================
\label{sec:concept:doc}

We know that the ARTS documentation is far from perfect. It is quite
complete in some areas, but patchy in others. It also contains bugs
and more serious errors. We are struggling to make it as good as
possible, but it is ongoing work, and we do not have any direct
funding for it. All help from users to extend or correct the
documentation is highly appreciated! Having said that, the
documentation that already is available for ARTS is described in the
following subsections.

\subsection{Guide documents}

\begin{description}
\item[ARTS User Guide:] This document.
\item[ARTS Developer Guide:] Guide for ARTS developers.
\item[ARTS Theory:] Describes the theoretical basis for some parts of ARTS.
\end{description}

\subsection{Articles}

\begin{description}
\item[\citet{buehler:artst:05}:] General description of the old ARTS
  version without scattering. Many basic features are still the same,
  so this article is relevant also for the current ARTS version.
\item[\citet{eriksson:arts2:11}:] Introduction and overview to ARTS-2.
\item[\citet{emde04:_doit_jgr}:] Describes the Discrete Ordinate Iterative
  method (DOIT) for handling scattering.
\item[\citet{davisetal:04}:] Describes the Monte Carlo scattering method.
\item[\citet{eriksson:06}:] Describes the calculation approach for the
  incorporation of sensor characteristics.
\item[\citet{buehler:effic:10}:] Describes a method to efficiently handle
  broadband infrared channels, that is implemented in ARTS.
\item[\citet{buehler:absor:11}:] Describes the absorption look-up table
  approach used inside ARTS.
\end{description}

\subsection{Built-in documentation}
\label{sec:built-in_doc}

ARTS contains \textindex{built-in documentation} for all functions and
variables that are directly visible to the user (in ARTS terminology called
workspace functions and workspace variables; they are explained in more depth
further below).
The easiest way to access this documentation is on the web page
\artsurl{\docserver}. Alternatively, start ARTS with
\begin{code}
  arts -s
\end{code}
or
\begin{code}
  arts --docserver
\end{code}
and then point your browser to \url{http://localhost:9000/}.

This user guide also contains links to the built-in
documentation.  If you are reading the pdf file on a computer, then
names of ARTS objects, such as \builtindoc{f\_grid}, will be links to
the corresponding entries in the built-in documentation. 

\subsection{Test and include controlfiles}

ARTS calculations are governed by controlfiles (see below). The ARTS
distribution already comes with a large number of controlfiles, which fall into
two categories: includes and tests. They are described in more detail below,
but already mentioned here as an important source of information for new users.
In particular, ARTS already comes with controlfiles to simulate some well-known
instruments, such as for example MHS or HIRS. \\
\FIXME{Control file structure changed, update this section!}

\subsection{Source code documentation}

All internal ARTS functions are documented with \textindex{DOXYGEN} at the
source code level. This documentation is intended mostly for those who want to
do ARTS development work. It can be accessed at
\artsurl{misc/arts-doc/doxygen/html/}.

\subsection{Build instructions}

Instructions on how to configure and compile the ARTS source code can
be found in the file \fileindex{README} in the top directory of the
ARTS distribution.

\subsection{Command line parameters}

ARTS offers a number of useful \textindex{command line parameters}. In
general, there is a short form and a long form for each parameter. The
short form consists of a minus sign and a single letter, whereas the
long form consists of two minus signs and a descriptive name. To get a
full list of available command line parameters, type
\begin{code}
  arts -h
\end{code}
or
\begin{code}
  arts --help
\end{code}

\section{ARTS as a scripting language}

One of the main goals in the ARTS development was to make the program
as flexible as possible, so that it can be used for a wide range of
applications and new features can be added in a relatively simple
manner. As a result, ARTS behaves like a \textindex{scripting
  language}. An ARTS \textindex{controlfile} contains a sequence of
instructions. When ARTS is executed, the controlfile is parsed, and
then the instructions are executed sequentially. Controlfile
\shortcode{somefile.arts} is executed by running
\begin{code}
  arts somefile.arts
\end{code}
A minimal ARTS controlfile example (the well-known `Hello World' program) is
given in Figure \ref{fig:hello}. In this example, the variable \shortcode{s} is
called a \emph{\textindex{workspace variable}}. We use this name to distinguish
it from the variables that appear internally in the ARTS source code.

\begin{figure}
\footnotesize
\begin{minipage}[t]{0.35\hsize}
\begin{code}
Arts2 {
StringCreate(s)
StringSet(s,
   "Hello World")
Print(s)
}
\end{code}
\end{minipage}
\hspace*{\fill}
\begin{minipage}[t]{0.5\hsize}
\begin{code}
arts-1.14.122
Executing Arts
{
- StringCreate
- StringSet
- Print
Hello World
}
This run took 0.03s (0.03s CPU time)
Everything seems fine. Goodbye.
\end{code}
\end{minipage}
\caption{Left: A minimal ARTS controlfile example. Right: ARTS output
  when running this controlfile. } 
\label{fig:hello}
\end{figure}

In a similar spirit, the functions \wsmindex{StringCreate},
\wsmindex{StringSet}, and \wsmindex{Print} in the example are called
\emph{\textindex{workspace methods}}. We use this name to distinguish
them from the functions that appear internally in the ARTS source
code. For brevity, we may sometimes drop the workspace qualifier and
refer to them just as \textindex{methods}.

ARTS consists roughly of three parts. Firstly, the ARTS core contains the
controlfile \textindex{parser}, and the \textindex{engine} that executes the controlfile. This part is
quite compact and constitutes only a small fraction of the total source code.
Secondly, there is a large collection of workspace methods that can be used to
carry out various sub-tasks (at the time of writing approximately 300).
Thirdly, there is a large number of predefined workspace variables (at the time
of writing more than 200). These predefined variables make it easier to set up
controlfiles, since they provide hints on how the different workspace methods
fit together.

ARTS has built-in documentation for all workspace methods and
variables, which can be accessed as described in Section
\ref{sec:built-in_doc}.  In this user guide, just clicking on the name
of a variable or method will take you directly to the built-in
documentation for that object.  Below, we will discuss workspace
variables and methods in some more detail and give more examples.

\subsection{Workspace variables}

Workspace variables (such as the variable \shortcode{s} in Figure
\ref{fig:hello}) are the variables that are manipulated by the
workspace methods during the execution of an ARTS
controlfile. Workspace variables belong to different \emph{groups}
(Index, String, Vector, Matrix, etc.). The built-in documentation
lists all \textindex{groups}, at the time of writing there are
approximately 60 of them.

As the example in Figure \ref{fig:hello} shows, \textindex{workspace
  variables} can be freely created by the user with methods like
\builtindoc{StringCreate}, \builtindoc{VectorCreate}, and so on.
Each group has its own create method.

However, in most cases it is not necessary to create new variables in
this way, since a lot of variables are predefined in ARTS. The
built-in documentation describes all \textindex{predefined
  variables}. As an example, Figure \ref{fig:f_grid} shows the
description for the variable \wsvindex{f\_grid}, which stores the
frequency grid and is used as input by many workspace methods, for
example those that calculate absorption coefficients. The built-in
documentation also lists all methods that take \builtindoc{f\_grid} as
input and all methods that produce \builtindoc{f\_grid} as output.

\begin{figure}
\begin{code}
*-------------------------------------------------------*
Workspace variable = f_grid
---------------------------------------------------------
The frequency grid for monochromatic pencil beam 
calculations.

Usage: Set by the user.
 
Unit:  Hz
---------------------------------------------------------
Group = Vector
*-------------------------------------------------------*
\end{code}
\caption{Built-in documentation for variable \builtindoc{f\_grid}, obtained by
  command `\shortcode{arts -d f\_grid}', or on page
  \artsurl{\docserver variables/f\_grid}.} 
\label{fig:f_grid}
\end{figure}

\subsection{Workspace methods}
%
As shown in Figure \ref{fig:hello}, names of \textindex{workspace methods} in an ARTS
controlfile are followed by their output and input arguments (workspace
variables) in parentheses. (`\shortcode{Print(s)}' prints the content of variable
\shortcode{s}.)

From the methods point of view, arguments can be \emph{output},
\emph{input}, or both, and additionally they can be either
\emph{\textindex{specific}} (= referring to a predefined variable) or
\emph{\textindex{generic}} (= not referring to a predefined
variable). To illustrate this, Figure \ref{fig:WriteXML} shows the
built-in documentation for method \wsmindex{WriteXML}, the most
common method to write ARTS variables to a file. The list at the
bottom of the documentation shows that
\wsvindex{output\_file\_format} is a specific input argument, and
that \shortcode{v} and \shortcode{filename} are generic input
arguments.

\begin{figure}
\footnotesize
\begin{code}
*---------------------------------------------------------*
Workspace method = WriteXML
-----------------------------------------------------------
Writes a workspace variable to an XML file.

This method can write variables of any group.

If the filename is omitted, the variable is written
to <basename>.<variable_name>.xml.

Synopsis:

WriteXML( output_file_format, v, filename )

Authors: Oliver Lemke

Variables:

IN    output_file_format (String): Output file format.
GIN   v (Any): Variable to be saved.
GIN   filename (String, Default: ""): Name of the XML file.
*---------------------------------------------------------*
\end{code}
\caption{Built-in documentation for method \builtindoc{WriteXML}, obtained by
  command `\shortcode{arts -d WriteXML}', or on page
  \artsurl{\docserver methods/WriteXML}.}
\label{fig:WriteXML}
\end{figure}

What this means is that \builtindoc{output\_file\_format} already automatically
exists as a variable, whereas \shortcode{v} and \shortcode{filename}
do not. The built-in documentation provides descriptions also of these generic
arguments and lists the allowed values.

The predefined variables, combined with specific method arguments, are meant to
help in combining methods into meaningful calculations. Predefined variables
are typically relevant for more than one method. For example, variable
\builtindoc{output\_file\_format} can be used to change the format of all produced
files at the same time. However, the use of a specific variable in the
controlfile is not mandatory, so `\shortcode{WriteXML( output\_file\_format, v,
  "test.xml")}', `\shortcode{WriteXML( "ascii", v, "test.xml" )}', and
`\shortcode{WriteXML( my\_format, v, "test.xml" )}' are all allowed. (But in the
last example the variable \shortcode{my\_format} must have been defined before.)

Besides the variable names, the built-in documentation also lists the allowed
variable \textindex{groups} (or types). In the example, the groups for workspace variable
\shortcode{v} are `Any', which means that \shortcode{v} can belong to any of the
known groups. The group for \shortcode{filename} is `String', which means that a
string is expected here. Method arguments can be a literal, as in
`\shortcode{WriteXML( "ascii", v, "test.xml" )}', or a variable, as in
`\shortcode{WriteXML( "ascii", v, s )}', where in the latter case variable
\shortcode{s} must be already defined.

The built-in documentation further states that argument
\shortcode{filename} has a \textindex{default value} (in this case the
empty string). Because of this, the argument can actually be omitted,
so `\shortcode{WriteXML( "ascii", v )}' will also work.

Alternatively, workspace methods can be called with named arguments. All omitted
arguments will be set to their default values. \shortcode{WriteXML( in=v )} is
equivalent to calling \shortcode{WriteXML( output\_file\_format, v )}. Note that
named arguments can not be mixed with positional arguments.

One additional rule has to be mentioned here. If all arguments to a
method are specific, and the user wants to use all the predefined
variables, then the entire argument list (including parentheses) may
be omitted. 


\subsection{Agendas}
\label{sec:agendas}
%
Agendas are a special group of workspace variables, which allow to
modify how a calculation is performed. A variable of group
\textindex{agenda} holds a list of workspace method calls. It can be
executed, which means that the method calls it contains are executed one
after another.

Figure \ref{fig:agendas} gives an example, for the agenda
\wsaindex{propmat\_clearsky\_agenda}. Several radiative transfer methods use this
agenda as input variable. When they need local absorption coefficients for a
point in the atmosphere, they execute the agenda with the local pressure,
temperature, and trace gas volume mixing ratio values as inputs. The agenda
then provides absorption coefficients as output.

\begin{figure}
\begin{code}
*-------------------------------------------------------------------*
Workspace variable = propmat_clearsky_agenda
---------------------------------------------------------------------

This agenda calculates absorption coefficients for all gas species
as a function of the given atmospheric state for one point in the
atmosphere. The result is returned in *propmat_clearsky*, the
atmospheric state has to be specified by *rtp_pressure*,
*rtp_temperature*, *rtp_mag*, and *rtp_vmr*.

The methods inside this agenda may require a lot of additional
input variables, such as *f_grid*, *species*, etc.
---------------------------------------------------------------------
Group  = Agenda
Output = propmat_clearsky
Input  = f_grid, rtp_doppler, rtp_mag, rtp_pressure,
         rtp_temperature, rtp_vmr
*-------------------------------------------------------------------*
\end{code}
\begin{minipage}[t]{0.48\hsize}
\begin{code}
AgendaSet(
propmat_clearsky_agenda)
{
  propmat_clearskyInit
  propmat_clearskyAddOnTheFly
  propmat_clearskyAddZeeman
}
\end{code}
\end{minipage}
\hspace*{\fill}
\rule[-75pt]{.5pt}{85pt}
\hspace*{\fill}
\begin{minipage}[t]{0.48\hsize}
\begin{code}
AgendaSet(
propmat_clearsky_agenda)
{
  Ignore(rtp_mag)
  propmat_clearskyInit
  propmat_clearskyAddFromLookup
}
\end{code}
\end{minipage}
\caption{Top: Built-in documentation for variable
  \builtindoc{propmat\_clearsky\_agenda}, obtained by command
  `\shortcode{arts -d propmat\_clearsky\_agenda}', or on page
  \artsurl{\docserver agendas/propmat\_clearsky\_agenda}. Bottom
  left: Controlfile agenda definition for line-by-line absorption
  calculation. Bottom right: Controlfile agenda definition to extract
  absorption from a pre-calculated lookup table.}
\label{fig:agendas}
\end{figure}

The bottom of Figure \ref{fig:agendas} shows two different ways how this agenda
could be defined in the controlfile. In the first case a line-by-line
absorption calculation is performed when the agenda is executed (every time
absorption coefficients are needed). In the second case the absorption
coefficients are extracted from a pre-calculated lookup table.

On invocation, an agenda executes its methods one after the
other. The inputs and outputs defined for the agenda must be satisfied
by the invoked workspace methods. E.g., if an agenda has
\builtindoc{propmat\_clearsky} in its list of output workspace
variables, at least one workspace method which generates
\builtindoc{propmat\_clearsky} must be added to the agenda in the
controlfile.

\section{Include controlfiles}

ARTS controlfiles can \emph{\textindex{include}} other ARTS
controlfiles, which is achieved by statements such as
\shortcode{INCLUDE "general.arts"}.  This mechanism is used to
predefine general default settings, settings for typical applications,
and/or settings for the simulation of well-known instruments.  A
variety of include controlfiles are collected in directory
\fileindex{controlfiles} of the ARTS distribution.

You should normally at least include the file \fileindex{general.arts}, which
contains general default settings. Because giving the full path for every
include file is inconvenient, ARTS will look for include files in a special
directory. This can be set by the command line option \shortcode{-I
  <includepath>}, or by the environment variable
\shortcode{\textindex{ARTS\_INCLUDE\_PATH}}. If none of these are set, ARTS
will assume the include path to point to the includes directory in the ARTS
distribution.
The file \fileindex{agendas.arts} is also useful, because it predefines
agendas with settings suitable for many applications.
\FIXME{Revise this section after general.arts has been changed.}


\section{Test controlfiles}

The directory \fileindex{controlfiles} in the ARTS distribution contains some
test and \textindex{example controlfiles}.  You should study them to learn more about how
the program works. You can run these controlfiles like this:
\begin{code}
  arts TestAbs.arts
\end{code}
This assumes that you are in the directory where the control
file resides, and that the \shortcode{arts} executable is in
your path. 

Alternatively, you can run a standard set of the test controlfiles by
going to the build directory, and say
\begin{code}
  make check
\end{code}
This standard set of test is run by us on every automatic build, that
means every time a new ARTS version is submitted to the GitHub
repository. If your ARTS installation is healthy, \shortcode{make
  check} should run through without any errors. (See file
\fileindex{README} in the ARTS distribution for detailed instructions
on how to build ARTS.)\\
\FIXME{The tests are now structured differently. Update the text.}



\section{Verbosity levels}

The command line parameter 
\begin{code}
  arts -r
\end{code}
or
\begin{code}
  arts --reporting
\end{code}
can be used to set how much output ARTS produces. You can supply a
three-digit integer here. Each digit can have a value between 0 and 3.

The last digit determines, how verbose ARTS is in its
\textindex{report file}. If it is 0, the report file will be empty, if
it is 3 it will be longest.

The middle digit determines, how verbose ARTS is on the screen
(stdout). The meaning of the values is exactly as for the report
file. 

The first digit is special. It determines how much you will see of the
output of agendas (other than the main program agenda). Normally, you
do not want to see this output, since many agendas are called over and
over again in a normal program run. 

The agenda \textindex{verbosity} applies in addition to the screen or
file verbosity. For example, if you set the \textindex{reporting
  level} to `123', you will get:
\begin{itemize}
\item From the main agenda: Level 1-2 outputs to the screen, and level
  1-3 outputs to the report file.
\item From all other agendas: Only level 1 outputs to both screen and
  report file.
\end{itemize}
As another example, if you set the reporting level to `120' the
report file will be empty.

The default setting for ARTS (if you do not use the command line flag)
is `010', i.e., only the important messages to the screen, nothing to
the report file, and no sub-agenda output.


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "uguide"
%%% End: 
