Gnuastro 0.19 released

posted by Roy Schestowitz on Oct 24, 2022

Dear all,


I am happy to announce the availability of GNU Astronomy Utilities
(Gnuastro) version 0.19. It is packed with many exciting new features
and bug fixes (see below).


Gnuastro is an official GNU package, consisting of various
command-line programs, C/C++ library functions and Makefile extensions
for the manipulation and analysis of (astronomical) data. All the
programs share the same basic command-line user interface (modeled on
GNU Coreutils). For the full list of Gnuastro's library, programs, and
several comprehensive tutorials (recommended place to start using
Gnuastro), please see the links below respectively:


  https://www.gnu.org/s/gnuastro/manual/html_node/Gnuastro-library.html


https://www.gnu.org/s/gnuastro/manual/html_node/Gnuastro-programs-list.html
  https://www.gnu.org/software/gnuastro/manual/html_node/Tutorials.html


The full list of improvements in this version is available in the NEWS
file within the source, and also in [1] below. The new features are so
exciting/useful that Pedram (author of the newly added Warp features)
has recorded a video to show the tip of the iceberg in the Warp and
ConvertType programs. Please watch it to get a hands-on feeling of the
power of these new capabilities using SDSS and J-PLUS images (and go
the manual for the rest!):


  https://peertube.stream/w/uq7SBDYZS1HRtJwCkbcDsz (17.5 minutes)


If you have any questions or ideas, or just want to stay up to date
with tips and other discussions, feel free to visit our Matrix chat:


  #gnuastro:openastronomy.org


Here is the compressed source and the GPG detached signature for this
release. To uncompress Lzip tarballs, see [2]. To check the validity
of the tarballs using the GPG detached signature (*.sig) see [3]:


  https://ftp.gnu.org/gnu/gnuastro/gnuastro-0.19.tar.lz     (4.3MB)
  https://ftp.gnu.org/gnu/gnuastro/gnuastro-0.19.tar.lz.sig (833B)
  https://ftp.gnu.org/gnu/gnuastro/gnuastro-0.19.tar.gz     (6.8MB)
  https://ftp.gnu.org/gnu/gnuastro/gnuastro-0.19.tar.gz.sig (833B)


You can use a mirror for higher download bandwidth:


  https://ftpmirror.gnu.org/gnuastro/gnuastro-0.19.tar.lz
  https://ftpmirror.gnu.org/gnuastro/gnuastro-0.19.tar.lz.sig
  https://ftpmirror.gnu.org/gnuastro/gnuastro-0.19.tar.gz
  https://ftpmirror.gnu.org/gnuastro/gnuastro-0.19.tar.gz.sig


Here are the SHA1 and SHA256 checksums (other ways to check if the
tarball you download is what we distributed). Just note that the
SHA256 checksum is base64 encoded, instead of the hexadecimal encoding
that most checksum tools default to (see [4] on how to generate it).
For the list of software used to bootstrap this tarball, see [5].


fb53193886ca967a17bd8daf85835e8ad2b57780  gnuastro-0.19.tar.lz
f3UQsxNt2P9AxyVfe6DWLWn/3LU0OQoZc7w6+kkcWKQ  gnuastro-0.19.tar.lz
adc6fcbf7ca476ffd3f6c4468527210ffeaff8b4  gnuastro-0.19.tar.gz
4bPNW0sSb/J34vSOit8BA9Z/wK0Hz5o9OqfgVSlDDjU  gnuastro-0.19.tar.gz


For their direct contribution to this version's source code, I am very
grateful to Pedram Ashofteh-Ardakani (9 commits), Sepideh Eskandarlou
(6), Raul Infante-Sainz (6), Faezeh Bidjarchian (4), Jash Shah (3),
Marjan Akbari (2) and Elham Saremi (1). I am also grateful to (in
alphabetical order) to Marjan Akbari, Faezeh Bidjarchian, Sepideh
Eskandarlou, Giulia Golini, Raul Infante-Sainz, Teet Kuutma, Irene
Pintos Castro, Nafise Sedighi and Richard Stallman for their good
suggestions or reported bugs that have been implemented in this
release.


If any of Gnuastro's programs or libraries are useful in your work,
please cite _and_ acknowledge them. For citation and acknowledgment
guidelines, run the relevant programs with a `--cite' option (it can
be different for different programs, so run it for all the programs
you use). Citations _and_ acknowledgments are vital for the continued
work on Gnuastro, so please don't forget to support us by doing so.


Best wishes,
Mohammad


--
Staff Researcher
Centro de Estudios de Física del Cosmos de Aragón (CEFCA),
Plaza San Juan 1, Planta 2, Teruel 44001, Spain


[1] Noteworthy updates in Gnuastro 0.19 (released on October 24th, 2022)


** New features


  Book:
  - Two new sections added to the "General program usage tutorial" for
    describing how to prepare a FITS image for high quality publication (in
    PDF), and drawing vector graphics marks from a catalog over it (for
    example to show your selected galaxies in the field of view). The use
    the newly added features of ConvertType.


  Arithmetic:
  - Added new type operators and physical constants. All are also available
    in Table's column arithmetic also).
    - e: Base of the natural logarithm (no units).
    - pi: Fraction of Circle cirumference to diameter (no units).
    - c: speed of light in vaccume (in units of m/s).
    - G: Gravitational constant (in units of m^3/kg/s^2).
    - h: Plank's constant (in units of J/Hz).
    - au: Astronomical Units (in units of meters).
    - ly: Light years (in units of meters).
    - avogadro: Avogadro's constant (in units of 1/mol).
    - fine-structure: Fine structure constant (no units).
    - counts-to-sb: convert counts to surface brightness (mag/arcsec^2).
    - sb-to-counts: convert surface brightness (mag/arcsec^2) to counts.
    - mag-to-sb: convert magnitudes to surface brightness over an area.
    - sb-to-mag: convert surface brightness to magnitudes over an area.
  - New operators that are specific to Arithmetic:
    - collapse-median: collapse input dataset by calculating the median
      along the given dimension.
    - collapse-sigclip-std: Collapse with sigma-clipped standard deviation.
    - collapse-sigclip-mean: Collapse with sigma-clipped mean.
    - collapse-sigclip-median: Collapse with sigma-clipped median.
- collapse-sigclip-number: Collapse with number remaining after sigma-clip.


  ConvertType:
  - It is now possible to draw vector graphics marks from a catalog over
    the output PDF images. The following options have been added to
    ConvertType for doing this. See the "General program usage tutorial"
    for a fully working example.
    --marks: name of table containing mark information.
    --markshdu: HDU of table if file given to '--marks' is FITS.
    --markcoords: name or number of two columns containing coordinates.
    --mode: if the coordinates are in 'img' (image) or 'wcs' (RA/Dec).
--markshape: name or number of column containing the shape of each mark. --markrotate: name or number of column containing rotation of each mark.
    --marksize: name or number of column containing the size of the mark.
    --sizeinpix: interpret the values in the size column as pixels.
    --sizeinarcsec: interpret the values in the size column as arc-seconds.
    --sizeinarcmin: interpret the values in the size column as arc-minutes.
    --marklinewidth: name or number of column containing mark's line width.
    --markcolor: name or number of column containing mark's color.
    --listcolors: List all the 140 available colors, and show the colors on
                  24-bit (true color) terminal.
    --marktext: name or number of column containing text under each mark.
    --marktextprecision: number of decimals to print as text when the text
                         column (given to '--marktext') is floating point.
    --markfont: name or number of column containing the font to use for the
                the mark text (given to '--marktext').
    --markfontsize: name or number of column containing the size of the
                    font to use for the mark text (given to '--marktext').
    --showfonts: build a demo PDF with one page per font to show the
                 various available fonts on the system.
    --listfonts: List the names of the available fonts on the terminal.
  - It is now possible to select the color of the border of images produced
    in vector graphics outputs (EPS and PDF) with '--bordercolor'.


  Fits:
  --pixelareaarcsec2: print the image pixel area in units of arcsec^2 to
    standard output. Among other things, this is useful in creating a
    surface brightness image using the new 'counts-to-sb' operator of
    Arithmetic.
  --pixelareaonwcs: Ouput an image with the same number of pixels as the
    input. But each pixel's value shows its area on the sky (in
    degrees-squared). This area is calculated after accounting for
    distortion, projection or rotation. Implemented by Pedram
    Ashofteh-Ardakani.
  --edgesampling: extra sampling along each pixel's edge used to configure
    the output of '--pixelareaonwcs', similar to Warp.


  Statistics:
  - Linear and Polynomial least squares fitting are now available and very
    easy to call on the command-line. They are wrappers over the respective
    least squares fitting functions of the GNU Scientific Library. The
    interface is pretty simple, like the example below:
       aststatistics table.fits -cX,Y,Yerr --fit=linear-weighted
    It is also possible to estimate values and errors of the fitted model
    on a new X axis. A complete example has been added to the newly added
    "Least squares fitting" section of the book (under the Statistics
    program documentation). Please see that tutorial to easily use this
    feature. The following new options have been added to the Statistics
    program for this purpose:
    --fit: the model to use. Currently the following models are supported:
           linear
           linear-weighted
           linear-no-constant
           linear-no-constant-weighted
           polynomial
           polynomial-weighted
           polynomial-robust
    --fitweight: nature of the "weight" column (default: standard dev).
    --fitmaxpower: maximum power of X in polynomial models.
    --fitrobust: weight function to use in the "robust" polynomial model.
    --fitestimate: File name, or number to estimate the fit on.
--fitestimatehdu: HDU containing table in file given to '--fitestimate'.
    --fitestimatecol: Column containing X axis values for '--fitestimate'.


  Table:
  - It is now possible to customize the format of floating point numbers in
    the plain-text outputs: when output is printed on the standard output
    (command-line) or in plain-text files. The following new options have
    been added for this new feature:
    --txtf32format (or '-f'): Format of 32-bit floating point columns. This
      can be either 'fixed' (for fixed-point notation) or 'exp' (for
      exponential/scientific notation).
    --txtf32precision (or '-d'): number of digits following the
      decimal-point of 32-bit floating point columns.
    --txtf64format (or '-p'): Format of 64-bit floating point columns. This
      can be either 'fixed' (for fixed-point notation) or 'exp' (for
      exponential/scientific notation).
    --txtf32precision (or '-B'): number of digits following the
      decimal-point of 32-bit floating point columns.


  Warp:
  - Can correct distortions (with any standard recognized by WCSLIB) and
    simultaneously align the image to the coordinate system. When no named
    linear operation (like '--rotate', '--scale' or etc) is requested, Warp
    will go into this mode. It is highly customizable through the following
    options. See the "Invoking Warp" section of the book for more. This
    feature has been written by Pedram Ashofteh-Ardakani.
    --center: RA, DEC of the center of the central pixel of output.
    --width: Width of output in degrees or pixels (see '--widthinpix').
    --widthinpix: interpret values of '--width' as pixels.
    --cdelt: Pixel scale of output ('CDELTi' keywords in FITS).
    --ctype: Coordinates and projection algorithm. Default: RA/Dec and
      Gnomonic or 'TAN').
    --edgesampling: extra sampling of pixel polygon to account for strong
      non-linear projection or distortion effects, when necessary.
    --gridfile: warp the input to the exact WCS and pixel grid of the file
      given to this option. This is very useful when matching images from
      differetn surveys. Using this option, you can also insert distortions
      in an image (for example on a mock image, to make it match an
      observed exposure with dithering+distortion).
    --gridhdu: HDU containing image to be matched in '--gridfile'.
    --checkmaxfrac: visualize the Moiré pattern of the warp in the
      second extension of the output. This is the maximum fraction of
      a single input pixel's area in the output pixel. When the output
      pixel scale is similar to the input, the Moiré pattern can cause
      varying artificial smoothing of the noise level. See the newly
      added "Moiré pattern and its correction" section of the book for
      more on its basics and how to reduce it in your outputs.
  - List of WCS projections available in WCSLIB 7.12 (and therefore in
    Gnuastro's Warp):
       AZP: Zenithal/azimuthal perspective.
       SZP: Slant zenithal perspective.
       TAN: Gnomonic (tangential).
       STG: Stereographic.
       SIN: Orthographic/synthesis.
       ARC: Zenithal/azimuthal equidistant.
       ZPN: Zenithal/azimuthal polynomial.
       ZEA: Zenithal/azimuthal equal area.
       AIR: Airy.
       CYP: Cylindrical perspective.
       CEA: Cylindrical equal area.
       CAR: Plate carree.
       MER: Mercator.
       SFL: Sanson-Flamsteed.
       PAR: Parabolic.
       MOL: Mollweide.
       AIT: Hammer-Aitoff.
       COP: Conic perspective.
       COE: Conic equal area.
       COD: Conic equidistant.
       COO: Conic orthomorphic.
       BON: Bonne.
       PCO: Polyconic.
       TSC: Tangential spherical cube.
       CSC: COBE spherical cube.
       QSC: Quadrilateralized spherical cube.
       HPX: HEALPix.
       XPH: HEALPix polar, aka "butterfly".
  - List of WCS distortions available in WCSLIB 7.12 (and therefore in
    Gnuastro's Warp):
       TPD: Template Polynomial Distortion.
       SIP: Simple Imaging Polynomial.
       TPV: Polynomial distortion for Gnomonic (TAN) projection.
       DSS: Digitized Sky Survey.
       WAT: Chebyshev or Legendre polynomials for TNX and ZPX projections.


  astscript-fits-view:
  --ds9colorbarmulti: show a separate color-bar for each image in DS9. By
    default this script will show a single color-bar for all the images to
    help save space on the monitor when there are many images.


  astscript-psf-stamp:
  - sub-pixel warping is applied to ensure that your coordinate is at the
    center of the central pixel of the output image. This results in a
    _major_ improvement when estimating the center of the PSF.
  --nocentering: disable sub-pixel warping when creating the PSF stamp. As
    described above, the sub-pixel warping is critical for the central part
    of the PSF, but for the outer parts it is statistically negligible. So
    to avoid slowing down you pipeline, you can disable sub-pixel warping
    with this option.
  --snthresh: if given, the value to this option is assumed to be a
    signal-to-noise ratio (S/N) threshold and all pixels below that S/N
    will be masked. This is useful because we are often forced to stack
    stars of differing magnitudes. While the fainter ones are good for the
    inner parts of the star, they degrade the stack's outer parts. With
    this option, the fainter stars won't harm the outer parts.


  GNU Make extensions (in a Makefile)
    It is now possible to use custom Gnuastro functions in GNU Make, using
    its extension facilities with Dynamic libraries. GNU Make is a very
    powerful workflow manager that is also used for data analysis (not just
    for compilation). With the Gnaustro Make functions, (astronomical) data
    analysis becomes even more easier and faster. In the following, you can
    see the first set of such functions (they all begin with 'ast-'). For
    more, see the newly added chapter in the Gnuastro manual.
    - ast-version-is: will return '1' if the running Gnuastro has the given
      version (argument of this function). This can be used to ensure
      reproducibility in combination with Make's conditional features, see
      the minimal working example in the manual.
    - ast-text-contains: will return space-separated words within a larger
      list that contain a certain string. The to-contain string can be
      anywhere within the words of the larger list.
    - ast-text-not-contains: will return space-separated words within a
      larger list that DO NOT contain a certain string. The to-not-contain
      string can be anywhere within the words of the larger list.
    - ast-fits-with-keyvalue: takes a keyword name, a list of keyword
      values, a HDU and a list of FITS files. It will return only those
      FITS files that have the requested value(s) in the requested keyword
      of the requested HDU.
    - ast-fits-unique-keyvalues: takes a keyword name, a HDU and a list of
      FITS files. It will return all the unique values given to that
      keyword within the FITS files.


  Library:
  - GAL_CONFIG_HAVE_PYTHON: non-zero if Python3+Numpy features included.
  - GAL_CONFIG_HAVE_GNUMAKE_H: non-zero if GNU Make extensions can be made.
  - gal_box_border_rotate_around_center: width of box after rotation.
  - gal_color_id_to_name: return the name of a color from its ID.
  - gal_color_in_rgb: return the fraction of red-green-blue in a color.
  - gal_color_name_to_id: return the ID of a color from its name.
  - gal_dimension_collapse_median: collapse input along dim. using median.
  - gal_dimension_collapse_sigclip_mean: collapse with sig-clipped mean.
  - gal_dimension_collapse_sigclip_std: collapse with sig-clipped STD.
- gal_dimension_collapse_sigclip_median: collapse with sig-clipped median. - gal_dimension_collapse_sigclip_number: collapse with num. after sig-clip.
  - gal_eps_shape_id_to_name: return the name of a shape from its ID.
  - gal_eps_shape_name_to_id: return the ID of a shape from its name.
  - gal_fit_name_to_id: Convert string name to ID of fitting model.
  - gal_fit_name_from_id: Convert ID of fitting model to string name.
  - gal_fit_name_robust_to_id: Convert name of robust weights to ID.
  - gal_fit_name_robust_from_id: Convert ID of robust weights to name.
  - gal_fit_1d_linear: linear fit of input columns.
  - gal_fit_1d_linear_no_constant: linear fit with no constant.
  - gal_fit_1d_linear_estimate: estimate a linear fit on a new X column.
  - gal_fit_1d_polynomial: polynomial fit of input columns.
  - gal_fit_1d_polynomial_robust: robust polynomial fit of input columns.
- gal_fit_1d_polynomial_estimate: estimate a polynomial fit on new X column.
  - gal_fits_unique_keyvalues: extract all unique values to a certain
    keyword in many files.
  - gal_fits_with_keyvalue: select FITS image with a certain key value.
  - gal_list_data_select_by_name: select a dataset from a list by its name.
- gal_list_str_cat: Concatenate (append) list to a space-separated string.
  - gal_list_str_extract: Extract space-separated tokens to a list.
  - gal_python_type_from_numpy: Convert Numpy's type id. to Gnuastro's.
  - gal_python_type_to_numpy: Convert Gnuastro's type id. to Numpy's.
  - gal_txt_contains_string: Check a certain string within in a larger one.
  - gal_units_counts_to_sb: SB from counts, zeropoint and area.
  - gal_units_mag_to_sb: surface brightness (SB) from magnitude and area.
  - gal_units_sb_to_counts: counts from SB, zeropoint and area.
  - gal_units_sb_to_mag: magnitude from SB and area.
  - gal_warp_pixelarea: return image of same size, but with area on sky.
  - gal_warp_wcsalign_init: initialize the WCS aligning structure.
  - gal_warp_wcsalign_onpix: Per-pixel filling of output.
  - gal_warp_wcsalign_onthread: function to give to pthreads.
  - gal_warp_wcsalign: high-level function to align input by its WCS.
- gal_warp_wcsalign_free: free the contents of the WCS aligning structure.
  - gal_wcs_free: free a WCS structure that is created or read by Gnuastro.


** Removed features


  Statistics:
  --refcol has been removed because it breaks the modularity principle
    (given that it is the job of Gnuastro's Table program to limit rows
    from a larger table based on many different criteria). The output of
    Table can be directly piped to Statistics to achieve the same (and much
    more feature-rich effect).


  Warp:
  --align: has been removed. This is because aligning an image (while
    correcting for any possible distortion) is now the default behavior of
    Warp (when no linear operations have been requested).


** Changed features


  Book:
  - The "General program usage tutorial" section is now the first section
    of the Tutorial chapter, since it introduces the tools at a more basic
    level. The "Sufi simulates a detection" (which was previously first)
    has been moved to the fourth section.


  Warp:
  - The short format of the '--centeroncorner' option has been removed. The
    '-c' is now the short format for the new '--center' option to Warp.


  astscript-psf-scale-factor:
  --widthinpix: new name for the old '--stampwidth' option. This was done
    to have the same name to a similar option in Crop and help in
    remembering.


  astscript-psf-stamp:
  --widthinpix: new name for the old '--stampwidth' option. This was done
    to have the same name to a similar option in Crop and help in
    remembering.


  Library
  - gal_eps_write: two new arguments have been added to draw marks, and to
        set the border color.
  - gal_pdf_write: similar to 'gal_eps_write'.
  - gal_fits_hdu_open: new argument to optionally exit program if HDU
        couldn't be opened.
  - GAL_TABLE_DISPLAY_FMT_FIXED: new name for GAL_TABLE_DISPLAY_FMT_FLOAT
    since it corresponds to the fixed-point notation of printing floating
    points in plain-text (the '_FLOAT' suffix was too generic and unclear).


** Bugs fixed
  bug #62861: '--printkeynames' of Fits program gets caught in an infinite
              loop on FITS files that have empty keywords before
              'END'. Found by Pedram Ashofteh-Ardakani.
  bug #62892: Installed scripts don't account for differing LANG and
              LC_NUMERIC. Found by Teet Kuutma and fixed by Raul
              Infante-Sainz.
  bug #62937: psf-scale-factor not being quiet, when requested. Found and
              fixed by Sepideh Eskandarlou.
  bug #62943: Couldn't read the value of width with '--snthresh' is called.
              Found and fixed by Sepideh Eskandarlou.
  bug #62944: No warning when the option value isn't immediately after the
              equal sign in long format. Found by Faezeh Bijarchian.
  bug #63013: Sigma clip segmentation fault when input has an integer type
              with values close to saturation-level.
  bug #63022: psf-scale-factor not saving the result in the output file,
              found and fixed by Raul Infante-Sainz.
  bug #63189: MakeProfiles custom profiles become NaN with a single row
              being NaN, reported by Nafise Sedighi.
  bug #63207: Match crashes when one input has no rows. Found by Sepideh
              Eskandarlou.
  bug #63257: Fits program's '--skycoverage' gives unreasonable outputs
              when image crosses the RA=0 hour circle. Found by Irene
              Pintos Castro.
  bug #63261: Radial profile script ignores central pixel in azimuthal
              profiles. Found and fixed by Sepideh Eskandarlou.


[2] Lzip has better compression ratio and archival features compared
to the `.gz' or `.xz' formats. Therefore Gnuastro's alpha/test
releases are only in this format, but for historical reasons we also
include `.gz' tarballs in the official releases. If you don't have
Lzip (you can check with `lzip --version' command), download and
install it from its webpage:


  https://www.nongnu.org/lzip/lzip.html


If Lzip is present and you use GNU Tar, then the single command below
should uncompress and un-pack the tarball:


  $ tar xf gnuastro-0.19.tar.lz


If the command above doesn't work, you have to un-compress and un-pack
it with two separate commands (or use a pipe to feed the output of the
first into the second: `lzip -cd gnuastro-0.19.tar.lz | tar -xf -'):


  $ lzip -d gnuastro-0.19.tar.lz
  $ tar xf gnuastro-0.19.tar


[3] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:


  gpg --verify gnuastro-0.19.tar.gz.sig


The signature should match the fingerprint of the following key:


  pub   rsa4096 2018-12-08 [SC]
        52B0 4484 D806 C90D CB52  7249 71E8 9901 2D17 4B66
  uid           [ unknown] Mohammad Akhlaghi 


If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.


  gpg --recv-keys 71E899012D174B66


As a last resort to find the key, you can try the official GNU
keyring:


  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify gnuastro-0.19.tar.gz.sig


[4] To get the base64 SHA256 checksum, you can use the command below
(assuming you already have the 'openssl' command-line program). Just
note that an extra '=' is printed as the last character that you
should ignore.


  cat gnuastro-0.18.71-c982.tar.lz \
      | openssl dgst -binary -sha256 | openssl base64 -A


[5] This tarball was bootstrapped (created) with the tools below. Note
that you don't need these to build Gnuastro from the tarball, these
are the tools that were used to make the tarball itself. They are only
mentioned here to be able to reproduce/recreate this tarball later.


  Texinfo 6.8
  Autoconf 2.71
  Automake 1.16.4
  Help2man 1.49.2
  ImageMagick 7.1.0-51
  Gnulib v0.1-5536-g0814a293a4
  Autoconf archives v2022.09.03-4-gda89908

Read on