Corrfunc 2.5.0

Pre-requisites

1. make >= 3.80

2. OpenMP capable compiler like icc, gcc>=4.6 or clang >= 3.7. If not available, please disable USE_OMP option option in theory.options and mocks.options. On a HPC cluster, consult the cluster documentation for how to load a compiler (often module load gcc or similar). If you are using Corrfunc with Anaconda Python, then conda install gcc (MAC/linux) should work. On MAC, (sudo) port install gcc5 is also an option.

3. gsl >= 2.4. On an HPC cluster, consult the cluster documentation (often module load gsl will work). With Anaconda Python, use conda install -c conda-forge gsl (MAC/linux). On MAC, you can use (sudo) port install gsl (MAC) if necessary.

4. python >= 2.7 or python>=3.4 for compiling the CPython extensions.

5. numpy>=1.7 for compiling the CPython extensions.

Method 1: Source Installation (Recommended)

$ git clone https://github.com/manodeep/Corrfunc.git
$ cd Corrfunc
$ make
$ make install
$ python -m pip install . [--user]

$ make tests  # run the C tests
$ python -m pip install pytest
$ python -m pytest  # run the Python tests

Assuming you have gcc in your PATH, make and make install should compile and install the C libraries + Python extensions within the source directory. If you would like to install the CPython extensions in your environment, then python -m pip install . [--user] should be sufficient. If you are primarily interested in the Python interface, you can condense all of the steps by using python -m pip install . [--user] --install-option="CC=yourcompiler" after git clone [...] and cd Corrfunc.

Method 2: pip installation

The Python package is directly installable via python -m pip install Corrfunc. However, in that case you will lose the ability to recompile the code. This usually fine if you are only using the Python interface and are on a single machine, like a laptop. For usage on a cluster or other environment with multiple CPU architectures, you may find it more useful to use the Source Installation method above in case you need to compile for a different architecture later.

$ python -m pytest --pyargs Corrfunc

OpenMP on OSX

Automatically detecting OpenMP support from the compiler and the runtime is a bit tricky. If you run into any issues compiling (or running) with OpenMP, please refer to the FAQ for potential solutions.

Input data

The input galaxies (or any discrete distribution of points) are derived from a simulation. For instance, the galaxies could be a result of an Halo Occupation Distribution (HOD) model, a Subhalo Abundance matching (SHAM) model, a Semi-Empirical model (SEM), or a Semi-Analytic model (SAM) etc. The input set of points can also be the dark matter halos, or the dark matter particles from a cosmological simulation. The input set of points are expected to have positions specified in Cartesian XYZ.

Types of available clustering statistics

All codes that work on cosmological boxes with co-moving positions are located in the theory directory. The various clustering measures are:

1. DD – Measures auto/cross-correlations between two boxes. The boxes do not need to be cubes.

2. xi – Measures 3-d auto-correlation in a cubic cosmological box. Assumes PERIODIC boundary conditions.

3. wp – Measures auto 2-d point projected correlation function in a cubic cosmological box. Assumes PERIODIC boundary conditions.

4. DDrppi – Measures the auto/cross correlation function between two boxes. The boxes do not need to be cubes.

5. DDsmu – Measures the auto/cross correlation function between two boxes. The boxes do not need to be cubes.

6. vpf – Measures the void probability function + counts-in-cells.

Input data

The input galaxies are typically observed galaxies coming from a large-scale galaxy survey. In addition, simulated galaxies that have been projected onto the sky (i.e., where observational systematics have been incorporated and on-sky positions have been generated) can also be used. We generically refer to both these kinds of galaxies as “mocks”.

The input galaxies are expected to have positions specified in spherical co-ordinates with at least right ascension (RA) and declination (DEC). For spatial correlation functions, an approximate “co-moving” distance (speed of light multiplied by redshift, CZ) is also required.

Types of available clustering statistics

All codes that work on mock catalogs (RA, DEC, CZ) are located in the mocks directory. The various clustering measures are:

1. DDrppi_mocks – The standard auto/cross correlation between two data sets. The outputs, DD, DR and RR can be combined using wprp to produce the Landy-Szalay estimator for wp(rp).

2. DDsmu_mocks – The standard auto/cross correlation between two data sets. The outputs, DD, DR and RR can be combined using the Python utility convert_3d_counts_to_cf to produce the Landy-Szalay estimator for xi(s, mu).

3. DDtheta_mocks – Computes angular correlation function between two data sets. The outputs from DDtheta_mocks need to be combined with wtheta to get the full omega(theta)

4. vpf_mocks – Computes the void probability function on mocks.

Theory (in theory.options)

1. PERIODIC (ignored in case of wp/xi) – switches periodic boundary conditions on/off. Enabled by default.

2. OUTPUT_RPAVG – switches on output of <rp> in each rp bin. Can be a massive performance hit (~ 2.2x in case of wp). Disabled by default.

Mocks (in mocks.options)

1. OUTPUT_RPAVG – switches on output of <rp> in each rp bin for DDrppi_mocks. Enabled by default.

2. OUTPUT_THETAAVG – switches on output of in each theta bin. Can be extremely slow (~5x) depending on compiler, and CPU capabilities. Disabled by default.

3. LINK_IN_DEC – creates binning in declination for DDtheta_mocks. Please check that for your desired limits \theta, this binning does not produce incorrect results (due to numerical precision). Generally speaking, if your \thetamax (the max. \theta to consider pairs within) is too small (probaly less than 1 degree), then you should check with and without this option. Errors are typically sub-percent level.

4. LINK_IN_RA – creates binning in RA once binning in DEC has been enabled for DDtheta_mocks. Same numerical issues as LINK_IN_DEC

5. FAST_ACOS – Relevant only when OUTPUT_THETAAVG is enabled for DDtheta_mocks. Disabled by default. An arccos is required to calculate <\theta>. In absence of vectorized arccos (intel compiler, icc provides one via intel Short Vector Math Library), this calculation is extremely slow. However, we can approximate arccos using polynomials (with Remez Algorithm). The approximations are taken from implementations released by Geometric Tools. Depending on the level of accuracy desired, this implementation of fast acos can be tweaked in the file utils/fast_acos.h. An alternate, less accurate implementation is already present in that file. Please check that the loss of precision is not important for your use-case.

6. COMOVING_DIST – Currently there is no support in Corrfunc for different cosmologies. However, for the mocks routines like, DDrppi_mocks and vpf_mocks, cosmology parameters are required to convert between redshift and co-moving distance. Both DDrppi_mocks and vpf_mocks expects to receive a redshift array as input; however, with this option enabled, the redshift array will be assumed to contain already converted co-moving distances. So, if you have redshifts and want to use an arbitrary cosmology, then convert the redshifts into co-moving distances, enable this option, and pass the co-moving distance array into the routines.

Using the command-line interface

Navigate to the correct directory. Make sure that the options, set in either theory.options or mocks.options in the root directory are what you want. If not, edit those two files (and possibly common.mk), and recompile. Then, you can use the command-line executables in each individual subdirectory corresponding to the clustering measure you are interested in. For example, if you want to compute the full 3-D correlation function, \xi(r), then run the executable theory/xi/xi. If you run executables without any arguments, the program will output a message with all the required arguments.

Calling from C

Look under the run_correlations.c and run_correlations_mocks.c to see examples of calling the C API directly. If you run the executables, run_correlations and run_correlations_mocks, the output will also show how to call the command-line interface for the various clustering measures.

Calling from Python

If all went well, the codes can be directly called from python. Please see call_correlation_functions.py and call_correlation_functions_mocks.py for examples on how to use the CPython extensions directly. Here are a few examples:

from __future__ import print_function
import os.path as path
import numpy as np
import Corrfunc
from Corrfunc.theory import wp

# Setup the problem for wp
boxsize = 500.0
pimax = 40.0
nthreads = 4

# Create a fake data-set.
Npts = 100000
x = np.float32(np.random.random(Npts))
y = np.float32(np.random.random(Npts))
z = np.float32(np.random.random(Npts))
x *= boxsize
y *= boxsize
z *= boxsize

# Setup the bins
rmin = 0.1
rmax = 20.0
nbins = 20

# Create the bins
rbins = np.logspace(np.log10(0.1), np.log10(rmax), nbins + 1)

# Call wp
wp_results = wp(boxsize, pimax, nthreads, rbins, x, y, z, verbose=True, output_rpavg=True)

# Print the results
print("#############################################################################")
print("##       rmin           rmax            rpavg             wp            npairs")
print("#############################################################################")
print(wp_results)