A Guide to the DiFX Software Correlator

Version 2.9

Walter Brisken

National Radio Astronomy Observatory

Contents:

• Introduction
  • Notation
• The DiFX correlator
  • NRAO-DiFX 1.0
  • NRAO-DiFX 1.1
  • DiFX 1.5.0
  • DiFX 1.5.1
  • DiFX 1.5.2
  • DiFX 1.5.3
  • DiFX 1.5.4
  • DiFX 2.0.0
  • DiFX 2.0.1
  • DiFX 2.1
  • DiFX 2.1.1
  • DiFX 2.2
  • DiFX 2.3
  • DiFX 2.4
  • DiFX 2.5.1
  • DiFX 2.5.2
  • DiFX 2.5.3
  • DiFX 2.5.4
  • DiFX 2.5.5
  • DiFX 2.6.1
  • DiFX 2.6.2
  • DiFX 2.6.3
  • DiFX 2.7.1
  • DiFX 2.8.1
• Features left to implement
  • DiFX and AIPS
• Hardware and operating system configuration
  • Cluster configuration
  • Environment variables
  • Directory structure and versioning
• Running DiFX at the VLBA correlator (VLBA Specific)
  • Software correlation based on vex files
  • Correlating data files
  • The VLBA database
  • Archiving
  • Ownership and permissions of files
  • Copying baseband data
  • Some comments on channels
• Data Sources
  • File
  • Mark5
  • Mark6
  • Network / eVLBI
  • Fake
• Baseband Data Formats
  • LBA
  • VLBA
  • Mark IV
  • Mark5B
  • VDIF
• DiFX and pulsars
  • Pulse ephemeris
  • Bin configuration file
  • Preparing correlator jobs
  • Making FITS files
• Conventions
  • Clock offsets and rates
  • Geometric delays and rates
  • Antenna coordinates
  • Baseline coordinates
  • Visibility phase
• Reference guide to programs and utilities
  • apd2antenna \(\mathrm{(package: difx2fits)}\)
  • avgDiFX \(\mathrm{(package: difxio)}\)
  • bp2antenna \(\mathrm{(package: difx2fits)}\)
  • calcif2
  • CalcServer
  • checkdir \(\mathrm{(package: mk5daemon)}\)
  • checkmpifxcorr \(\mathrm{(package: mpifxcorr)}\)
  • cleanVDIF \(\mathrm{(package: vdifio)}\) [sec:printVDIF]
  • condition \(\mathrm{(package: nrao\_difx\_db)}\)
  • condition_watch \(\mathrm{(package: nrao\_difx\_db)}\)
  • countVDIFpackets \(\mathrm{(package: vdifio)}\) [sec:countVDIFpackets]
  • cpumon \(\mathrm{(package: difxmessage)}\)
  • diffDiFX.py \(\mathrm{(package: vis2screen)}\)
  • difx2fits
  • difx2mark4
  • difxarch \(\mathrm{(package: nrao\_difx\_db)}\)
  • difxbuild
  • difxcalc11
  • difxcalculator \(\mathrm{(package: difxio)}\)
  • difxclean \(\mathrm{(package: nrao\_difx\_db)}\)
  • difxcopy \(\mathrm{(package: misc\_utils)}\)
  • difxdiagnosticmon \(\mathrm{(package: difxmessage)}\)
  • difxlog \(\mathrm{(package: difxmessage)}\)
  • difxqueue \(\mathrm{(package: nrao\_difx\_db)}\)
  • difxsniff \(\mathrm{(package: SniffPlots)}\)
  • difxspeed \(\mathrm{(package: vex2difx)}\)
  • difxusage \(\mathrm{(package: nrao\_difx\_db)}\)
  • difxvmf \(\mathrm{(package: calcif2)}\)
  • difxwatch \(\mathrm{(package: difxmessage)}\)
  • DiFX Operator Interface
  • e2ecopy \(\mathrm{(package: nrao\_difx\_db)}\)
  • errormon \(\mathrm{(package: difxmessage)}\)
  • extractSingleVDIFThread \(\mathrm{(package: vdifio)}\)
  • extractVDIFThreads \(\mathrm{(package: vdifio)}\)
  • fakemultiVDIF \(\mathrm{(package: vdifio)}\)
  • fileto5c \(\mathrm{(package: mark5daemon)}\)
  • filterVDIF \(\mathrm{(package: vdifio)}\)
  • generateVDIF \(\mathrm{(package: vdifio)}\)
  • genmachines \(\mathrm{(package: mpifxcorr)}\)
  • getshelf \(\mathrm{(package: nrao\_difx\_db)}\)
  • jobdisks \(\mathrm{(package: mpifxcorr)}\)
  • joblist \(\mathrm{(package: mpifxcorr)}\)
  • jobstatus \(\mathrm{(package: mpifxcorr)}\)
  • listcpus \(\mathrm{(package: mk5daemon)}\)
  • makefits \(\mathrm{(package: difx2fits)}\)
  • makemark4 \(\mathrm{(package: difxdb)}\)
  • m5bstate \(\mathrm{(package: mark5access)}\)
  • m5d \(\mathrm{(package: mark5access)}\)
  • m5findformats \(\mathrm{(package: mark5access)}\)
  • m5fold \(\mathrm{(package: mark5access)}\)
  • m5pcal \(\mathrm{(package: mark5access)}\)
  • m5slice \(\mathrm{(package: mark5access)}\)
  • m5spec \(\mathrm{(package: mark5access)}\)
  • m5test \(\mathrm{(package: mark5access)}\)
  • m5time\(\mathrm{(package: mark5access)}\)
  • m5timeseries \(\mathrm{(package: mark5access)}\)
  • m5tsys \(\mathrm{(package: mark5access)}\)
  • mk5cat \(\mathrm{(package: mk5daemon)}\)
  • mk5control \(\mathrm{(package: mk5daemon)}\)
  • mk5cp \(\mathrm{(package: mk5daemon)}\)
  • mk5daemon \(\mathrm{(package: mk5daemon)}\)
  • mk5dir \(\mathrm{(package: mark5daemon)}\)
  • mk5erase \(\mathrm{(package: mark5daemon)}\)
  • mk5mon \(\mathrm{(package: difxmessage)}\)
  • mk6cp \(\mathrm{(package: mark6sg)}\) [sec:mk6cp]
  • mk6gather \(\mathrm{(package: mark6sg)}\) [sec:mk6gather]
  • mk6ls \(\mathrm{(package: vdifio)}\) [sec:mk6ls]
  • mk6mon \(\mathrm{(package: difxmessage)}\)
  • mk6summary \(\mathrm{(package: mark6sg)}\)
  • mk6vmux \(\mathrm{(package: vdifio)}\)
  • mpifxcorr
  • oms2v2d \(\mathrm{(package: vex2difx)}\) [sec:oms2v2d]
  • padVDIF \(\mathrm{(package: vdifio)}\) [sec:padVDIF]
  • plotapd \(\mathrm{(package: SniffPlots)}\) [sec:plotapd]
  • plotbp \(\mathrm{(package: SniffPlots)}\) [sec:plotbp]
  • plotwt \(\mathrm{(package: SniffPlots)}\) [sec:plotwt]
  • printDiFX.py \(\mathrm{(package: vis2screen)}\) [sec:printDiFX]
  • printVDIF \(\mathrm{(package: vdifio)}\) [sec:printVDIF]
  • printVDIFgaps \(\mathrm{(package: vdifio)}\)
  • printVDIFheader \(\mathrm{(package: vdifio)}\)
  • psrflag \(\mathrm{(package: difxio)}\)
  • record5c \(\mathrm{(package: mark5daemon)}\)
  • recover \(\mathrm{(package: mk5daemon)}\)
  • reducepoly \(\mathrm{(package: difxio)}\)
  • searchVDIF \(\mathrm{(package: vdifio)}\)
  • splitVDIFbygap \(\mathrm{(package: vdifio)}\)
  • startdifx \(\mathrm{(package: mpifxcorr)}\)
  • statemon \(\mathrm{(package: difxmessage)}\)
  • stopmpifxcorr \(\mathrm{(package: mpifxcorr)}\)
  • stripVDIF \(\mathrm{(package: vdifio)}\) [sec:stripVDIF]
  • tabulatedelays \(\mathrm{(package: difxio)}\)
  • testdifxinput \(\mathrm{(package: difxio)}\)
  • testdifxmessagereceive\(\mathrm{(package: difxmessage)}\)
  • testmod \(\mathrm{(package: mk5daemon)}\)
  • testseqnumbers \(\mathrm{(package: difxmessage)}\)
  • vdif2to8 \(\mathrm{(package: vdifio)}\)
  • vdifbstate \(\mathrm{(package: vdifio)}\)
  • vdifChanSelect \(\mathrm{(package: vdifio)}\)
  • vdifd \(\mathrm{(package: vdifio)}\)
  • vdiffold \(\mathrm{(package: vdifio)}\)
  • vdifspec \(\mathrm{(package: vdifio)}\)
  • vex2difx
  • vexpeek \(\mathrm{(package: vex2difx)}\)
  • vlog \(\mathrm{(package: vex2difx)}\)
  • vmux \(\mathrm{(package: vdifio)}\)
  • vsn \(\mathrm{(package: mk5daemon)}\)
  • vsum \(\mathrm{(package: vdifio)}\)
  • zerocorr \(\mathrm{(package: mark5access)}\)
• Description of various files
  • .aapd
  • .abp
  • .acb
  • .apc
  • .apd
  • .bandpass
  • .binconfig
  • .bootstrap
  • .cablecal
  • cal.vlba
  • .calc
  • .difx/
  • .difxlog
  • .speed
  • .speed.out
  • $DIFX_MACHINES
  • .dir
  • .filelist
  • .FITS
  • .fitslist
  • .flag
  • .<antId>.flag
  • .channelflags
  • flag
  • .im
  • .input
  • .joblist
  • .jobmatrix
  • .lag
  • .log
  • .machines
  • .mark4list
  • .oms
  • .params
  • pcal
  • .polyco
  • .shelf
  • .threads
  • tsys
  • weather
  • .wts
  • .v2d
  • .xcb
  • .vex, .skd, .vex.obs, & .skd.obs
  • .vis
  • .zerocorr
• XML message types
  • DifxAlertMessage
  • DifxCommand
  • DifxLoadMessage
  • DifxParameter
  • DifxSmartMessage
  • DifxTransientMessage
  • DifxStart
  • DifxStatusMessage
  • DifxStop
  • Mark5DriveStatsMessage
  • Mark5StatusMessage
  • Mark5VersionMessage
• Database tables
  • The DIFXQUEUE table
  • The DIFXLOG table
  • The CONDITION table
• DiFX alert messages
  • Fatal
  • Severe
  • Error
  • Warning
  • Info
  • Verbose
  • Debug
• Installation and upgrade guide
  • Installation with difxbuild
  • Manual installation
• Acknowledgements
• References
• Internals for Developers
  • Data Layout in Mpifxcorr class Core

Indices and tables

• Index

• Module Index

• Search Page