Getting started¶

About¶

PyHum - a Python framework for reading and processing data from a Humminbird low-cost sidescan sonar

PyHum is an open-source project dedicated to provide a generic Python framework for reading and exporting data from Humminbird(R) instruments, carrying out rudimentary radiometric corrections to the data, classify bed texture, and produce some maps on aerial photos and kml files for google-earth

The software is designed to read Humminbird data (.SON, .IDX, and .DAT files) and works on both sidescan and downward-looking echosounder data, where available.

Some aspects of the program are detailed in:

Buscombe, D., Grams, P.E., and Smith, S. (2015) “Automated riverbed sediment classification using low-cost sidescan sonar”, Journal of Hydraulic Engineering, 10.1061/(ASCE)HY.1943-7900.0001079, 06015019. Download here

Full documentation of the procedures behind the program is forthcoming in the following publication:

Buscombe, D., submitted, Processing and georeferencing recreational-grade sidescan-sonar data to support the democratization of acoustic imaging in shallow water. LIMNOLOGY AND OCEANOGRAPHY: METHODS.

For the source code visit the project github site

Primary Developer | Daniel Buscombe —— | —————

Grand Canyon Monitoring and Research Center | United States Geological Survey | Flagstaff, AZ 86001 | dbuscombe@usgs.gov

Co-Developer | Daniel Hamill —— | —————

Department of Watershed Sciences | Utah State University | Logan, UT 84322 | dhamill@usgs.gov

License¶

This software is in the public domain because it contains materials that originally came from the United States Geological Survey, an agency of the United States Department of Interior. For more information, see the official USGS copyright policy

Any use of trade, product, or firm names is for descriptive purposes only and does not imply endorsement by the U.S. government.

This software is issued under the GNU Lesser General Public License, Version 3

Thanks to Barb Fagetter (blueseas@oceanecology.ca) for some format info, Dan Hamill (Utah State University) and Paul Anderson (Quest Geophysical Asia) for debugging and suggestions for improvements

Some aspects of the program are detailed in: 1. Buscombe, D., Grams, P.E., and Smith, S. (2015) “Automated riverbed sediment classification using low-cost sidescan sonar”, Journal of Hydraulic Engineering, 10.1061/(ASCE)HY.1943-7900.0001079, 06015019.

Buscombe, D., submitted, Processing and georeferencing recreational-grade sidescan-sonar data to support the democratization of acoustic imaging in shallow water. LIMNOLOGY AND OCEANOGRAPHY: METHODS.

Full documentation of the program is forthcoming

Setup on Anaconda for Windows¶

PyHum currently has only been tested with Python 2.7, so you’ll need that version of Anaconda

Step 1. Before installing PyHum, install Basemap using:

conda install basemap

Step 2. Install pyproj. pip seems to have a bug with pyproj depending on what c-compiler your python distribution uses. Therefore, you may have to install pyproj (and other dependencies) from here

download the .whl file. Choose the file associated with python 2.7 (“cp27”) and the architecture you are using, i.e. 32-bit (win32) or 64-bit (amd64)
then move that file to your root Anaconda directory (when you open an Anaconda command prompt it’s the directory that’s listed before the prompt ‘>’)
then use pip to install it, e.g.:

pip install pyproj-1.9.4-cp27-none-win_amd64.whl

Step 4. Install simplekml, using:

pip install simplekml

Step 3. Assuming a Anaconda distribution which comes with almost all required program dependencies:

pip uninstall PyHum (removes any previous installation)
pip install PyHum

If you get C++ compiler errors (such as “Unable to find vcvarsall.bat”), you will need to install the Microsoft Visual C++ compiler from here

(Advanced) If you have git installed (from here), you can install the latest ‘bleeding edge’ (pre-release) version directly from github:

pip install git+https://github.com/dbuscombe-usgs/PyHum.git

Setup¶

Automatic Installation from PyPI:

pip uninstall PyHum (removes previous installation)
pip install PyHum

Automatic Installation from github:

git clone git@github.com:dbuscombe-usgs/PyHum.git
cd PyHum
python setup.py install

or a local installation:

python setup.py install --user

or with admin privileges, e.g.:

sudo python setup.py install

This software has been tested with Python 2.7 on 1. Windows 7 2. Linux Fedora (16 & 20) 3. Linux Ubuntu (12.4 & 13.4 & 14.4)

This software has (so far) been used only with Humminbird 798, 898, 998, 1198 and 1199 series instruments.

Virtual environment¶

You could try before you install, using a virtual environment:

virtualenv venv
source venv/bin/activate
pip install numpy
pip install cython
pip install scipy
pip install joblib
pip install simplekml
pip install pyproj
pip install scikit-learn
pip install Pillow
pip install matplotlib
pip install basemap --allow-external basemap --allow-unverified basemap
pip install pyresample
pip install toolz
pip install dask
pip install pandas
pip install PyHum
python -c "import PyHum; PyHum.test()"
deactivate #(or source venv/bin/deactivate)

The results will live in “venv/lib/python2.7/site-packages/PyHum”

Manual installation¶

Python libraries you need to have installed to use PyHum:

All of the above are available through pip and easy_install

Installation on Amazon Linux EC-2 instance¶

It’s best to install numpy, scipy, cython and matplotlib through the OS package manager:

sudo yum install gcc gcc-c++
sudo yum install python27-numpy python27-Cython python27-scipy python27-matplotlib

Then install geos libraries using yum and Basemap using pip:

sudo yum install geos geos-devel geos-python27
sudo pip install basemap --allow-external basemap --allow-unverified basemap

Then PyHum using pip (which will install Pillow, pyproj, simplekml, joblib and scikit-learn):

sudo pip install PyHum

Test¶

A test can be carried out by running the supplied script:

python -c "import PyHum; PyHum.test.dotest()"

which carries out the following operations:

# general settings
humfile = os.path.normpath(os.path.join(os.path.expanduser("~"),'pyhum_test','test.DAT'))
sonpath = os.path.normpath(os.path.join(os.path.expanduser("~"),'pyhum_test'))

doplot = 1 #yes

# reading specific settings
cs2cs_args = "epsg:26949" #arizona central state plane
bedpick = 1 # auto bed pick
c = 1450 # speed of sound fresh water
t = 0.108 # length of transducer
draft = 0.3 # draft in metres
flip_lr = 1 # flip port and starboard
model = 998 # humminbird model
calc_bearing = 1 #1=yes
filt_bearing = 1 #1=yes
chunk = 'd100' # distance, 100m
#chunk = 'p1000' # pings, 1000
#chunk = 'h10' # heading deviation, 10 deg

# correction specific settings
maxW = 1000 # rms output wattage
dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans)
correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)
ph = 7.0 # acidity on the pH scale
temp = 10.0 # water temperature in degrees Celsius
salinity = 0.0

# for shadow removal
shadowmask = 0 #automatic shadow removal
win = 31

# for texture calcs
shift = 10 # pixel shift
density =win/2 # win/2
numclasses = 4 # number of discrete classes for contouring and k-means
maxscale = 20 # Max scale as inverse fraction of data length (for wavelet analysis)
notes = 4 # Notes per octave (for wavelet analysis)

# for mapping
res = 0.25 #99 # grid resolution in metres
# if res==99, the program will automatically calc res from the spatial res of the scans
mode = 1 # gridding mode (simple nearest neighbour)
#mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
#mode = 3 # gridding mode (gaussian weighted nearest neighbour)
use_uncorrected = 0

nn = 64 #number of nearest neighbours for gridding (used if mode > 1)
##influence = 1 #Radius of influence used in gridding. Cut off distance in meters
numstdevs = 5 #Threshold number of standard deviations in sidescan intensity per grid cell up to which to accept

# for downward-looking echosounder echogram (e1-e2) analysis
beam = 20.0
transfreq = 200.0 # frequency (kHz) of downward looking echosounder
integ = 5
numclusters = 3 # number of acoustic classes to group observations

## read data in SON files into PyHum memory mapped format (.dat)
PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, chunk)

## correct scans and remove water column
PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)

## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)
PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot)

win = 100 # pixel window

## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
PyHum.texture(humfile, sonpath, win, shift, doplot, density, numclasses, maxscale, notes)

## grid and map the scans
PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected) #dowrite,

res = 1 # grid resolution in metres
numstdevs = 5

## grid and map the texture lengthscale maps
PyHum.map_texture(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs)

## calculate and map the e1 and e2 acoustic coefficients from the downward-looking sonar
PyHum.e1e2(humfile, sonpath, cs2cs_args, ph, temp, salinity, beam, transfreq, integ, numclusters, doplot)

Getting Started¶

Inputs to the program are a .DAT file (e.g. R0089.DAT) and a folder of .SON and .IDX files (e.g. /my/folder/R0089). The program will read the .SON files with or without the accompanying .IDX files, but will be faster if the .IDX files are present.

PyHum is modular so can be called from within a python or ipython console, from an IDE (such as IDLE or Spyder), or by running a script.

The following example script:

import sys, getopt

from Tkinter import Tk
from tkFileDialog import askopenfilename, askdirectory

import PyHum
import os

if __name__ == '__main__':

    argv = sys.argv[1:]
    humfile = ''; sonpath = ''

    # parse inputs to variables
    try:
       opts, args = getopt.getopt(argv,"hi:s:")
    except getopt.GetoptError:
         print 'error'
         sys.exit(2)
    for opt, arg in opts:
       if opt == '-h':
         print 'help'
         sys.exit()
       elif opt in ("-i"):
          humfile = arg
       elif opt in ("-s"):
          sonpath = arg

    # prompt user to supply file if no input file given
    if not humfile:
       print 'An input file is required!!!!!!'
       Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
       humfile = askopenfilename(filetypes=[("DAT files","*.DAT")])

    # prompt user to supply directory if no input sonpath is given
    if not sonpath:
       print 'A *.SON directory is required!!!!!!'
       Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
       sonpath = askdirectory()

    # print given arguments to screen and convert data type where necessary
    if humfile:
       print 'Input file is %s' % (humfile)

    if sonpath:
       print 'Son files are in %s' % (sonpath)

    # general settings
    doplot = 1 #yes

    # reading specific settings
    cs2cs_args = "epsg:32100" #NAD83 / Montana
    bedpick = 2 # manual bed pick
    c = 1450 # speed of sound fresh water
    t = 0.108 # length of transducer
    draft = 0.3 # draft in metres
    flip_lr = 1 # flip port and starboard
    model = 1199 # humminbird model
    dowrite = 0 #disable writing of point cloud data to file
    chunk = 'd100' # distance, 100m
    #chunk = 'p1000' # pings, 1000
    #chunk = 'h10' # heading deviation, 10 deg

    # correction specific settings
    maxW = 1000 # rms output wattage
    dofilt = 1 # apply a phase preserving filter (WARNING!! takes a very long time for large scans)
    correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)

    # for shadow removal
    shadowmask = 1 #manual shadow removal
    win = 100

    # for mapping
    res = 99 # grid resolution in metres
    # if res==99, the program will automatically calc res from the spatial res of the scans
    mode = 1 # gridding mode (simple nearest neighbour)
    #mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
    #mode = 3 # gridding mode (gaussian weighted nearest neighbour)
    dowrite = 0 #disable writing of point cloud data to file

    PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, f, bedpick, flip_lr, chunk_size, model)

    PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater)

    PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot)

    PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected) #dowrite,

could be saved as, for example “proc_mysidescandata.py” and run from the command line using:

python proc_mysidescandata.py -i C:\MyData\R0087.DAT -s C:\MyData\R0087

or from within ipython (with a GUI prompt to navigate to the files):

%run proc_mysidescandata.py

If you are in bash (or git bash) you might want to automate through a folder of subfolders like this:

for k in $(find $PWD -type d -maxdepth 1 -mindepth 1); do python proc_mysidescandata.py -i "$k/${k##*/}.DAT" -s $k; done

which assumes the .DAT file is in the folder with the same root (such as a folder called R00123 which contains SON and IDX files as well as a file called R00123.DAT)

Using the GUI¶

From the command line (terminal):

python -c "import PyHum; PyHum.gui()"

Support¶

This is a new project written and maintained by Daniel Buscombe. Bugs are expected - please report them, I will fix them quickly. Feedback and suggestions for improvements are very welcome

Please download, try, report bugs, fork, modify, evaluate, discuss, collaborate. Please use the ‘Issues’ tab in github here

Thanks for stopping by!

Trouble Shooting¶

1. Problem: pyhum read hangs for a long time (several minutes) on the test script. Try this: uninstall joblib and install an older version:

pip uninstall joblib
pip install joblib==0.7.1

2. Problem: you get an “invalid mode or file name” error. Try this: construct file paths using raw strings e.g.:

r'C:\Users\me\mydata\R0089'

or using os, e.g.:

import os
os.path.abspath(os.path.join('C:\Users','me','mydata','R0089'))

Problem: on Linux, PyHum is using an older version of scipy than 0.16, as revealed by:

python -c ‘import scipy;print(scipy.__version__)’

Try this: remove a system installed file e.g.:

sudo apt-get remove python-scipy ##(Debian based)
yum remove scipy ##(Fedora based)

Problem: do I have the latest version of PyHum installed? Check your version using this:

python -c ‘import PyHum;print(PyHum.__version__)’

Check this against the latest bleeding-edge’ version `here (line 47)