Getting started¶
About¶
PyHum - a Python framework for reading and processing data from a Humminbird low-cost sidescan sonar
PyHum is an open-source project dedicated to provide a generic Python framework for reading and exporting data from Humminbird(R) instruments, carrying out rudimentary radiometric corrections to the data, classify bed texture, and produce some maps on aerial photos and kml files for google-earth
The software is designed to read Humminbird data (.SON, .IDX, and .DAT files) and works on both sidescan and downward-looking echosounder data, where available.
Some aspects of the program are detailed in:
Buscombe, D., Grams, P.E., and Smith, S. (2015) “Automated riverbed sediment classification using low-cost sidescan sonar”, Journal of Hydraulic Engineering, 10.1061/(ASCE)HY.1943-7900.0001079, 06015019. Download here
Full documentation of the procedures behind the program is forthcoming in the following publication:
Buscombe, D., submitted, Processing and georeferencing recreational-grade sidescan-sonar data to support the democratization of acoustic imaging in shallow water. LIMNOLOGY AND OCEANOGRAPHY: METHODS.
For the source code visit the project github site
Primary Developer | Daniel Buscombe —— | —————
Grand Canyon Monitoring and Research Center | United States Geological Survey | Flagstaff, AZ 86001 | dbuscombe@usgs.govCo-Developer | Daniel Hamill —— | —————
Department of Watershed Sciences | Utah State University | Logan, UT 84322 | dhamill@usgs.gov
License¶
This software is in the public domain because it contains materials that originally came from the United States Geological Survey, an agency of the United States Department of Interior. For more information, see the official USGS copyright policy
Any use of trade, product, or firm names is for descriptive purposes only and does not imply endorsement by the U.S. government.
This software is issued under the GNU Lesser General Public License, Version 3
Thanks to Barb Fagetter (blueseas@oceanecology.ca) for some format info, Dan Hamill (Utah State University) and Paul Anderson (Quest Geophysical Asia) for debugging and suggestions for improvements
Some aspects of the program are detailed in: 1. Buscombe, D., Grams, P.E., and Smith, S. (2015) “Automated riverbed sediment classification using low-cost sidescan sonar”, Journal of Hydraulic Engineering, 10.1061/(ASCE)HY.1943-7900.0001079, 06015019.
- Buscombe, D., submitted, Processing and georeferencing recreational-grade sidescan-sonar data to support the democratization of acoustic imaging in shallow water. LIMNOLOGY AND OCEANOGRAPHY: METHODS.
Full documentation of the program is forthcoming
Setup on Anaconda for Windows¶
PyHum currently has only been tested with Python 2.7, so you’ll need that version of Anaconda
Step 1. Before installing PyHum, install Basemap using:
conda install basemap
Step 2. Install pyproj. pip seems to have a bug with pyproj depending on what c-compiler your python distribution uses. Therefore, you may have to install pyproj (and other dependencies) from here
- download the .whl file. Choose the file associated with python 2.7 (“cp27”) and the architecture you are using, i.e. 32-bit (win32) or 64-bit (amd64)
- then move that file to your root Anaconda directory (when you open an Anaconda command prompt it’s the directory that’s listed before the prompt ‘>’)
- then use pip to install it, e.g.:
pip install pyproj-1.9.4-cp27-none-win_amd64.whl
Step 4. Install simplekml, using:
pip install simplekml
Step 3. Assuming a Anaconda distribution which comes with almost all required program dependencies:
pip uninstall PyHum (removes any previous installation)
pip install PyHum
If you get C++ compiler errors (such as “Unable to find vcvarsall.bat”), you will need to install the Microsoft Visual C++ compiler from here
(Advanced) If you have git installed (from here), you can install the latest ‘bleeding edge’ (pre-release) version directly from github:
pip install git+https://github.com/dbuscombe-usgs/PyHum.git
Setup¶
Automatic Installation from PyPI:
pip uninstall PyHum (removes previous installation)
pip install PyHum
Automatic Installation from github:
git clone git@github.com:dbuscombe-usgs/PyHum.git
cd PyHum
python setup.py install
or a local installation:
python setup.py install --user
or with admin privileges, e.g.:
sudo python setup.py install
This software has been tested with Python 2.7 on 1. Windows 7 2. Linux Fedora (16 & 20) 3. Linux Ubuntu (12.4 & 13.4 & 14.4)
This software has (so far) been used only with Humminbird 798, 898, 998, 1198 and 1199 series instruments.
Virtual environment¶
You could try before you install, using a virtual environment:
virtualenv venv
source venv/bin/activate
pip install numpy
pip install cython
pip install scipy
pip install joblib
pip install simplekml
pip install pyproj
pip install scikit-learn
pip install Pillow
pip install matplotlib
pip install basemap --allow-external basemap --allow-unverified basemap
pip install pyresample
pip install toolz
pip install dask
pip install pandas
pip install PyHum
python -c "import PyHum; PyHum.test()"
deactivate #(or source venv/bin/deactivate)
The results will live in “venv/lib/python2.7/site-packages/PyHum”
Manual installation¶
Python libraries you need to have installed to use PyHum:
- SciPy
- Numpy
- pyresample
- Matplotlib
- cython
- joblib
- Scikit-learn
- Python Image LIbrary (PIL)
- simplekml
- pyproj
- basemap
- pyresample
- toolz
- dask
- pandas
All of the above are available through pip and easy_install
Installation on Amazon Linux EC-2 instance¶
It’s best to install numpy, scipy, cython and matplotlib through the OS package manager:
sudo yum install gcc gcc-c++
sudo yum install python27-numpy python27-Cython python27-scipy python27-matplotlib
Then install geos libraries using yum and Basemap using pip:
sudo yum install geos geos-devel geos-python27
sudo pip install basemap --allow-external basemap --allow-unverified basemap
Then PyHum using pip (which will install Pillow, pyproj, simplekml, joblib and scikit-learn):
sudo pip install PyHum
Test¶
A test can be carried out by running the supplied script:
python -c "import PyHum; PyHum.test.dotest()"
which carries out the following operations:
# general settings
humfile = os.path.normpath(os.path.join(os.path.expanduser("~"),'pyhum_test','test.DAT'))
sonpath = os.path.normpath(os.path.join(os.path.expanduser("~"),'pyhum_test'))
doplot = 1 #yes
# reading specific settings
cs2cs_args = "epsg:26949" #arizona central state plane
bedpick = 1 # auto bed pick
c = 1450 # speed of sound fresh water
t = 0.108 # length of transducer
draft = 0.3 # draft in metres
flip_lr = 1 # flip port and starboard
model = 998 # humminbird model
calc_bearing = 1 #1=yes
filt_bearing = 1 #1=yes
chunk = 'd100' # distance, 100m
#chunk = 'p1000' # pings, 1000
#chunk = 'h10' # heading deviation, 10 deg
# correction specific settings
maxW = 1000 # rms output wattage
dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans)
correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)
ph = 7.0 # acidity on the pH scale
temp = 10.0 # water temperature in degrees Celsius
salinity = 0.0
# for shadow removal
shadowmask = 0 #automatic shadow removal
win = 31
# for texture calcs
shift = 10 # pixel shift
density =win/2 # win/2
numclasses = 4 # number of discrete classes for contouring and k-means
maxscale = 20 # Max scale as inverse fraction of data length (for wavelet analysis)
notes = 4 # Notes per octave (for wavelet analysis)
# for mapping
res = 0.25 #99 # grid resolution in metres
# if res==99, the program will automatically calc res from the spatial res of the scans
mode = 1 # gridding mode (simple nearest neighbour)
#mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
#mode = 3 # gridding mode (gaussian weighted nearest neighbour)
use_uncorrected = 0
nn = 64 #number of nearest neighbours for gridding (used if mode > 1)
##influence = 1 #Radius of influence used in gridding. Cut off distance in meters
numstdevs = 5 #Threshold number of standard deviations in sidescan intensity per grid cell up to which to accept
# for downward-looking echosounder echogram (e1-e2) analysis
beam = 20.0
transfreq = 200.0 # frequency (kHz) of downward looking echosounder
integ = 5
numclusters = 3 # number of acoustic classes to group observations
## read data in SON files into PyHum memory mapped format (.dat)
PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, chunk)
## correct scans and remove water column
PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)
## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)
PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot)
win = 100 # pixel window
## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
PyHum.texture(humfile, sonpath, win, shift, doplot, density, numclasses, maxscale, notes)
## grid and map the scans
PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected) #dowrite,
res = 1 # grid resolution in metres
numstdevs = 5
## grid and map the texture lengthscale maps
PyHum.map_texture(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs)
## calculate and map the e1 and e2 acoustic coefficients from the downward-looking sonar
PyHum.e1e2(humfile, sonpath, cs2cs_args, ph, temp, salinity, beam, transfreq, integ, numclusters, doplot)
Getting Started¶
Inputs to the program are a .DAT file (e.g. R0089.DAT) and a folder of .SON and .IDX files (e.g. /my/folder/R0089). The program will read the .SON files with or without the accompanying .IDX files, but will be faster if the .IDX files are present.
PyHum is modular so can be called from within a python or ipython console, from an IDE (such as IDLE or Spyder), or by running a script.
The following example script:
import sys, getopt
from Tkinter import Tk
from tkFileDialog import askopenfilename, askdirectory
import PyHum
import os
if __name__ == '__main__':
argv = sys.argv[1:]
humfile = ''; sonpath = ''
# parse inputs to variables
try:
opts, args = getopt.getopt(argv,"hi:s:")
except getopt.GetoptError:
print 'error'
sys.exit(2)
for opt, arg in opts:
if opt == '-h':
print 'help'
sys.exit()
elif opt in ("-i"):
humfile = arg
elif opt in ("-s"):
sonpath = arg
# prompt user to supply file if no input file given
if not humfile:
print 'An input file is required!!!!!!'
Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
humfile = askopenfilename(filetypes=[("DAT files","*.DAT")])
# prompt user to supply directory if no input sonpath is given
if not sonpath:
print 'A *.SON directory is required!!!!!!'
Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
sonpath = askdirectory()
# print given arguments to screen and convert data type where necessary
if humfile:
print 'Input file is %s' % (humfile)
if sonpath:
print 'Son files are in %s' % (sonpath)
# general settings
doplot = 1 #yes
# reading specific settings
cs2cs_args = "epsg:32100" #NAD83 / Montana
bedpick = 2 # manual bed pick
c = 1450 # speed of sound fresh water
t = 0.108 # length of transducer
draft = 0.3 # draft in metres
flip_lr = 1 # flip port and starboard
model = 1199 # humminbird model
dowrite = 0 #disable writing of point cloud data to file
chunk = 'd100' # distance, 100m
#chunk = 'p1000' # pings, 1000
#chunk = 'h10' # heading deviation, 10 deg
# correction specific settings
maxW = 1000 # rms output wattage
dofilt = 1 # apply a phase preserving filter (WARNING!! takes a very long time for large scans)
correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)
# for shadow removal
shadowmask = 1 #manual shadow removal
win = 100
# for mapping
res = 99 # grid resolution in metres
# if res==99, the program will automatically calc res from the spatial res of the scans
mode = 1 # gridding mode (simple nearest neighbour)
#mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
#mode = 3 # gridding mode (gaussian weighted nearest neighbour)
dowrite = 0 #disable writing of point cloud data to file
PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, f, bedpick, flip_lr, chunk_size, model)
PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater)
PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot)
PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected) #dowrite,
could be saved as, for example “proc_mysidescandata.py” and run from the command line using:
python proc_mysidescandata.py -i C:\MyData\R0087.DAT -s C:\MyData\R0087
or from within ipython (with a GUI prompt to navigate to the files):
%run proc_mysidescandata.py
If you are in bash (or git bash) you might want to automate through a folder of subfolders like this:
for k in $(find $PWD -type d -maxdepth 1 -mindepth 1); do python proc_mysidescandata.py -i "$k/${k##*/}.DAT" -s $k; done
which assumes the .DAT file is in the folder with the same root (such as a folder called R00123 which contains SON and IDX files as well as a file called R00123.DAT)
Support¶
This is a new project written and maintained by Daniel Buscombe. Bugs are expected - please report them, I will fix them quickly. Feedback and suggestions for improvements are very welcome
Please download, try, report bugs, fork, modify, evaluate, discuss, collaborate. Please use the ‘Issues’ tab in github here
Thanks for stopping by!
Trouble Shooting¶
1. Problem: pyhum read hangs for a long time (several minutes) on the test script. Try this: uninstall joblib and install an older version:
pip uninstall joblib
pip install joblib==0.7.1
2. Problem: you get an “invalid mode or file name” error. Try this: construct file paths using raw strings e.g.:
r'C:\Users\me\mydata\R0089'
or using os, e.g.:
import os
os.path.abspath(os.path.join('C:\Users','me','mydata','R0089'))
Problem: on Linux, PyHum is using an older version of scipy than 0.16, as revealed by:
python -c ‘import scipy;print(scipy.__version__)’
Try this: remove a system installed file e.g.:
sudo apt-get remove python-scipy ##(Debian based)
yum remove scipy ##(Fedora based)
Problem: do I have the latest version of PyHum installed? Check your version using this:
python -c ‘import PyHum;print(PyHum.__version__)’
Check this against the latest bleeding-edge’ version `here (line 47)
