Updates to exporter

This commit is contained in:
Jamie Mac 2024-05-27 16:55:53 +01:00
parent d8cd536f26
commit 4707830953
12 changed files with 289 additions and 119 deletions

View File

@ -1,5 +1,73 @@
This is the main code repository for the PAMGuard software.
# PAMGuard
PAMGuard is a bioacoustics analysis program designed for use in real time research contexts and for the processing of large datasets. PAMGuard provides users access to a suite of state-of-the-art auotmated analysis algorithms alongside displays for visualisation data and a comprehensive data management systems.
This repository was created on 7 January 2022 from sourceforge SVN repository at https://sourceforge.net/p/pamguard/svn/HEAD/tree/ revision r6278.
# Why do we need PAMGuard?
PAMGuard fufills two main requirements within marine bioacoustics
1) **Real time operation** - Almost all PAMGuard features and modules work in real time - this allows scientists and industry to detect, classify and loclaise animals in real time on a standard consumer laptop, enabling mitigations and research survey without expensive bespoke software solutions and the transparncy of open source software.
2) **Processing and visuslisation of large datasets** -
## Installation
PAMGuard is available on Windows and can be downloaded from the [PAMGuard website](www.pamguard.org). Note that we are considering MacOS installers but they are not available at this time.
## Tutorial
PAMGuard is a modular program with two modes; real-time and viewer. Typically a user will start with real-time model, either in the field collecting data or post processing sound files from a recorder. Once data are processed, users move on to viewer mode where data can be explored and further processed.
Upon opening PAMGuard for the first time you are greeted with a blank screen. You must add a series of modules to create the desired acosutic workflow. For example if processing sound files then first add the Sound Acquisition module **_File->Add Modules->Sound Processing->Sound Acquisition_**. Then add the desired detection algorothms e.g. **_File->Add Modules->Detector->Click Detectors_**. Some modules (such as the click detector) have their own displays, others are added to more generalised displays. For example, the whislte and moan detector module shows detections on a spectrgram display. First add a new tab using **_File->Add Modules->Displays->User Display**. Click on the user display tab and then from the top menu select **_User display-> New Spectrgram_**. Right click on the added spectrgram and select whistle and moan contours to show whistle detections overlaid on the raw spectrgram.
Make sure to add the database and binary file storage modules **_File->Add Modules->Utilities->..._**) to save data then press the run button (red button) and data will process. PAMGuard can handle huge datasets so runing might take hours or even days. Progress is shown on the bottom of the screen.
## Features
### Hardware integration
PAMGuard connects with hardware such as various GPS and AIS systems and a multitude of different sound cards (e.g. [National Instruments](www.ni.com) devices, [SAIL DAQ cards](www.smruconsulting.com/contact-us), almost all ASIO sound cards and standard computer sound cards) for real time data collection and processing. PAMGuard also works with some very bespoke hardware such as [DIFAR Sonobuoys]();
### Real time operation
PAMGuard takes advanatge of multi-core processors to run multiple signal processing automatic analysis algorithms in real time to detect whales, dolphins, bats etc. Data are shown in different displayes, including interactive spectrograms and maps. You might be using PAMGuard for simply viewing a spectrgram and making recordings or running deep learning algorithms for multiple species and loclaising the results to view locations on a map. Whatever acosutic workflow a user creates, PAMGuard can run it in real time.
### Support for compressed audio
PAMGuard supports processing audio data from standard files (e.g. wav, aif) and also compressed files (e.g. .flac and .sud). Notew that sud files are created on SoundTraps widely used marine recorders and can be read by PAMGuard without decompressing - PAMGuard will automtically import click detections if present in sud files. PAMGuard also supports importing detection data from CPODs and FPODs.
### Comprehensive data management system
PAMGuard is designed to collect/process data from large acosutic datasets. PAMGuard stores data in an SQLite databases and "Binary" files. The database stores important metadata such as when data has been processed and some low volume data streams such as GPS. Binary files are not human readbale but efficient to access - PAMGuard stores detection data (e.g. clicks, whistles, noise, etc) in these files. this allows PAMGuard to rapidly access data from large datasets. Data from binary files can be viewed in PAMGuard viewer mode or can be exported to MATLAB using the PAMGuard-MATLAB library or the exported to R using the R PAMBinaries package.
### Access to detection and classification algorithms
PAMGuard allows users to inegrate automated detection and classification algorithms directly into their acosutic workflow. There are a multitude of differwent algorothms to choose from, including a basic click detector, whislte and moan detector, GPL detector, click train detectors and many others. The idea behind PAMGuard is allow researchers to access open source state-of-the-art algorithms devleoped within the scientific community - if you want to contribute and get your algorithm into PAMGuard get in touch.
###Localisation
PAMGuard has a mutltude of different options for acoustic loclaisation. There's a comprehesnive beam forming module for beam forming arrays, a large aperture localiser for 3D loclaisation and target motion analysis for towed hydrophone arrays.
###Soundscape analysis
PAMGuard has a noise band (which supports third octave noise bands) and long term spectral average module for soundscape analysis.
### GIS
Almsot all detection data can be visualised on a map. PAMGaurd also supports plotting GPS and AIS data.
### Suite of data visualisation tools
An important aspect of PAMGuard is the ability for users to explore porcessed data. This is
### Advanced manual annotation
The displays within PAMGuard support a variety of manual annottion tools. A simple spectrogram
### Deep learning integration
### Meatadata standard and Tethys compatibility
## Feature roadmap
There's lots of features we would like to add to PAMGuard. If you want to add a feature you can either code it up yourself in Java and submit a pull request or get in touch with us to discuss how to it might be integrated. Some smaller features might be in our roadmap anyway but larger features usually require funding. Some features we are thinking about (but do not necassarily have time for yet) are;
* Support for decidecade noise bands (base 10 filter bank) in noise band monitor to meet Euopean standards
* Capabaility to export data directly from PAMGaurd e.g. as MAT files.
* Automated test suite to make releases more stable. Note that unit and integration tests are also being slowly incorporated.
## Development
This is the main code repository for the PAMGuard software and was created on 7 January 2022 from a [sourceforge SVN repository](https://sourceforge.net/p/pamguard/svn/HEAD/tree/) revision r6278.
If you are a PAMGuard developer, you should clone and branch this repository and share with any collaborators in your own workspace. When your work is ready, contact the PAMGuard team to have your changes merged back into this repo.
PAMGuard uses Maven as build tool.
# Organisation and License
PAMGuard is open source under an MIT license. It is currently primarily managed by the Sea Mammal Research Unit within the [University of St Andrews](https://www.st-andrews.ac.uk/). Please get in touch if you have any questions.

View File

@ -3,13 +3,16 @@ package PamUtils;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.NoSuchElementException;
import java.util.TreeMap;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.math.linear.Array2DRowRealMatrix;
import org.apache.commons.math.linear.RealMatrix;
import PamguardMVC.PamDataUnit;
import us.hebi.matlab.mat.types.Matrix;
/**
@ -177,6 +180,22 @@ public class PamArrayUtils {
return median;
}
/**
* Get the data unit with the lowest time in millis from a data unit list.
* @param dataUnits - a data unit list.
* @return the data unit with the lowest time in millis.
*/
public static PamDataUnit getMinTimeMillis(List<PamDataUnit> dataUnits) {
// then
PamDataUnit minByTime = dataUnits
.stream()
.min(Comparator.comparing(PamDataUnit::getTimeMilliseconds))
.orElseThrow(NoSuchElementException::new);
return minByTime;
}
/**
* Calculate the median value of an array
* @param numArray - the number array

View File

@ -35,4 +35,10 @@ public class CSVExportManager implements PamDataUnitExporter{
return "CSV Export";
}
@Override
public void close() {
// TODO Auto-generated method stub
}
}

View File

@ -10,6 +10,7 @@ import java.util.List;
import java.util.NoSuchElementException;
import java.util.zip.Deflater;
import PamUtils.PamArrayUtils;
import PamUtils.PamCalendar;
import PamguardMVC.PamDataUnit;
import export.PamDataUnitExporter;
@ -32,9 +33,9 @@ public class MLDetectionsManager implements PamDataUnitExporter {
public static final String extension = "mat";
// Creating date format
public static SimpleDateFormat dataFormat = new SimpleDateFormat(
"yyyyMMdd_HHmmss_SSS");
// Creating date format
public static SimpleDateFormat dataFormat = new SimpleDateFormat(
"yyyyMMdd_HHmmss_SSS");
/**
*
@ -81,14 +82,11 @@ public class MLDetectionsManager implements PamDataUnitExporter {
Struct dataUnitsStruct = dataUnits2MAT(dataUnits);
// then
PamDataUnit minByTime = dataUnits
.stream()
.min(Comparator.comparing(PamDataUnit::getTimeMilliseconds))
.orElseThrow(NoSuchElementException::new);
// then
PamDataUnit minByTime = PamArrayUtils.getMinTimeMillis(dataUnits);
//matlab struct must start with a letter.
Date date = new Date(minByTime.getTimeMilliseconds());
Date date = new Date(minByTime.getTimeMilliseconds());
String entryName = "det_" + dataFormat.format( date);
//is there an existing sink? Is that sink writing to the correct file?
@ -104,7 +102,7 @@ public class MLDetectionsManager implements PamDataUnitExporter {
//create the Mat File - gets all the headers right etc.
Mat5File matFile = Mat5.newMatFile();
matFile.addArray(entryName, dataUnitsStruct);
// matFile.addArray("two", Mat5.newScalar(2));
// matFile.addArray("two", Mat5.newScalar(2));
matFile.writeTo(sink);
@ -118,7 +116,7 @@ public class MLDetectionsManager implements PamDataUnitExporter {
writer
.writeArray(entryName, dataUnitsStruct)
.setDeflateLevel(Deflater.NO_COMPRESSION);
// .writeArray("three", Mat5.newScalar(2));
// .writeArray("three", Mat5.newScalar(2));
writer.flush();
}
@ -198,7 +196,7 @@ public class MLDetectionsManager implements PamDataUnitExporter {
}
}
if (n>=1) {
if (n>0) {
list.set(mlDataUnitsExport.get(i).getName(),mlStructure);
list.set(mlDataUnitsExport.get(i).getName()+"_sR", Mat5.newScalar(sampleRate));
}
@ -245,11 +243,11 @@ public class MLDetectionsManager implements PamDataUnitExporter {
Mat5.writeToFile(matFile, fileName);
// Sink sink = Sinks.newMappedFile(new File(fileName), Casts.sint32(1000000));
//
// matFile.writeTo(sink);
//
// sink.close();
// Sink sink = Sinks.newMappedFile(new File(fileName), Casts.sint32(1000000));
//
// matFile.writeTo(sink);
//
// sink.close();
} catch (IOException e) {
// TODO Auto-generated catch block
@ -257,6 +255,20 @@ public class MLDetectionsManager implements PamDataUnitExporter {
}
}
@Override
public void close() {
//handled in the mian funtion
if (sink!=null) {
try {
sink.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

View File

@ -46,5 +46,10 @@ public interface PamDataUnitExporter {
*/
public String getName();
/**
* Clsoe the exporter
*/
public void close();
}

View File

@ -74,15 +74,15 @@ public class PamExporterManager {
if (dataUnit==null) {
if (force) {
System.out.println("Write data 1!!" + dataUnitBuffer.size());
System.out.println("Write data 1!!" + dataUnitBuffer.size() );
//finish off saving any buffered data
exportOK = pamExporters.get(exportParams.exportChoice).exportData(currentFile, dataUnitBuffer, true);
dataUnitBuffer.clear();
}
return true;
}
//if the data unit is null then save everything to the buffer.
//if file is null or too large create another a file for saving.
if (currentFile == null || isFileSizeMax(currentFile)) {
Date date = new Date(dataUnit.getTimeMilliseconds());
@ -98,11 +98,10 @@ public class PamExporterManager {
dataUnitBuffer.add(dataUnit);
// System.out.println("Write data unit " + dataUnitBuffer.size() + " to: "+ currentFile);
System.out.println("Write data unit " + dataUnitBuffer.size() + " to: "+ currentFile);
if (dataUnitBuffer.size()>=BUFFER_SIZE || force) {
System.out.println("Write data 2!!" + dataUnitBuffer.size());
exportOK = pamExporters.get(exportParams.exportChoice).exportData(currentFile, dataUnitBuffer, true);
dataUnitBuffer.clear();
}
@ -111,6 +110,11 @@ public class PamExporterManager {
}
public void close() {
pamExporters.get(exportParams.exportChoice).close();
}
/**
* Check whether the current file is greater than the maximum allowed file size.
* @param currentFile2 - the current file
@ -171,4 +175,6 @@ public class PamExporterManager {
}
}

View File

@ -17,7 +17,7 @@ public class RClickExport extends RRawExport{
ClickDetection clickDetection = (ClickDetection) dataUnit;
super.addDetectionSpecificFields(rData, null, index);
super.addDetectionSpecificFields(rData, dataUnit, index);
//add some basic click measurements
rData.add("triggerMap", new DoubleArrayVector(clickDetection.getTriggerList()));

View File

@ -4,16 +4,22 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.zip.GZIPOutputStream;
import org.renjin.eval.Context;
import org.renjin.primitives.io.serialization.RDataWriter;
import org.renjin.sexp.DoubleArrayVector;
import org.renjin.sexp.IntArrayVector;
import org.renjin.sexp.ListVector;
import org.renjin.sexp.PairList;
import org.renjin.sexp.PairList.Builder;
import PamUtils.PamArrayUtils;
import PamguardMVC.PamDataUnit;
import export.PamDataUnitExporter;
import export.MLExport.MLDetectionsManager;
import pamViewFX.fxNodes.pamDialogFX.PamDialogFX;
/**
@ -28,74 +34,76 @@ public class RExportManager implements PamDataUnitExporter {
*
* All the possible RDataUnit export classes.
*/
ArrayList<RDataUnitExport> mlDataUnitsExport = new ArrayList<RDataUnitExport>();
private FileOutputStream fos;
private GZIPOutputStream zos;
ArrayList<RDataUnitExport> rDataExport = new ArrayList<RDataUnitExport>();
private File currentFileName ;
private RDataWriter writer;
private Context context;
private Builder allData;
public RExportManager(){
/***Add more options here to export data units****/
mlDataUnitsExport.add(new RClickExport());
mlDataUnitsExport.add(new RWhistleExport());
mlDataUnitsExport.add(new RRawExport()); //should be last in case raw data holders have specific exporters
rDataExport.add(new RClickExport());
rDataExport.add(new RWhistleExport());
rDataExport.add(new RRawExport()); //should be last in case raw data holders have specific exporters
}
@Override
public boolean exportData(File fileName, List<PamDataUnit> dataUnits, boolean append) {
//convert the data units to
RData dataUnitsR = dataUnits2R(dataUnits);
/**
* Note - there is no way to save data units to R files wothout loading the file into memory.
* So everything is stored in memory until saved.
*/
// System.out.println("Export R file!!" + dataUnits.size());
// then
PamDataUnit minByTime = PamArrayUtils.getMinTimeMillis(dataUnits);
//now write the file
try {
//matlab struct must start with a letter.
Date date = new Date(minByTime.getTimeMilliseconds());
String entryName = "det_" + MLDetectionsManager.dataFormat.format( date);
//is there an existing writer? Is that writer writing to the correct file?
if (zos==null || fileName.equals(currentFileName)) {
// System.out.println("Save R data! "+ dataUnits.size());
if (zos!=null) {
zos.close();
writer.close();
}
// System.out.println("Export R file!!" + dataUnits.size());
currentFileName = fileName;
//is there an existing writer? Is that writer writing to the correct file?
if (allData==null || !fileName.equals(currentFileName)) {
// System.out.println("MLDATA size: "+ mlData.size());
// System.out.println("---MLArray----");
// for (int i=0; i<mlData.size(); i++) {
// System.out.println(mlData.get(i));
// System.out.println("-------");
// }
// System.out.println("--------------");
context = Context.newTopLevelContext();
fos = new FileOutputStream(fileName);
zos = new GZIPOutputStream(fos);
return true;
if (allData!=null) {
writeRFile();
}
writer = new RDataWriter(context, zos);
writer.save(dataUnitsR.rData.build());
allData = new PairList.Builder();
currentFileName = fileName;
}
return true;
//convert the data units to R and save to the PairList builder
dataUnits2R(dataUnits, entryName, allData);
return true;
}
private void writeRFile() {
Context context = Context.newTopLevelContext();
try {
FileOutputStream fos = new FileOutputStream(currentFileName);
GZIPOutputStream zos = new GZIPOutputStream(fos);
RDataWriter writer = new RDataWriter(context, zos);
writer.save(allData.build());
writer.close();
zos.close();
}
catch (IOException e1) {
e1.printStackTrace();
return false;
}
}
/**
@ -113,11 +121,11 @@ public class RExportManager implements PamDataUnitExporter {
@Override
public boolean hasCompatibleUnits(Class dataUnitType) {
for (int i=0; i<mlDataUnitsExport.size(); i++){
for (int i=0; i<rDataExport.size(); i++){
//check whether the same. ;
//System.out.println(" dataUnits.get(j).getClass(): " + dataUnits.get(j).getClass());
//System.out.println(" mlDataUnitsExport.get(i).getUnitClass(): " + mlDataUnitsExport.get(i).getUnitClass());
if (mlDataUnitsExport.get(i).getUnitClass().isAssignableFrom(dataUnitType)) {
if (rDataExport.get(i).getUnitClass().isAssignableFrom(dataUnitType)) {
//System.out.println("FOUND THE DATA UNIT!");
return true;
}
@ -131,21 +139,36 @@ public class RExportManager implements PamDataUnitExporter {
* @return list of list of R strucutures ready for saving to .RData file.
*/
public RData dataUnits2R(List<PamDataUnit> dataUnits){
PairList.Builder allData = new PairList.Builder();
return dataUnits2R(dataUnits, null, allData);
}
/**
* Sort a list of data units into lists of the same type of units. Convert to a list of structures.
* @param dataUnits - a list of data units to convert to matlab structures.
* @param - a name for the structure.
* @return list of list of R strucutures ready for saving to .RData file.
*/
public RData dataUnits2R(List<PamDataUnit> dataUnits, String name, PairList.Builder allData) {
//if there's a mixed bunch of data units then we want separate arrays of structures. So a structure of arrays of structures.
//so, need to sort compatible data units.
PairList.Builder allData = new PairList.Builder();
ArrayList<String> dataUnitTypes = new ArrayList<String>();
//keep a track of the data units that have been transcribed. This means data units that are multiple types
//(e.g. a raw data holder and click) are not added to two different list of structures.
boolean[] alreadyStruct = new boolean[dataUnits.size()];
//iterate through possible export functions.
for (int i=0; i<mlDataUnitsExport.size(); i++){
for (int i=0; i<rDataExport.size(); i++){
//first need to figure out how many data units there are.
int n=0;
for (int j=0; j<dataUnits.size(); j++){
//check whether the same.
if (mlDataUnitsExport.get(i).getUnitClass().isAssignableFrom(dataUnits.get(j).getClass())) {
if (rDataExport.get(i).getUnitClass().isAssignableFrom(dataUnits.get(j).getClass())) {
n++;
}
}
@ -154,22 +177,35 @@ public class RExportManager implements PamDataUnitExporter {
if (n==0) continue; //no need to do anything else. There are no data units of this type.
ListVector.NamedBuilder dataListArray = new ListVector.NamedBuilder();
ListVector.NamedBuilder dataList;
n=0;
double sampleRate = 0.;
//allocate the class now.
for (int j=0; j<dataUnits.size(); j++){
//check whether the same.
if (mlDataUnitsExport.get(i).getUnitClass().isAssignableFrom(dataUnits.get(j).getClass())) {
dataList=mlDataUnitsExport.get(i).detectionToStruct(dataUnits.get(j), n);
dataListArray.add((mlDataUnitsExport.get(i).getName() + "_" + dataUnits.get(j).getUID()), dataList);
if (rDataExport.get(i).getUnitClass().isAssignableFrom(dataUnits.get(j).getClass()) && !alreadyStruct[j]) {
dataList=rDataExport.get(i).detectionToStruct(dataUnits.get(j), n);
dataListArray.add((rDataExport.get(i).getName() + "_" + dataUnits.get(j).getUID()), dataList);
sampleRate = dataUnits.get(j).getParentDataBlock().getSampleRate();
n++;
alreadyStruct[j] = true;
}
}
if (n>1) {
allData.add(mlDataUnitsExport.get(i).getName(), dataListArray.build());
dataUnitTypes.add(mlDataUnitsExport.get(i).getName());
if (n>0) {
String dataName;
if (name==null) {
dataName = rDataExport.get(i).getName();
}
else dataName = name + "_" + rDataExport.get(i).getName();
allData.add(dataName, dataListArray.build());
allData.add(rDataExport.get(i).getName()+"_sR", new DoubleArrayVector(sampleRate));
dataUnitTypes.add(rDataExport.get(i).getName());
}
}
@ -195,7 +231,7 @@ public class RExportManager implements PamDataUnitExporter {
public PairList.Builder rData;
/**
* List of the names of the types of data units whihc were saved.
* List of the names of the types of data units which were saved.
*/
public ArrayList<String> dataUnitTypes;
}
@ -218,5 +254,13 @@ public class RExportManager implements PamDataUnitExporter {
}
@Override
public void close() {
if (allData!=null) {
writeRFile();
}
}
}

View File

@ -73,7 +73,7 @@ public class RRawExport extends RDataUnitExport<PamDataUnit> {
@Override
public String getName() {
return "pam_data_units";
return "raw_data_units";
}
}

View File

@ -60,8 +60,10 @@ public class ExportTask extends OfflineTask<PamDataUnit<?,?>>{
@Override
public void loadedDataComplete() {
System.out.println("EXPORTER: loaded data complete");
//force the exporter so save any renaming data units in the buffer
exporter.exportDataUnit(null, true);
exporter.exportDataUnit(null, true);
exporter.close();
exporter.setCurrentFile(null);
}

View File

@ -459,6 +459,14 @@ public class WavFileExportManager implements PamDataUnitExporter {
@Override
public void close() {
// TODO Auto-generated method stub
}
// hello(){

View File

@ -264,7 +264,7 @@ public class PlotPane extends PamBorderPane {
rightPane.prefWidthProperty().bind(yAxisRightPane.widthProperty());
rightPane.minWidthProperty().bind(yAxisRightPane.widthProperty());
horzHolder.getChildren().addAll(leftPane, axisPane);
horzHolder.getChildren().addAll(leftPane, axisPane, rightPane);
//axisPane.toFront(); this changes the order of children in a PamHBox.
HBox.setHgrow(axisPane, Priority.ALWAYS);