Updates to exporter

readme.md

@@ -1,5 +1,73 @@
-This is the main code repository for the PAMGuard software. 
+# PAMGuard
+PAMGuard is a bioacoustics analysis program designed for use in real time research contexts and for the processing of large datasets. PAMGuard provides users access to a suite of state-of-the-art auotmated analysis algorithms alongside displays for visualisation data and a comprehensive data management systems. 
 
-This repository was created on 7 January 2022 from sourceforge SVN repository at https://sourceforge.net/p/pamguard/svn/HEAD/tree/ revision r6278.
+# Why do we need PAMGuard?
+PAMGuard fufills two main requirements within marine bioacoustics
+
+1) **Real time operation** - Almost all PAMGuard features and modules work in real time - this allows scientists and industry to detect, classify and loclaise animals in real time on a standard consumer laptop, enabling mitigations and research survey without expensive bespoke software solutions and the transparncy of open source software.  
+
+2) **Processing and visuslisation of large datasets** - 
+
+ 
+## Installation
+PAMGuard is available on Windows and can be downloaded from the [PAMGuard website](www.pamguard.org). Note that we are considering MacOS installers but they are not available at this time. 
+
+## Tutorial
+PAMGuard is a modular program with two modes; real-time and viewer. Typically a user will start with real-time model, either in the field collecting data or post processing sound files from a recorder. Once data are processed, users move on to viewer mode where data can be explored and further processed. 
+
+Upon opening PAMGuard for the first time you are greeted with a blank screen. You must add a series of modules to create the desired acosutic workflow. For example if processing sound files then first add the Sound Acquisition module **_File->Add Modules->Sound Processing->Sound Acquisition_**. Then add the desired detection algorothms e.g.  **_File->Add Modules->Detector->Click Detectors_**. Some modules (such as the click detector) have their own displays, others are added to more generalised displays. For example, the whislte and moan detector module shows detections on a spectrgram display. First add a new tab using  **_File->Add Modules->Displays->User Display**. Click on the user display tab and then from the top menu select **_User display-> New Spectrgram_**. Right click on the added spectrgram and select whistle and moan contours to show whistle detections overlaid on the raw spectrgram. 
+
+Make sure to add the database and binary file storage modules **_File->Add Modules->Utilities->..._**) to save data then press the run button (red button) and data will process. PAMGuard can handle huge datasets so runing might take hours or even days. Progress is shown on the bottom of the screen. 
+
+## Features
+
+### Hardware integration
+PAMGuard connects with hardware such as various GPS and AIS systems and a multitude of different sound cards (e.g. [National Instruments](www.ni.com) devices, [SAIL DAQ cards](www.smruconsulting.com/contact-us), almost all ASIO sound cards and standard computer sound cards) for real time data collection and processing. PAMGuard also works with some very bespoke hardware such as [DIFAR Sonobuoys]();
+
+### Real time operation
+PAMGuard takes advanatge of multi-core processors to run multiple signal processing automatic analysis algorithms in real time to detect whales, dolphins, bats etc. Data are shown in different displayes, including interactive spectrograms and maps. You might be using PAMGuard for simply viewing a spectrgram and making recordings or running deep learning algorithms for multiple species and loclaising the results to view locations on a map. Whatever acosutic workflow a user creates, PAMGuard can run it in real time. 
+
+### Support for compressed audio
+PAMGuard supports processing audio data from standard files (e.g. wav, aif) and also compressed files (e.g. .flac and .sud). Notew that sud files are created on SoundTraps widely used marine recorders and can be read by PAMGuard without decompressing - PAMGuard will automtically import click detections if present in sud files. PAMGuard also supports importing detection data from CPODs and FPODs. 
+
+### Comprehensive data management system
+PAMGuard is designed to collect/process data from large acosutic datasets. PAMGuard stores data in an SQLite databases and "Binary" files. The database stores important metadata such as when data has been processed and some low volume data streams such as GPS. Binary files are not human readbale but efficient to access - PAMGuard stores detection data (e.g. clicks, whistles, noise, etc) in these files. this allows PAMGuard to rapidly access data from large datasets. Data from binary files can be viewed in PAMGuard viewer mode or can be exported to MATLAB using the PAMGuard-MATLAB library or the exported to R using the R PAMBinaries package. 
+
+### Access to detection and classification algorithms
+PAMGuard allows users to inegrate automated detection and classification algorithms directly into their acosutic workflow. There are a multitude of differwent algorothms to choose from, including a basic click detector, whislte and moan detector, GPL detector, click train detectors and many others. The idea behind PAMGuard is allow researchers to access open source state-of-the-art algorithms devleoped within the scientific community - if you want to contribute and get your algorithm into PAMGuard get in touch. 
+
+###Localisation
+PAMGuard has a mutltude of different options for acoustic loclaisation. There's a comprehesnive beam forming module for beam forming arrays, a large aperture localiser for 3D loclaisation and target motion analysis for towed hydrophone arrays. 
+
+###Soundscape analysis
+PAMGuard has a noise band (which supports third octave noise bands) and long term spectral average module for soundscape analysis. 
+
+### GIS
+Almsot all detection data can be visualised on a map. PAMGaurd also supports plotting GPS and AIS data. 
+
+### Suite of data visualisation tools
+An important aspect of PAMGuard is the ability for users to explore porcessed data. This is 
+
+### Advanced manual annotation
+The displays within PAMGuard support a variety of manual annottion tools. A simple spectrogram 
+
+### Deep learning integration
+
+### Meatadata standard and Tethys compatibility
+
+## Feature roadmap
+There's lots of features we would like to add to PAMGuard. If you want to add a feature you can either code it up yourself in Java and submit a pull request or get in touch with us to discuss how to it might be integrated. Some smaller features might be in our roadmap anyway but larger features usually require funding. Some features we are thinking about (but do not necassarily have time for yet) are;
+
+* Support for decidecade noise bands (base 10 filter bank) in noise band monitor to meet Euopean standards
+* Capabaility to export data directly from PAMGaurd e.g. as MAT files.
+* Automated test suite to make releases more stable. Note that unit and integration tests are also being slowly incorporated. 
+
+## Development 
+This is the main code repository for the PAMGuard software and was created on 7 January 2022 from a [sourceforge SVN repository](https://sourceforge.net/p/pamguard/svn/HEAD/tree/) revision r6278.
 
 If you are a PAMGuard developer, you should clone and branch this repository and share with any collaborators in your own workspace. When your work is ready, contact the PAMGuard team to have your changes merged back into this repo.
+
+PAMGuard uses Maven as build tool. 
+
+# Organisation and License
+PAMGuard is open source under an MIT license. It is currently primarily managed by the Sea Mammal Research Unit within the [University of St Andrews](https://www.st-andrews.ac.uk/). Please get in touch if you have any questions. 

src/PamUtils/PamArrayUtils.java

@@ -3,13 +3,16 @@ package PamUtils;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.List;
+import java.util.NoSuchElementException;
 import java.util.TreeMap;
 
 import org.apache.commons.lang3.ArrayUtils;
 import org.apache.commons.math.linear.Array2DRowRealMatrix;
 import org.apache.commons.math.linear.RealMatrix;
 
+import PamguardMVC.PamDataUnit;
 import us.hebi.matlab.mat.types.Matrix;
 
 /**
@@ -177,6 +180,22 @@ public class PamArrayUtils {
 		return median;
 	}
 	
+	
+	/**
+	 * Get the data unit with the lowest time in millis from a data unit list. 
+	 * @param dataUnits - a data unit list.
+	 * @return the data unit with the lowest time in millis.
+	 */
+	public static PamDataUnit getMinTimeMillis(List<PamDataUnit> dataUnits) {
+	    // then
+		PamDataUnit minByTime = dataUnits
+	      .stream()
+	      .min(Comparator.comparing(PamDataUnit::getTimeMilliseconds))
+	      .orElseThrow(NoSuchElementException::new);
+		
+		return minByTime;
+	}
+
 	/**
 	 * Calculate the median value of an array 
 	 * @param numArray - the number array 

src/export/CSVExport/CSVExportManager.java

@@ -35,4 +35,10 @@ public class CSVExportManager implements PamDataUnitExporter{
 		return "CSV Export";
 	}
 
+	@Override
+	public void close() {
+		// TODO Auto-generated method stub
+		
+	}
+
 }

src/export/MLExport/MLDetectionsManager.java

@@ -10,6 +10,7 @@ import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.zip.Deflater;
 
+import PamUtils.PamArrayUtils;
 import PamUtils.PamCalendar;
 import PamguardMVC.PamDataUnit;
 import export.PamDataUnitExporter;
@@ -82,10 +83,7 @@ public class MLDetectionsManager implements PamDataUnitExporter {
 			Struct dataUnitsStruct = dataUnits2MAT(dataUnits);
 
 			// then
-			PamDataUnit minByTime = dataUnits
-		      .stream()
-		      .min(Comparator.comparing(PamDataUnit::getTimeMilliseconds))
-		      .orElseThrow(NoSuchElementException::new);
+			PamDataUnit minByTime = PamArrayUtils.getMinTimeMillis(dataUnits);
 
 			//matlab struct must start with a letter. 
 			Date date = new Date(minByTime.getTimeMilliseconds());
@@ -198,7 +196,7 @@ public class MLDetectionsManager implements PamDataUnitExporter {
 				}
 			}
 
-			if (n>=1) {
+			if (n>0) {
 				list.set(mlDataUnitsExport.get(i).getName(),mlStructure);
 				list.set(mlDataUnitsExport.get(i).getName()+"_sR", Mat5.newScalar(sampleRate)); 
 			}
@@ -257,6 +255,20 @@ public class MLDetectionsManager implements PamDataUnitExporter {
 		}
 	}
 
+	@Override
+	public void close() {
+		//handled in the mian funtion
+		if (sink!=null) {
+			try {
+				sink.close();
+			} catch (IOException e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+		}
+
+	}
+
 
 
 

src/export/PamDataUnitExporter.java

@@ -46,5 +46,10 @@ public interface PamDataUnitExporter {
 	 */
 	public String getName();
 
+	/**
+	 * Clsoe the exporter
+	 */
+	public void close();
+
 
 }

src/export/PamExporterManager.java

@@ -75,14 +75,14 @@ public class PamExporterManager {
 		if (dataUnit==null) {
 			if (force) {
 				System.out.println("Write data 1!!" + dataUnitBuffer.size() ); 
+				//finish off saving any buffered data
 				exportOK = pamExporters.get(exportParams.exportChoice).exportData(currentFile, dataUnitBuffer, true);
 				dataUnitBuffer.clear();
 			}
 			return true;
 		}
 		
-		//if the data unit is null then save everything to the buffer.
-
+		//if file is null or too large create another a file for saving. 
 		if (currentFile == null || isFileSizeMax(currentFile)) {
 			Date date = new Date(dataUnit.getTimeMilliseconds());
 			
@@ -98,11 +98,10 @@ public class PamExporterManager {
 
 		dataUnitBuffer.add(dataUnit);
 
-//		System.out.println("Write data unit " + dataUnitBuffer.size() + " to: "+ currentFile); 
+		System.out.println("Write data unit " + dataUnitBuffer.size() + " to: "+ currentFile); 
 		
 		if (dataUnitBuffer.size()>=BUFFER_SIZE || force) {
 			System.out.println("Write data 2!!" + dataUnitBuffer.size()); 
-
 			exportOK = pamExporters.get(exportParams.exportChoice).exportData(currentFile, dataUnitBuffer, true);
 			dataUnitBuffer.clear();
 		}
@@ -111,6 +110,11 @@ public class PamExporterManager {
 
 	}
 	
+	public void close() {
+		pamExporters.get(exportParams.exportChoice).close();
+		
+	}
+
 	/**
 	 * Check whether the current file is greater than the maximum allowed file size. 
 	 * @param currentFile2 - the current file
@@ -171,4 +175,6 @@ public class PamExporterManager {
 	}
 
 
+
+
 }

src/export/RExport/RClickExport.java

@@ -17,7 +17,7 @@ public class RClickExport extends RRawExport{
 		
 		ClickDetection clickDetection = (ClickDetection) dataUnit; 
   		
-		super.addDetectionSpecificFields(rData, null, index); 
+		super.addDetectionSpecificFields(rData, dataUnit, index); 
 
 		//add some basic click measurements
 		rData.add("triggerMap", new DoubleArrayVector(clickDetection.getTriggerList())); 

src/export/RExport/RExportManager.java

@@ -4,16 +4,22 @@ import java.io.File;
 import java.io.FileOutputStream;
 import java.io.IOException;
 import java.util.ArrayList;
+import java.util.Date;
 import java.util.List;
 import java.util.zip.GZIPOutputStream;
 
 import org.renjin.eval.Context;
 import org.renjin.primitives.io.serialization.RDataWriter;
+import org.renjin.sexp.DoubleArrayVector;
+import org.renjin.sexp.IntArrayVector;
 import org.renjin.sexp.ListVector;
 import org.renjin.sexp.PairList;
+import org.renjin.sexp.PairList.Builder;
 
+import PamUtils.PamArrayUtils;
 import PamguardMVC.PamDataUnit;
 import export.PamDataUnitExporter;
+import export.MLExport.MLDetectionsManager;
 import pamViewFX.fxNodes.pamDialogFX.PamDialogFX;
 
 /**
@@ -28,74 +34,76 @@ public class RExportManager implements PamDataUnitExporter {
 	 * 
 	 * All the possible RDataUnit export classes. 
 	 */
-	ArrayList<RDataUnitExport> mlDataUnitsExport = new ArrayList<RDataUnitExport>();
-
-	private FileOutputStream fos;
-
-	private GZIPOutputStream zos;
+	ArrayList<RDataUnitExport> rDataExport = new ArrayList<RDataUnitExport>();
 
 	private File currentFileName ;
 
-	private RDataWriter writer;
 
-	private Context context; 
+	private Builder allData; 
 
 
 	public RExportManager(){
 		/***Add more options here to export data units****/
-		mlDataUnitsExport.add(new RClickExport()); 
-		mlDataUnitsExport.add(new RWhistleExport()); 
-		mlDataUnitsExport.add(new RRawExport()); //should be last in case raw data holders have specific exporters
+		rDataExport.add(new RClickExport()); 
+		rDataExport.add(new RWhistleExport()); 
+		rDataExport.add(new RRawExport()); //should be last in case raw data holders have specific exporters
 	}
 
 
 	@Override
 	public boolean exportData(File fileName, List<PamDataUnit> dataUnits, boolean append) {
 
-		//convert the data units to 
-		RData dataUnitsR = dataUnits2R(dataUnits);
+		/**
+		 * Note - there is no way to save data units to R files wothout loading the file into memory. 
+		 * So everything is stored in memory until saved. 
+		 */
+
+		// then
+		PamDataUnit minByTime = PamArrayUtils.getMinTimeMillis(dataUnits);
+
+		//matlab struct must start with a letter. 
+		Date date = new Date(minByTime.getTimeMilliseconds());
+		String entryName = "det_" + MLDetectionsManager.dataFormat.format( date);
+
+		//		System.out.println("Save R data! "+ dataUnits.size());
 
 		//		System.out.println("Export R file!!" + dataUnits.size());
 
-		//now write the file
+		//is there an existing writer? Is that writer writing to the correct file?
+		if (allData==null || !fileName.equals(currentFileName)) {
+
+			if (allData!=null) {
+				writeRFile();
+			}
+
+			allData = new PairList.Builder();
+			currentFileName = fileName;
+		}
+
+		//convert the data units to R and save to the PairList builder
+		dataUnits2R(dataUnits, entryName, allData);
+
+		return true;
+
+	}
+
+	private void writeRFile() {
+		Context context = Context.newTopLevelContext();
 		try {
 
-			//is there an existing writer? Is that writer writing to the correct file?
-			if (zos==null || fileName.equals(currentFileName)) {
+			FileOutputStream fos = new FileOutputStream(currentFileName);
+			GZIPOutputStream zos = new GZIPOutputStream(fos);
 			
-				if (zos!=null) {
-					zos.close();
+			RDataWriter writer = new RDataWriter(context, zos);	
+
+			writer.save(allData.build());
 			writer.close();
-				}
-
-				currentFileName = fileName;
-
-				//			System.out.println("MLDATA size: "+ mlData.size());
-				//			System.out.println("---MLArray----");
-				//			for (int i=0; i<mlData.size(); i++) {
-				//				System.out.println(mlData.get(i));
-				//				System.out.println("-------");
-				//			}
-				//			System.out.println("--------------");
-
-
-				context = Context.newTopLevelContext(); 
-
-				fos = new FileOutputStream(fileName);
-				zos = new GZIPOutputStream(fos);
-
-				return true;
-			}
-			
-			writer = new RDataWriter(context, zos);	
-			writer.save(dataUnitsR.rData.build());	
-			
-			return true;
+			zos.close();
 		}
 		catch (IOException e1) {
 			e1.printStackTrace();
-			return false;
 		}
+
 	}
 
 	/**
@@ -113,11 +121,11 @@ public class RExportManager implements PamDataUnitExporter {
 
 	@Override
 	public boolean hasCompatibleUnits(Class dataUnitType) {
-		for (int i=0; i<mlDataUnitsExport.size(); i++){
+		for (int i=0; i<rDataExport.size(); i++){
 			//check whether the same. ;
 			//System.out.println(" dataUnits.get(j).getClass(): " + dataUnits.get(j).getClass());
 			//System.out.println(" mlDataUnitsExport.get(i).getUnitClass(): " + mlDataUnitsExport.get(i).getUnitClass());
-			if (mlDataUnitsExport.get(i).getUnitClass().isAssignableFrom(dataUnitType)) {
+			if (rDataExport.get(i).getUnitClass().isAssignableFrom(dataUnitType)) {
 				//System.out.println("FOUND THE DATA UNIT!");
 				return true; 
 			}
@@ -131,21 +139,36 @@ public class RExportManager implements PamDataUnitExporter {
 	 * @return list of list of R strucutures ready for saving to .RData file. 
 	 */
 	public RData dataUnits2R(List<PamDataUnit> dataUnits){
+		PairList.Builder allData = new PairList.Builder();
+		return dataUnits2R(dataUnits,  null, allData);
+	}
+
+
+	/**
+	 * Sort a list of data units into lists of the same type of units. Convert to a list of structures. 
+	 * @param dataUnits - a list of data units to convert to matlab structures. 
+	 * @param - a name for the structure. 
+	 * @return list of list of R strucutures ready for saving to .RData file. 
+	 */
+	public RData dataUnits2R(List<PamDataUnit> dataUnits, String name, 	PairList.Builder allData) {
 
 		//if there's a mixed bunch of data units then we want separate arrays of structures. So a structure of arrays of structures.
 		//so, need to sort compatible data units.  		
 
-		PairList.Builder allData = new PairList.Builder();
 		ArrayList<String> dataUnitTypes = new ArrayList<String>(); 
 
+		//keep a track of the data units that have been transcribed. This means data units that are multiple types
+		//(e.g. a raw data holder and click) are not added to two different list of structures. 
+		boolean[] alreadyStruct = new boolean[dataUnits.size()];
+
 		//iterate through possible export functions. 
-		for (int i=0; i<mlDataUnitsExport.size(); i++){
+		for (int i=0; i<rDataExport.size(); i++){
 
 			//first need to figure out how many data units there are. 
 			int n=0; 
 			for (int j=0; j<dataUnits.size(); j++){
 				//check whether the same. 
-				if (mlDataUnitsExport.get(i).getUnitClass().isAssignableFrom(dataUnits.get(j).getClass())) {
+				if (rDataExport.get(i).getUnitClass().isAssignableFrom(dataUnits.get(j).getClass())) {
 					n++;
 				}
 			}
@@ -154,22 +177,35 @@ public class RExportManager implements PamDataUnitExporter {
 			if (n==0) continue; //no need to do anything else. There are no data units of this type. 
 
 			ListVector.NamedBuilder dataListArray = new ListVector.NamedBuilder();
-
 			ListVector.NamedBuilder dataList;
+
 			n=0; 
+			double sampleRate = 0.;
 			//allocate the class now. 
 			for (int j=0; j<dataUnits.size(); j++){
 				//check whether the same. 
-				if (mlDataUnitsExport.get(i).getUnitClass().isAssignableFrom(dataUnits.get(j).getClass())) {
-					dataList=mlDataUnitsExport.get(i).detectionToStruct(dataUnits.get(j), n); 
-					dataListArray.add((mlDataUnitsExport.get(i).getName() + "_" + dataUnits.get(j).getUID()), dataList);	
+				if (rDataExport.get(i).getUnitClass().isAssignableFrom(dataUnits.get(j).getClass()) && !alreadyStruct[j]) {
+					dataList=rDataExport.get(i).detectionToStruct(dataUnits.get(j), n); 
+					dataListArray.add((rDataExport.get(i).getName() + "_" + dataUnits.get(j).getUID()), dataList);	
+
+					sampleRate = dataUnits.get(j).getParentDataBlock().getSampleRate(); 
 					n++; 
+					alreadyStruct[j] = true;
 				}
 			}
 
-			if (n>1) {
-				allData.add(mlDataUnitsExport.get(i).getName(), dataListArray.build());
-				dataUnitTypes.add(mlDataUnitsExport.get(i).getName()); 
+			if (n>0) {
+
+				String dataName; 
+				if (name==null) {
+					dataName = rDataExport.get(i).getName();
+				}
+				else dataName = name + "_" + rDataExport.get(i).getName();
+
+				allData.add(dataName, dataListArray.build());
+				allData.add(rDataExport.get(i).getName()+"_sR",  new DoubleArrayVector(sampleRate));
+
+				dataUnitTypes.add(rDataExport.get(i).getName()); 
 			}
 
 		}
@@ -195,7 +231,7 @@ public class RExportManager implements PamDataUnitExporter {
 		public PairList.Builder rData; 
 
 		/**
-		 * List of the names of the types of data units whihc were saved. 
+		 * List of the names of the types of data units which were saved. 
 		 */
 		public ArrayList<String> dataUnitTypes; 
 	}
@@ -218,5 +254,13 @@ public class RExportManager implements PamDataUnitExporter {
 	}
 
 
+	@Override
+	public void close() {
+		if (allData!=null) {
+			writeRFile();
+		}
+	}
+
+
 
 }

src/export/RExport/RRawExport.java

@@ -73,7 +73,7 @@ public class RRawExport extends RDataUnitExport<PamDataUnit> {
 
 	@Override
 	public String getName() {
-		return "pam_data_units";
+		return "raw_data_units";
 	}
 
 }

src/export/swing/ExportTask.java

@@ -60,8 +60,10 @@ public class ExportTask extends OfflineTask<PamDataUnit<?,?>>{
 	@Override
 	public void loadedDataComplete() {
 		System.out.println("EXPORTER: loaded data complete"); 
+		
 		//force the exporter so save any renaming data units in the buffer
 		exporter.exportDataUnit(null,  true);
+		exporter.close();
 		exporter.setCurrentFile(null); 
 		
 	}

src/export/wavExport/WavFileExportManager.java

@@ -459,6 +459,14 @@ public class WavFileExportManager implements PamDataUnitExporter  {
 
 
 
+	@Override
+	public void close() {
+		// TODO Auto-generated method stub
+		
+	}
+
+
+
 
 
 	//	hello(){

src/pamViewFX/fxPlotPanes/PlotPane.java

@@ -264,7 +264,7 @@ public class PlotPane extends PamBorderPane {
 		rightPane.prefWidthProperty().bind(yAxisRightPane.widthProperty());
 		rightPane.minWidthProperty().bind(yAxisRightPane.widthProperty());
 
-		horzHolder.getChildren().addAll(leftPane, axisPane);
+		horzHolder.getChildren().addAll(leftPane, axisPane, rightPane);
 		//axisPane.toFront(); this changes the order of children in a PamHBox. 
 		HBox.setHgrow(axisPane, Priority.ALWAYS);