Sud bug fix and updates to CPOD module. (#162)

* Update exporter_help.md

Updated help for exporter

* Update .gitignore

* Bug fixes for deep learning classifier.

Updated the symbol options to make sure opacity is passed through the symbol chooser.

* Updates to the data map FX display

* Updates to DelphinID and Data ModelFX

Used a new writable image for much faster drawing in FX

* bb

* Delete C:\Users\Jamie Macaulay\MATLAB Drive\MATLAB\PAMGUARD\deep_learning\delphinID\whistle_image_example_java.mat

* Updates to DelphinID

* Updates to ReadMe and JavaFX GUI

* Update readme.md

Updates to ReadMe

* Updates to CPOD module and also delphinID classifier

* Fix colour scaling in FX data map

* Create cpod_help.md

* Update cpod_help.md

* Update cpod_help.md

* Updates to CPOD module

Added ability to export CPOD clicks
Added some extra features to data selector
Swing GUI for data selector (in progress)

* Updates to data map FX GUI

* Updates to CPOD module

* Added help files resources for CPOD

* Add CPOD resources for help

* Add CPOD resources for help

* Add CPOD resources for help

* Add CPOD resources for help

* Add CPOD resources for help

* Update cpod_help.md

* Update cpod_help.md

* Bug fixes for deep learning classifier.

Updated the symbol options to make sure opacity is passed through the symbol chooser.

* Updates to the data map FX display

* Updates to DelphinID and Data ModelFX

Used a new writable image for much faster drawing in FX

* Updates to ReadMe and JavaFX GUI

* bb

* Delete C:\Users\Jamie Macaulay\MATLAB Drive\MATLAB\PAMGUARD\deep_learning\delphinID\whistle_image_example_java.mat

* Updates to DelphinID

* Update readme.md

Updates to ReadMe

* Fix colour scaling in FX data map

* Updates to CPOD module and also delphinID classifier

* Updates to CPOD module

Added ability to export CPOD clicks
Added some extra features to data selector
Swing GUI for data selector (in progress)

* Create cpod_help.md

* Update cpod_help.md

* Update cpod_help.md

* Updates to data map FX GUI

* Updates to CPOD module

* Added help files resources for CPOD

* Add CPOD resources for help

* Add CPOD resources for help

* Add CPOD resources for help

* Add CPOD resources for help

* Add CPOD resources for help

* Update cpod_help.md

* Update cpod_help.md

* Bug fix to BT display

The BT display was not selecting clicks properly.

* Import bug fix

* Updates to data map FX GUI

* Updates to datamap FX

* Got date axis working properly.

* Added some arrows to the scroll bar for data map FX pane.

* Working on getting datagrams in FX saving

* Updates to DataMapFX and exporting annotations

* Update MLAnnotationsManager.java

* Working on adding annotations to exporter

* Bug fix for processing files and annotations for exporter

Rebase with main
Bug fix for processing files - stops only the last file processing when "Start normally" selected on data processing
R and MATLAB export of data annotation added.

* Add a ttoltip to the exporter

* Minor text change

.classpath

@@ -6,7 +6,7 @@
 			<attribute name="maven.pomderived" value="true"/>
 		</attributes>
 	</classpathentry>
-	<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER">
+	<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/Amazon Coretto 21">
 		<attributes>
 			<attribute name="module" value="true"/>
 			<attribute name="maven.pomderived" value="true"/>

.settings/org.eclipse.jdt.core.prefs

@@ -11,9 +11,9 @@ org.eclipse.jdt.core.codeComplete.staticFinalFieldPrefixes=
 org.eclipse.jdt.core.codeComplete.staticFinalFieldSuffixes=
 org.eclipse.jdt.core.compiler.codegen.inlineJsrBytecode=enabled
 org.eclipse.jdt.core.compiler.codegen.methodParameters=do not generate
-org.eclipse.jdt.core.compiler.codegen.targetPlatform=19
+org.eclipse.jdt.core.compiler.codegen.targetPlatform=18
 org.eclipse.jdt.core.compiler.codegen.unusedLocal=preserve
-org.eclipse.jdt.core.compiler.compliance=19
+org.eclipse.jdt.core.compiler.compliance=18
 org.eclipse.jdt.core.compiler.debug.lineNumber=generate
 org.eclipse.jdt.core.compiler.debug.localVariable=generate
 org.eclipse.jdt.core.compiler.debug.sourceFile=generate
@@ -22,5 +22,5 @@ org.eclipse.jdt.core.compiler.problem.enablePreviewFeatures=disabled
 org.eclipse.jdt.core.compiler.problem.enumIdentifier=error
 org.eclipse.jdt.core.compiler.problem.forbiddenReference=warning
 org.eclipse.jdt.core.compiler.problem.reportPreviewFeatures=warning
-org.eclipse.jdt.core.compiler.release=disabled
+org.eclipse.jdt.core.compiler.release=enabled
-org.eclipse.jdt.core.compiler.source=19
+org.eclipse.jdt.core.compiler.source=18

readme.md

@@ -3,14 +3,14 @@
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.13378951.svg)](https://doi.org/10.5281/zenodo.13378951)
 
 ## Introduction
-PAMGuard is a bioacoustics analysis program designed for use in real time research contexts and for the processing of large datasets. PAMGuard provides users access to a suite of state-of-the-art auotmated analysis algorithms alongside displays for visualisation data and a comprehensive data management systems. 
+PAMGuard is a bioacoustics analysis program designed for use in real time research contexts and for the processing of large datasets. PAMGuard provides users access to a suite of state-of-the-art automated analysis algorithms alongside displays for visualisation data and a comprehensive data management systems. 
 
 ## Why do we need PAMGuard?
-PAMGuard fufills two main requirements within marine bioacoustics
+PAMGuard fulfils two main requirements within marine bioacoustics
 
-1) **Real time operation**: Almost all PAMGuard features and modules work in real time - this allows scientists and industry to detect, classify and localise the position of animals in real time on a standard consumer laptop, enabling mitigation and research surveys without expensive bespoke software solutions and the transparncy of open source software.  
+1) **Real time operation**: Almost all PAMGuard features and modules work in real time - this allows scientists and industry to detect, classify and localise the position of animals in real time on a standard consumer laptop, enabling mitigation and research surveys without expensive bespoke software solutions and the transparency of open source software.  
 
-2) **Processing and visuslisation of large datasets**: Off-the-shelf autonomous PAM devices, large scale surveys involving drifters, towed arrays, glidders, bottom mounted devices and real time continuous monitoring system all generate huge volumes of data whcih requires automated analysis approaches. PAMGuard allows the processing of very large passive acoustic datasets using automated algorothms and crucially provides comprehensive visualisation tools for a manual analyst to check the results.
+2) **Processing and visualisation of large datasets**: Off-the-shelf autonomous PAM devices, large scale surveys involving drifters, towed arrays, gliders, bottom mounted devices and real time continuous monitoring system all generate huge volumes of data which requires automated analysis approaches. PAMGuard allows the processing of very large passive acoustic datasets using automated algorithms and crucially provides comprehensive visualisation tools for a manual analyst to check the results.
 
 ## Installation
 PAMGuard is available on Windows and can be downloaded from the [PAMGuard website](www.pamguard.org). Note that we are considering MacOS and Linux installers but they are not available at this time. 
@@ -18,54 +18,63 @@ PAMGuard is available on Windows and can be downloaded from the [PAMGuard websit
 ## Quick start tutorial
 PAMGuard is a modular program with two modes; real-time and viewer. Typically a user will start with real-time model, either in the field collecting data or post processing sound files from a recorder. Once data are processed, users move on to viewer mode where data can be explored and further processed. 
 
-Upon opening PAMGuard for the first time you are greeted with a blank screen. You must add a series of modules to create the desired acosutic workflow - this is referred to as PAMGuard *data model*. For example if processing sound files then first add the Sound Acquisition module **_File->Add Modules->Sound Processing->Sound Acquisition_**. Then add the desired detection algorothms e.g.  **_File->Add Modules->Detector->Click Detectors_**. Some modules (such as the click detector) have their own bespoke displays, others are added to more generalised displays. For example, the whistle and moan detector module shows detections on a spectrgram display, time base display, map etc.. First add a new tab using  **_File->Add Modules->Displays->User Display**. Click on the user display tab and then from the top menu select **_User display-> New Spectrogram_**. Right click on the added spectrgram and select whistle and moan contours to show whistle detections overlaid on the raw spectrgram. 
+Upon opening PAMGuard for the first time you are greeted with a blank screen. You must add a series of modules to create the desired acoustic workflow - this is referred to as PAMGuard *data model*. For example if processing sound files then first add the Sound Acquisition module **_File->Add Modules->Sound Processing->Sound Acquisition_**. Then add the desired detection algorithms e.g.  **_File->Add Modules->Detector->Click Detectors_**. Some modules (such as the click detector) have their own bespoke displays, others are added to more generalised displays. For example, the whistle and moan detector module shows detections on a spectrogram display, time base display, map etc. First add a new tab using  **_File->Add Modules->Displays->User Display_**. Click on the user display tab and then from the top menu select **_User display-> New Spectrogram_**. Right click on the added spectrogram and select whistle and moan contours to show whistle detections overlaid on the raw spectrogram. 
 
 Make sure to add the database and binary file storage modules **_File->Add Modules->Utilities->..._**) to save data then press the run button (red button) and data will process. PAMGuard can handle huge datasets so running might take hours or even days. Progress is shown on the bottom of the screen. 
 
-Once the data has run, open PAMGuard viewer mode. Select the database you used to process the data along and thebianry file storage path and PAMGuard will open, showing an overview of the dataset in a new _Data map_ tab. Right click anywhere on the data map and select "Center data here" - PAMGuard will load the data for the selected period which can be explored using whichever displays have been added to the data model. 
+Once the data has run, open PAMGuard viewer mode. Select the database you used to process the data along and the binary file storage path and PAMGuard will open, showing an overview of the dataset in a new _Data map_ tab. Right click anywhere on the data map and select "Center data here" - PAMGuard will load the data for the selected period which can be explored using whichever displays have been added to the data model. 
 
 ## Features
 ### Hardware integration
 PAMGuard connects with hardware such as various GPS and AIS systems and a multitude of different sound cards (e.g. [National Instruments](www.ni.com) devices, [SAIL DAQ cards](www.smruconsulting.com/contact-us), almost all ASIO sound cards and standard computer sound cards) for real time data collection and processing. PAMGuard also works with some very bespoke hardware such as [DIFAR Sonobuoys]();
 
 ### Real time operation
-PAMGuard takes advanatge of multi-core processors to run multiple signal processing automatic analysis algorithms in real time to detect whales, dolphins, bats etc. Data are shown in different displayes, including interactive spectrograms and maps. You might be using PAMGuard for simply viewing a spectrgram and making recordings or running deep learning algorithms for multiple species and loclaising the results to view locations on a map. Whatever acosutic workflow a user creates, PAMGuard can run it in real time. 
+PAMGuard takes advantage of multi-core processors to run multiple signal processing automatic analysis algorithms in real time to detect whales, dolphins, bats etc. Data are shown in different displays, including interactive spectrograms and maps. You might be using PAMGuard for simply viewing a spectrogram and making recordings or running deep learning algorithms for multiple species and localising the results to view locations on a map. Whatever acoustic workflow a user creates, PAMGuard can run it in real time. 
 
 ### Support for compressed audio
-PAMGuard supports processing audio data from standard files (e.g. wav, aif) and also compressed files (e.g. .flac and .sud). Notew that sud files are created on SoundTraps widely used marine recorders and can be read by PAMGuard without decompressing - PAMGuard will automtically import click detections if present in sud files. PAMGuard also supports importing detection data from CPODs and FPODs. 
+PAMGuard supports processing audio data from standard files (e.g. wav, aif) and also compressed files (e.g. .flac and .sud). Note that sud files are created on SoundTraps widely used marine recorders and can be read by PAMGuard without decompressing - PAMGuard will automatically import click detections if present in sud files. PAMGuard also supports importing detection data from CPODs and FPODs. 
 
 ### Comprehensive data management system
-PAMGuard is designed to collect/process data from large acosutic datasets. PAMGuard stores data in an SQLite databases and "Binary" files. The database stores important metadata such as when data has been processed and some low volume data streams such as GPS. Binary files are not human readbale but efficient to access - PAMGuard stores detection data (e.g. clicks, whistles, noise, etc) in these files. this allows PAMGuard to rapidly access data from large datasets. Data from binary files can be viewed in PAMGuard viewer mode or can be exported to MATLAB using the PAMGuard-MATLAB library or the exported to R using the R PAMBinaries package. 
+PAMGuard is designed to collect/process data from large acoustic datasets. PAMGuard stores data in an SQLite databases and "Binary" files. The database stores important metadata such as when data has been processed and some low volume data streams such as GPS. Binary files are not human readable but efficient to access - PAMGuard stores detection data (e.g. clicks, whistles, noise, etc) in these files. This allows PAMGuard to rapidly access data from large datasets. Data from binary files can be viewed in PAMGuard viewer mode or can be exported to MATLAB using the [PAMGuard-MATLAB](https://github.com/PAMGuard/PAMGuardMatlab) library or the exported to R using the R [PAMBinaries](https://github.com/TaikiSan21/PamBinaries) package. 
 
 ### Access to detection and classification algorithms
-PAMGuard allows users to inegrate automated detection and classification algorithms directly into their acosutic workflow. There are a multitude of differwent algorothms to choose from, including a basic click detector, whislte and moan detector, GPL detector, click train detectors and many others. The idea behind PAMGuard is allow researchers to access open source state-of-the-art algorithms devleoped within the scientific community - if you want to contribute and get your algorithm into PAMGuard get in touch. 
+PAMGuard allows users to integrate automated detection and classification algorithms directly into their acoustic workflow. There are a multitude of different algorithms to choose from, including a basic click detector, whistle and moan detector, GPL detector, click train detectors and many others. The idea behind PAMGuard is allow researchers to access open source state-of-the-art algorithms developed within the scientific community - if you want to contribute and get your algorithm into PAMGuard get in touch. 
+
+![PAMGuard click detector](./src/Resources/screenshots/Click_detector_screenshot.png)
+
+_The PAMGuard click detector can be used to detect transient sounds such as echolocation clicks. It also automatically localises click bearings for closely spaced hydrophones._
 
 ### Localisation
-PAMGuard has a mutltude of different options for acoustic loclaisation. There's a comprehesnive beam forming module for beam forming arrays, a large aperture localiser for 3D loclaisation and target motion analysis for towed hydrophone arrays. 
+PAMGuard has a multitude of different options for acoustic localisation. There's a comprehensive beam forming module for beam forming arrays, a large aperture localiser for 3D localisation and target motion analysis for towed hydrophone arrays. 
 
 ### Soundscape analysis
 PAMGuard has a noise band (which supports third octave noise bands) and long term spectral average module for soundscape analysis. 
 
 ### GIS
-Almsot all detection data can be visualised on a map. PAMGaurd also supports plotting GPS and AIS data. 
+Almost all detection data can be visualised on a map. PAMGuard also supports plotting GPS and AIS data. 
 
 ### Suite of data visualisation tools
-An important aspect of PAMGuard is the ability for users to explore porcessed data. PAMGuard allows users to visualise data at multiple different times scales, from inspecting individual waveforms microseconds long to datagrams showing detector output or soundscape metrics over days, weeks or even years.
+An important aspect of PAMGuard is the ability for users to explore processed data. PAMGuard allows users to visualise data at multiple different times scales, from inspecting individual waveforms microseconds long to datagrams showing detector output or soundscape metrics over days, weeks or even years.
+
+![Example of a data map](./src/Resources/screenshots/Datamap_screenshot.png)
+
+_The PAMGuard data map provides an overview of the entire processed dataset over days, weeks and even years._
+
 
 ### Advanced manual annotation
 The displays within PAMGuard support a variety of manual annotation tools. Users can add data to annotations in multiple ways, from simple text annotations to complex forms created by users. 
 
 ### Deep learning integration
-PAMGuard allows users to run their own deep learning models using the deep learning module. AI can therfore be integrated into PAMGuard workflows, allowing for more efficient analysis of data. 
+PAMGuard allows users to run their own deep learning models using the deep learning module. AI can therefore be integrated into PAMGuard workflows, allowing for more efficient analysis of data. 
 
 ### Metadata standard and Tethys compatibility
-PAMGuard Integrates with Tethys database. Users can export processed PAMGuard data to a Tethys database seamlessly; this ifeature is great for large scale projects or organisatiosn with long term datasets. 
+PAMGuard Integrates with Tethys database. Users can export processed PAMGuard data to a Tethys database seamlessly; this feature is great for large scale projects or organisations with long term datasets. 
 
 ## Feature roadmap
-There's lots of features we would like to add to PAMGuard. If you want to add a feature you can either code it up yourself in Java and submit a pull request or get in touch with us to discuss how to it might be integrated. Some smaller features might be in our roadmap anyway but larger features usually require funding. Some features we are thinking about (but do not necassarily have time for yet) are;
+There's lots of features we would like to add to PAMGuard. If you want to add a feature you can either code it up yourself in Java and submit a pull request or get in touch with us to discuss how to it might be integrated. Some smaller features might be in our roadmap anyway but larger features usually require funding. Some features we are thinking about (but do not necessarily have time for yet) are;
 
 * Support for decidecade noise bands (base 10 filter bank) in noise band monitor to meet Euopean standards
-* Capabaility to export data directly from PAMGaurd e.g. as MAT files (in progress).
+* Capability to export data directly from PAMGaurd e.g. as MAT files (in progress).
 * Automated test suite to make releases more stable. Note that unit and integration tests are also being slowly incorporated.
 * Support for ARM based computers (in progress). 
 * A graphical user interface and Python library for training PAMGuard compatible deep learning models. 

src/Acquisition/FolderInputSystem.java

@@ -981,6 +981,7 @@ public class FolderInputSystem extends FileInputSystem implements PamSettings, D
 
 	@Override
 	public boolean setAnalysisStartTime(long startTime) {
+				
 		/**
 		 * Called from the reprocess manager just before PAMGuard starts with a time
 		 * we want to process from. This should be equal to the start of one of the files

src/PamController/fileprocessing/ReprocessManager.java

@@ -184,10 +184,14 @@ public class ReprocessManager {
 		}
 		InputStoreInfo inputInfo = null;
 		boolean OK = true;
+		long procStartTime = deleteFrom;
+		if (choice == ReprocessStoreChoice.STARTNORMAL) {
+			procStartTime = 0;
+		}
 		for (PamControlledUnit aPCU : inputStores) {
 			DataInputStore inputStore = (DataInputStore) aPCU;
-			OK &= inputStore.setAnalysisStartTime(deleteFrom);
+			OK &= inputStore.setAnalysisStartTime(procStartTime);
-//			System.out.println("Input store info: " + inputInfo);
+			System.out.println("Input store info: " + inputInfo);
 		}
 		return OK;
 	}

src/PamModel/PamModel.java

@@ -524,7 +524,7 @@ final public class PamModel implements PamSettings {
 		mi = PamModuleInfo.registerControlledUnit("cpod.CPODControl2", "CPOD Detector Import");
 		mi.setModulesMenuGroup(sensorsGroup);
 		mi.setToolTipText("Imports CPOD data");
-		mi.setHidden(!SMRUEnable.isEnable());
+		//mi.setHidden(SMRUEnable.isEnable() == false);
 		mi.addGUICompatabilityFlag(PamGUIManager.FX); //has FX enabled GUI.
 
 		/*

src/PamUtils/PamArrayUtils.java

@@ -196,6 +196,41 @@ public class PamArrayUtils {
 		return minByTime;
 	}
 	
+	
+	   /**
+     * Finds the closest number to the given target in a sorted array.
+     *
+     * @param arr the sorted array of doubles
+     * @param target the target number
+     * @return the index of the closest number in the array
+     */
+    public static int findClosest(double[] arr, double target) {        
+    	
+    	   if(target < arr[0]) {
+               return 0;
+           }
+           if(target > arr[arr.length-1]) {
+               return arr.length-1;
+           }
+
+           int lo = 0;
+           int hi = arr.length - 1;
+
+           while (lo <= hi) {
+               int mid = (hi + lo) / 2;
+
+               if (target < arr[mid]) {
+                   hi = mid - 1;
+               } else if (target > arr[mid]) {
+                   lo = mid + 1;
+               } else {
+                   return mid;
+               }
+           }
+           // lo == hi + 1
+           return (arr[lo] - target) < (target - arr[hi]) ? lo : hi;
+    }
+
 	/**
 	 * Calculate the median value of an array 
 	 * @param numArray - the number array 

src/cpod/CPODClassification.java

@@ -16,7 +16,7 @@ public class CPODClassification {
 		public enum CPODSpeciesType{NBHF, DOLPHIN, SONAR, UNKNOWN}
 		
 		/**
-		 * A unique ID for the click trian within the file
+		 * A unique ID for the click train within the file
 		 */
 		public int clicktrainID = 0;
 		

src/cpod/CPODClick.java

@@ -55,6 +55,8 @@ public class CPODClick extends PamDataUnit<PamDataUnit,SuperDetection> implement
 		short endF = shortData[6];
 		short spl = shortData[7];
 		short slope = shortData[8];
+		
+		
 		CPODClick cpodClick = new CPODClick(baseData.getTimeMilliseconds(), baseData.getStartSample(), 
 				nCyc, bw, kHz, endF, spl, slope, shortData);
 //		cpodClick.setDurationInMilliseconds(baseData.getMillisecondDuration());
@@ -115,7 +117,7 @@ public class CPODClick extends PamDataUnit<PamDataUnit,SuperDetection> implement
 				(short) FPODReader.IPItoKhz(fpodData.IPIatMax),  (short) FPODReader.IPItoKhz(fpodData.EndIPI), 
 				(short) fpodData.MaxPkExtnd, (short) 0, null);
 		
-		//durartion is measured more accurately in FPOD data
+		//DURATION is measured more accurately in FPOD data
 		cpodClick.setDurationInMilliseconds((fpodData.duration*5.)/1000.);
 		cpodClick.setSampleDuration((long) ((fpodData.duration*5./1e6)*CPODClickDataBlock.CPOD_SR));
 		
@@ -223,6 +225,8 @@ public class CPODClick extends PamDataUnit<PamDataUnit,SuperDetection> implement
 	}
 
 	/**
+	 * 
+	 * Get the bandwidth in kHz
 	 * @return the bw
 	 */
 	public short getBw() {
@@ -334,7 +338,7 @@ public class CPODClick extends PamDataUnit<PamDataUnit,SuperDetection> implement
 		str += String.format("End Freq: %dkHz<p>", getEndF());
 		str += String.format("Slope: %d<p>", getSlope());
 		str += String.format("SPL: %d", getSpl());
-		if (rawData.length == 40) {
+		if (rawData != null && rawData.length == 40) {
 			str += String.format("<p>QClass %d, SpClass %d", CPODUtils.getBits(rawData[19], (short) 0x3), 
 					CPODUtils.getBits(rawData[19], (short) 0b11100));
 			str += String.format("<p>Train %d, %d click", rawData[20], rawData[23]);

src/cpod/CPODClickTrainDataBlock.java

@@ -6,7 +6,6 @@ import PamguardMVC.PamProcess;
 import PamguardMVC.dataOffline.OfflineDataLoadInfo;
 import PamguardMVC.debug.Debug;
 import PamguardMVC.superdet.SuperDetDataBlock;
-import clickTrainDetector.CTDataUnit;
 import pamScrollSystem.ViewLoadObserver;
 
 /**
@@ -32,7 +31,7 @@ public class CPODClickTrainDataBlock extends SuperDetDataBlock<CPODClickTrainDat
 
 
 	public CPODClickTrainDataBlock(CPODControl2 cpodControl, PamProcess parentProcess, String name, int channelMap) {
-		super(CTDataUnit.class, name, parentProcess, channelMap, SuperDetDataBlock.ViewerLoadPolicy.LOAD_OVERLAPTIME);
+		super(CPODClickTrainDataUnit.class, name, parentProcess, channelMap, SuperDetDataBlock.ViewerLoadPolicy.LOAD_OVERLAPTIME);
 		this.cpodControl = cpodControl;
 	}
 	

src/cpod/CPODClickTrainDataUnit.java

@@ -9,7 +9,6 @@ import detectiongrouplocaliser.DetectionGroupDataUnit;
 
 /**
  * Base class for a click train data unit. 
- * Note this must implrement PamDetection to work with the clip generator. 
  * 
  * @author Jamie Macaulay 
  *
@@ -42,8 +41,6 @@ public class CPODClickTrainDataUnit extends DetectionGroupDataUnit implements Pa
 	 * @return
 	 */
 	public String getStringInfo() {
-		
-		
 		return String.format("Species: %s Confidence %d is echo? %b", getSpecies(), getConfidence(), isEcho());
 	}
 	

src/cpod/CPODSymbolManager.java

@@ -18,7 +18,7 @@ public class CPODSymbolManager extends StandardSymbolManager {
 	private PamControlledUnit cpodControl;
 	
 	
-	private static SymbolData defaultSymbol = new SymbolData(PamSymbolType.SYMBOL_CIRCLE, 10, 10, true, Color.BLACK, Color.BLACK);
+	public static SymbolData defaultSymbol = new SymbolData(PamSymbolType.SYMBOL_CIRCLE, 10, 10, true, Color.BLACK, Color.BLACK);
 	
 //	/**
 //	 * Flag to colour clicks by their frequency. It has to be one higher than the other options. 

src/cpod/cpod_help.md

@@ -0,0 +1,69 @@
+# CPOD module help
+
+## Introduction  
+CPODs and FPODs are click data loggers widely used in research and industry developed by [Chelonia Ltd](www.chelonia.co.uk). PAMGuard's CPOD module allows users to import CPOD _and_ FPOD data into PAMGuard so it can be viewed using PAMGuard's visualisation tools. A common use case for this module is to display both CPOD/FPOD data and a lower frequency recording device together - for example a typical PAM setup is to use a CPOD to record high frequency echolocation clicks then a lower frequency recorder (e.g. 96 kHz sample rate) to record dolphin whistles, noise, fish sounds etc.
+
+## Quick overview of CPOD/FPOD data
+CPODs and FPODs do not record raw audio - they run a simple on board click detector and then save some basic metrics on each detected click e.g. peak frequency, bandwidth, end frequency etc. FPODs, the successor of CPODs, also record a little extra waveform data on a small subset of detected clicks. Once a CPOD or FPOD has been recovered, it is processed using CPOD.exe or FPOD.exe software respectively which runs a click train classifier. The click train classifier extracts sequences of successive clicks that are likely from the same source, e.g a dolphin, porpoise or echo sounder. It then assigns a likely species to a click train or classes it as unknown. The outputs from the click train detector are saved to a CP3 file which is essentially a file that contains only clicks that have been assigned to click trains. 
+
+## Adding the CPOD module
+To add the CPOD module go to **_File->Add Modules->Sensors->CPOD_**. The module requires the binary storage module in PAMGuard **_File-> Add modules->Utilities->Binary File_**. Once both the CPOD and Binary file storage modules have been added open the CPOD settings using **_Settings->CPOD importer_**. 
+
+## Importing CPOD/FPOD data
+The module has three possible modes of importing data 
+- Import raw detection data i.e. CP1 or FP1 data. PAMGuard will display the CPOD detection but no click train ID data is available.
+- Import just the click trains data i.e. CP3 or FP3 files - PAMGuard will only import clicks which are part of click trains
+- Import both both raw and click train data (recommended). PAMGuard imports the raw detections then uses the click train data to assign detections to click trains.
+
+Users can use the file button to select a single file (e.g. an FP3 file) or the folder button to select a folder of CPOD/FPOD files. If the folder button is used and there are both CP1/FP1 (detections) and CP3/FP3 (click trains) files then PAMGuard will automatically load all files and assign detections to click trains. Once files have imported select **_Import_** and the data will be imported into PAMGuard - note this can take some time, especially if importing CP1/FP1 files.
+
+<p align="center">
+  <img width="750" height="400" src = "resources/cpod_settings_description.png">
+</p>
+
+_The CPOD module allow users to import CPOD/FPOD data by selecting either individual files or a folder of files_
+
+## Visualising CPOD/FPOD data
+A broad overview of CPOD data is shown in PAMGuard's data map which shows a datagram similar to the click detector. The datagram shows the frequency density of CPOD/FPOD clicks constructed from the peak frequency parameter for each click The datagram can be useful for quickly navigating to sections of data that may contain porpoises and/or dolphins.
+
+Individual CPOD detections can be viewed in the Time base display in PAMGuard. Add the CPOD module to a plot and then data can be viewed with time using a selection y-axis options such as Amplitude, Amplitude (stem), Frequency, bandwidth etc. 
+
+<p align="center">
+  <img width="750" height="380" src = "resources/cpod_time_display_stem.png">
+</p>
+<p align="center">
+  <img width="750" height="380" src = "resources/cpod_time_display_amplitude2.png">
+</p>
+
+_FPOD data visualised in the time base display as a stem plots like FPOD.exe (top) and as a scatter plot of click amplitudes (bottom)_
+
+## Data selector
+The CPOD module has a custom data selector which provides a unified interface for users to select subsets of CPOD/FPOD data. The data selector has sliders which set the range of various parameters, such as peak frequency and also allows the selection of CPOD/FPOD which belong to a click and those gave been classified to a particular species. Users can use the data selector to, for example, only export clicks classified as dolphin or perhaps plot only clicks with a peak frequency between 100 and 150 kHz on the time display.
+
+CPOD data selector             |  CPOD data selector in time display
+:-------------------------:|:-------------------------:
+![](resources/cpod_data_selector_swing.png)  |  ![](resources/cpod_data_selector_fx.png)
+
+_Screenshots of the data selector user interface used throughtout PAMGuard. Although the style can change, the data selector is the same throughout PAMGuard_
+
+## Exporting CPOD/FPOD data
+CPOD and FPOD data can be exported to .RData and .mat using PAMGuard's exporter. 
+
+<p align="center">
+  <img width="450" height="600" src = "resources/cpod_exporter_dialog_swing.png">
+</p>
+
+_PAMGuard's exporter can export CPOD/FPOD clicks to .mat or .RData files. These can be openend in MATLAB/Python and R respectively_
+
+The fields saved by the exporter are the same as a standard PAMGuard detection [(see exporter help)](./src/export/exporter_help.md). The additional fields for CPODs and FPODs are 
+
+ - _bandwidth_: the bandwidth of the click in Hz
+ - _numcycles_: the number of cycles fo the click in Hz
+ - _peakfreq_: the peak frequency of the click in Hz
+ - _endfreq_: the end frequency of the click in Hz
+ - _SPL_: the CPOD measure of sound pressure level which is an integer between 0 and 255. 
+ - _slope_: the slope which is a paramter measured by the CPOD and FPOD.
+ - _wave_: this will be empty for most clicks but some clicks from FPODs will have a waveform. Note that this is reconstructed from zero corssings and is NOT a clip from the raw sound data.
+ - _species_: if the CPOD is part of a click train then species will be 0 for UNKNOWN, 1 for NBHF, 2 for DOLPHIN and 3 for SONAR. -1 indicates a click is not part of a click train. 
+ - _clicktrainID_: indicates the click train that the click belongs to. This can be cross referenced with UID column in the PAMGuard database which stores click trains and/or used to as an identifier to associate different clicks together into trains. 
+

src/cpod/dataPlotFX/CPODPlotInfoFX.java

@@ -15,6 +15,7 @@ import PamguardMVC.PamDataUnit;
 import PamguardMVC.dataSelector.DataSelector;
 import cpod.CPODClick;
 import cpod.CPODClickDataBlock;
+import cpod.CPODSymbolManager;
 import dataPlotsFX.TDManagedSymbolChooserFX;
 import dataPlotsFX.TDSymbolChooserFX;
 import dataPlotsFX.data.TDDataInfoFX;
@@ -381,17 +382,37 @@ public class CPODPlotInfoFX extends GenericDataPlotInfo {
 		public PamSymbolFX getPamSymbol(PamDataUnit dataUnit, int type) {
 			PamSymbolFX symbol =  super.getPamSymbol(dataUnit, type);
 			
+			//FIXME - don't know why this needs reset for every symbol...
+			symbol.setWidth(CPODSymbolManager.defaultSymbol.width);
+			symbol.setHeight(CPODSymbolManager.defaultSymbol.height);
+			
+			
+			//if there is a waveform then highlight as a larger symbol
 			if (((CPODClick) dataUnit).getWaveData()!=null && type!=TDSymbolChooserFX.HIGHLIGHT_SYMBOL) {
+				symbol = symbol.clone();
 				symbol.setLineColor(symbol.getFillColor().darker());
 				symbol.setHeight(symbol.getHeight()*1.5);
 				symbol.setWidth(symbol.getWidth()*1.5);
+
 				return symbol;
 			}
 			
+			if (type==TDSymbolChooserFX.HIGHLIGHT_SYMBOL) {
+				symbol = symbol.clone();
+				symbol.setHeight(CPODSymbolManager.defaultSymbol.width+4);;
+				symbol.setWidth(CPODSymbolManager.defaultSymbol.height+4);
+			}
+						
 			return symbol;
 
 		}
 		
 	}
 	
+	@Override
+	protected void setDefaultOpacity(ParameterType dataType) {
+
+	}
+
+
 }

src/cpod/dataSelector/CPODDatSelectorParams.java

@@ -4,6 +4,7 @@ package cpod.dataSelector;
 import java.util.ArrayList;
 
 import PamguardMVC.dataSelector.DataSelectParams;
+import cpod.CPODClassification.CPODSpeciesType;
 
 /**
  * Parameters for the CPOD data data selectors. Thus allows users to filter
@@ -22,6 +23,16 @@ public class CPODDatSelectorParams extends DataSelectParams {
 	//create the default list of params; 
 	public ArrayList<StandardCPODFilterParams> cpodDataFilterParams;
 
+	/**
+	 * Select only click trains
+	 */
+	public boolean selectClickTrain = false;
+	
+	/**
+	 * The species to select. Null indicates all species. 
+	 */
+	public  CPODSpeciesType speciesID = null;
+	
 	public CPODDatSelectorParams(){
 		cpodDataFilterParams = new 	ArrayList<StandardCPODFilterParams>(); 
 		

src/cpod/dataSelector/CPODDataSelector.java

@@ -2,15 +2,14 @@ package cpod.dataSelector;
 
 import PamController.PamControlledUnit;
 import PamView.dialog.PamDialogPanel;
-import PamguardMVC.PamDataBlock;
 import PamguardMVC.PamDataUnit;
 import PamguardMVC.dataSelector.DataSelectParams;
 import PamguardMVC.dataSelector.DataSelector;
-import clickDetector.alarm.ClickAlarmParameters;
 import cpod.CPODClick;
 import cpod.CPODClickDataBlock;
-import cpod.CPODControl;
+import cpod.CPODClickTrainDataUnit;
 import cpod.fx.CPODDataSelectorPane;
+import cpod.fx.CPODDataSelectorPanel;
 import pamViewFX.fxSettingsPanes.DynamicSettingsPane;
 
 /**
@@ -35,7 +34,9 @@ public class CPODDataSelector extends DataSelector {
 	/**
 	 * The cpod data selector pane. 
 	 */
-	private CPODDataSelectorPane cPODDataSelectorPane;
+	private CPODDataSelectorPane cPODDataSelectorPaneFX;
+
+	private CPODDataSelectorPanel cPODDataSelectorPanel;
 
 	public CPODDataSelector(PamControlledUnit cpodControl, CPODClickDataBlock cpodDataBlock, String selectorName,
 			boolean allowScores) {
@@ -60,16 +61,18 @@ public class CPODDataSelector extends DataSelector {
 
 	@Override
 	public PamDialogPanel getDialogPanel() {
-		// TODO Auto-generated method stub
+		if (cPODDataSelectorPanel==null) {
-		return null;
+			cPODDataSelectorPanel = new CPODDataSelectorPanel(this);
+		}
+		return cPODDataSelectorPanel;
 	}
 
 	@Override
 	public DynamicSettingsPane<Boolean> getDialogPaneFX() {
-		if (cPODDataSelectorPane==null) {
+		if (cPODDataSelectorPaneFX==null) {
-			cPODDataSelectorPane = new CPODDataSelectorPane(this);
+			cPODDataSelectorPaneFX = new CPODDataSelectorPane(this);
 		}
-		return cPODDataSelectorPane;
+		return cPODDataSelectorPaneFX;
 	}
 
 	@Override
@@ -85,6 +88,16 @@ public class CPODDataSelector extends DataSelector {
 			//e.g. the FPOD data has waveforms so when that is implemented may want extra bits and pieces. 
 		}
 		
+		if (dataSelectorParams.selectClickTrain) {
+			CPODClickTrainDataUnit cpodClickTrain = cpodClick.getCPODClickTrain();
+			if (cpodClickTrain==null) return 0;
+			
+			if (dataSelectorParams.speciesID != null && !dataSelectorParams.speciesID.equals(cpodClickTrain.getSpecies())) {
+				return 0;
+			}
+			
+		}
+
 		return 1;
 	}
 

src/cpod/fx/CPODDataSelectorPane.java

@@ -2,15 +2,26 @@ package cpod.fx;
 
 import java.util.ArrayList;
 
+import javax.swing.JCheckBox;
+import javax.swing.JComboBox;
+
 import org.controlsfx.control.RangeSlider;
 
+import PamView.panel.PamPanel;
+import cpod.CPODClassification.CPODSpeciesType;
+import cpod.CPODUtils;
 import cpod.dataSelector.CPODDataSelector;
 import cpod.dataSelector.StandardCPODFilterParams;
+import export.MLExport.MLCPODExport;
+import javafx.geometry.Pos;
 import javafx.scene.Node;
+import javafx.scene.control.ComboBox;
 import javafx.scene.control.Label;
 import javafx.scene.layout.BorderPane;
 import javafx.scene.layout.Pane;
+import pamViewFX.fxNodes.PamHBox;
 import pamViewFX.fxNodes.PamVBox;
+import pamViewFX.fxNodes.utilityPanes.PamToggleSwitch;
 import pamViewFX.fxSettingsPanes.DynamicSettingsPane;
 
 /**
@@ -33,6 +44,10 @@ public class CPODDataSelectorPane extends DynamicSettingsPane<Boolean> {
 
 	Pane mainPane;
 
+	private PamToggleSwitch clcikTrainCheckBox;
+
+	private ComboBox<String> speciesSelectBox; 
+
 	public CPODDataSelectorPane(CPODDataSelector cpodDataSelector) {
 		super(null);
 		this.cpodDataSelector = cpodDataSelector;
@@ -51,6 +66,35 @@ public class CPODDataSelectorPane extends DynamicSettingsPane<Boolean> {
 			vBox.getChildren().add(standardCPODFilterPanes.get(i)); 
 		}
 		
+		
+		PamHBox speciesBox = new PamHBox(); 
+		speciesBox.setSpacing(5);
+		speciesBox.setAlignment(Pos.CENTER_LEFT);
+		
+		this.clcikTrainCheckBox = new PamToggleSwitch("Select click trains only: Species"); 
+		clcikTrainCheckBox.selectedProperty().addListener((obsVal, oldVal, newVal)->{
+			speciesSelectBox.setDisable(!newVal); 
+			this.getParams(true);
+			notifySettingsListeners(); 
+		});
+		this.speciesSelectBox = new ComboBox<String>(); 
+		speciesSelectBox.setOnAction((action)->{
+			this.getParams(true);
+			notifySettingsListeners(); 
+
+		});
+
+		//CPODs and FPODs have set species identifiers.
+		this.speciesSelectBox.getItems().add("All");
+		this.speciesSelectBox.getItems().add("Unknown");
+		this.speciesSelectBox.getItems().add("NBHF");
+		this.speciesSelectBox.getItems().add("Dolphins");
+		this.speciesSelectBox.getItems().add("Sonar");
+		
+		speciesBox.getChildren().addAll(clcikTrainCheckBox, speciesSelectBox); 
+		
+		vBox.getChildren().add(speciesBox); 
+
 		return vBox; 
 	}
 
@@ -61,6 +105,10 @@ public class CPODDataSelectorPane extends DynamicSettingsPane<Boolean> {
 			standardCPODFilterPanes.get(i).getParams(cpodDataSelector.getParams().cpodDataFilterParams.get(i)); 
 		}
 		
+		cpodDataSelector.getParams().selectClickTrain = clcikTrainCheckBox.isSelected();
+		 
+		cpodDataSelector.getParams().speciesID = getSpecies(speciesSelectBox.getSelectionModel().getSelectedIndex());
+		
 		return true;
 	}
 
@@ -70,6 +118,22 @@ public class CPODDataSelectorPane extends DynamicSettingsPane<Boolean> {
 			standardCPODFilterPanes.get(i).setParams(cpodDataSelector.getParams().cpodDataFilterParams.get(i)); 
 		}
 		
+		clcikTrainCheckBox.setSelected(cpodDataSelector.getParams().selectClickTrain);
+		speciesSelectBox.getSelectionModel().select(getSpeciesIndex(cpodDataSelector.getParams().speciesID));
+		
+		speciesSelectBox.setDisable(!cpodDataSelector.getParams().selectClickTrain); 
+		
+	}
+	
+	protected static int getSpeciesIndex(CPODSpeciesType speciesType) {
+		if (speciesType==null) return 0;
+		else return MLCPODExport.getCPODSpecies(speciesType)+1;
+	}
+
+	
+	protected static CPODSpeciesType getSpecies(int selectedIndex) {
+		if (selectedIndex==0) return null;
+		else return CPODUtils.getCPODSpecies((short) (selectedIndex-1)); 
 	}
 
 	@Override
@@ -169,7 +233,7 @@ public class CPODDataSelectorPane extends DynamicSettingsPane<Boolean> {
 		}
 
 		public void setParams(StandardCPODFilterParams standardCPODFilterParams) {
-			System.out.println("StandardCPODFilterPane. SET PARAMS: min: " + standardCPODFilterParams.min + "  " + standardCPODFilterParams.max); 
+//			System.out.println("StandardCPODFilterPane. SET PARAMS: min: " + standardCPODFilterParams.min + "  " + standardCPODFilterParams.max); 
 			//set the parameters. 
 			rangeSlider.setHighValue(standardCPODFilterParams.max);
 			rangeSlider.setLowValue(standardCPODFilterParams.min);

src/cpod/fx/CPODDataSelectorPanel.java

@@ -0,0 +1,279 @@
+package cpod.fx;
+
+import java.awt.BorderLayout;
+import java.awt.Color;
+import java.awt.GridBagConstraints;
+import java.awt.GridBagLayout;
+import java.util.ArrayList;
+
+import javax.swing.JCheckBox;
+import javax.swing.JComboBox;
+import javax.swing.JComponent;
+import javax.swing.JLabel;
+
+import PamView.dialog.PamDialogPanel;
+import PamView.dialog.PamGridBagContraints;
+import PamView.panel.PamPanel;
+import PamView.sliders.PamRangeSlider;
+import cpod.dataSelector.CPODDataSelector;
+import cpod.dataSelector.StandardCPODFilterParams;
+
+/**
+ * Swing data selector 
+ */
+public class CPODDataSelectorPanel implements PamDialogPanel {
+
+	private JComponent mainPane;
+
+	private CPODDataSelector cpodDataSelector;
+	
+	private ArrayList<StandardCPODFilterPanel> standardCPODFilterPanes;
+	
+	private JCheckBox clcikTrainCheckBox;
+
+	private JComboBox<String> speciesSelectBox;
+
+//	private CPODDataSelectorPane cpodDataSlectorPane;
+
+	public CPODDataSelectorPanel(CPODDataSelector cpodDataSelelctor) {
+		this.cpodDataSelector=cpodDataSelelctor; 
+		mainPane = createPanel();
+//		createFXPanel();
+	}
+	
+	
+	private JComponent createPanel() {
+		PamPanel vBox = new PamPanel();
+		vBox.setLayout(new GridBagLayout());
+		
+		PamGridBagContraints c = new PamGridBagContraints(); 
+		
+		c.gridx = 0;
+		c.gridy = 0;
+		c.gridwidth=4;
+
+		
+		standardCPODFilterPanes = new ArrayList<StandardCPODFilterPanel>();
+		ArrayList<StandardCPODFilterParams> standParams = cpodDataSelector.getParams().cpodDataFilterParams; 
+		
+		for (int i=0; i<standParams.size(); i++) {
+			standardCPODFilterPanes.add(new StandardCPODFilterPanel( cpodDataSelector.getParams().cpodDataFilterParams.get(i))); 
+			
+			PamPanel.addComponent(vBox, standardCPODFilterPanes.get(i), c);
+			c.gridx = 0;
+
+			c.gridy++;
+		}
+		
+		this.clcikTrainCheckBox = new JCheckBox("Select click trains only: Species"); 
+		clcikTrainCheckBox.addActionListener((action)->{
+			speciesSelectBox.setEnabled(clcikTrainCheckBox.isSelected()); 
+		});
+		this.speciesSelectBox = new JComboBox<String>(); 
+		
+		c.gridy++;
+		c.gridwidth=2;
+
+		PamPanel.addComponent(vBox, clcikTrainCheckBox, c);
+		c.gridx = 2;
+		PamPanel.addComponent(vBox, speciesSelectBox, c);
+
+		//CPODs and FPODs have set species identifiers.
+		this.speciesSelectBox.addItem("All");
+		this.speciesSelectBox.addItem("Unknown");
+		this.speciesSelectBox.addItem("NBHF");
+		this.speciesSelectBox.addItem("Dolphins");
+		this.speciesSelectBox.addItem("Sonar");
+		
+		vBox.validate();
+
+
+		return vBox; 
+	}
+	
+	
+	
+
+//	private void createFXPanel() {
+//		// This method is invoked on Swing thread
+//		final JFXPanel fxPanel = new JFXPanel();
+//		mainPane.add(fxPanel); 
+//
+//
+//		Platform.runLater(new Runnable() {
+//			@Override
+//			public void run() {
+//				initFX(fxPanel);
+//			}
+//		});
+//	}
+//
+//	private void initFX(JFXPanel fxPanel) {
+//		cpodDataSlectorPane = new CPODDataSelectorPane(cpodDataSelelctor); 
+//		// This method is invoked on JavaFX thread
+//		Group  root  =  new  Group();
+//		Scene  scene  =  new  Scene(root);
+//
+//		root.getChildren().add(cpodDataSlectorPane.getContentNode());
+//
+//		fxPanel.setScene(scene);
+//	}
+
+
+
+
+	@Override
+	public JComponent getDialogComponent() {
+		return mainPane;
+	}
+
+	@Override
+	public void setParams() {
+		
+		System.out.println("CPOD SET PARAMS"); 
+		
+		for (int i=0; i<cpodDataSelector.getParams().cpodDataFilterParams.size(); i++) {
+			standardCPODFilterPanes.get(i).setParams(cpodDataSelector.getParams().cpodDataFilterParams.get(i)); 
+		}
+		
+		clcikTrainCheckBox.setSelected(cpodDataSelector.getParams().selectClickTrain);
+		speciesSelectBox.setSelectedIndex(CPODDataSelectorPane.getSpeciesIndex(cpodDataSelector.getParams().speciesID));
+		
+		speciesSelectBox.setEnabled(cpodDataSelector.getParams().selectClickTrain); 
+
+	}
+
+	@Override
+	public boolean getParams() {
+		
+		for (int i=0; i<cpodDataSelector.getParams().cpodDataFilterParams.size(); i++) {
+			standardCPODFilterPanes.get(i).getParams(cpodDataSelector.getParams().cpodDataFilterParams.get(i)); 
+		}
+		
+		cpodDataSelector.getParams().selectClickTrain = clcikTrainCheckBox.isSelected();
+		 
+		cpodDataSelector.getParams().speciesID = CPODDataSelectorPane.getSpecies(speciesSelectBox.getSelectedIndex());		
+		
+		return true;
+	}
+	
+	
+	/**
+	 * The CPOD data filter pane.
+	 * @author Jamie Macaulay
+	 *
+	 */
+	public class StandardCPODFilterPanel extends PamPanel {
+
+		/**
+		 * The range slider. 
+		 */
+		private PamRangeSlider rangeSlider;
+
+		public StandardCPODFilterPanel(StandardCPODFilterParams params) {
+
+			rangeSlider = new PamRangeSlider(); 
+			JLabel topLabel = new JLabel(); 
+
+			rangeSlider.setMinimum(0);
+			rangeSlider.setMaximum(256);
+			
+			//dodger blue instead of obnoxius green
+			rangeSlider.setRangeColour(new Color(0, 90, 156));
+
+			rangeSlider.setPaintTicks(true);
+			rangeSlider.setPaintLabels(true);
+
+			rangeSlider.setMajorTickSpacing(40);
+			rangeSlider.setMinorTickSpacing(10);
+
+			String unit = "kHz";
+			switch (params.dataType) {
+			case StandardCPODFilterParams.AMPLITUDE:
+				unit = "dB re 1\u03BCPa";
+				topLabel.setText("Amplitude ("+ unit+ ")");
+				rangeSlider.setMinimum(80);
+				rangeSlider.setMaximum(170);
+				break;
+			case StandardCPODFilterParams.PEAK_FREQ:
+				topLabel.setText("Peak Frequency ("+ unit+ ")");
+				break;
+			case StandardCPODFilterParams.BW:
+				topLabel.setText("Bandwidth ("+ unit+ ")");
+				rangeSlider.setMinimum(0);
+				rangeSlider.setMaximum(100);
+
+				break;
+			case StandardCPODFilterParams.END_F:
+				topLabel.setText("End Frequency ("+ unit+ ")");
+
+				break;
+
+			case StandardCPODFilterParams.NCYCLES:
+				unit="";
+				topLabel.setText("Number cycles");
+				rangeSlider.setMinimum(0);
+				rangeSlider.setMaximum(40);
+				break;
+			}
+
+			this.setLayout(new BorderLayout());
+			
+			//create  pane to show the min and the max values
+			JLabel valueLabel = new JLabel();
+			
+			final String unit2 = unit;
+			rangeSlider.addChangeListener((a)->{
+				valueLabel.setText(String.format("%d to %d %s",rangeSlider.getValue() ,rangeSlider.getUpperValue() , unit2));
+			});
+
+			PamPanel labePanel = new PamPanel();
+			labePanel.setLayout(new BorderLayout());
+			labePanel.add(topLabel, BorderLayout.WEST);
+			labePanel.add(valueLabel, BorderLayout.EAST);
+
+			this.add(labePanel, BorderLayout.NORTH);
+			this.add(rangeSlider, BorderLayout.CENTER);
+			
+			this.setParams(params);
+
+
+		}
+
+		public void setParams(StandardCPODFilterParams standardCPODFilterParams) {
+//			System.out.println("StandardCPODFilterPane. SET PARAMS: min: " + standardCPODFilterParams.min + "  " + standardCPODFilterParams.max  +  "  " + rangeSlider.getMaximum()); 
+			//set the parameters. 
+			
+			rangeSlider.setValue((int) standardCPODFilterParams.min);
+			rangeSlider.setUpperValue((int) standardCPODFilterParams.max);
+			
+		}
+
+		public void getParams(StandardCPODFilterParams standardCPODFilterParams) {
+			//standardCPODFilterParams.max = rangeSlider.getMax();
+			
+			//if the range sliders are maxed out then all values are used. 
+			if (rangeSlider.getUpperValue()==rangeSlider.getMinimum()) {
+				standardCPODFilterParams.max = Double.POSITIVE_INFINITY; 
+			}
+			else {
+				standardCPODFilterParams.max = rangeSlider.getUpperValue();
+			}
+
+			if (rangeSlider.getValue()==rangeSlider.getMinimum()) {
+				standardCPODFilterParams.min = Double.NEGATIVE_INFINITY; 
+			}
+			else {
+				standardCPODFilterParams.min = rangeSlider.getValue();
+			}
+			
+//			standardCPODFilterParams.max = rangeSlider.getHighValue();
+//			standardCPODFilterParams.min = rangeSlider.getLowValue();
+	
+		}
+
+	}
+
+
+
+}

src/dataMap/DataMapParameters.java

@@ -11,7 +11,6 @@ public class DataMapParameters implements Cloneable, Serializable, ManagedParame
 
 	protected static final long serialVersionUID = 1L;
 	
-	
 	public int vScaleChoice = OfflineDataMap.SCALE_PERHOUR;
 	
 	public boolean vLogScale = true;
@@ -20,7 +19,9 @@ public class DataMapParameters implements Cloneable, Serializable, ManagedParame
 	 * Scale factor for horizontal axis. 
 	 */
 	public int hScaleChoice = 4;
+	
 	public static final double[] hScaleChoices = {0.1, 0.5, 1, 2, 5, 10, 20, 60, 180, 600};
+	
 	public double getPixeslPerHour() {
 		return hScaleChoices[hScaleChoice];
 	}

src/dataMap/layoutFX/DataMapInfo.java

@@ -0,0 +1,45 @@
+package dataMap.layoutFX;
+
+import java.io.Serializable;
+
+/**
+ * Holds information whcih can identify a data map. 
+ */
+public class DataMapInfo implements Serializable {
+
+	private static final long serialVersionUID = 1L;
+
+	private String name;
+	
+	private String longDataName;
+
+	public String getName() {
+		return name;
+	}
+	
+	public String getLongtName() {
+		return longDataName;
+	}
+
+	public void setLongName(String longDataName) {
+		this.longDataName=longDataName; 
+	}
+
+	public void setName(String dataName) {
+		this.name=dataName;
+	}
+	
+	@Override
+	public String toString() {
+		return name; 
+	}
+
+	@Override
+	public boolean equals(Object dataMapInfo1) {
+		if (dataMapInfo1==null) return false;
+		DataMapInfo dataMapInfo = (DataMapInfo) dataMapInfo1;
+		if (this.longDataName.equals(dataMapInfo.longDataName) && this.name.equals(dataMapInfo.name)) return true;
+		return false;
+	}
+	
+}

src/dataMap/layoutFX/DataMapPaneFX.java

@@ -1,9 +1,10 @@
 package dataMap.layoutFX;
 
-import java.awt.Dimension;
+import java.io.Serializable;
 
-import PamController.PamController;
+import PamController.PamControlledUnitSettings;
 import PamController.PamControllerInterface;
+import PamController.PamSettings;
 import dataMap.DataMapControl;
 import javafx.application.Platform;
 import javafx.geometry.Insets;
@@ -33,7 +34,7 @@ import userDisplayFX.UserDisplayNodeParams;
  * @author Jamie Macaulay
  *
  */
-public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
+public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX, PamSettings {
 
 	private static final double HIDE_PANE_WIDTH = 400;
 
@@ -48,8 +49,6 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 	public ScrollingDataPaneFX scrollingDataPanel;
 
 
-	private Dimension graphDimension;
-
 	private SummaryPaneFX summaryPane;
 
 	/**
@@ -74,6 +73,11 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 	 */
 	private PamDateAxis dateAxis;
 
+	/**
+	 * The parameters for the data map 
+	 */
+	private DataMapParametersFX dataMapParamsFX = new DataMapParametersFX();
+
 	public DataMapPaneFX(DataMapControl dataMapControl){
 		this.dataMapControl=dataMapControl; 
 		createDataMapPaneFX();
@@ -90,6 +94,15 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 		scrollingDataPanel= new ScrollingDataPaneFX(dataMapControl, this); 
 
 		dataMapSettingsPane=new DataMapSettingsPane(dataMapControl,this);
+		dataMapSettingsPane.addSettingsListener(()->{
+			//make sure paramters updates whenever user changes a setting. 
+			dataMapParamsFX = dataMapSettingsPane.getParams(dataMapParamsFX);
+			
+			if (scrollingDataPanel !=null) {
+				//forces things like the date axis to change. 
+				scrollingDataPanel.notifyScrollChange();
+			}
+		});
 
 		//create the setting spane
 		settingsPane=new PamVBox(); 
@@ -105,7 +118,7 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 		//topHolder.prefHeightProperty().bind(summaryPane.prefHeightProperty());
 
 		//hiding summary pane
-		hidingSettingsPane=new HidingPane(Side.RIGHT, settingsPane, this, true);
+		hidingSettingsPane=new HidingPane(Side.RIGHT, settingsPane, scrollingDataPanel, true);
 		hidingSettingsPane.setVisibleImmediatly(false); 
 		hidingSettingsPane.showHidePane(true);
 		hidingSettingsPane.getStyleClass().add("pane-trans");
@@ -160,7 +173,6 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 	 */
 	private void setGraphDimensions() {
 		long totalLength = dataMapControl.getLastTime() - dataMapControl.getFirstTime();
-		graphDimension = new Dimension(2000, 100);
 	}
 
 	public void repaintAll() {
@@ -174,17 +186,17 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 		//hidingSummaryPane.resetHideAnimation();
 	}
 
-	/**
+//	/**
-	 * Called from ScalePanel when anything 
+//	 * Called from ScalePanel when anything 
-	 * to do with scaling changes. 
+//	 * to do with scaling changes. 
-	 */
+//	 */
-	public void scaleChanged() {
+//	public void scaleChanged() {
-		if (dataMapSettingsPane == null || scrollingDataPanel == null) {
+//		if (dataMapSettingsPane == null || scrollingDataPanel == null) {
-			return;
+//			return;
-		}
+//		}
-		dataMapSettingsPane.getParams(dataMapControl.dataMapParameters);
+//		dataMapSettingsPane.getParams(dataMapParamsFX);
 //		scrollingDataPanel.scaleChange();
-	}
+//	}
 
 	@Override
 	public Region getNode() {
@@ -206,7 +218,7 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 	@Override
 	public boolean isResizeableDisplay() {
 		//Cannot resize the display
-		return true;
+		return false;
 	}
 
 	@Override
@@ -227,30 +239,30 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 		switch (changeType) {
 		case PamControllerInterface.INITIALIZATION_COMPLETE:
 			scrollingDataPanel.updateScrollBar();
-			dataMapSettingsPane.setParams(dataMapControl.dataMapParameters);
+			dataMapSettingsPane.setParams(dataMapParamsFX);
 			this.repaintAll();
 			break;
 		case PamControllerInterface.CHANGED_OFFLINE_DATASTORE:
 			scrollingDataPanel.updateScrollBar();
 			dataMapSettingsPane.checkDataGramPane();
-			dataMapSettingsPane.setParams(dataMapControl.dataMapParameters);
+			dataMapSettingsPane.setParams(dataMapParamsFX);
 			break;
 		case PamControllerInterface.ADD_CONTROLLEDUNIT:
 		case PamControllerInterface.REMOVE_CONTROLLEDUNIT:
 			dataMapSettingsPane.checkDataGramPane();
-			dataMapSettingsPane.setParams(dataMapControl.dataMapParameters);
+			dataMapSettingsPane.setParams(dataMapParamsFX);
 			break;
 		case PamControllerInterface.INITIALIZE_LOADDATA:
 		case PamControllerInterface.EXTERNAL_DATA_IMPORTED:
 			scrollingDataPanel.updateScrollBar();
 			dataMapSettingsPane.checkDataGramPane();
-			dataMapSettingsPane.setParams(dataMapControl.dataMapParameters);
+			dataMapSettingsPane.setParams(dataMapParamsFX);
 			this.repaintAll();
 			break;
 		case PamControllerInterface.OFFLINE_DATA_LOADED:
 			scrollingDataPanel.updateScrollBar();
 			dataMapSettingsPane.checkDataGramPane();
-			dataMapSettingsPane.setParams(dataMapControl.dataMapParameters);
+			dataMapSettingsPane.setParams(dataMapParamsFX);
 			this.repaintAll();
 			break;
 		case PamControllerInterface.DATA_LOAD_COMPLETE:
@@ -293,7 +305,7 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 
 	@Override
 	public UserDisplayNodeParams getDisplayParams() {
-		// TODO Auto-generated method stub
+		// the datamap is a display which cannot be moved or changed so this is not needed. 
 		return null;
 	}
 
@@ -323,8 +335,58 @@ public class DataMapPaneFX extends PamBorderPane implements UserDisplayNodeFX {
 	 * @return the data stream pane or null if the index is out of bounds. 
 	 */
 	public DataStreamPaneFX getDataStreamPane(int n) {
-		return this.scrollingDataPanel.getDataSyreamPane( n);
+		return this.scrollingDataPanel.getDataStreamPane( n);
 
 	}
 
+	public DataStreamPaneFX getDataStreamPane(DataMapInfo selectedItem) {
+		return scrollingDataPanel.getDataStreamPane(selectedItem);
+	}
+	
+	/**
+	 * Set the data map parameters. 
+	 * @param dataMapParamsFX
+	 */
+	public void setDataMapParams(DataMapParametersFX dataMapParamsFX) {
+		 this.dataMapParamsFX = dataMapParamsFX;
+	}
+
+	/**
+	 * Get the data map parameters associated with the FX GUI. Note these are
+	 * separate from the parameters in the DataMapControls which are for the default
+	 * swing display (not great)
+	 * 
+	 * @return the current data map parameters.
+	 */
+	public DataMapParametersFX getDataMapParams() {
+		return this.dataMapParamsFX;
+	}
+
+	@Override
+	public String getUnitName() {
+		return this.dataMapControl.getUnitName();
+	}
+
+	@Override
+	public String getUnitType() {
+		return "data_map_paneFX";
+	}
+
+	@Override
+	public Serializable getSettingsReference() {
+		return this.dataMapParamsFX;
+	}
+
+	@Override
+	public long getSettingsVersion() {
+		return  DataMapParametersFX.serialVersionUID;
+
+	}
+
+	@Override
+	public boolean restoreSettings(PamControlledUnitSettings pamControlledUnitSettings) {
+		dataMapParamsFX = ((DataMapParametersFX) pamControlledUnitSettings.getSettings()).clone();
+		return (dataMapParamsFX != null);
+	}
+
 }

src/dataMap/layoutFX/DataMapParametersFX.java

@@ -0,0 +1,65 @@
+package dataMap.layoutFX;
+
+import java.io.Serializable;
+import java.util.HashMap;
+
+import PamModel.parametermanager.ManagedParameters;
+import PamModel.parametermanager.PamParameterSet;
+import PamModel.parametermanager.PamParameterSet.ParameterSetType;
+import dataMap.OfflineDataMap;
+import dataPlotsFX.scrollingPlot2D.PlotParams2D;
+
+public class DataMapParametersFX implements Cloneable, Serializable, ManagedParameters {
+
+	protected static final long serialVersionUID = 1L;
+
+	/**
+	 * The data maps which are collapsed. 
+	 */
+	public HashMap<DataMapInfo, Boolean> dataMapsCollapsed = new HashMap<DataMapInfo, Boolean>();
+
+	
+	/**
+	 * The data maps which are shown. 
+	 */
+	public HashMap<DataMapInfo, Boolean> dataMapsShown = new HashMap<DataMapInfo, Boolean>();
+
+	
+	/**
+	 * A has map of colours for each data gram. 
+	 */
+	public HashMap<DataMapInfo, PlotParams2D> datagramColours = new HashMap<DataMapInfo, PlotParams2D>();
+	
+	/**
+	 * The log scale. 
+	 */
+	public int vScaleChoice = OfflineDataMap.SCALE_PERHOUR;
+	
+	/**
+	 * Use a log scale. 
+	 */
+	public boolean vLogScale = true;
+	
+	/**
+	 * The selected dfatagranm for colour changes. 
+	 */
+	public int selectedColourDatagram = 0; 
+	
+
+	@Override
+	protected DataMapParametersFX clone() {
+		try {
+			return (DataMapParametersFX) super.clone();
+		} catch (CloneNotSupportedException e) {
+			e.printStackTrace();
+			return null;
+		}
+	}
+
+	@Override
+	public PamParameterSet getParameterSet() {
+		PamParameterSet ps = PamParameterSet.autoGenerate(this, ParameterSetType.DISPLAY);
+		return ps;
+	}
+
+}

src/dataMap/layoutFX/DataMapScrollBar.java

@@ -11,7 +11,8 @@ import javafx.scene.paint.Color;
 import pamViewFX.fxNodes.pamScrollers.acousticScroller.ScrollBarPane;
 
 /**
- * A scroll bar whihc shows a summary fo the data. 
+ * A scroll bar which shows a summary of the data. This allows users to scroll to different sections of the data, 
+ * 
  */
 public class DataMapScrollBar extends ScrollBarPane {
 

src/dataMap/layoutFX/DataMapSettingsPane.java

@@ -4,7 +4,10 @@ import dataGram.DatagramManager;
 import dataGram.DatagramScaleInformation;
 import dataGram.DatagramSettings;
 import dataMap.DataMapControl;
-import dataMap.DataMapParameters;
+import dataPlotsFX.scrollingPlot2D.PlotParams2D;
+
+import org.controlsfx.control.CheckComboBox;
+
 import PamController.PamController;
 import PamUtils.PamCalendar;
 import binaryFileStorage.BinaryStore;
@@ -18,6 +21,7 @@ import javafx.geometry.Pos;
 import javafx.scene.Node;
 import javafx.scene.control.ComboBox;
 import javafx.scene.control.Label;
+import javafx.scene.control.Tooltip;
 import javafx.scene.layout.BorderPane;
 import javafx.scene.layout.ColumnConstraints;
 import javafx.scene.layout.GridPane;
@@ -26,6 +30,7 @@ import javafx.scene.layout.Priority;
 import pamViewFX.PamGuiManagerFX;
 import pamViewFX.fxNodes.PamBorderPane;
 import pamViewFX.fxNodes.PamGridPane;
+import pamViewFX.fxNodes.PamHBox;
 import pamViewFX.fxNodes.PamVBox;
 import pamViewFX.fxNodes.comboBox.ColorComboBox;
 import pamViewFX.fxNodes.pamDialogFX.PamDialogFX;
@@ -41,7 +46,7 @@ import pamViewFX.fxSettingsPanes.DynamicSettingsPane;
  * @author Jamie Macaulay
  *
  */
-public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters> {
+public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParametersFX> {
 
 	/*
 	 * Reference to the data map control. 
@@ -100,7 +105,7 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 
 	private Label dataGramLabel;
 
-	private ComboBox<String> dataGramBox;
+	private ComboBox<DataMapInfo> dataGramColorBox;
 
 	/**
 	 * Holdes datagram settings. 
@@ -111,6 +116,11 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 
 	private PamBorderPane mainPain;
 
+	private Label colourLabel;
+
+	private Pane dataMapChoicePane;
+
+	private CheckComboBox<String> dataMapCheckComboBox; 
 
 
 	public DataMapSettingsPane(DataMapControl dataMapControl, DataMapPaneFX dataMapPane) {
@@ -129,7 +139,12 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 		holder=new PamVBox(); 
 		holder.setSpacing(5);
 
-		Label scaleLabel = new Label("Data Scale");
+		Label dataLabel = new Label("Show data maps");
+		PamGuiManagerFX.titleFont2style(dataLabel);
+		holder.getChildren().addAll(dataLabel, dataMapChoicePane = createDataMapPane());
+		updateDataMapBinBox();
+
+		Label scaleLabel = new Label("Data scale");
 		PamGuiManagerFX.titleFont2style(scaleLabel);
 		holder.getChildren().add(scaleLabel);
 		holder.getChildren().add(scaleSettingsPane = createScalePane());
@@ -141,12 +156,35 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 		mainPain.setCenter(holder); 
 
 		//set params for the pane		
-		setParams(dataMapControl.dataMapParameters);
+		setParams(dataMapPane.getDataMapParams());
 		checkDataGramPane(); // create datagram pane if a binary store already added. 
-		sayHScale();
 
 	}
 
+	/**
+	 * Create the data mmpa choice pane. 
+	 * @return
+	 */
+	private Pane createDataMapPane() {
+
+		dataMapCheckComboBox = new 	CheckComboBox<String>(); 
+
+		PamHBox hBox = new PamHBox(); 
+		hBox.setSpacing(5.);
+		hBox.getChildren().addAll(dataMapCheckComboBox);
+
+		return hBox;
+	}
+
+	/**
+	 * Update the combo box which allows users to select which datagram are shown. 
+	 */
+	private void updateDataMapBinBox() {
+		dataMapCheckComboBox.getItems().clear();
+		for (int i=0; i<this.dataMapPane.getNumDataStreamPanes(); i++) {
+			dataMapCheckComboBox.getItems().add(dataMapPane.getDataStreamPane(i).getDataName().getName()); 
+		}
+	}
 	/**
 	 * Adds a settings pane for the datagram if a binary store is present. 
 	 */
@@ -228,13 +266,18 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 		ComboBox<String> datagramBinsBox = createDataGramBinPane(dataGramManager); 
 
 		//Pane for colouring datagrams. 
-		dataGramBox=new ComboBox<String> (); 
+		dataGramColorBox=new ComboBox<DataMapInfo> (); 
 
 		//find all datagrams. 
-		updateDataStreamBox();
+		updateDatagramColorBox();
 
-		dataGramBox.setOnAction((action)->{
+		dataGramColorBox.setOnAction((action)->{
-			dataGramColPane.setDataStreamPanel(dataMapPane.getDataStreamPane(dataGramBox.getSelectionModel().getSelectedIndex()));
+			if (dataGramColorBox.getSelectionModel().getSelectedItem()==null) return;
+			dataGramColPane.setDataStreamPanel(dataMapPane.getDataStreamPane(dataGramColorBox.getSelectionModel().getSelectedItem()));
+			colourLabel.setText(String.format("Colours for %s " , dataGramColorBox.getSelectionModel().getSelectedItem().getName())); 
+			colourLabel.setTooltip(new Tooltip(String.format("Colours for %s " , dataGramColorBox.getSelectionModel().getSelectedItem().getName()))); 
+			
+			notifySettingsListeners();
 		});
 
 		//holds settings for the datagram
@@ -243,17 +286,25 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 		PamGridPane holder = new PamGridPane(); 
 		holder.setHgap(5);
 		holder.setVgap(5);
+
 		//		holder.setGridLinesVisible(true);
 
 		int row = 0;
 
 		holder.add(dataGramLabel, 0,row);
 		holder.add(datagramBinsBox, 1, row);
+		GridPane.setColumnSpan(datagramBinsBox, 2);
 
 		row++;
 
 		holder.add(new Label("Select datagram"), 0, row);
-		holder.add(dataGramBox, 1, row);
+		holder.add(dataGramColorBox, 1, row);
+		GridPane.setColumnSpan(dataGramColorBox, 2);
+
+		row++;
+
+		holder.add(colourLabel = new Label("Colour for"), 0, row);
+		GridPane.setColumnSpan(colourLabel, 3);
 
 		row++;
 
@@ -263,116 +314,35 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 		//dunno why this always had to be set after the child has been added to work. 
 		GridPane.setColumnSpan(dataGramColPane, 3);
 
-		//hack to make sure the third column of the gris expands to fit the pane
+		//hack to make sure the third column of the grid expands to fit the pane
 		ColumnConstraints rightCol = new ColumnConstraints();
 		rightCol.setHgrow(Priority.ALWAYS);
-        holder.getColumnConstraints().addAll(new ColumnConstraints(),  new ColumnConstraints(), rightCol);
+
+		holder.getColumnConstraints().addAll(new ColumnConstraints(150),  new ColumnConstraints(150), rightCol);
+
+		dataGramColPane.setDataStreamPanel(dataMapPane.getDataStreamPane(0));
+		colourLabel.setText(String.format("Colours for %s " , dataMapPane.getDataStreamPane(0))); 
+
 
 		return holder; 
-
 	}
 
-	private void updateDataStreamBox() {
+
-		System.out.println("UPDATE DATA STREAM BOX: " + this.dataMapPane.getNumDataStreamPanes());
+
+	/**
+	 * Update the datagram colours combo box thjat allows users to select which datagram to select for changing colour settings.. 
+	 */
+	private void updateDatagramColorBox() {
+		dataGramColorBox.getItems().clear();
 		for (int i=0; i<this.dataMapPane.getNumDataStreamPanes(); i++) {
-			if (dataMapPane.getDataStreamPane(i).getScaleType() == DatagramScaleInformation.PLOT_3D) {
+			//only include if the data block has a datagram and the data gram is a color plot - some datagrams can be lines e.g. filtered noise meaurements. 
-				dataGramBox.getItems().add(dataMapPane.getDataStreamPane(i).getDataName().getName()); 
+			if (dataMapPane.getDataStreamPane(i).isHasDatagram() && dataMapPane.getDataStreamPane(i).getScaleType() == DatagramScaleInformation.PLOT_3D) {
+				dataGramColorBox.getItems().add(dataMapPane.getDataStreamPane(i).getDataName()); 
 			}
 		}
 	}
 
 
-
-	/** 
-	 * Allows the user to change datagram colours. 
-	 * 
-	 */
-	public class DataGramColPane extends PamBorderPane {
-
-
-		private ColourRangeSlider colourSlider;
-
-		private Label ampLabel;
-
-		private ColorComboBox colorBox;
-
-		private ColourArrayType colorArray = ColourArrayType.HSV; 
-		
-		private DataStreamPaneFX dataStreamPane; 
-
-		public DataGramColPane() {
-
-			//create colour slider
-			//colour slider 
-			colourSlider=new ColourRangeSlider();
-			colourSlider.setShowTickMarks(false);
-			colourSlider.setShowTickLabels(false);
-			colourSlider.setOrientation(Orientation.HORIZONTAL);
-			//amplifier label
-			//				String dBRef = GlobalMedium.getdBRefString(PamController.getInstance().getGlobalMediumManager().getCurrentMedium());
-			Label ampLabel = new Label("Colour scale"); 
-			
-			colorBox=new ColorComboBox(ColorComboBox.COLOUR_ARRAY_BOX);
-			colorBox.setPrefWidth(80);
-			
-			colourSlider.highValueProperty().addListener((obsVal, oldVal, newVal)->{
-				
-			});
-
-			//Change the colour of the colour slider when combo box is changed. 
-			colorBox.valueProperty().addListener(new ChangeListener<String>() {
-				@Override public void changed(ObservableValue<? extends String> ov, String t, String t1) {                
-					//change the colour of the colour range slider.     
-					setColours();
-				}
-			});
-			
-
-			colorBox.setValue(ColourArray.getName(getColourArrayType()));
-
-			//need to set up alignment properly. //FIXME- a bit messy 
-			BorderPane.setAlignment(colorBox, Pos.CENTER);
-			//sliderPane.setPadding(new Insets(10,0,0,0));
-			BorderPane.setMargin(colorBox, new Insets(0,5,0,5));
-			
-			this.setCenter(colourSlider);
-			this.setRight(colorBox);
-		
-			//set up so the correct color
-			colorBox.setValue(ColourArray.getName(getColourArrayType()));
-			colourSlider.setColourArrayType(getColourArrayType()); 
-
-		}
-
-
-		public void setDataStreamPanel(DataStreamPaneFX selectedItem) {
-			this.dataStreamPane=selectedItem;
-			//TODO - set the colours here. 
-			
-		}
-
-
-		/**
-		 * Set colours depending on current colour selection in combo box. 
-		 */
-		private void setColours(){
-			this.setColourArrayType(ColourArray.getColourArrayType(colorBox.getValue()));
-			colourSlider.setColourArrayType(getColourArrayType());
-		}
-		
-	
-		public ColourArrayType getColourArrayType() {
-			return colorArray;
-		}
-
-		public void setColourArrayType(ColourArrayType colourArrayType) {
-			this.colorArray = colourArrayType; 
-		}
-
-
-	}
-
-
 	/**
 	 * Create a pane to change vertical scales. 
 	 * @return pamne with controls to change vertical scales. 
@@ -413,58 +383,136 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 		scaleBox.getItems().add("per Minute");
 		scaleBox.getItems().add("per Hour");
 		scaleBox.getItems().add("per Day");
+		GridPane.setColumnSpan(scaleBox, 2);
+
+		Label showDetLabel  = new Label("Show detections ");
 
 		controlPane.add(new Label("Show detections "),0,0);
 		controlPane.add(scaleBox,1,0);
 		//		scaleBox.setPrefWidth(200);
 
 		scaleBox.valueProperty().addListener((ov, oldVal, newVal)->{
-			dataMapPane.scaleChanged();
+			notifySettingsListeners(); 
 		}); 
 
 
 		logScaleToggle=new PamToggleSwitch("Log Scale"); 
 		logScaleToggle.selectedProperty().addListener((obsVal, oldVal, newVal)->{
-			dataMapPane.scaleChanged();
+			notifySettingsListeners(); 
 		});
 
+		ColumnConstraints rightCol = new ColumnConstraints();
+		rightCol.setHgrow(Priority.ALWAYS);
+		controlPane.getColumnConstraints().addAll(new ColumnConstraints(150),  new ColumnConstraints(150), rightCol);
 
 		controlPane.add(logScaleToggle,0,1);
 
-
 		return controlPane;
 	}
 
-	/**
-	 * Show the horizontal scale. 
-	 */
-	private void sayHScale() {
-//		double hChoice = timeScaleChoices[this.];
-//		timeScaleLabel.setText(String.format("%s pixs/hour", new Double(timeScaleChoices[(int) hChoice]).toString()));
-	}
 
-	//HACK use setting flag to avoid immediate callback which overwrites changes 2 and 3 ! 
+
+	// use setting flag to avoid immediate callback which overwrites changes 2 and 3 ! 
 	boolean setting = false;
 
-	public void setParams(DataMapParameters dataMapParameters) {
+	public void setParams(DataMapParametersFX dataMapParameters) {
+		System.out.println("----SET DATAMAP PARAMS");
+
+		// ----- Datagram bin size ----- //
 		setting = true;
 		//		timeSlider.setValue(dataMapParameters.hScaleChoice);
 		scaleBox.getSelectionModel().select(dataMapParameters.vScaleChoice);
 		logScaleToggle.setSelected(dataMapParameters.vLogScale);
 
-		//make sure the combo box has correct datastreams
+		// ----- Datagram bin size ----- //
-		updateDataStreamBox();
+		//make sure combo box for datamaps  is sorted
+		updateDataMapBinBox();
 
+		if (BinaryStore.findBinaryStoreControl()!=null){
+			DatagramManager dataGramManager=BinaryStore.findBinaryStoreControl().getDatagramManager();
+			dataGramComboBox.getSelectionModel().select(durationToString(dataGramManager.getDatagramSettings().datagramSeconds*1000L));
+
+			// ----- Datagram colours ----- //
+
+			//make sure the combo box has correct data streams
+			updateDatagramColorBox();
+
+
+			//get colour data from Hash tables	
+			DataStreamPaneFX dataStreamPane;
+			for (int i=0; i<dataGramColorBox.getItems().size(); i++) {
+
+				dataStreamPane = dataMapPane.getDataStreamPane(dataGramColorBox.getItems().get(i));
+
+				PlotParams2D plotCols =  dataMapParameters.datagramColours.get(dataGramColorBox.getItems().get(i));
+
+				System.out.println("------Set colours: " + (plotCols == null ? null : plotCols.getAmplitudeLimits()[0] + "  " +  plotCols.getAmplitudeLimits()[1]) + "  " + dataGramColorBox.getItems().get(i)); 
+
+				if (plotCols!=null) {
+					dataStreamPane.setMinMaxColourLimits(plotCols.getAmplitudeLimits()[0].get(), plotCols.getAmplitudeLimits()[1].get());
+					dataStreamPane.setColourArrayType(plotCols.getColourMap());
+				}
+				else {
+					dataStreamPane.setMinMaxColourLimits(-80, 80);
+					dataStreamPane.setColourArrayType(ColourArrayType.HSV);
+				}
+
+			}
+			
+			System.out.println("------Set colour box: " + dataMapParameters.selectedColourDatagram);
+
+
+			//select the correct colour box - not this should also change the colours and limits of the colour settings pane. 
+			if (this.dataGramColorBox.getItems().size()>dataMapParameters.selectedColourDatagram && dataMapParameters.selectedColourDatagram>=0) {
+				dataGramColorBox.getSelectionModel().select(dataMapParameters.selectedColourDatagram);
+			}
+			else if (this.dataGramColorBox.getItems().size()>0 && this.dataGramColorBox.getSelectionModel().getSelectedIndex()<0){
+				dataGramColorBox.getSelectionModel().select(0);
+			}
+
+		}
 		setting = false;
 	}
 
 
-	public DataMapParameters getParams(DataMapParameters dataMapParameters) {
+	public DataMapParametersFX getParams(DataMapParametersFX dataMapParameters) {
 		if (setting) return dataMapParameters;
+
+		System.out.println("---GET DATAMAP PARAMS");
+
 		//		dataMapParameters.hScaleChoice = (int) timeSlider.getValue(); 
+
 		dataMapParameters.vScaleChoice = scaleBox.getSelectionModel().getSelectedIndex();
 		dataMapParameters.vLogScale = logScaleToggle.isSelected();
 
+		//could have just cleared the hash map here but didn't want to change the setting object for 
+		//data stream panes everytime this is called. 
+
+		if (BinaryStore.findBinaryStoreControl()!=null){
+
+
+			//set the data in the hash map.
+			DataStreamPaneFX dataStreamPane = null;
+			for (int i=0; i<dataGramColorBox.getItems().size(); i++) {
+
+				dataStreamPane = dataMapPane.getDataStreamPane(dataGramColorBox.getItems().get(i));
+
+				PlotParams2D plotCols =  dataMapParameters.datagramColours.get(dataGramColorBox.getItems().get(i)) ;
+				if (plotCols==null) {
+					plotCols = new PlotParams2D();
+					dataMapParameters.datagramColours.put(dataGramColorBox.getItems().get(i), plotCols);
+				}
+
+				plotCols.getAmplitudeLimits()[0].set(dataStreamPane.getMinColourLimit());
+				plotCols.getAmplitudeLimits()[1].set(dataStreamPane.getMaxColourLimit());
+				plotCols.setColourMap(dataStreamPane.getColourMapArray());
+			}
+
+			//TODO get rid of entries no longer used in HasMap?
+			dataMapParameters.selectedColourDatagram = dataGramColorBox.getSelectionModel().getSelectedIndex();
+
+		}
+
 		return dataMapParameters;
 	}
 
@@ -475,15 +523,126 @@ public class DataMapSettingsPane extends DynamicSettingsPane<DataMapParameters>
 
 	@Override
 	public Node getContentNode() {
-		// TODO Auto-generated method stub
 		return mainPain;
 	}
 
 	@Override
 	public void paneInitialized() {
-		// TODO Auto-generated method stub
 
 	}
 
 
+	/** 
+	 * Allows the user to change datagram colours. 
+	 * 
+	 */
+	public class DataGramColPane extends PamBorderPane {
+
+		private ColourRangeSlider colourSlider;
+
+		private Label ampLabel;
+
+		private ColorComboBox colorBox;
+
+		private DataStreamPaneFX dataStreamPane; 
+
+		public DataGramColPane() {
+
+			//create colour slider
+			//colour slider 
+			colourSlider=new ColourRangeSlider();
+			colourSlider.setShowTickMarks(false);
+			colourSlider.setShowTickLabels(false);
+			colourSlider.setOrientation(Orientation.HORIZONTAL);
+			colourSlider.setMin(-100);
+			colourSlider.setMax(100);
+			//amplifier label
+			//				String dBRef = GlobalMedium.getdBRefString(PamController.getInstance().getGlobalMediumManager().getCurrentMedium());
+			//			Label ampLabel = new Label("Colour scale"); 
+
+			colorBox=new ColorComboBox(ColorComboBox.COLOUR_ARRAY_BOX);
+			colorBox.setPrefWidth(80);
+
+			colourSlider.lowValueProperty().addListener((obsVal, oldVal, newVal)->{
+				setColours();
+			});
+
+			colourSlider.highValueProperty().addListener((obsVal, oldVal, newVal)->{
+				setColours();
+			});
+
+			//Change the colour of the colour slider when combo box is changed. 
+			colorBox.valueProperty().addListener(new ChangeListener<String>() {
+				@Override public void changed(ObservableValue<? extends String> ov, String t, String t1) {                
+					//change the colour of the colour range slider.     
+					setColours();
+				}
+			});
+
+
+			//need to set up alignment properly. //FIXME- a bit messy 
+			BorderPane.setAlignment(colorBox, Pos.CENTER);
+			//sliderPane.setPadding(new Insets(10,0,0,0));
+			BorderPane.setMargin(colorBox, new Insets(0,5,0,5));
+
+			this.setCenter(colourSlider);
+			this.setRight(colorBox);
+
+			//set up so the correct color
+			colorBox.setValue(ColourArrayType.HOT);
+			colourSlider.setColourArrayType(ColourArrayType.HOT);
+
+		}
+
+
+		public void setDataStreamPanel(DataStreamPaneFX selectedItem) {
+			this.dataStreamPane=selectedItem;
+
+			if (dataStreamPane==null) return;
+
+			//keep a record of the colour limits because the slider has listeners that sets
+			//them in the datastream
+			double minCol = dataStreamPane.getMinColourLimit(); 
+			double maxCol = dataStreamPane.getMaxColourLimit(); 
+			
+			System.out.println("Data stream colour limits: " + minCol + "  " + maxCol);
+
+			
+			//the settings will already ahve been set in the data stream pane so 
+			//make sure those are reflected in the controls. 
+			colorBox.setValue(dataStreamPane.getColourMapArray()); 
+			
+
+			colourSlider.lowValueProperty().set(minCol);
+			colourSlider.highValueProperty().set(maxCol);
+			
+		}
+
+
+		/**
+		 * Set colours depending on current colour selection in combo box. 
+		 */
+		private void setColours(){
+			
+			System.out.println("Set colours: " + colourSlider.getLowValue() + "  " + colourSlider.getHighValue());
+
+
+			ColourArrayType colourArrayType = ColourArray.getColourArrayType(colorBox.getValue());
+
+			colourSlider.setColourArrayType(colourArrayType);
+
+			if (dataStreamPane==null) return;
+
+			dataStreamPane.setColourArrayType(colourArrayType); 
+			dataStreamPane.setMinMaxColourLimits(colourSlider.getLowValue(), colourSlider.getHighValue()); 
+			
+			//want to be efficient here so that we do not repaint all datagrams whenever these settings are changed so
+			//get the params but not through notify setting listeners. 
+			dataMapPane.setDataMapParams(getParams(dataMapPane.getDataMapParams()));
+		}
+
+
+
+	}
+
 }

src/dataMap/layoutFX/DataStreamPaneFX.java

@@ -1,5 +1,6 @@
 package dataMap.layoutFX;
 
+import java.nio.IntBuffer;
 import java.util.Arrays;
 import java.util.Iterator;
 
@@ -13,15 +14,20 @@ import dataMap.DataMapControl;
 import dataMap.DataMapDrawing;
 import dataMap.OfflineDataMap;
 import dataMap.OfflineDataMapPoint;
+import dataPlotsFX.scrollingPlot2D.PlotParams2D;
+import dataPlotsFX.scrollingPlot2D.StandardPlot2DColours;
 import javafx.animation.KeyFrame;
 import javafx.animation.Timeline;
 import javafx.geometry.Orientation;
 import javafx.geometry.Pos;
 import javafx.scene.canvas.Canvas;
 import javafx.scene.canvas.GraphicsContext;
+import javafx.scene.control.Button;
 import javafx.scene.control.Label;
-import javafx.scene.image.PixelWriter;
+import javafx.scene.image.PixelBuffer;
+import javafx.scene.image.PixelFormat;
 import javafx.scene.image.WritableImage;
+import javafx.scene.image.WritablePixelFormat;
 import javafx.scene.input.ScrollEvent;
 import javafx.scene.layout.Pane;
 import javafx.scene.paint.Color;
@@ -36,6 +42,8 @@ import pamViewFX.fxNodes.PamHBox;
 import pamViewFX.fxNodes.pamAxis.PamAxisFX;
 import pamViewFX.fxNodes.pamAxis.PamAxisPane;
 import pamViewFX.fxNodes.utilsFX.ColourArray;
+import pamViewFX.fxNodes.utilsFX.ColourArray.ColourArrayType;
+import pamViewFX.fxNodes.utilsFX.PamUtilsFX;
 
 public class DataStreamPaneFX extends PamBorderPane {
 	
@@ -69,7 +77,7 @@ public class DataStreamPaneFX extends PamBorderPane {
 	/**
 	 * Name associated with the data stream.
 	 */
-	private DataName dataName;
+	private DataMapInfo dataName;
 	
 	private int totalDatas, maxDatas;
 
@@ -137,8 +145,9 @@ public class DataStreamPaneFX extends PamBorderPane {
 		hasDatagram = (dataBlock.getDatagramProvider() != null);
 		dataGraph = new DataGraphFX();
 		dataGraph.setupAxis();
-		dataName = new DataName();
+		dataName = new DataMapInfo();
 		dataName.setName(dataBlock.getDataName()); 
+		dataName.setLongName(dataBlock.getLongDataName()); 
 
 		this.setTop(topPane=createTopPane());
 		this.setCenter(dataGraph);
@@ -195,7 +204,7 @@ public class DataStreamPaneFX extends PamBorderPane {
 	/**
 	 * @return the dataName
 	 */
-	public DataName getDataName() {
+	public DataMapInfo getDataName() {
 		return dataName;
 	}
 	
@@ -231,9 +240,9 @@ public class DataStreamPaneFX extends PamBorderPane {
 		
 		private static final int NCOLOURPOINTS = 100;
 
-		//for 3D data gram 
+		private static final double MAX_LOG_LOOKUP = 1.;
 		
-		private ColourArray datagramColours;
+		//for 3D data gram 
 		
 		/**
 		 * The wheel scroll factor. 
@@ -272,11 +281,21 @@ public class DataStreamPaneFX extends PamBorderPane {
 
 		private long lastTime;
 		
+		private StandardPlot2DColours plotColours2D = new StandardPlot2DColours(); 
+
+		/**
+		 * A lookup table for log values to speed up image drawing for 3D images
+		 */
+		private double[] logimagetable;
+
 
 		private DataGraphFX() {
 			createDataGraph();
 			addDataGraphMouse();
 			
+			//set some sensible default amplitude limits. 
+			plotColours2D.getAmplitudeLimits()[0].set(-80);
+			plotColours2D.getAmplitudeLimits()[0].set(80);
 
 		}
 		
@@ -395,6 +414,7 @@ public class DataStreamPaneFX extends PamBorderPane {
 			
 			this.setLeft(axisPane);
 			this.setCenter(canvasHolder);
+			
 		}
 		
 		
@@ -480,7 +500,7 @@ public class DataStreamPaneFX extends PamBorderPane {
 		private void paintPlotCanvas(GraphicsContext gc){
 			setupAxis();
 			gc.clearRect(0, 0, plotCanvas.getWidth(), plotCanvas.getHeight());
-			if (hasDatagram && showDatagram) {
+			if (isHasDatagram() && showDatagram) {
 				datagramPaint(gc);
 			}
 			else {
@@ -494,7 +514,7 @@ public class DataStreamPaneFX extends PamBorderPane {
 		public void setupAxis(){
 			sortScales();
 			//use log scale only if datagram is null. 
-			if (findDatagramScaleInfo()==null)datastreamAxis.setLogScale(dataMapControl.dataMapParameters.vLogScale);
+			if (findDatagramScaleInfo()==null)datastreamAxis.setLogScale(scrollingDataPanel.getDataMapParams().vLogScale);
 			datastreamAxis.setRange(getYScaleMin(), getYScaleMax());
 			datastreamAxis.setLabel(getScaleUnits());
 			paintAxis();
@@ -508,6 +528,7 @@ public class DataStreamPaneFX extends PamBorderPane {
 		}
 		
 		private void datagramPaint(GraphicsContext g) {
+//			System.out.println("PAINT DATA STREAM BOX: " + getScaleType() + "  " + getDataName().getName());
 			if (getScaleType() == DatagramScaleInformation.PLOT_3D) {
 				datagramPaint3D(g);
 			}
@@ -611,15 +632,24 @@ public class DataStreamPaneFX extends PamBorderPane {
 				return;
 			}
 			
-			DatagramImageData datagramImageData = datagramManager.getImageData(dataBlock, startMillis, endMillis, (int) getWidth());
+//			long time1 = System.currentTimeMillis();
+			
+			//JavaFX rendering works better with less pixels. 
+			DatagramImageData datagramImageData = datagramManager.getImageData(dataBlock, startMillis, endMillis, (int) Math.min(getWidth(), 1000.));
+			
 			double[][] imageData = datagramImageData.imageData;
 			
-			if (datagramColours == null) {
+			if (logimagetable==null) {
-				datagramColours = ColourArray.createHotArray(NCOLOURPOINTS);
+				//now generate a log lookup table for the image because Math.log takes a long time!
+				//a bit complicate because, if we have a huge value then the lookup table is useless
+				//so need to us a median
+				generateLogImageTable(2000); 
 			}
+						
 			int nXPoints = imageData.length;
 			int nYPoints = imageData[0].length;
 			
+		
 			double[] minMaxValue;
 			if (getMinMaxValues(imageData, false)!=null){
 				minMaxValue = Arrays.copyOf(minMaxVal,2);
@@ -628,15 +658,6 @@ public class DataStreamPaneFX extends PamBorderPane {
 				return;
 			}
 
-			minMaxValue[1] *= wheelScrollFactor;
-			minMaxValue[0] = Math.log(minMaxValue[0]);
-			minMaxValue[1] = Math.log(minMaxValue[1]);
-			/* 
-			 * now fudge the scale a little since black is zero and we want 
-			 * anything > 0 to be significantly away from black.
-			 */
-			double scaleRange = (minMaxValue[1] - minMaxValue[0]) * 1.2;
-			minMaxValue[0] = minMaxValue[1]-scaleRange;
 
 			int iCol, y;
 			int x1, x2;
@@ -645,38 +666,66 @@ public class DataStreamPaneFX extends PamBorderPane {
 			if (imageData.length == 0 || imageData[0].length == 0) {
 				return;
 			}
-			datagramImage = new WritableImage(imageData.length, imageData[0].length);
+			
-			PixelWriter writableRaster = datagramImage.getPixelWriter();
+			//use a pixel buffer for rendering an image as it's much faster!
+			IntBuffer buffer = IntBuffer.allocate(nXPoints * nYPoints);
+			int[] pixels = buffer.array();
+			PixelBuffer<IntBuffer> pixelBuffer = new PixelBuffer<>(nXPoints, nYPoints, buffer, PixelFormat.getIntArgbPreInstance());
+
+			datagramImage = new WritableImage(pixelBuffer);
 			g.setFill(Color.LIGHTGRAY);
 			g.fillRect(0, 0, nXPoints, nYPoints);
+			
+			Color col = null;
+			int colorARGB = 0;
+					
+//			long time2 = System.currentTimeMillis();
 			for (int i = 0; i < nXPoints; i++) {
 				for (int j = 0; j < nYPoints; j++) {
 					y = nYPoints-j-1;
-					if (imageData[i][j] < 0) {
+					if (imageData[i][y] < 0) {
 						//writableRaster.setColor(i,y,Color.LIGHTGRAY);
 					}
-					else if (imageData[i][j] == 0) {
+					else if (imageData[i][y] == 0) {
 						//writableRaster.setColor(i,y, Color.LIGHTGRAY);
 					}
 					else {
-						iCol = (int) (NCOLOURPOINTS * (Math.log(imageData[i][j]) - minMaxValue[0]) / scaleRange);
-						iCol = Math.min(Math.max(0, iCol), NCOLOURPOINTS-1);
-						writableRaster.setColor(i, y, datagramColours.getColour(iCol));
-						//						datagramImage.setRGB(i, y, 0x0000FF);
-					}
-				}
-			}
-			/*
-			 * Now finally paint that into the full window ...
-			 * 
-			 */
-			double imageWidth = getWidth();
 						
 						
+//						iCol = (int) (NCOLOURPOINTS * (approximateLogLookup(imageData[i][y]) - minMaxValue[0]) / scaleRange);
+//						iCol = Math.min(Math.max(0, iCol), NCOLOURPOINTS-1);
+						
+						col =plotColours2D.getColours(20*approximateLogLookup(imageData[i][y]) ); 
+						
+						
+//						if (i==0) {
+//						System.out.println("dB image: " + 20*approximateLogLookup(imageData[i][y])+ "  " + plotColours2D.getAmplitudeLimits()[0].get()); 
+//						}
+					
+						colorARGB = (255 << 24) | (((int) (col.getRed()*255.))  << 16) | (((int) (col.getGreen()*255.)) << 8) | (int) (col.getBlue()*255.);
+
+ 
+						pixels[(i % nXPoints) + (j * nXPoints)] =  colorARGB;
+												
+//						writableRaster.setColor(i, y, datagramColours.getColour(iCol));
+						//						datagramImage.setRGB(i, y, 0x0000FF);
+						
+					}
+				}
+			}
+			
+			// tell the buffer that the entire area needs redrawing
+			pixelBuffer.updateBuffer(b -> null);
+			
+//			long time3 = System.currentTimeMillis();		
+
 			//javafx version using a writable image (remember reversed compared to swing)
 			g.drawImage(datagramImage, 0, 0, nXPoints, nYPoints,
 					x1, 0,  x2-x1, getHeight());
 			
+//			long time4 = System.currentTimeMillis();
+			//System.out.println("Datagram Image data size: " + nXPoints +  "  " + nYPoints + " total time: " + (time4-time1) + " time pixel writer: " + (time3-time2) );
+
 			//swing version for ref
 			//g.drawImage(datagramImage, x1, 0, x2, getHeight(), 0, 0, nXPoints, nYPoints, null);
 
@@ -694,6 +743,65 @@ public class DataStreamPaneFX extends PamBorderPane {
 
 		}
 		
+		private void generateLogImageTable(int N) {
+			logimagetable=new double[N];
+			double min = 0.001;
+			double max = MAX_LOG_LOOKUP;
+			generateLogImageTable( min,  max,  N);
+		}
+		
+		private void generateLogImageTable(double min, double max, int N) {
+			logimagetable=new double[N];
+			double bin = (max-min)/N; 
+		    for (int i = 0; i < N; i++) {
+		        logimagetable[i] = Math.log10(min + i*bin);
+		    }
+		}
+	
+
+		private double approximateLogLookup(double x) {
+//		    int index = (int) (x * 10000);
+			if (x>MAX_LOG_LOOKUP) {
+				//System.out.println("X>LOG: " + x); 
+				return Math.log10(x);
+			}
+			
+			double bin = (MAX_LOG_LOOKUP-0.001)/logimagetable.length; 
+
+		    //int index  = PamArrayUtils.findClosest(logimagetable, x);
+			
+			int index = (int) Math.round((x -  0.001)/bin);
+			
+			index = index < 0 ? 0 : index;
+			index = index >=logimagetable.length ? (logimagetable.length-1): index; 
+			
+		    //System.out.println("Index: " + index + " x: "  + x + " " + minMaxVal[0] + "  " + minMaxVal[1]);
+		    return logimagetable[index];
+		}
+
+		
+		
+//		public double approximateLog(double x) {
+//		    if (x <= 0) {
+//		        return 0;
+//		    }
+//
+//		    double result = 0;
+//		    double term = x;
+//		    double denominator = 1;
+//
+//		    int count=0;
+//		    while (Math.abs(term) > 1e-3 && count<10000) {
+//		        result += term;
+//		        term *= -x;
+//		        denominator++;
+//		        term /= denominator;
+//		        count++;
+//		    }
+//
+//		    return result;
+//		}
+		
 		public void drawDataRate(GraphicsContext g,
 				OfflineDataMap offlineDataMap, Color dataColour) {
 			OfflineDataMapPoint mapPoint;
@@ -749,8 +857,8 @@ public class DataStreamPaneFX extends PamBorderPane {
 		 */
 		public double getYCoord(double count, long itemDuration) {
 			double plotHeight = getHeight();
-			boolean logScale = dataMapControl.dataMapParameters.vLogScale;
+			boolean logScale = scrollingDataPanel.getDataMapParams().vLogScale;
-			int scaleType = dataMapControl.dataMapParameters.vScaleChoice;
+			int scaleType = scrollingDataPanel.getDataMapParams().vScaleChoice;
 			double value = OfflineDataMap.scaleData(count, itemDuration, scaleType);
 			if (logScale) {
 				if (value <= 0) {
@@ -860,7 +968,7 @@ public class DataStreamPaneFX extends PamBorderPane {
 
 
 			yScaleMin = 0;
-			yScaleMax = highestMap.getHighestPoint(dataMapControl.dataMapParameters.vScaleChoice);
+			yScaleMax = highestMap.getHighestPoint(scrollingDataPanel.getDataMapParams().vScaleChoice);
 			if (dataMapControl.dataMapParameters.vLogScale) {
 				yScaleMin = highestMap.getLowestNonZeroPoint(dataMapControl.dataMapParameters.vScaleChoice);
 				//			if (yScaleMin > 0) {
@@ -990,19 +1098,6 @@ public class DataStreamPaneFX extends PamBorderPane {
 		return (long) (xPos / pixelsPerMilli) + scrollingDataPanel.getScreenStartMillis();
 	}
 	
-	public class DataName {
-
-		private String name;
-
-		public String getName() {
-			return name;
-		}
-
-		public void setName(String dataName) {
-			this.name=dataName;
-		}
-		
-	}
 	
 	/**
 	 * @return the yScaleMin
@@ -1060,7 +1155,7 @@ public class DataStreamPaneFX extends PamBorderPane {
 	 * @return
 	 */
 	private DatagramScaleInformation findDatagramScaleInfo() {
-		if (hasDatagram == false) {
+		if (isHasDatagram() == false) {
 			return null;
 		}
 		DatagramProvider dp = dataBlock.getDatagramProvider();
@@ -1129,4 +1224,52 @@ public class DataStreamPaneFX extends PamBorderPane {
 	}
 
 	
+	/**
+	 * Set the colour array for the data stream panel. 
+	 * @param colourArrayType
+	 */
+	public void setColourArrayType(ColourArrayType colourArrayType) {
+		dataGraph.plotColours2D.setColourMap(colourArrayType);
+		this.repaint(0);
+	}
+
+	/**
+	 * Get the colour map for the datagram.
+	 * @return the colour map for the datagram.
+	 */
+	public ColourArrayType getColourMapArray() {
+		return dataGraph.plotColours2D.getColourMap();
+	}
+
+	/**
+	 * Set the minimum and maximum colour limits for the datagram
+	 * @param lowValue - the low colour limit in dB
+	 * @param highValue - the high colour limit in dB
+	 */
+	public void setMinMaxColourLimits(double lowValue, double highValue) {
+		dataGraph.plotColours2D .getAmplitudeLimits()[0].set(lowValue);
+		dataGraph.plotColours2D .getAmplitudeLimits()[1].set(highValue);
+		this.repaint(50);
+	}
+	
+	
+	/**
+	 * Get the minimum dB for the colour array
+	 * @return - the minimum dB for the colour array. 
+	 */
+	public double getMinColourLimit() {
+		return dataGraph.plotColours2D.getAmplitudeLimits()[0].get();
+	}
+	
+	
+	public double getMaxColourLimit() {
+		return dataGraph.plotColours2D .getAmplitudeLimits()[1].get();
+	}
+
+	public boolean isHasDatagram() {
+		return hasDatagram;
+	}
+
+
+
 }

src/dataMap/layoutFX/ScrollingDataPaneFX.java

@@ -11,18 +11,20 @@ import javafx.geometry.Insets;
 import javafx.geometry.Side;
 import javafx.scene.Node;
 import javafx.scene.control.Label;
-import javafx.scene.control.ScrollBar;
 import javafx.scene.control.ScrollPane.ScrollBarPolicy;
 import javafx.scene.paint.Color;
 import javafx.util.Duration;
 import PamguardMVC.PamDataBlock;
 import dataMap.DataMapControl;
+import pamViewFX.PamGuiManagerFX;
+import pamViewFX.fxGlyphs.PamGlyphDude;
 import pamViewFX.fxNodes.PamBorderPane;
+import pamViewFX.fxNodes.PamButton;
 import pamViewFX.fxNodes.PamColorsFX;
 import pamViewFX.fxNodes.PamScrollPane;
 import pamViewFX.fxNodes.PamVBox;
 import pamViewFX.fxNodes.pamAxis.PamDateAxis;
-import pamViewFX.fxNodes.pamScrollers.acousticScroller.ScrollBarPane;
+
 
 public class ScrollingDataPaneFX extends PamBorderPane {
 
@@ -37,6 +39,8 @@ public class ScrollingDataPaneFX extends PamBorderPane {
 	 */
 	private static final int DATASTREAMPANE_HEIGHT = 220;
 
+	private static final double SCROLL_BAR_INCREMENT = 0.05; //10% movement when scroll bnutton pressed
+
 	/**
 	 * Reference to the DataMapControl.
 	 */
@@ -58,7 +62,7 @@ public class ScrollingDataPaneFX extends PamBorderPane {
 	private ArrayList<DataStreamPaneFX> dataStreamPanels = new ArrayList<DataStreamPaneFX>();
 
 	/**
-	 * Split pane whihc holds different graphs. 
+	 * Split pane which holds different graphs. 
 	 */
 	private PamVBox dataPanePanes;
 
@@ -230,9 +234,46 @@ public class ScrollingDataPaneFX extends PamBorderPane {
 		holder.setCenter(timeScrollBar);
 		holder.setBottom(timeLabelPane);
 		
+		PamButton scrollLeft = new PamButton(); 
+		scrollLeft.setOnAction((a)->{
+			 /*
+			  * The distance is a percentage of the
+			 * visible distance. So 1. means the scroll bar will move by the visible
+			 * distance. 0.5 means half the visible distance and so on. Negative means move
+			 * left and positive means move right/
+			 */
+			moveScrollBar(-timeScrollBar.getVisibleAmount()*SCROLL_BAR_INCREMENT);
+		});
+		scrollLeft.setGraphic(PamGlyphDude.createPamIcon("mdi2c-chevron-left", PamGuiManagerFX.iconSize));
+		scrollLeft.prefHeightProperty().bind(timeScrollBar.heightProperty());
+
+		PamButton scrollRight = new PamButton(); 
+		scrollRight.setGraphic(PamGlyphDude.createPamIcon("mdi2c-chevron-right", PamGuiManagerFX.iconSize));
+		scrollRight.setOnAction((a)->{
+			moveScrollBar(timeScrollBar.getVisibleAmount()*SCROLL_BAR_INCREMENT);
+		});
+		scrollRight.prefHeightProperty().bind(timeScrollBar.heightProperty());
+		
+		holder.setLeft(scrollLeft);
+		holder.setRight(scrollRight);
+
 		return holder; 
 	}
 
+	/**
+	 * Move the scroll bar by a a number fo seconds. Positive means move forward in
+	 * time, negative means move back in time.
+	 * 
+	 * @param the distance to move as a percentage of the visible amount
+	 */
+	private void moveScrollBar(double seconds) {
+		
+		double newValue =  timeScrollBar.getCurrentValue()+seconds;
+	
+		//this should trigger all relevant repaint listeners etc. 
+		timeScrollBar.setCurrentValue(newValue);
+	}
+
 	/**
 	 * Calculate the start and millis based on scroll position and screen seconds. 
 	 */
@@ -262,10 +303,11 @@ public class ScrollingDataPaneFX extends PamBorderPane {
 	public void notifyScrollChange() {
 		// tell all panlettes to repaint. 
 		for (int i = 0; i < dataStreamPanels.size(); i++) {
+			if (!dataStreamPanels.get(i).isCollapsed()) {
 				dataStreamPanels.get(i).scrollChanged();
 			}
+		}
 //		settingsStrip.scrollChanged();
-
 		updateDateAxis();
 	}
 
@@ -442,9 +484,7 @@ public class ScrollingDataPaneFX extends PamBorderPane {
 	public double getPixelsPerHour() {
 //		System.out.println("Pixels per hour: " + dataMapControl.dataMapParameters.getPixeslPerHour() + " " + this.getPlotWidth()/(this.timeScrollBar.getVisibleAmount()/1000./3600.));
 		//return dataMapControl.dataMapParameters.getPixeslPerHour();
-
 		return this.getPlotWidth()/(this.timeScrollBar.getVisibleAmount()/1000./3600.);
-
 	}
 
 	/**
@@ -487,14 +527,9 @@ public class ScrollingDataPaneFX extends PamBorderPane {
 		return dataMapPaneFX;
 	}
 
-//	int lastHScaleChoice=-1; 
+	public void scaleChange() {
-//	public void scaleChange() {
+		this.notifyScrollChange();
-//		if (lastHScaleChoice != dataMapControl.dataMapParameters.hScaleChoice) {
+	}
-//			lastHScaleChoice = dataMapControl.dataMapParameters.hScaleChoice;
-//			setupScrollBar();
-//		}
-//		this.notifyScrollChange();
-//	}
 
 	/**
 	 * Get the current number of data stream panes
@@ -509,11 +544,32 @@ public class ScrollingDataPaneFX extends PamBorderPane {
 	 * @param n - the index of the data stream pane
 	 * @return the data stream pane or null if the index is out of bounds. 
 	 */
-	public DataStreamPaneFX getDataSyreamPane(int n) {
+	public DataStreamPaneFX getDataStreamPane(int n) {
-		if (n<this.dataStreamPanels.size()) {
+		if (n>=0 && n<this.dataStreamPanels.size()) {
 			return dataStreamPanels.get(n);
 		}
 		else return null;
 	}
 
+	public DataStreamPaneFX getDataStreamPane(DataMapInfo selectedItem) {
+		if (selectedItem==null) return null;
+		for (int i=0; i<dataStreamPanels.size(); i++) {
+			if (selectedItem.equals(dataStreamPanels.get(i).getDataName())) {
+				return dataStreamPanels.get(i);
+			}
+		}
+		return null;
+	}
+
+	/**
+	 * Get the data map parameters associated with the FX GUI. Note these are
+	 * separate from the parameters in the DataMapControls which are for the default
+	 * swing display (not great)
+	 * 
+	 * @return the current data map parameters.
+	 */
+	public DataMapParametersFX getDataMapParams() {
+		return this.dataMapPaneFX.getDataMapParams();
+	}
+	
 }

src/dataModelFX/DataMapParametersFX.java

@@ -0,0 +1,40 @@
+package dataModelFX;
+
+import java.io.Serializable;
+
+import PamModel.parametermanager.ManagedParameters;
+import PamModel.parametermanager.PamParameterSet;
+import PamModel.parametermanager.PamParameterSet.ParameterSetType;
+import dataMap.OfflineDataMap;
+
+public class DataMapParametersFX implements Cloneable, Serializable, ManagedParameters {
+
+	protected static final long serialVersionUID = 1L;
+	
+	public int vScaleChoice = OfflineDataMap.SCALE_PERHOUR;
+	
+	public boolean vLogScale = true;
+	
+	/*
+	 * Scale factor for horizontal axis. 
+	 */
+	public int hScaleChoice = 4;
+	
+
+	@Override
+	protected DataMapParametersFX clone() {
+		try {
+			return (DataMapParametersFX) super.clone();
+		} catch (CloneNotSupportedException e) {
+			e.printStackTrace();
+			return null;
+		}
+	}
+
+	@Override
+	public PamParameterSet getParameterSet() {
+		PamParameterSet ps = PamParameterSet.autoGenerate(this, ParameterSetType.DISPLAY);
+		return ps;
+	}
+
+}

src/dataPlotsFX/TDManagedSymbolChooserFX.java

@@ -55,6 +55,7 @@ public class TDManagedSymbolChooserFX implements TDSymbolChooserFX{
 		else if (type==TDSymbolChooserFX.HIGHLIGHT_SYMBOL) {
 			symbol=ClickSymbolChooserFX.highLightClick;
 		}
+		
 		return symbol; 
 	}
 	

src/dataPlotsFX/data/generic/GenericDataPlotInfo.java

@@ -366,6 +366,7 @@ public class GenericDataPlotInfo extends TDDataInfoFX {
 	
 	protected void setDefaultOpacity(ParameterType dataType) {
 		if (dataType.equals(ParameterType.FREQUENCY)) {
+			//we set the default opacity because often frequency is shown as a box on a spectrogram. 
 			this.genericSettingsPane.setDefaultFillOpacity(DEFAULT_FILL_OPACITY);
 		}
 		else {

src/dataPlotsFX/layout/TDControlPaneFX.java

@@ -110,7 +110,7 @@ public class TDControlPaneFX extends Pane  {
 		this.tdDisplay=tdMainDsiplay;
 
 		//CSS styling- pane has the standard PAMGUARD settings pane styling. 		  
-		this.getStylesheets().addAll(tdMainDsiplay.getCSSSettingsResource());
+		//this.getStylesheets().addAll(tdMainDsiplay.getCSSSettingsResource());
 		this.getStyleClass().add("pane");
 
 		PamBorderPane mainPane=new PamBorderPane();

src/dataPlotsFX/scrollingPlot2D/PlotParams2D.java

@@ -25,6 +25,11 @@ public class PlotParams2D implements Serializable, Cloneable, ManagedParameters
 	 */
 	public double[] maxAmplitudeLimits={30,180}; 
 	
+	/**
+	 * Limits of amplitude range- this is a double property which can be bound to colour sliders.
+	 */
+	private transient DoubleProperty[] amplitudeLimits= {new SimpleDoubleProperty(amplitudeLimitsSerial[0]), new SimpleDoubleProperty(amplitudeLimitsSerial[1])};
+	
 
 	/**
 	 * @return the amplitudeLimitsSerial
@@ -62,11 +67,6 @@ public class PlotParams2D implements Serializable, Cloneable, ManagedParameters
 		return amplitudeLimits;
 	}
 
-	/**
-	 * Limits of amplitude range- this is a double property which can be bound to colour sliders.
-	 */
-	private transient DoubleProperty[] amplitudeLimits= {new SimpleDoubleProperty(amplitudeLimitsSerial[0]), new SimpleDoubleProperty(amplitudeLimitsSerial[1])};
-	
 	public PlotParams2D() {
 		super();
 		//make sure the amplitude limits are set to the correct air/water to begin with. 

src/dataPlotsFX/scrollingPlot2D/StandardPlot2DColours.java

@@ -15,9 +15,12 @@ import pamViewFX.fxNodes.utilsFX.ColourArray.ColourArrayType;
  */
 public class StandardPlot2DColours implements Plot2DColours {
 	
+	private static final int NCOLOUR_DEFUALT = 256;
+
 	private ColourArrayType colourArrayType=ColourArrayType.HOT; 
 	
-	private Color[] colourArray=ColourArray.createStandardColourArray(256, colourArrayType=ColourArrayType.HOT).getColours();
+	private Color[] colourArray=ColourArray.createStandardColourArray(NCOLOUR_DEFUALT, colourArrayType=ColourArrayType.HOT).getColours();
+	
 	
 	/**
 	 * The amplitude limits 
@@ -118,12 +121,24 @@ public class StandardPlot2DColours implements Plot2DColours {
 	
 	/**
 	 * Set the current colour map. 
-	 * @param colourMap
+	 * @param colourMap - the colour map to set.
 	 */
 	public void setColourMap(ColourArrayType colourMap) {
 		this.colourArrayType=colourMap; 
 		if (spectParams!=null) spectParams.setColourMap(colourMap);
-		ColourArray colourArray = ColourArray.createStandardColourArray(256, colourArrayType);
+		ColourArray colourArray = ColourArray.createStandardColourArray(NCOLOUR_DEFUALT, colourArrayType);
+		this.colourArray=colourArray.getColours();
+	}
+
+	/**
+	 * Set the current colour map. 
+	 * @param ncolours - the number of colours in the colour map. 
+	 * @param colourMap - the colour map to set.
+	 */
+	public void setColourMap(ColourArrayType colourMap, int ncolours) {
+		this.colourArrayType=colourMap; 
+		if (spectParams!=null) spectParams.setColourMap(colourMap);
+		ColourArray colourArray = ColourArray.createStandardColourArray(ncolours, colourArrayType);
 		this.colourArray=colourArray.getColours();
 	}
 
@@ -143,4 +158,5 @@ public class StandardPlot2DColours implements Plot2DColours {
 	public  DoubleProperty[]  getAmplitudeLimits() {
 		return this.amplitudeLimits;
 	}
+
 }

src/detectionPlotFX/DetectionGroupDisplayFX.java

@@ -77,6 +77,7 @@ public class DetectionGroupDisplayFX extends DetectionGroupDisplay  implements U
 	@Override
 	public void closeNode() {}
 	
+
 	@Override
 	public DetectionPlotParams getDisplayParams() {		
 
@@ -98,8 +99,17 @@ public class DetectionGroupDisplayFX extends DetectionGroupDisplay  implements U
 			detectionPlotParams.sizeX=internalFrame.getInternalRegion().getWidth();
 			detectionPlotParams.sizeY=internalFrame.getInternalRegion().getHeight();
 		}
+
+		if (this.getDetectionDisplay().getCurrentDataInfo()!=null) {
+			//save the axis so the same axis appear when PAMGuard is restrated. 
+			String detectionPlotName = 	this.getDetectionDisplay().getCurrentDataInfo().getCurrentDetectionPlot().getName();
+			detectionPlotParams.dataAxisMap.put(detectionPlotParams.dataSource, detectionPlotName);
 		}
 
+	}
+
+
+
 
 
 	@Override
@@ -118,7 +128,6 @@ public class DetectionGroupDisplayFX extends DetectionGroupDisplay  implements U
 
 	private void showSettingsPane(boolean b) {
 		this.detectionDisplay.getHidingPane(Side.RIGHT).showHidePane(b);
-		
 	}
 
 	@Override
@@ -162,6 +171,7 @@ public class DetectionGroupDisplayFX extends DetectionGroupDisplay  implements U
 
 		//		System.out.println("SAVE DETECTION DISPLAY DATA SOURCE: " + detectionPlotParams.dataSource);
 		//		System.out.println("SAVE DETECTION DISPLAY TAB NAME: " + detectionPlotParams.tabName);
+		//		System.out.println("SAVE DETECTION DISPLAY DATA  AXIS: " + detectionPlotParams.dataAxisMap);
 
 		return detectionPlotParams;	
 	}
@@ -192,10 +202,10 @@ public class DetectionGroupDisplayFX extends DetectionGroupDisplay  implements U
 
 		//		System.out.println("LOAD DETECTION DISPLAY DATA SOURCE: " + settings.dataSource);
 		//		System.out.println("LOAD DETECTION DISPLAY DATA SOURCE: " + settings.tabName);
+		//		System.out.println("LOAD DETECTION DISPLAY DATA  AXIS: " + detectionPlotParams.dataAxisMap);
 
 		this.detectionPlotParams = settings.clone();	
 
-		
 		this.setEnableScrollBar(detectionPlotParams.showScrollBar);
 
 		return true;

src/detectionPlotFX/layout/DetectionPlotDisplay.java

@@ -385,7 +385,7 @@ public class DetectionPlotDisplay extends PamBorderPane  {
 	
 		}
 		else {
-			System.out.println("Min scroll limit: " + detectionPlotProjector.getMinScrollLimit() + "max: " + detectionPlotProjector.getMaxScrollLimit());
+//			System.out.println("Min scroll limit: " + detectionPlotProjector.getMinScrollLimit() + "max: " + detectionPlotProjector.getMaxScrollLimit());
 			//need this to ensure the axis change when scroll bar is not longer displayed. 
 			detectionPlotProjector.setAxisMinMax(detectionPlotProjector.getMinScrollLimit(), 
 					detectionPlotProjector.getMaxScrollLimit(), detectionPlotProjector.getScrollAxis());

src/detectionPlotFX/plots/RawFFTPlot.java

@@ -155,7 +155,7 @@ public abstract class RawFFTPlot<D extends PamDataUnit> extends FFTPlot<D> {
 		}
 		else {
 			//repaint the image!!
-			System.out.println("PAINT the image for: " +dataUnit.getUID());
+			//System.out.println("PAINT the image for: " +dataUnit.getUID());
 			if (detectionPlotDisplay.isViewer()) paintSpecImage(graphicsContext,  rectangle, projector);
 			paintDetections(dataUnit,  graphicsContext,  rectangle, projector) ;
 		}

src/export/MLExport/MLAnnotationsManager.java

@@ -1,11 +1,34 @@
 package export.MLExport;
 
+import java.util.List;
+
+import org.jamdev.jdl4pam.utils.DLMatFile;
+
+import PamUtils.PamArrayUtils;
+import PamguardMVC.PamDataBlock;
 import PamguardMVC.PamDataUnit;
+import annotation.DataAnnotation;
 import annotation.DataAnnotationType;
+import bearinglocaliser.annotation.BearingAnnotation;
+import bearinglocaliser.annotation.BearingAnnotationType;
+import clickDetector.ClickClassifiers.annotation.ClickClassificationType;
+import clickDetector.ClickClassifiers.annotation.ClickClassifierAnnotation;
+import matchedTemplateClassifer.annotation.MatchedClickAnnotation;
+import matchedTemplateClassifer.annotation.MatchedClickAnnotationType;
+import rawDeepLearningClassifier.logging.DLAnnotation;
+import rawDeepLearningClassifier.logging.DLAnnotationType;
+import us.hebi.matlab.mat.format.Mat5;
+import us.hebi.matlab.mat.types.Array;
+import us.hebi.matlab.mat.types.Matrix;
 import us.hebi.matlab.mat.types.Struct;
 
 /**
- * Adds annotations to data units. 
+ * Adds annotations to data units. Annotations need to be managed differently. 
+ * 
+ * There can be different annotation for different data units and data units of the same
+ * type won't necessarily have the same annotations. Therefore the annotation types are 
+ * used to figure out which type of annotations are available and these are added
+ * to the structure irrespective if whether there is data or not. 
  * 
  * @author Jamie Macaulay 
  *
@@ -14,20 +37,221 @@ public class MLAnnotationsManager {
 
 	/**
 	 * Add annotations to a MATLAB structure. 
-	 * @param mlStruct
+	 * @param mlStruct - the structure to add to.
-	 * @param index 
+	 * @param index  - index of the struct .
-	 * @param dataUnit
+	 * @param dataUnit - the data unit with annotations to add. 
 	 */
 	public void addAnnotations(Struct mlStruct, int index, PamDataUnit dataUnit) {
-		for (int i=0; i<dataUnit.getNumDataAnnotations(); i++) {
+
-			DataAnnotationType type = dataUnit.getDataAnnotation(i).getDataAnnotationType(); 
+		//first we need to find out which data annotation the data unit data block has - need to ensure we add all the annotation
-			//would be better to have the mlstruct updated by the annotation type but don't
+		//even if they don't exist- otherwise we may get dissimilar structure exceptions. 
-			//want to be spreading MATLAB code throughout PAMGuard so keep here for now. 
+		PamDataBlock parentblock = dataUnit.getParentDataBlock();
-			//TODO
+
+		DataAnnotationType annotType;
+		List<DataAnnotationType<?>> annotationTypes = parentblock.getAnnotationHandler().getAvailableAnnotationTypes();
+
+
+		for (int i=0; i<parentblock.getAnnotationHandler().getNumUsedAnnotationTypes(); i++) {
+
+			annotType = annotationTypes.get(i);
+
+			//now iterate through the data annotations within the data unit and find the data annotation	
+			//Maybe not necessary but much safer than assuming data type list is same as annotatio list. 
+			DataAnnotation dataAnnotation;
+			for (int j=0; j<dataUnit.getNumDataAnnotations(); j++) {
+				dataAnnotation=  dataUnit.getDataAnnotation(j); 
+				if (dataAnnotation.getDataAnnotationType().getAnnotationName().equals(annotType.getAnnotationName())){
+					//add the annotation even if it null
+					addAnnotations( mlStruct,  index,  dataUnit,  dataAnnotation, annotType);
+					break;
+				} 
+			}
+
 		}; 
 
 	}
 
+	/**
+	 * Get the name for the annotation i.e. the name to be used in the struct. 
+	 * @param annotationType - the annotation type
+	 * @return the anme to be used in the struct for the annotation
+	 */
+	public static String getAnnotationNameMAT( DataAnnotationType annotationType) {
+
+		String name = null;
+		switch (annotationType.getAnnotationName()){
+
+		case BearingAnnotationType.NAME:
+			name="bearing";
+			break;
+
+		case ClickClassificationType.NAME:
+			name="classification";
+			break;
+
+		case MatchedClickAnnotationType.NAME:
+			name="mclassification";
+			break;
+
+		case DLAnnotationType.NAME:
+			name="dlclassification";
+			break;
+		}
+
+		return name;
+
+	}
+
+	/**
+	 * Add an annotation to an existing MATLAB structure. 
+	 * @param mlStruct
+	 * @param index
+	 * @param dataUnit
+	 * @param dataAnnotation
+	 * @param annotationType
+	 */
+	public void addAnnotations(Struct mlStruct, int index, PamDataUnit dataUnit, DataAnnotation dataAnnotation, DataAnnotationType annotationType) {
+
+
+		//if we don't have an annotation
+		if (dataAnnotation == null) {
+			mlStruct.set(getAnnotationNameMAT(annotationType), index, Mat5.newMatrix(0, 0));
+			return;
+		}
+
+		Array annotation =null;
+		switch (dataAnnotation.getDataAnnotationType().getAnnotationName()){
+
+		case BearingAnnotationType.NAME:
+			BearingAnnotation bearingAnnotation = (BearingAnnotation) dataAnnotation;
+			annotation = bearingAnnotation2MAT(bearingAnnotation);
+			break;
+
+		case ClickClassificationType.NAME:
+			ClickClassifierAnnotation clkClassifierAnnotation = (ClickClassifierAnnotation) dataAnnotation;
+			annotation = clkClassification2MAT(clkClassifierAnnotation);
+			break;
+
+		case MatchedClickAnnotationType.NAME:
+			MatchedClickAnnotation matchAnnotation = (MatchedClickAnnotation) dataAnnotation;
+			annotation = matchAnnotation2MAT(matchAnnotation);
+			break;
+
+		case DLAnnotationType.NAME:
+			DLAnnotation dlAnnotation = (DLAnnotation) dataAnnotation;
+			annotation = dlAnnoation2MAT(dlAnnotation);
+			break;
+
+		default:
+			System.out.println("MLAnnotationsManager: Annotation: " + dataAnnotation.getDataAnnotationType().getAnnotationName() 
+					+ " for " + dataUnit + " not yet supported: ");
+			return;
+
+		}
+		
+		mlStruct.set(getAnnotationNameMAT(annotationType),index,annotation);
+
+
+	}
+
+
+	private Array matchAnnotation2MAT(MatchedClickAnnotation matchAnnotation) {
+		//now write the matched template classifier results. Results form each template are written. 
+		double threshold;
+		double matchCorr;
+		double rejectCorr; 
+
+		double[][] macthAnnotationM = new double[matchAnnotation.getMatchedTemplateResult().size()][];
+
+		for (int i = 0; i<matchAnnotation.getMatchedTemplateResult().size(); i++) {
+			threshold  = matchAnnotation.getMatchedTemplateResult().get(i).threshold;
+			matchCorr  = matchAnnotation.getMatchedTemplateResult().get(i).matchCorr;
+			rejectCorr = matchAnnotation.getMatchedTemplateResult().get(i).rejectCorr;
+
+			macthAnnotationM[i] = new double[] {threshold, matchCorr, rejectCorr};
+
+		}
+		return DLMatFile.array2Matrix(macthAnnotationM);
+	}
+
+
+	private Struct bearingAnnotation2MAT(BearingAnnotation bearingAnnotation) {
+
+
+		//extract the data needed for the bearing annotation
+		int hydrophones  = bearingAnnotation.getBearingLocalisation().getReferenceHydrophones(); 
+		int arrayType  = bearingAnnotation.getBearingLocalisation().getSubArrayType();
+		int localisationContent  = bearingAnnotation.getBearingLocalisation().getLocContents().getLocContent();
+		int nAngles  = bearingAnnotation.getBearingLocalisation().getAngles().length;
+		double[] angles  = bearingAnnotation.getBearingLocalisation().getAngles();
+		int nErrors  = bearingAnnotation.getBearingLocalisation().getAngleErrors().length;
+		double[] errors  = bearingAnnotation.getBearingLocalisation().getAngleErrors();
+		double[] refAngles =  bearingAnnotation.getBearingLocalisation().getReferenceAngles();
+
+		Struct bearingStruct = Mat5.newStruct(); 
+
+		bearingStruct.set("hydrophones", Mat5.newScalar(hydrophones));
+		bearingStruct.set("arrayType", Mat5.newScalar(arrayType));
+		bearingStruct.set("localisationContent", Mat5.newScalar(localisationContent));
+		bearingStruct.set("nAngles", Mat5.newScalar(nAngles));
+		bearingStruct.set("nErrors", Mat5.newScalar(nErrors));
+
+		bearingStruct.set("angles", DLMatFile.array2Matrix(angles));
+		bearingStruct.set("errors", DLMatFile.array2Matrix(errors));
+		bearingStruct.set("refAngles", DLMatFile.array2Matrix(refAngles));
+
+		//		for (int i=0; i<predictions.length; i++) {
+		//			predictions[i] = dlAnnotation.getModelResults().get(i).getPrediction();
+		//			decision[i] = dlAnnotation.getModelResults().get(i).isBinaryClassification();
+		//		}
+		//		
+		//		Struct dlAnnotationMat = Mat5.newStruct(); 
+		//		
+		//		dlAnnotationMat.set("predictions", DLMatFile.array2Matrix(PamArrayUtils.float2Double(predictions)));
+		//		
+		//		Matrix matrix = Mat5.newMatrix(decision.length, 1);
+		//		for (int i=0; i<decision.length; i++) {
+		//				matrix.setBoolean(i, decision[i]);
+		//		}
+		//		dlAnnotationMat.set("isdecision", matrix);
+
+		return bearingStruct;
+	}
+
+	private Struct clkClassification2MAT(ClickClassifierAnnotation clkClassifyAnnot) {
+
+		Struct clkclsfrAnnotation = Mat5.newStruct(); 
+		clkclsfrAnnotation.set("classify_set", DLMatFile.array2Matrix(clkClassifyAnnot.getClassiferSet()));
+
+		return clkclsfrAnnotation; 
+
+	}
+
+
+
+	private Struct dlAnnoation2MAT(DLAnnotation dlAnnotation) {
+		float[][] predictions = new float[dlAnnotation.getModelResults().size()][];
+		boolean[] decision = new boolean[dlAnnotation.getModelResults().size()];
+
+		for (int i=0; i<predictions.length; i++) {
+			predictions[i] = dlAnnotation.getModelResults().get(i).getPrediction();
+			decision[i] = dlAnnotation.getModelResults().get(i).isBinaryClassification();
+		}
+
+		Struct dlAnnotationMat = Mat5.newStruct(); 
+
+		dlAnnotationMat.set("predictions", DLMatFile.array2Matrix(PamArrayUtils.float2Double(predictions)));
+
+		Matrix matrix = Mat5.newMatrix(decision.length, 1);
+		for (int i=0; i<decision.length; i++) {
+			matrix.setBoolean(i, decision[i]);
+		}
+		dlAnnotationMat.set("isdecision", matrix);
+
+		return dlAnnotationMat;
+	}
+
+
 
 
 

src/export/MLExport/MLCPODExport.java

@@ -0,0 +1,114 @@
+package export.MLExport;
+
+import org.jamdev.jdl4pam.utils.DLMatFile;
+
+import PamguardMVC.PamDataUnit;
+import cpod.CPODClick;
+import cpod.CPODClickTrainDataUnit;
+import cpod.CPODClassification.CPODSpeciesType;
+import us.hebi.matlab.mat.format.Mat5;
+import us.hebi.matlab.mat.types.Matrix;
+import us.hebi.matlab.mat.types.Struct;
+
+
+/**
+ * Export data for any data unit which implements raw data holder. 
+ * @author Jamie Macaulay
+ *
+ */
+public class MLCPODExport extends MLDataUnitExport<PamDataUnit>{
+
+	@Override
+	public Struct addDetectionSpecificFields(Struct mlStruct, int index, PamDataUnit dataUnit) {
+
+		if (dataUnit instanceof CPODClickTrainDataUnit) {
+			System.out.println("CPOD click train datablock!!"); 
+		}
+
+		CPODClick fpodClick = (CPODClick) dataUnit; 
+
+		Matrix wave =  Mat5.newMatrix(new int[] {0,0});
+		if (fpodClick.getWaveData()!=null) {
+			//the waveform
+			wave = DLMatFile.array2Matrix(fpodClick.getWaveData());
+		}
+
+		//the duration - repeat of duration in main data unit. Keeping here so strcut is the same as the struct from binary files 
+		Matrix duration = Mat5.newScalar(dataUnit.getDurationInMilliseconds()); 
+
+		mlStruct.set("durationmillis",index, duration);
+		mlStruct.set("wave", index, wave);
+
+		//		Matrix duration = Mat5.newScalar(fpodClick.getDurationInMilliseconds()); 
+
+		Matrix nCyc = Mat5.newScalar(fpodClick.getnCyc()); 
+		Matrix bw = Mat5.newScalar(fpodClick.getBw()*1000.); 
+		Matrix peakfreq = Mat5.newScalar(fpodClick.getkHz()*1000.); 
+		Matrix endF = Mat5.newScalar(fpodClick.getEndF()*1000.); 
+		Matrix spl = Mat5.newScalar(fpodClick.getSpl()); 
+		Matrix slope = Mat5.newScalar(fpodClick.getSlope()); 
+
+		mlStruct.set("numcycles",index, nCyc);
+		mlStruct.set("bandwidth", index, bw);
+		mlStruct.set("peakfreq", index, peakfreq);
+		mlStruct.set("endfreq", index, endF);
+		mlStruct.set("SPL", index, spl);
+		mlStruct.set("slope", index, slope);
+
+		short species = -1;
+		long clicktrainID = -1;
+
+		if (fpodClick.getCPODClickTrain()!=null) {
+			species =  getCPODSpecies(fpodClick.getCPODClickTrain().getSpecies());
+			clicktrainID = fpodClick.getCPODClickTrain().getUID();
+		}
+
+		
+		Matrix speciesm = Mat5.newScalar(species); 
+		Matrix clicktrainIDm = Mat5.newScalar(clicktrainID); 
+		
+		mlStruct.set("species", index, speciesm);
+		mlStruct.set("clcikTrainID", index, clicktrainIDm);
+		return mlStruct;
+	}
+
+	/**
+	 * Get the CPOD species from an int flag 
+	 * @param species - integer flag representing the species
+	 * @return the ENUM species type. 
+	 */
+	public static short getCPODSpecies(CPODSpeciesType species) {
+		short type= -1;
+		switch (species) {
+		case UNKNOWN:
+			type = 0;
+			break;
+		case NBHF:
+			type = 1;
+			break;
+		case  DOLPHIN:
+			type =2;
+			break;
+		case SONAR:
+			type = 4;
+			break;
+		default:
+			break;
+		}
+		return type;
+	}
+
+	@Override
+	public Class<?> getUnitClass() {
+		return CPODClick.class;
+	}
+
+	@Override
+	public String getName() {
+		return "raw_data_units";
+	}
+
+}
+
+
+

src/export/MLExport/MLDetectionsManager.java

@@ -49,6 +49,7 @@ public class MLDetectionsManager implements PamDataUnitExporter {
 	public MLDetectionsManager(){
 		mlDataUnitsExport.add(new MLClickExport()); 
 		mlDataUnitsExport.add(new MLWhistleMoanExport()); 
+		mlDataUnitsExport.add(new MLCPODExport()); 
 		mlDataUnitsExport.add(new MLRawExport()); 
 	}
 
@@ -56,8 +57,10 @@ public class MLDetectionsManager implements PamDataUnitExporter {
 	public boolean hasCompatibleUnits(Class dataUnitType) {
 		for (int i=0; i<mlDataUnitsExport.size(); i++){
 			//check whether the same. ;
-			//System.out.println(" dataUnits.get(j).getClass(): " + dataUnits.get(j).getClass());
+//			System.out.println("------");
-			//System.out.println(" mlDataUnitsExport.get(i).getUnitClass(): " + mlDataUnitsExport.get(i).getUnitClass());
+//			System.out.println(" dataUnitType " + dataUnitType);
+//			System.out.println(" mlDataUnitsExport.get(i).getUnitClass() " + mlDataUnitsExport.get(i).getUnitClass() 
+//					+ " " +  (mlDataUnitsExport.get(i).getUnitClass().isAssignableFrom(dataUnitType)));
 			if (mlDataUnitsExport.get(i).getUnitClass().isAssignableFrom(dataUnitType)) {
 				//System.out.println("FOUND THE DATA UNIT!");
 				return true; 
@@ -69,7 +72,7 @@ public class MLDetectionsManager implements PamDataUnitExporter {
 	@Override
 	public boolean exportData(File fileName, List<PamDataUnit> dataUnits, boolean append) {
 
-		System.out.println("Export: " + dataUnits.size() + " data units " + append);
+		System.out.println("Export: " + dataUnits.size() + " data units to mat append: " + append);
 
 		if (dataUnits==null || dataUnits.size()<1) {
 			//nothing to write but no error. 
@@ -90,7 +93,7 @@ public class MLDetectionsManager implements PamDataUnitExporter {
 			//is there an existing sink? Is that sink writing to the correct file?
 			if (sink==null || !fileName.equals(currentFile)) {
 
-				System.out.println("Export: " + dataUnitsStruct.getNumDimensions() + entryName);
+				//System.out.println("Export: " + dataUnitsStruct.getNumDimensions() + entryName);
 
 				currentFile = fileName;
 

src/export/MLExport/MLRawExport.java

@@ -28,7 +28,7 @@ public class MLRawExport extends MLDataUnitExport<PamDataUnit>{
 		//the number of channels
 		Matrix nChan = Mat5.newScalar(PamUtils.getNumChannels(dataUnit.getChannelBitmap())); 
 
-		//the duration - repeat of duration in main data unit. Keeping here so strcut is the same as the struct from binary files 
+		//the duration - repeat of duration in main data unit. Keeping here so struct is the same as the struct from binary files 
 		Matrix duration = Mat5.newScalar(dataUnit.getSampleDuration()); 
 
 		mlStruct.set("nChan", index, nChan);

src/export/RExport/RAnnotationManager.java

@@ -0,0 +1,191 @@
+package export.RExport;
+
+import java.util.List;
+
+import org.renjin.sexp.BooleanArrayVector;
+import org.renjin.sexp.DoubleArrayVector;
+import org.renjin.sexp.IntArrayVector;
+import org.renjin.sexp.ListVector;
+import org.renjin.sexp.Vector;
+import org.renjin.sexp.ListVector.NamedBuilder;
+
+import PamUtils.PamArrayUtils;
+import PamguardMVC.PamDataBlock;
+import PamguardMVC.PamDataUnit;
+import annotation.DataAnnotation;
+import annotation.DataAnnotationType;
+import bearinglocaliser.annotation.BearingAnnotation;
+import bearinglocaliser.annotation.BearingAnnotationType;
+import clickDetector.ClickClassifiers.annotation.ClickClassificationType;
+import clickDetector.ClickClassifiers.annotation.ClickClassifierAnnotation;
+import export.MLExport.MLAnnotationsManager;
+import matchedTemplateClassifer.annotation.MatchedClickAnnotation;
+import matchedTemplateClassifer.annotation.MatchedClickAnnotationType;
+import rawDeepLearningClassifier.logging.DLAnnotation;
+import rawDeepLearningClassifier.logging.DLAnnotationType;
+
+
+
+
+/**
+ * Adds data unit annotation to RData frame. 
+ . 
+ * @author Jamie Macaulay
+ *
+ */
+public class RAnnotationManager {
+
+
+	public NamedBuilder addDataAnnotations(NamedBuilder rData, PamDataUnit dataUnit, int index) {
+
+		//first we need to find out which data annotation the data unit data block has - need to ensure we add all the annotation
+		//even if they don't exist- otherwise we may get dissimilar structure exceptions. 
+		PamDataBlock parentblock = dataUnit.getParentDataBlock();
+
+		DataAnnotationType annotType;
+		List<DataAnnotationType<?>> annotationTypes = parentblock.getAnnotationHandler().getAvailableAnnotationTypes();
+
+		for (int i=0; i<parentblock.getAnnotationHandler().getNumUsedAnnotationTypes(); i++) {
+
+			annotType = annotationTypes.get(i);
+			//now iterate through the data annotations within the data unit and find the data annotation	
+			//Maybe not necessary but much safer than assuming data type list is same as annotatio list. 
+			DataAnnotation dataAnnotation;
+			for (int j=0; j<dataUnit.getNumDataAnnotations(); j++) {
+				dataAnnotation=  dataUnit.getDataAnnotation(j); 
+				if (dataAnnotation.getDataAnnotationType().getAnnotationName().equals(annotType.getAnnotationName())){
+					//add the annotation even if it null
+					//add the annotation even if it is null
+					addAnnotations( rData,  index,  dataUnit, dataAnnotation, annotType);
+					break;
+				}
+			}
+		}; 
+
+		return rData;
+
+	}
+
+
+	private void addAnnotations(NamedBuilder rData, int index, PamDataUnit dataUnit, DataAnnotation dataAnnotation,
+			DataAnnotationType annotType) {
+
+		NamedBuilder rDataAnnot = new ListVector.NamedBuilder();
+
+
+		//if we don't have an annotation
+		if (dataAnnotation == null) {
+			rData.add(MLAnnotationsManager.getAnnotationNameMAT(annotType), rDataAnnot);
+			return;
+		}
+
+		switch (dataAnnotation.getDataAnnotationType().getAnnotationName()){
+
+		case BearingAnnotationType.NAME:
+			BearingAnnotation bearingAnnotation = (BearingAnnotation) dataAnnotation;
+			bearingAnnotation2R(bearingAnnotation, rDataAnnot);
+			break;
+
+		case ClickClassificationType.NAME:
+			ClickClassifierAnnotation clkClassifierAnnotation = (ClickClassifierAnnotation) dataAnnotation;
+			clkClassification2R(clkClassifierAnnotation, rDataAnnot);
+			break;
+
+		case MatchedClickAnnotationType.NAME:
+			MatchedClickAnnotation matchAnnotation = (MatchedClickAnnotation) dataAnnotation;
+			matchAnnotation2R(matchAnnotation, rDataAnnot);
+			break;
+
+		case DLAnnotationType.NAME:
+			DLAnnotation dlAnnotation = (DLAnnotation) dataAnnotation;
+			dlAnnoation2R(dlAnnotation, rDataAnnot);
+			break;
+
+		default:
+			System.out.println("RAnnotationsManager: Annotation: " + dataAnnotation.getDataAnnotationType().getAnnotationName() 
+					+ " for " + dataUnit + " not yet supported: ");
+		}
+
+		rData.add(MLAnnotationsManager.getAnnotationNameMAT(annotType), rDataAnnot); 
+
+
+	}
+
+
+	/**
+	 * Add deep learning annotations to rData.
+	 * @param dlAnnotation - the deep learning annotation.
+	 * @param rData - rData frame to add to.
+	 */
+	private void dlAnnoation2R(DLAnnotation dlAnnotation, NamedBuilder rData) {
+
+		float[][] predictions = new float[dlAnnotation.getModelResults().size()][];
+		boolean[] decision = new boolean[dlAnnotation.getModelResults().size()];
+
+		for (int i=0; i<predictions.length; i++) {
+			predictions[i] = dlAnnotation.getModelResults().get(i).getPrediction();
+			decision[i] = dlAnnotation.getModelResults().get(i).isBinaryClassification();
+		}
+
+		Vector predictionsR = RDataUnitExport.doubleArr2R(PamArrayUtils.float2Double(predictions));
+		rData.add("predictions",predictionsR);
+
+		BooleanArrayVector decisionR =  BooleanArrayVector.unsafe(decision);
+		rData.add("isdecision",decisionR);
+		
+	}
+
+
+	private void matchAnnotation2R(MatchedClickAnnotation matchAnnotation, NamedBuilder rData) {
+		
+		//now write the matched template classifier results. Results form each template are written. 
+		double threshold;
+		double matchCorr;
+		double rejectCorr; 
+
+		double[][] macthAnnotationM = new double[matchAnnotation.getMatchedTemplateResult().size()][];
+
+		for (int i = 0; i<matchAnnotation.getMatchedTemplateResult().size(); i++) {
+			threshold  = matchAnnotation.getMatchedTemplateResult().get(i).threshold;
+			matchCorr  = matchAnnotation.getMatchedTemplateResult().get(i).matchCorr;
+			rejectCorr = matchAnnotation.getMatchedTemplateResult().get(i).rejectCorr;
+
+			macthAnnotationM[i] = new double[] {threshold, matchCorr, rejectCorr};
+		}
+		
+		Vector matchAnnotationsR = RDataUnitExport.doubleArr2R(macthAnnotationM);
+		
+		rData.add("matchcorr",matchAnnotationsR);
+
+	}
+
+
+	private void clkClassification2R(ClickClassifierAnnotation clkClassifierAnnotation, NamedBuilder rData) {
+		rData.add("classify_set",new IntArrayVector(clkClassifierAnnotation.getClassiferSet()));
+	}
+
+
+	private void bearingAnnotation2R(BearingAnnotation bearingAnnotation, NamedBuilder rData) {
+		//extract the data needed for the bearing annotation
+		int hydrophones  = bearingAnnotation.getBearingLocalisation().getReferenceHydrophones(); 
+		int arrayType  = bearingAnnotation.getBearingLocalisation().getSubArrayType();
+		int localisationContent  = bearingAnnotation.getBearingLocalisation().getLocContents().getLocContent();
+		int nAngles  = bearingAnnotation.getBearingLocalisation().getAngles().length;
+		double[] angles  = bearingAnnotation.getBearingLocalisation().getAngles();
+		int nErrors  = bearingAnnotation.getBearingLocalisation().getAngleErrors().length;
+		double[] errors  = bearingAnnotation.getBearingLocalisation().getAngleErrors();
+		double[] refAngles =  bearingAnnotation.getBearingLocalisation().getReferenceAngles();
+
+		rData.add("hydrophones", hydrophones);
+		rData.add("arrayType",arrayType);
+		rData.add("localisationContent", localisationContent);
+		rData.add("nAngles", nAngles);
+		rData.add("nErrors", nErrors);
+
+		rData.add("angles", DoubleArrayVector.unsafe(angles));
+		rData.add("errors", DoubleArrayVector.unsafe(errors));
+		rData.add("refAngles",   DoubleArrayVector.unsafe(refAngles));
+	}
+
+
+}

src/export/RExport/RCPODExport.java

@@ -0,0 +1,89 @@
+package export.RExport;
+
+import org.renjin.sexp.DoubleArrayVector;
+import org.renjin.sexp.Vector;
+import org.renjin.sexp.ListVector.NamedBuilder;
+
+import PamguardMVC.PamDataUnit;
+import PamguardMVC.RawDataHolder;
+import cpod.CPODClick;
+import export.MLExport.MLCPODExport;
+
+
+public class RCPODExport extends RDataUnitExport<PamDataUnit> {
+
+	@Override
+	public NamedBuilder addDetectionSpecificFields(NamedBuilder rData, PamDataUnit dataUnit, int index) {
+		// add click detection specific fields to rData units. 
+		//the waveform
+//		DoubleArrayVector[] wavefrom = new DoubleArrayVector[dataUnit.getWaveLength()]; 
+//		for (int i = 0; i<dataUnit.getWaveLength();i++) {
+//			wavefrom[i] = new DoubleArrayVector(dataUnit.getWaveData()[i]);
+//		}
+		
+		
+		CPODClick fpodClick = (CPODClick) dataUnit;
+		
+		if (fpodClick==null) return null; 
+
+
+		//add the raw wave data 
+		if (fpodClick.getWaveData()!=null && fpodClick.getWaveData().length>0) {
+			int nbins =fpodClick.getWaveData().length*fpodClick.getWaveData()[0].length;
+			int n=0;
+			double[] concatWaveform  = new double[nbins];
+			//System.out.println("Number of bins: " + nbins);
+			for (int i=0; i<fpodClick.getWaveData().length; i++) {
+				for (int j=0; j<fpodClick.getWaveData()[i].length; j++) {
+//					System.out.println("Current: " + i + " "+ j 
+//							+ " nchan: " + dataUnit.getNChan() + "  wave size: " 
+//							+ dataUnit.getWaveLength() +"len concat: " + concatWaveform.length);
+					concatWaveform[n++] = fpodClick.getWaveData()[i][j];
+				}
+			}
+
+			Vector newMatrix = DoubleArrayVector.newMatrix(concatWaveform, fpodClick.getWaveData()[0].length, fpodClick.getWaveData().length); 
+			rData.add("wave", newMatrix); 
+		}
+		else {
+			rData.add("wave", DoubleArrayVector.newMatrix(new double[] {}, 0, 0)); 
+		}
+		
+		
+		rData.add("durationmillis", dataUnit.getDurationInMilliseconds()); 
+
+//		Matrix duration = Mat5.newScalar(fpodClick.getDurationInMilliseconds()); 
+		
+		rData.add("numcycles", fpodClick.getnCyc()); 
+		rData.add("bandwidth", fpodClick.getBw()*1000.); 
+		rData.add("peakfreq", fpodClick.getkHz()*1000.); 
+		rData.add("endfreq", fpodClick.getEndF()*1000); 
+		rData.add("SPL", fpodClick.getSpl()); 
+		rData.add("slope", fpodClick.getSlope()); 
+		
+		short species = -1;
+		long clicktrainID = -1;
+
+		if (fpodClick.getCPODClickTrain()!=null) {
+			species =  MLCPODExport.getCPODSpecies(fpodClick.getCPODClickTrain().getSpecies());
+			clicktrainID = fpodClick.getCPODClickTrain().getUID();
+		}
+		
+		rData.add("species", species); 
+		rData.add("clicktrainID", clicktrainID); 
+
+		
+		return rData;
+	}
+
+	@Override
+	public Class<?> getUnitClass() {
+		return CPODClick.class;
+	}
+
+	@Override
+	public String getName() {
+		return "cpod_clicks";
+	}
+
+}

src/export/RExport/RDataUnitExport.java

@@ -2,9 +2,12 @@ package export.RExport;
 
 import org.renjin.sexp.DoubleArrayVector;
 import org.renjin.sexp.ListVector;
+import org.renjin.sexp.Vector;
+import org.renjin.sexp.ListVector.NamedBuilder;
 
 import PamUtils.PamCalendar;
 import PamguardMVC.PamDataUnit;
+import export.MLExport.MLAnnotationsManager;
 
 /**
  * Exports a data unit to a List in R. Specific data units should subclass this. 
@@ -15,11 +18,17 @@ import PamguardMVC.PamDataUnit;
  */
 public abstract class RDataUnitExport<T extends PamDataUnit<?, ?>> {
 	
+	private RAnnotationManager rAnnotationsManager;
+
+	public RDataUnitExport() {
+		this.rAnnotationsManager = new RAnnotationManager(); 
+	}
+	
 	
 	/**
-	 * Create a MATLAB structure which contains all information for a data unit. 
+	 * Create a R data frame which contains all information for a data unit. 
-	 * @param dataUnit - the data unit to convert to a MATLAB structure
+	 * @param dataUnit - the data unit to convert to a R structure
-	 * @return detection data MATLAB structure ready to be exported to a .mat file or added to a ArrayList<MLArray>. 
+	 * @return detection data R structure ready to be exported to a .mat file or added to a ArrayList<MLArray>. 
 	 */
 	public ListVector.NamedBuilder detectionToStruct(T dataUnit, int index){
 		
@@ -57,6 +66,8 @@ public abstract class RDataUnitExport<T extends PamDataUnit<?, ?>> {
 		//add detection specific data 
 		rData= addDetectionSpecificFields(rData, dataUnit, index);
 		
+		rAnnotationsManager.addDataAnnotations(rData, dataUnit, index);
+				
 		return rData; 
 	}
 
@@ -80,4 +91,29 @@ public abstract class RDataUnitExport<T extends PamDataUnit<?, ?>> {
 	 */
 	public abstract String getName();
 	
+	/**
+	 * Convert a 2D double array to a vector which can be added to an RData frame.
+	 * @param arr - the array
+	 * @return Vector containing the array data. 
+	 */
+	public static Vector doubleArr2R(double[][] arr) {
+		int nbins =arr.length*arr[0].length;
+		int n=0;
+		double[] concatWaveform  = new double[nbins];
+		//System.out.println("Number of bins: " + nbins);
+		for (int i=0; i<arr.length; i++) {
+			for (int j=0; j<arr[i].length; j++) {
+//				System.out.println("Current: " + i + " "+ j 
+//						+ " nchan: " + dataUnit.getNChan() + "  wave size: " 
+//						+ dataUnit.getWaveLength() +"len concat: " + concatWaveform.length);
+				concatWaveform[n++] = arr[i][j];
+			}
+		}
+
+		Vector newMatrix = DoubleArrayVector.newMatrix(concatWaveform, arr[0].length, arr.length); 
+		
+		return newMatrix;
+	}
+
+
 }

src/export/RExport/RExportManager.java

@@ -49,6 +49,7 @@ public class RExportManager implements PamDataUnitExporter {
 		/***Add more options here to export data units****/
 		rDataExport.add(new RClickExport()); 
 		rDataExport.add(new RWhistleExport()); 
+		rDataExport.add(new RCPODExport()); 
 		rDataExport.add(new RRawExport()); //should be last in case raw data holders have specific exporters
 	}
 
@@ -136,7 +137,7 @@ public class RExportManager implements PamDataUnitExporter {
 
 	/**
 	 * Sort a list of data units into lists of the same type of units. Convert to a list of structures. 
-	 * @param dataUnits - a list of data units to convert to matlab structures. 
+	 * @param dataUnits - a list of data units to convert to R data frames. 
 	 * @return list of list of R strucutures ready for saving to .RData file. 
 	 */
 	public RData dataUnits2R(List<PamDataUnit> dataUnits){

src/export/RExport/RRawExport.java

@@ -29,20 +29,7 @@ public class RRawExport extends RDataUnitExport<PamDataUnit> {
 
 		//add the raw wave data 
 		if (dataholder.getWaveData()!=null && dataholder.getWaveData().length>0) {
-			int nbins =dataholder.getWaveData().length*dataholder.getWaveData()[0].length;
+			Vector newMatrix = doubleArr2R(dataholder.getWaveData()); 
-			int n=0;
-			double[] concatWaveform  = new double[nbins];
-			//System.out.println("Number of bins: " + nbins);
-			for (int i=0; i<dataholder.getWaveData().length; i++) {
-				for (int j=0; j<dataholder.getWaveData()[i].length; j++) {
-//					System.out.println("Current: " + i + " "+ j 
-//							+ " nchan: " + dataUnit.getNChan() + "  wave size: " 
-//							+ dataUnit.getWaveLength() +"len concat: " + concatWaveform.length);
-					concatWaveform[n++] = dataholder.getWaveData()[i][j];
-				}
-			}
-
-			Vector newMatrix = DoubleArrayVector.newMatrix(concatWaveform, dataholder.getWaveData()[0].length, dataholder.getWaveData().length); 
 			rData.add("wave", newMatrix); 
 		}
 		
@@ -66,6 +53,8 @@ public class RRawExport extends RDataUnitExport<PamDataUnit> {
 		return rData;
 	}
 
+
+
 	@Override
 	public Class<?> getUnitClass() {
 		return RawDataHolder.class;
@@ -76,4 +65,6 @@ public class RRawExport extends RDataUnitExport<PamDataUnit> {
 		return "raw_data_units";
 	}
 	
+	
+
 }

src/export/swing/ExportProcessDialog.java

@@ -346,7 +346,7 @@ public class ExportProcessDialog {
 
 		@Override
 		public boolean getParams() {
-			System.out.println("EXPORT: GET PARAMS:");
+//			System.out.println("EXPORT: GET PARAMS:");
 
 			//make sure we update the current paramters before processing starts. 
 			this.currentParams = getExportParams();

src/export/wavExport/WavDetExportManager.java

@@ -2,32 +2,13 @@ package export.wavExport;
 
 import java.awt.Component;
 import java.io.File;
-import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.List;
 
-import javax.sound.sampled.AudioFormat;
-import javax.swing.filechooser.FileSystemView;
-
-import PamController.PamController;
-import PamDetection.RawDataUnit;
-import PamUtils.PamCalendar;
-import PamUtils.PamUtils;
-import PamView.paneloverlay.overlaymark.OverlayMark;
-import PamguardMVC.LoadObserver;
 import PamguardMVC.PamDataUnit;
-import PamguardMVC.PamObservable;
-import PamguardMVC.PamObserver;
-import PamguardMVC.PamObserverAdapter;
-import PamguardMVC.PamRawDataBlock;
 import PamguardMVC.RawDataHolder;
-import PamguardMVC.dataOffline.OfflineDataLoading;
+import cpod.CPODClick;
-import dataMap.OfflineDataMapPoint;
-import detectiongrouplocaliser.DetectionGroupSummary;
 import export.PamDataUnitExporter;
 import javafx.scene.layout.Pane;
-import wavFiles.Wav16AudioFormat;
-import wavFiles.WavFileWriter;
 
 /**
  * Writes data units and/or ordered raw data to a wav file. Has functions to
@@ -69,7 +50,12 @@ public class WavDetExportManager implements PamDataUnitExporter  {
 	@Override
 	public boolean hasCompatibleUnits(Class dataUnitType) {
 		//        boolean implementsInterface = Arrays.stream(dataUnitType.getInterfaces()).anyMatch(i -> i == RawDataHolder.class);
+		//don't allow folk to export FPOD data as wav files or even make it seem CPOD data can be export as wav - all sorts of confusion arises from here. 
+		if (CPODClick.class.isAssignableFrom(dataUnitType)) return false; 
+		
+		//if a raw data holder we can likely export detections to wav files. 
 		if (RawDataHolder.class.isAssignableFrom(dataUnitType)) return true;
+		
 		return false;
 	}
 

src/export/wavExport/WavOptionsPanel.java

@@ -38,7 +38,7 @@ public class WavOptionsPanel extends PamPanel {
 		
 		this.setLayout((new GridBagLayout()));
 
-		addComponent(this, zeroPad = new JRadioButton("Zero Pad"), c);
+		addComponent(this, zeroPad = new JRadioButton("Zero pad"), c);
 		
 		c.gridx++;
 		

src/offlineProcessing/OLProcessDialog.java

@@ -166,6 +166,7 @@ public class OLProcessDialog extends PamDialog {
 			c.gridx = 0;
 			aTask = taskGroup.getTask(i);
 			addComponent(tasksPanel, taskCheckBox[i] = new JCheckBox(aTask.getName()), c);
+			taskCheckBox[i].setToolTipText(aTask.getLongName());
 			taskCheckBox[i].addActionListener(new SelectionListener(aTask, taskCheckBox[i]));
 			c.gridx++;
 			if (aTask.hasSettings()) {

src/pamViewFX/PamGuiFX.java

@@ -19,6 +19,7 @@ import javafx.scene.Node;
 import javafx.scene.control.Button;
 import javafx.scene.control.MenuButton;
 import javafx.scene.control.MenuItem;
+import javafx.scene.layout.Pane;
 import javafx.scene.layout.StackPane;
 import javafx.scene.paint.Color;
 import javafx.stage.Stage;
@@ -409,6 +410,10 @@ public class PamGuiFX extends StackPane implements PamViewInterface {
 		
 	}
 	
+	private boolean isViewer() {
+		return PamController.getInstance().getRunMode() == PamController.RUN_PAMVIEW;
+	}
+	
 
 	/**
 	 * Create the tool bar pane. The top of each tab content node contains a tool
@@ -430,10 +435,6 @@ public class PamGuiFX extends StackPane implements PamViewInterface {
 		 */
 		private PamButton recordButton;
 	
-		/**
-		 *Play/pause button. Plays back sound in real time/viewer mode. 
-		 */
-		private PamButton playButton;
 		
 		/**
 		 * Holds all extra controls in the toolbar. 
@@ -472,48 +473,25 @@ public class PamGuiFX extends StackPane implements PamViewInterface {
 			this.pamGuiTab=pamGuiTab;
 		
 			//create record and play buttons. 
-			PamHBox playControls=new PamHBox();
+			Pane playControls;
-			recordButton=new PamButton();
+			
-//			recordButton.setGraphic(PamGlyphDude.createPamGlyph(FontAwesomeIcon.CIRCLE, Color.LIMEGREEN, PamGuiManagerFX.iconSize));
+			if (isViewer()) {
-			recordButton.setGraphic(PamGlyphDude.createPamIcon("mdi2c-checkbox-blank-circle", Color.LIMEGREEN, PamGuiManagerFX.iconSize));
+				playControls = createViewerControls();
-			recordButton.getStyleClass().add("transparent-button");
+
-			recordButton.setStyle(" -fx-background-radius: 50;");
-			recordButton.setOnAction((action)->{
-				if (PamController.getInstance().getPamStatus()==PamController.PAM_RUNNING){
-					PamController.getInstance().pamStop();
-					pamGuiManagerFX.setPamRunning(false);
 			}
 			else {
-					PamController.getInstance().pamStart();
+				playControls = createRealTimeControls();
-					pamGuiManagerFX.setPamRunning(true);
 			}
-			});
-
-			playButton=new PamButton();
-//			playButton.setGraphic(PamGlyphDude.createPamGlyph(FontAwesomeIcon.PLAY, Color.BLACK, PamGuiManagerFX.iconSize));
-			playButton.setGraphic(PamGlyphDude.createPamIcon("mdi2p-play", Color.BLACK, PamGuiManagerFX.iconSize));
-			playButton.getStyleClass().add("transparent-button");
-			playButton.setStyle(" -fx-background-radius: 50;");
-			playButton.setOnAction((action)->{
-				//TODO
-				//start pamguard
-				//PamController.getInstance().pamStart();
-			});
 			
 			
-			playControls.getChildren().addAll(recordButton,playButton);
+
-			playControls.setSpacing(10);
-			playControls.setPadding(new Insets(0,10,0,20));
-			playControls.getStyleClass().add("pane-opaque");
-			playControls.setPrefHeight(prefHeight);
-			playControls.setAlignment(Pos.CENTER);
 			
 			//create window editing button. This holds a toggle to edit windows and options. 
 			rightHBox=new PamHBox();
 			rightHBox.setAlignment(Pos.CENTER_LEFT);
 			rightHBox.setPadding(new Insets(0,10,0,20));
 			rightHBox.setSpacing(5);
-			rightHBox.getStyleClass().add("pane-opaque");
+//			rightHBox.getStyleClass().add("pane-opaque");
 			
 			editWindows=new ToggleSwitch("Resize"); 
 			//HACK,
@@ -563,7 +541,7 @@ public class PamGuiFX extends StackPane implements PamViewInterface {
 			this.setRight(rightHBox);
 			
 			this.setPrefHeight(prefHeight);
-			this.getStyleClass().add("pane-opaque");
+//			this.getStyleClass().add("pane-opaque");
 
 
 			//this.setPadding(new Insets(0,0,0,0));
@@ -573,6 +551,95 @@ public class PamGuiFX extends StackPane implements PamViewInterface {
 		}
 		
 		
+		private Pane createViewerControls() {
+			
+			//create record and play buttons. 
+			PamHBox playControls = new PamHBox();
+			
+			PamButton reProcess=new PamButton("Reprocess");
+			reProcess.setGraphic(PamGlyphDude.createPamIcon("mdi2p-play", PamGuiManagerFX.iconSize));
+			reProcess.setOnAction((action)->{
+				//Open reprocess dialog. 
+			});
+			
+			
+			PamButton exportButton = new PamButton("Export data");
+			exportButton.setGraphic(PamGlyphDude.createPamIcon("mdi2d-database-export", PamGuiManagerFX.iconSize));
+			exportButton.setOnAction((action)->{
+				//export dialog
+				
+			});
+			
+			PamButton importButton = new PamButton("Import data");
+			importButton.setGraphic(PamGlyphDude.createPamIcon("mdi2d-database-import", PamGuiManagerFX.iconSize));
+			importButton.setOnAction((action)->{
+				//iport dialog
+			});
+
+
+			
+			playControls.getChildren().addAll(reProcess, exportButton, importButton);
+			playControls.setSpacing(5);
+			playControls.setPadding(new Insets(0,10,0,50));
+//			playControls.getStyleClass().add("pane-opaque");
+			playControls.setPrefHeight(prefHeight);
+			playControls.setAlignment(Pos.CENTER_LEFT);
+			
+			return playControls;
+			
+		}
+		
+		
+		/**
+		 * Create controls for PAMGuard for real time processing. 
+		 * @return
+		 */
+		private Pane createRealTimeControls() {
+			
+			//create record and play buttons. 
+			PamHBox playControls=new PamHBox();
+			recordButton=new PamButton("Process");
+//			recordButton.setGraphic(PamGlyphDude.createPamGlyph(FontAwesomeIcon.CIRCLE, Color.LIMEGREEN, PamGuiManagerFX.iconSize));
+			recordButton.setGraphic(PamGlyphDude.createPamIcon("mdi2r-record-circle", Color.RED, 30));
+			recordButton.getStyleClass().add("transparent-button");
+			recordButton.setStyle(" -fx-padding: 1 1 1 1");
+			
+			recordButton.setOnAction((action)->{
+				if (PamController.getInstance().getPamStatus()==PamController.PAM_RUNNING){
+					PamController.getInstance().pamStop();
+					pamGuiManagerFX.setPamRunning(false);
+//					recordButton.setGraphic(PamGlyphDude.createPamIcon("mdi2r-record-circle", Color.RED, PamGuiManagerFX.iconSize));
+
+				}
+				else {
+					PamController.getInstance().pamStart();
+					pamGuiManagerFX.setPamRunning(true);
+//					recordButton.setGraphic(PamGlyphDude.createPamIcon("mdi2p-pause", Color.DARKGRAY, PamGuiManagerFX.iconSize));
+				}
+			});
+
+//			playButton=new PamButton();
+////			playButton.setGraphic(PamGlyphDude.createPamGlyph(FontAwesomeIcon.PLAY, Color.BLACK, PamGuiManagerFX.iconSize));
+//			playButton.setGraphic(PamGlyphDude.createPamIcon("mdi2p-play", Color.BLACK, PamGuiManagerFX.iconSize));
+//			playButton.getStyleClass().add("transparent-button");
+////			playButton.setStyle(" -fx-background-radius: 50;");
+//			playButton.setOnAction((action)->{
+//				//TODO
+//				//start pamguard
+//				//PamController.getInstance().pamStart();
+//			});
+			
+			playControls.getChildren().addAll(recordButton);
+			playControls.setSpacing(10);
+			playControls.setPadding(new Insets(0,10,0,20));
+//			playControls.getStyleClass().add("pane-opaque");
+			playControls.setPrefHeight(prefHeight);
+			playControls.setAlignment(Pos.CENTER);
+			
+			return playControls;
+		}
+		
+		
 		/**
 		 * Get the content HBox. This holds any extra controls in the top tool bar. 
 		 * @return HBox to add extra toolbar content to. 

src/pamViewFX/PamGuiManagerFX.java

@@ -537,21 +537,20 @@ public class PamGuiManagerFX implements PAMControllerGUI, PamSettings {
 	 * Set the GUI to show PAMGUARD has started or stopped. 
 	 */
 	public void setPamRunning(boolean running){
-		Glyph graphic;
+//		Glyph graphic;
-		for (int i=0; i<this.stages.size(); i++){
+//		for (int i=0; i<this.stages.size(); i++){
-			ArrayList<Button> recordButtons=stages.get(i).getRecordButtons();
+//			ArrayList<Button> recordButtons=stages.get(i).getRecordButtons();
-			for (int j=0; j<recordButtons.size(); j++){
+//			for (int j=0; j<recordButtons.size(); j++){
-				if (running){
+//				if (running){
-					graphic=Glyph.create("FontAwesome|SQUARE").size(22).color(Color.BLACK);
+//					graphic=Glyph.create("FontAwesome|SQUARE").size(22).color(Color.BLACK);
-				}
+//				}
-				else {
+//				else {
-					graphic=Glyph.create("FontAwesome|CIRCLE").size(22).color(Color.LIMEGREEN);
+//					graphic=Glyph.create("FontAwesome|CIRCLE").size(22).color(Color.LIMEGREEN);
-				}
+//				}
-				//now set all run buttons to correct format. 
+//				//now set all run buttons to correct format. 
-				recordButtons.get(j).setGraphic(graphic);
+//				recordButtons.get(j).setGraphic(graphic);
-
+//			}
-			}
+//		}
-		}
 	}
 
 	public void pamStop(WindowEvent e) {

src/pamViewFX/PamSettingsMenuPane.java

@@ -16,10 +16,12 @@ import PamController.soundMedium.GlobalMedium.SoundMedium;
 import PamModel.PamModuleInfo;
 import PamUtils.PamFileFilter;
 import binaryFileStorage.BinaryStore;
+import dataModelFX.connectionNodes.ModuleIconFactory;
 import generalDatabase.DBControlUnit;
 import javafx.geometry.Insets;
 import javafx.geometry.Pos;
 import javafx.geometry.Side;
+import javafx.scene.Node;
 import javafx.scene.control.Label;
 import javafx.scene.control.Labeled;
 import javafx.scene.control.MenuButton;
@@ -163,7 +165,14 @@ public class PamSettingsMenuPane extends PamVBox {
 			for (int i=0; i<nUnits ;i++){
 				final int n=i; //need to make n final- OK since menu re-makes itself every time opened. 
 				menuItem=new MenuItem(PamController.getInstance().getControlledUnit(i).getUnitName());
+				
+//				Node icon = ModuleIconFactory.getInstance().getModuleNode(PamController.getInstance().getControlledUnit(i).getPamModuleInfo().getClassName());
+//				icon.setScaleX(0.25);
+//				icon.setScaleY(0.25);
+//				menuItem.setGraphic(icon);
+				
 				settings.getItems().add(menuItem);
+				
 				menuItem.setOnAction((event) -> { 
 					openSettingsDialog( PamController.getInstance().getControlledUnit(n));
 				});

src/pamViewFX/fxNodes/pamAxis/PamDateAxis.java

@@ -20,6 +20,7 @@ import javafx.util.converter.TimeStringConverter;
 
 import java.lang.reflect.InvocationTargetException;
 import java.lang.reflect.Method;
+import java.time.Instant;
 import java.util.*;
 
 /*
@@ -34,6 +35,8 @@ public class PamDateAxis extends ValueAxis<Long> {
 	/** We use these for auto ranging to pick a user friendly tick unit. (must be increasingly bigger)*/
 	private static final double[] TICK_UNIT_DEFAULTS = {
 			3600000,       // 1 hour
+			14400000,       // 4 hours
+			21600000,     // 6 hours
 			86400000,       // 1 day
 			172800000,      // 2 days
 			259200000,      // 3 days
@@ -64,20 +67,22 @@ public class PamDateAxis extends ValueAxis<Long> {
 
 	/** These are matching date formatter strings */
 	private static final String[] TICK_UNIT_FORMATTER_DEFAULTS = {
-        "HH:mm:SS",     // 1 hour
+			"E HH:mm:ss",     // 1 hour
-        "HH:mm:SS",     // 1 day
+			"E HH:mm:ss",     // 4 hours
-        "MM/dd/yy",     // 2 days
+			"dd/MM/yy HH:mm",     // 6 hours
-        "MM/dd/yy",     // 3 days
+			"dd/MM/yy",     // 1 day
-        "MM/dd/yy",     // 4 days
+			"dd/MM/yy",     // 2 days
-        "MM/dd/yy",     // 5 days
+			"dd/MM/yy",     // 3 days
-        "MM/dd/yy",     // 6 days
+			"dd/MM/yy",     // 4 days
-        "MM/dd/yy",     // 7 days
+			"dd/MM/yy",     // 5 days
-        "MM/dd/yy",     // 8 days
+			"dd/MM/yy",     // 6 days
-        "MM/dd/yy",     // 9 days
+			"dd/MM/yy",     // 7 days
-        "MM/dd/yy",     // 10 days
+			"dd/MM/yy",     // 8 days
-        "MM/dd/yy",     // 15 days
+			"dd/MM/yy",     // 9 days
-        "MM/dd/yy",     // 20 days
+			"dd/MM/yy",     // 10 days
-        "MM/dd/yy",     // 25 days
+			"dd/MM/yy",     // 15 days
+			"dd/MM/yy",     // 20 days
+			"dd/MM/yy",     // 25 days
 			"MMM-yyyy",     // 31 days ~ 1 month
 			"MMM-yyyy",     // 41 days
 			"MMM-yyyy",     // 51 days
@@ -157,6 +162,24 @@ public class PamDateAxis extends ValueAxis<Long> {
 	 */
 	public PamDateAxis() {
 		forceZeroInRange.set(false);
+
+		//make sure that the tick marks update whenever the axis is manually set. 
+		this.upperBoundProperty().addListener((obsval, oldVal, newVal)->{
+			setTickDateFormatter();
+		});
+
+		this.lowerBoundProperty().addListener((obsval, oldVal, newVal)->{
+			setTickDateFormatter();
+		});
+	}
+
+	private void setTickDateFormatter() {
+		double labelSize = getTickLabelFont().getSize() * 2;
+		double[] rangeProps =  (double[]) autoRange(getLowerBound(),getUpperBound(),getWidth(),labelSize);
+		final double rangeIndex = rangeProps[4];
+		currentRangeIndexProperty.set((int)rangeIndex);
+		//    	System.out.println("Set data formatter: " + currentRangeIndexProperty.get());
+		defaultFormatter.setFormatter(DefaultFormatter.getFormatter(currentRangeIndexProperty.get()));
 	}
 
 	/**
@@ -183,6 +206,8 @@ public class PamDateAxis extends ValueAxis<Long> {
 		super(lowerBound, upperBound);
 		setTickUnit(tickUnit);
 		setLabel(axisLabel);
+
+
 	}
 
 	// -------------- PROTECTED METHODS --------------------------------------------------------------------------------
@@ -289,6 +314,7 @@ public class PamDateAxis extends ValueAxis<Long> {
 	 * @param animate If true animate the change in range
 	 */
 	@Override protected void setRange(Object range, boolean animate) {
+
 		final double[] rangeProps = (double[]) range;
 		final double lowerBound = rangeProps[0];
 		final double upperBound = rangeProps[1];
@@ -468,6 +494,7 @@ public class PamDateAxis extends ValueAxis<Long> {
 				double tickUnitDefault = TICK_UNIT_DEFAULTS[i];
 				if (tickUnitDefault > tickUnit) {
 					tickUnitRounded = tickUnitDefault;
+					//                    System.out.println("Found range index: " + i + " " + tickUnit); 
 					rangeIndex = i;
 					break;
 				}
@@ -558,6 +585,14 @@ public class PamDateAxis extends ValueAxis<Long> {
 	 */
 	public static class DefaultFormatter extends StringConverter<Long> {
 		private TimeStringConverter formatter;
+		public TimeStringConverter getFormatter() {
+			return formatter;
+		}
+
+		public void setFormatter(TimeStringConverter formatter) {
+			this.formatter = formatter;
+		}
+
 		private String prefix = null;
 		private String suffix = null;
 
@@ -575,11 +610,13 @@ public class PamDateAxis extends ValueAxis<Long> {
 		 */
 		public DefaultFormatter(final PamDateAxis axis) {
 			formatter = getFormatter(axis.isAutoRanging()? axis.currentRangeIndexProperty.get() : -1);
+
 			final ChangeListener axisListener = new ChangeListener() {
 				@Override public void changed(ObservableValue observable, Object oldValue, Object newValue) {
 					formatter = getFormatter(axis.isAutoRanging()? axis.currentRangeIndexProperty.get() : -1);
 				}
 			};
+
 			axis.currentRangeIndexProperty.addListener(axisListener);
 			axis.autoRangingProperty().addListener(axisListener);
 		}
@@ -598,11 +635,18 @@ public class PamDateAxis extends ValueAxis<Long> {
 		}
 
 		private static TimeStringConverter getFormatter(int rangeIndex) {
+			//        	System.out.println("Get fomatter range: " + rangeIndex);
+
 			if (rangeIndex < 0) {
 				return new TimeStringConverter("MM/dd/yy");
 			} else if(rangeIndex >= TICK_UNIT_FORMATTER_DEFAULTS.length) {
 				return new TimeStringConverter(TICK_UNIT_FORMATTER_DEFAULTS[TICK_UNIT_FORMATTER_DEFAULTS.length-1]);
 			} else {
+
+				TimeStringConverter timestringConverter = new TimeStringConverter(TICK_UNIT_FORMATTER_DEFAULTS[rangeIndex]);
+				timestringConverter.toString(Date.from(Instant.now()));
+
+				//            	System.out.println("Get fomatter range 2: " + TICK_UNIT_FORMATTER_DEFAULTS[rangeIndex]+ "  " + timestringConverter.toString(Date.from(Instant.now())));
 				return new TimeStringConverter(TICK_UNIT_FORMATTER_DEFAULTS[rangeIndex]);
 			}
 		}
@@ -648,6 +692,7 @@ public class PamDateAxis extends ValueAxis<Long> {
 	}
 
 
+
 	public static void main (String [] args)
 	{
 		// Date construction test

src/pamViewFX/fxNodes/pamScrollers/acousticScroller/AcousticScrollerFX.java

@@ -127,12 +127,12 @@ public class AcousticScrollerFX extends AbstractPamScrollerFX {
 	private ExecutorService executorService;
 
 	/**
-	 * The left arrow
+	 * The left arrow (or top if vertical).
 	 */
 	private PamButton arrowBottomLeft;
 
 	/**
-	 * The right arrow. 
+	 * The right arrow (or top if vertical).
 	 */
 	private PamButton arrowTopRight;
 

src/rawDeepLearningClassifier/DLZipUtils.java

@@ -54,6 +54,11 @@ public class DLZipUtils {
 	 */
 	public static File extractFile(URI zipPackage, String fileToBeExtracted, String outFolder) throws IOException {
 		File fileOut = new File(outFolder, fileToBeExtracted); 
+		
+		//need this incase the file is within a folder within the zip file.
+		if (new File(fileOut.getParent()).exists() == false) {
+			new File(fileOut.getParent()).mkdir();
+		}
         OutputStream out = new FileOutputStream(fileOut);
         FileInputStream fileInputStream = new FileInputStream(new File(zipPackage));
         BufferedInputStream bufferedInputStream = new BufferedInputStream(fileInputStream );

src/rawDeepLearningClassifier/dataSelector/DLPredictionPanel.java

@@ -102,7 +102,7 @@ public class DLPredictionPanel implements PamDialogPanel {
 		c.ipadx =5;
 
 		c.gridwidth=2;
-		contentPanel.add(new Label("Min. prediciton for each class"), c);
+		contentPanel.add(new Label("Min. prediction for each class"), c);
 		c.gridwidth=1;
 		c.gridy++;
 

src/rawDeepLearningClassifier/dlClassification/DLClassifierChooser.java

@@ -46,6 +46,7 @@ public class DLClassifierChooser {
 					//there's a settings file - does it contain a metadata field describing which
 					//type of classifier it belongs to. 
 					String outFolder = System.getProperty("user.home") +  File.separator + "PAMGuard_temp";
+					
 					new File(outFolder).mkdir();
 
 					File file = DLZipUtils.extractFile(modelURI, settingsFile, outFolder);

src/rawDeepLearningClassifier/dlClassification/archiveModel/ArchiveModelWorker.java

@@ -102,6 +102,8 @@ public class ArchiveModelWorker extends GenericModelWorker {
 			//read the JSON string from the the file. 
 			String jsonString  = DLTransformsParser.readJSONString(new File(dlModel.getAudioReprFile()));
 			
+			System.out.println("Archive model params: \n"+ jsonString);
+			
 
 			//convert the JSON string to a parameters object. 
 			GenericModelParams modelParams = makeModelParams( jsonString);

src/rawDeepLearningClassifier/dlClassification/archiveModel/SimpleArchiveModel.java

@@ -43,7 +43,7 @@ public class SimpleArchiveModel extends ArchiveModel {
 			String model = null;
 			model = getRelFilePath(zipFolder, ".pb");
 			if (model==null) model = getRelFilePath(zipFolder, ".py");
-			//System.out.println("MODEL PATH: " +model);
+			System.out.println("MODEL PATH: " +model + "  " + zipFolder);
 			return model;
 
 		} catch (IOException e) {
@@ -59,7 +59,7 @@ public class SimpleArchiveModel extends ArchiveModel {
 	}
 
 	/**
-	 * Gte the relative path of file within a zip folder. 
+	 * Get the relative path of file within a zip folder. 
 	 * @param zipFolder
 	 * @param fileEnd
 	 * @return
@@ -68,7 +68,14 @@ public class SimpleArchiveModel extends ArchiveModel {
 	private static String getRelFilePath(String zipFolder, String fileEnd) throws IOException {
 		  try (Stream<Path> walk = Files.walk(Paths.get(zipFolder)))  {
 			  List<String> result = walk
-		              .filter(p -> !Files.isDirectory(p))   // not a directory
+		              .filter(p -> {
+						try {
+							return (!Files.isDirectory(p) && !Files.isHidden(p));
+						} catch (IOException e) {
+							e.printStackTrace();
+							return false;
+						}
+					})   // not a directory
 		              .map(p -> p.toString()) // convert path to string
 		              .filter(f -> f.endsWith(fileEnd))       // check end with
 		              .collect(Collectors.toList());        // collect all matched to a List

src/rawDeepLearningClassifier/dlClassification/delphinID/DelphinIDTest.java

@@ -1,18 +1,26 @@
 package rawDeepLearningClassifier.dlClassification.delphinID;
 
+import java.awt.image.BufferedImage;
+import java.awt.image.Raster;
+import java.io.File;
 import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
 import java.util.ArrayList;
 
+import org.jamdev.jdl4pam.transforms.DLTransform;
+import org.jamdev.jdl4pam.transforms.FreqTransform;
+import org.jamdev.jdl4pam.transforms.DLTransform.DLTransformType;
 import org.jamdev.jdl4pam.utils.DLMatFile;
-
+import org.jamdev.jpamutils.JamArr;
+import org.jamdev.jpamutils.spectrogram.SpecTransform;
 import PamUtils.PamArrayUtils;
-import PamguardMVC.DataUnitBaseData;
 import PamguardMVC.PamDataUnit;
-import rawDeepLearningClassifier.dlClassification.animalSpot.StandardModelParams;
+import ai.djl.MalformedModelException;
+import rawDeepLearningClassifier.dlClassification.archiveModel.SimpleArchiveModel;
 import rawDeepLearningClassifier.dlClassification.genericModel.StandardPrediction;
 import rawDeepLearningClassifier.segmenter.SegmenterDetectionGroup;
 import us.hebi.matlab.mat.format.Mat5;
-import us.hebi.matlab.mat.format.Mat5File;
 import us.hebi.matlab.mat.types.MatFile;
 import us.hebi.matlab.mat.types.Matrix;
 import us.hebi.matlab.mat.types.Struct;
@@ -28,18 +36,39 @@ import whistlesAndMoans.AbstractWhistleDataUnit;
 public class DelphinIDTest {
 
 
+	/**
+	 * Main class for running the test. 
+	 * @param args - the arguments
+	 */
+	public static void main(String args[]) {
+		
+		testDelphinIDArray(null);
+
+		//
+		//		String matImageSave = "C:\\Users\\Jamie Macaulay\\MATLAB Drive\\MATLAB\\PAMGUARD\\deep_learning\\delphinID\\whistleimages_4s_415.mat";
+		//		matImageSave = null;
+		//		testDelphinIDModel(matImageSave);
+
+		//		//test a single image. 
+		//		String imagePathOut = "C:\\Users\\Jamie Macaulay\\MATLAB Drive\\MATLAB\\PAMGUARD\\deep_learning\\delphinID\\whistle_image_python_java.mat";
+		//		String imagePathOut = null;
+		//		testDelphinIDImage(imagePathOut);
+
+
+
+	}
+
+
 	public static class DelphinIDWorkerTest extends DelphinIDWorker {
 
 		private float[][][] lastModelInput;
 
-
 		public float[][][] dataUnits2ModelInput(ArrayList<? extends PamDataUnit> dataUnits, float sampleRate, int iChan){
 
 			float[][][] data = super.dataUnits2ModelInput(dataUnits, sampleRate, iChan);
 
 			this.lastModelInput = data;
 
-			
 			return data;
 		}
 
@@ -49,28 +78,30 @@ public class DelphinIDTest {
 
 	}
 
-	/**
-	 * Main class for running the test. 
-	 * @param args - the arguments
-	 */
-	public static void main(String args[]) {
 
+
+
+
+	/**
+	 * The 
+	 * @param matImageSave - the MATLAB image
+	 * @return true of the test is passed
+	 */
+	private static boolean testDelphinID2DModel(String matImageSave) {
 		double segLen = 4000.;
 		double segHop = 1000.0;
 		float sampleRate =96000;
+		double startSeconds = 9.565; //seconds to start segments (so we can compare to Python)
 		//unix time from sound file
 		long dataStartMillis = 1340212413000L;
 
 		//path to the .mat containing whistle contours. 
-		String whistleContourPath = "D:/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/testencounter415/whistle_contours.mat";
+		String whistleContourPath = "./src/test/resources/rawDeepLearningClassifier/DelphinID/SI20120620_171333_whistle_contours.mat";
 
 		//the path to the model
 		//String modelPath = "D:/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/testencounter415/whistle_model_2/whistle_4s_415.zip";
-		String modelPath = "D:/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/testencounter415/whistle_model_3/whistle_4s_415_f5.zip";
+		String modelPath = "/Users/au671271/Library/CloudStorage/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/delphinIDmodels/Dde415/whistle_4s_415_model.zip";
-
+		//		String modelPath =  "./src/test/resources/rawDeepLearningClassifier/DelphinID/whistle_4s_415_model.zip";
-		
-		//the path to the model
-		String matImageSave = "C:/Users/Jamie Macaulay/MATLAB Drive/MATLAB/PAMGUARD/deep_learning/delphinID/whistleimages.mat";
 
 		//create MatFile for saving the image data to. 
 		MatFile matFile = Mat5.newMatFile();
@@ -79,7 +110,8 @@ public class DelphinIDTest {
 		ArrayList<AbstractWhistleDataUnit> whistleContours = DelphinIDUtils.getWhistleContoursMAT(whistleContourPath);
 
 		//segment the whistle detections
-		ArrayList<SegmenterDetectionGroup> segments =  DelphinIDUtils.segmentWhsitleData(whistleContours,  (long) (dataStartMillis+(9.565*1000.)), 
+		//Note, delphinID starts from the first whistle and NOT the first file. 
+		ArrayList<SegmenterDetectionGroup> segments =  DelphinIDUtils.segmentWhsitleData(whistleContours,  (long) (dataStartMillis+(startSeconds*1000.)), 
 				segLen,  segHop);
 
 		for (int i=0; i<segments.size(); i++) {
@@ -87,7 +119,8 @@ public class DelphinIDTest {
 		}
 
 		//prepare the model - this loads the zip file and loads the correct transforms. 
-		DelphinIDWorkerTest model = DelphinIDUtils.prepDelphinIDModel(modelPath);
+		Path path = Paths.get(modelPath);
+		DelphinIDWorkerTest model = DelphinIDUtils.prepDelphinIDModel(path.toAbsolutePath().toString());
 		model.setEnableSoftMax(false);
 
 
@@ -100,17 +133,19 @@ public class DelphinIDTest {
 			ArrayList<SegmenterDetectionGroup> aSegment = new  ArrayList<SegmenterDetectionGroup>();
 			aSegment.add(segments.get(i)); 
 
-			//the prediciton. 
+			//the prediction. 
 			ArrayList<StandardPrediction> predicition = model.runModel(aSegment, sampleRate, 1);		
 
 			float[] output =  predicition.get(0).getPrediction();
 
 			System.out.println();
-			System.out.print("Segment: " + i + " " + (aSegment.get(0).getSegmentStartMillis()-dataStartMillis)/1000. + "s ");
+			System.out.print(String.format("Segment: %d %.2f s" , i ,((aSegment.get(0).getSegmentStartMillis()-dataStartMillis)/1000. - startSeconds)));
 			for (int j=0; j<output.length; j++) {
-				System.out.print("  " + output[j]); 
+				System.out.print(String.format( " %.3f" , output[j])); 
 			}
 
+
+
 			Matrix image = DLMatFile.array2Matrix(PamArrayUtils.float2Double(model.getLastModelInput()[0]));
 			imageStruct.set("image", i, image);
 			imageStruct.set("startmillis", i, Mat5.newScalar(aSegment.get(0).getSegmentStartMillis()));
@@ -120,6 +155,8 @@ public class DelphinIDTest {
 		}
 
 		matFile.addArray("whistle_images", imageStruct);
+
+		if (matImageSave!=null) {
 			// Serialize to disk using default configurations
 			try {
 				Mat5.writeToFile(matFile,matImageSave);
@@ -127,12 +164,265 @@ public class DelphinIDTest {
 				// TODO Auto-generated catch block
 				e.printStackTrace();
 			}
-		
+		}
 		//		for (int i=0; i<whistleContours.size(); i++) {
 		//			System.out.println("Whislte: " + i);
 		//			PamArrayUtils.printArray(whistleContours.get(i).getFreqsHz());
 		//		}
-
+		return true;
 	}
 
+	public static double[][] whistleScatter2Image(double[][] whistleValues, double startseg, double seglen) {
+
+		//now perform the image transform in Java 
+		double[] freqLimits = new double[] {2000., 20000.};
+		double[] size = new double[] {496., 369.};
+
+		ArrayList<double[][]> whistleImageArr = new ArrayList<double[][]>();
+		whistleImageArr.add(whistleValues);
+
+		BufferedImage canvas = Whistles2Image.makeScatterImage(whistleImageArr, size, new double[]{startseg, startseg + seglen}, freqLimits,  6.);
+
+		double[][] imaged = new double[(int) size[0]][(int) size[1]];
+
+		float[] color = new float[3];
+		Raster raster = canvas.getData();
+		for (int i=0; i<imaged.length; i++) {
+			for (int j=0; j<imaged[0].length; j++) {
+				color = raster.getPixel(i,j, color);
+				imaged[i][j] = (255-color[0])/255.; //normalize
+			}
+		}
+
+		//this is useful because it allows us to play around with transforms required to make this all work. 
+		//create the model transforms
+		ArrayList<DLTransform> modelTransforms = new ArrayList<DLTransform>();
+		modelTransforms.add(new FreqTransform(DLTransformType.SPECFLIP));
+		//		modelTransforms.add(new FreqTransform(DLTransformType.SPECNORMALISE_MINIMAX));
+		modelTransforms.add(new FreqTransform(DLTransformType.SPECRESIZE, new Number[] {Integer.valueOf(48), Integer.valueOf(62), SpecTransform.RESIZE_BICUBIC}));
+		modelTransforms.add(new FreqTransform(DLTransformType.GAUSSIAN_FILTER, new Number[] {Double.valueOf(0.5)}));
+
+		SpecTransform specTransform = new SpecTransform(); 
+		specTransform.setSpecData(imaged);
+		specTransform.setSampleRate((float) (freqLimits[1]*2)); 
+
+		//set the spec transform
+		((FreqTransform) modelTransforms.get(0)).setSpecTransfrom(specTransform);
+
+		//process all the transforms. 
+		DLTransform transform = modelTransforms.get(0); 
+		for (int i =0; i<modelTransforms.size(); i++) {
+			transform = modelTransforms.get(i).transformData(transform); 
+		}
+
+		double[][] transformedData2 = ((FreqTransform) transform).getSpecTransfrom().getTransformedData(); 
+
+		return transformedData2;
+	}
+
+
+	private static double[] whistle2AverageArray(double[][] whistleValues, double startseg, double seglen) {
+
+		//now perform the image transform in Java 
+		double[] freqLimits = new double[] {2000., 20000.};
+		double[] size = new double[] {496., 369.};
+
+		double freqBin = 100.;
+
+		int nbins = (int) ((freqLimits[1] -freqLimits[0])/freqBin);
+
+		System.out.println("Number of bins: " + nbins);
+
+		double[] peakBins = new double[nbins];
+		double minFreq, maxFreq;
+		int n;
+		int ntot = 0;
+		for (int i=0; i<nbins; i++) {
+			
+			minFreq = i*freqBin+freqLimits[0];
+			maxFreq = (i+1)*freqBin+freqLimits[0];
+
+			n=0;
+			for (int j=0; j<whistleValues.length ; j++) {
+				if (whistleValues[j][1]>= minFreq && whistleValues[j][1]< maxFreq && whistleValues[j][0]>=startseg && whistleValues[j][0]<(startseg+seglen)) {
+					n++;
+				}
+			}
+			
+			ntot=ntot+n;
+
+			System.out.println("Bin: " + minFreq + " Hz   " + n); 
+			peakBins[i]=n;
+		}
+		
+		for (int i=0; i<nbins; i++) {
+			peakBins[i]=peakBins[i]/ntot;
+		}
+		
+		return peakBins;
+	}
+
+	/**
+	 * This test runs delphinID on one 4 second window from whistle contours saved
+	 * in a mat file. The mat file also contains an image from Python. The test
+	 * compares the Python image to to the image generated by exporting both images
+	 * to a .mat file. The model is run on both images and results are compared
+	 * 
+	 * @return 
+	 */
+	public static boolean testDelphinIDImage(String imagePathOut) {
+
+		System.out.println("------DelphinID image comparison test---------");
+
+		double seglen = 4;
+
+		//test the model
+		//String modelPath =  "/Users/au671271/Library/CloudStorage/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/delphinIDmodels/Dde415/whistle_4s_415_model.zip";
+		String modelPath = "/Users/au671271/Library/CloudStorage/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/delphinIDmodels/Dde415/whistle_4s_415_model.zip";
+		//		String modelPath = "./src/test/resources/rawDeepLearningClassifier/DelphinID/whistle_4s_415_model.zip";
+
+		File file = new File(modelPath);
+		System.out.println("File exists: ? " + file.exists()); 
+
+		//the image to test
+		String relMatPath = "./src/test/resources/rawDeepLearningClassifier/DelphinID/Dde_415_s10_SI20120620_171333_d042_50.mat";
+
+		//Dde_415_s10_SI20120620_171333_d042_50
+		//		double[] expectedOutput = new double[]{0.998737633, 0.998737633,	0.000146952,	1.49E-10,	0.001111862, 1.64E-10,	1.66E-08,	3.53E-06};
+
+		//Dde_415_s10_SI20120620_171333_d042_27
+		//		double[] expectedOutput = new double[]{0.8434083	3.48E-05	8.71E-05	0.14855734	9.86E-07	0.002373327	0.005538126};
+		try {
+
+			Path path = Paths.get(modelPath);
+
+			//load the model
+			SimpleArchiveModel model = new SimpleArchiveModel(new File(path.toString()));
+
+			path = Paths.get(relMatPath);
+
+			// Create MAT file with a scalar in a nested struct
+			MatFile matFile = Mat5.readFromFile(path.toString());
+			Matrix array = matFile.getArray("tfvalues");
+
+			//the values for the whistle detector.
+			double[][] whistleValues = DLMatFile.matrix2array(array);
+
+			//the image after compression 
+			array = matFile.getArray("whistle_image_gray_python");
+			double[][] compressedWhistleImage = DLMatFile.matrix2array(array);
+
+			array = matFile.getArray("timeseg");
+
+			//the values for the whistle detector.
+			double[][] pamguardWhistleImage = 	whistleScatter2Image(whistleValues, array.getDouble(0), seglen);
+
+			//IMPORTANT - WE MUST TRANPOSE THE MATRIC HERE. 
+			pamguardWhistleImage=PamArrayUtils.transposeMatrix(pamguardWhistleImage);
+
+			//System.out.println("Size python: " + compressedWhistleImage.length + " x " + compressedWhistleImage[0].length);
+			System.out.println("----Model outputs---");
+
+			float[][][] input = new float[1][][];
+			input[0] =  JamArr.doubleToFloat(compressedWhistleImage);
+
+			System.out.println("Size Python: " + 	input[0].length + " x " + 	input[0][0].length);
+
+			float[] outputPython = model.runModel(input);
+
+			input[0] =   JamArr.doubleToFloat(pamguardWhistleImage);
+
+			System.out.println("Size Java: " + 	input[0].length + " x " + 	input[0][0].length);
+
+			//a bit ugly but works.
+			//				transformedData2 = JamArr.transposeMatrix(transformedData2);
+
+
+			float[] outputJava = model.runModel(input);
+
+			boolean outputOk = false;
+			for (int i=0; i<outputPython.length; i++) {
+				System.out.println(String.format("Output Python: %.6f Java: %.6f",outputPython[i],outputJava[i] )); 
+				if (Math.abs(outputPython[i] - outputJava[i])>0.05) outputOk=false;
+			}
+
+			if (imagePathOut!=null){
+				MatFile matFileWrite = Mat5.newMatFile()
+						.addArray("imagePython",DLMatFile.array2Matrix(compressedWhistleImage))
+						.addArray("imageJava",DLMatFile.array2Matrix(pamguardWhistleImage));
+
+
+				Mat5.writeToFile(matFileWrite, imagePathOut);
+			}
+
+			return outputOk; 
+
+			//			JamArr.printArray(output);
+
+		} 
+		catch (MalformedModelException | IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+
+			return false;
+		} 
+	}
+	
+	
+	/**
+	 * This test runs delphinID on one 4 second window from whistle contours saved
+	 * in a mat file
+	 * 
+	 * @return true if the test is passed. 
+	 */
+	public static boolean testDelphinIDArray(String arrayPathOut) {
+
+		System.out.println("------DelphinID 1D array comparison test---------");
+
+		double seglen = 4;
+
+		//test the model
+		//String modelPath =  "/Users/au671271/Library/CloudStorage/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/delphinIDmodels/Dde415/whistle_4s_415_model.zip";
+		String modelPath = "/Users/au671271/Library/CloudStorage/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/delphinIDmodels/Ggr242/whistleclassifier.zip";
+		//		String modelPath = "./src/test/resources/rawDeepLearningClassifier/DelphinID/whistle_4s_415_model.zip";
+
+		File file = new File(modelPath);
+		System.out.println("File exists: ? " + file.exists()); 
+
+		//the image to test
+		String relMatPath = "./src/test/resources/rawDeepLearningClassifier/DelphinID/Dde_415_s10_SI20120620_171333_d042_50.mat";
+	
+		try {
+
+			Path path = Paths.get(modelPath);
+
+			//load the model
+			SimpleArchiveModel model = new SimpleArchiveModel(new File(path.toString()));
+			
+			path = Paths.get(relMatPath);
+
+			// Create MAT file with a scalar in a nested struct
+			MatFile matFile = Mat5.readFromFile(path.toString());
+			Matrix array = matFile.getArray("tfvalues");
+
+			//the values for the whistle detector.
+			double[][] whistleValues = DLMatFile.matrix2array(array);
+			
+			//the values for the whistle detector.
+			double[] whistleArray = 	whistle2AverageArray(whistleValues, array.getDouble(0), seglen);
+			
+		} 
+		catch (MalformedModelException | IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+
+			return false;
+		}
+		
+		
+		return false;
+	}
+
+
+
 }

src/rawDeepLearningClassifier/dlClassification/delphinID/DelphinIDWorker.java

@@ -59,6 +59,8 @@ public class DelphinIDWorker extends  ArchiveModelWorker {
 
 		//now have to read the whsitle2image transform to get correct parameters for that. 
 		String jsonString  = DLTransformsParser.readJSONString(new File(this.getModel().getAudioReprFile()));
+		
+		
 		whistleImageParams = readWhistleImageTransform(new JSONObject(jsonString)) ;
 		if (whistleImageParams==null) {
 			System.err.println("Error: could not find whistle2image transform in DelphinID JSON file. Model will not work.");
@@ -96,12 +98,15 @@ public class DelphinIDWorker extends  ArchiveModelWorker {
 				freqLimits[1] = jsonObjectParams.getFloat("maxfreq"); 
 				size[0] = jsonObjectParams.getInt("widthpix"); 
 				size[1] = jsonObjectParams.getInt("heightpix"); 
+				double minfragmillis = jsonObjectParams.getDouble("minfragmillis"); 
+
 				double lineWidth = jsonObjectParams.getDouble("linewidthpix"); 
 
 				Whistle2ImageParams whistle2ImageParmas = new Whistle2ImageParams();
 				whistle2ImageParmas.freqLimits = freqLimits;
 				whistle2ImageParmas.size = size;
 				whistle2ImageParmas.lineWidth = lineWidth;
+				whistle2ImageParmas.minFragSize = minfragmillis;
 
 				return whistle2ImageParmas;
 			}
@@ -186,7 +191,7 @@ public class DelphinIDWorker extends  ArchiveModelWorker {
 			
 			transformedData2 = ((FreqTransform) transform).getSpecTransfrom().getTransformedData(); 
 
-			//a bit ugly but works.
+			//a bit ugly but works - it is very important we tranpose the matrix!!
 			transformedData2 = JamArr.transposeMatrix(transformedData2);
 
 			//System.out.println("DelphinID input image: " + transformedData2.length + " x " + transformedData2[0].length  );

src/rawDeepLearningClassifier/dlClassification/delphinID/Whistles2Image.java

@@ -10,7 +10,6 @@ import java.util.ArrayList;
 import org.jamdev.jdl4pam.transforms.FreqTransform;
 import org.jamdev.jpamutils.spectrogram.SpecTransform;
 
-import PamUtils.PamArrayUtils;
 import rawDeepLearningClassifier.segmenter.SegmenterDetectionGroup;
 import whistlesAndMoans.AbstractWhistleDataUnit;
 
@@ -36,7 +35,7 @@ public class Whistles2Image extends FreqTransform {
 		//		double[] freqLimits = 	new double[] {params[0].doubleValue(), params[1].doubleValue()};
 		//		double[] size =			new double[] {params[2].doubleValue(), params[3].doubleValue()};
 
-		SpecTransform specTransform = whistleGroupToImage( whistleGroup,  params.freqLimits, params.size, params.lineWidth);
+		SpecTransform specTransform = whistleGroupToImage( whistleGroup,  params.freqLimits, params.size, params.lineWidth, params.minFragSize);
 
 		this.setSpecTransfrom(specTransform);
 		this.setFreqlims(params.freqLimits);
@@ -48,12 +47,12 @@ public class Whistles2Image extends FreqTransform {
 	 * Convert a group of whistles 
 	 * @param whistleGroup - the whistle groups
 	 * @param freqLimits - the frequency limits
+	 * @paramn minFragSize - the minimum fragment size in milliseconds
 	 * @return the spectrogram transform. 
 	 */
-	private SpecTransform whistleGroupToImage(SegmenterDetectionGroup whistleGroup, double[] freqLimits, double[] size, double lineWidth) {
+	private SpecTransform whistleGroupToImage(SegmenterDetectionGroup whistleGroup, double[] freqLimits, double[] size, double lineWidth, double minFragSize) {
 
 		SpecTransform specTransform = new SpecTransform(); 
-
 		/*
 		 * All time-frequency points are saved as a scatterplot with x-axis spanning 0-4
 		 * seconds in time and y-axis spanning 0-20 kHz in frequency. - Matplotlib was
@@ -62,7 +61,7 @@ public class Whistles2Image extends FreqTransform {
 		 * 4.8 inches as default, axes removed before saving using plt.axes(‘off’))
 		 **/
 
-		ArrayList<double[][]> points = whistContours2Points(whistleGroup);
+		ArrayList<double[][]> points = whistContours2Points(whistleGroup, minFragSize);
 
 		//does not work becaue it has to be on the AWT thread. 
 		BufferedImage canvas = makeScatterImage(points, size, new double[]{0, whistleGroup.getSegmentDuration()/1000.}, freqLimits, lineWidth);
@@ -77,7 +76,7 @@ public class Whistles2Image extends FreqTransform {
 				imaged[i][j] = (255-color[0])/255.; //normalize
 			}
 		}
-//		
+		////		
 		//		System.out.println("Original image: "); 
 		//		PamArrayUtils.printArray(imaged);
 
@@ -87,12 +86,23 @@ public class Whistles2Image extends FreqTransform {
 		return specTransform;
 	}
 	
+	
 	/**
 	 * Convert a list of whistle contours to a list of time and frequency points. 
 	 * @param whistleGroup - list of whistle contours within a detection group. 
 	 * @return an array with time (seconds from start of group) and frequency (Hz)
 	 */
 	public static ArrayList<double[][]> whistContours2Points(SegmenterDetectionGroup whistleGroup) {
+		return  whistContours2Points( whistleGroup,  0.); 
+	}
+
+	/**
+	 * Convert a list of whistle contours to a list of time and frequency points. 
+	 * @param whistleGroup - list of whistle contours within a detection group. 
+	 * @param minFragSize - the minimum fragment size in millis. Fragments shorter than this are discarded and not added to the image.   
+	 * @return an array with time (seconds from start of group) and frequency (Hz)
+	 */
+	public static ArrayList<double[][]> whistContours2Points(SegmenterDetectionGroup whistleGroup, double minFragSize) {
 
 		ArrayList<double[][]> contours = new ArrayList<double[][]>();
 
@@ -124,6 +134,8 @@ public class Whistles2Image extends FreqTransform {
 
 			whistleContour = (AbstractWhistleDataUnit) whistleGroup.getSubDetection(i);
 
+			if (whistleContour.getDurationInMilliseconds() >=minFragSize) {
+
 				//			System.out.println("Whistle start time: " + (whistleContour.getTimeMilliseconds()-segStart)/1000. +  " end: " + 
 				//			(whistleContour.getTimeMilliseconds() - (segStart + whistleContour.getDurationInMilliseconds()))/1000.  
 				//			+ "  millis: " + whistleContour.getTimeMilliseconds() +  " first slice: " + whistleContour.getTimesInSeconds()[0]);
@@ -135,6 +147,7 @@ public class Whistles2Image extends FreqTransform {
 				}
 				contours.add(contourD);
 			}
+		}
 
 		return contours;
 	}
@@ -217,7 +230,12 @@ public class Whistles2Image extends FreqTransform {
 		/**
 		 * The line width to draw in pixels
 		 */
-		public double lineWidth = 10;
+		public double lineWidth = 10.;
+
+		/**
+		 * The minimum fragment length in millis. 
+		 */
+		public double minFragSize = 200.; //ms
 
 	}
 

src/rawDeepLearningClassifier/layoutFX/dlTransfroms/DLTransformsPane.java

@@ -164,11 +164,8 @@ public class DLTransformsPane extends PamBorderPane {
 			}
 
 			transformPane = DataTransformPaneFactory.getSettingsPane(dlTransforms.get(i)); 
-			
-			
 			//System.out.println("Set DL transfroms: " + dlTransforms.get(i).getDLTransformType() + " " + ((SimpleTransform) dlTransforms.get(i)).getParams()); 
 
-
 			//there must be a transform pane or else this will break. 
 			dlTransformPanes.add(transformPane); 
 			dlTransformPanes.get(i).setParams(dlTransforms.get(i));

src/rawDeepLearningClassifier/segmenter/SegmenterProcess.java

@@ -169,6 +169,11 @@ public class SegmenterProcess extends PamProcess {
 		}
 		
 		
+		//reset segments - important otherwise PG will crash if the wav file is reset. 
+		segmentStart=-1;
+		segmenterEnd=-1; 
+
+
 		//set up connection to the parent
 		PamDataBlock rawDataBlock = null;
 
@@ -254,7 +259,7 @@ public class SegmenterProcess extends PamProcess {
 
 		//TODO
 		//this contains no raw data so we are branching off on a completely different processing path here.
-		//Whislte data units are saved to a buffer and then fed to the deep learning algorithms
+		//Whistle data units are saved to a buffer and then fed to the deep learning algorithms
 		
 		int[] chanGroups = dlControl.getDLParams().groupedSourceParams.getChannelGroups();
 		
@@ -266,8 +271,8 @@ public class SegmenterProcess extends PamProcess {
 			}
 		}
 
-		//FIXME - TWEMP
+//		//FIXME - TWEMP
-		index =0;
+//		index =0;
 		
 //		System.out.println("Whistle data: " + ((dataUnit.getTimeMilliseconds()-firstClockUpdate)/1000.) + "s " + chanGroups.length +  "  " +  index + "  " + dataUnit.getChannelBitmap());
 //		PamArrayUtils.printArray(chanGroups);
@@ -287,6 +292,7 @@ public class SegmenterProcess extends PamProcess {
 			}
 			
 			while(!detectionInSegment(dataUnit,  segmentStart,  segmenterEnd)) {
+				System.out.println("Detection in segment: " + segmentStart); 
 				nextGroupSegment(index);
 			}
 		}

src/rocca/RoccaTrainClassifier.java

@@ -28,6 +28,7 @@ import java.io.File;
 import java.io.FileReader;
 import java.io.FilenameFilter;
 import java.util.Date;
+import java.util.Enumeration;
 
 import javax.swing.JFileChooser;
 import javax.swing.filechooser.FileNameExtensionFilter;

src/test/rawDeepLearningClassifier/DelphinIDTest.java

@@ -1,5 +1,6 @@
 package test.rawDeepLearningClassifier;
 
+import static org.junit.Assert.assertTrue;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 
 import java.awt.image.BufferedImage;
@@ -22,6 +23,7 @@ import org.jamdev.jpamutils.interpolation.NearestNeighbor;
 import org.jamdev.jpamutils.spectrogram.SpecTransform;
 import org.junit.jupiter.api.Test;
 
+import PamUtils.PamArrayUtils;
 import ai.djl.MalformedModelException;
 import rawDeepLearningClassifier.dlClassification.archiveModel.SimpleArchiveModel;
 import rawDeepLearningClassifier.dlClassification.delphinID.Whistles2Image;
@@ -31,230 +33,118 @@ import us.hebi.matlab.mat.types.Matrix;
 
 public class DelphinIDTest { 
 	
-	
+//	@Test
-	private static double[][] whistleScatter2Image(double[][] whistleValues) {
+//	public void whistle2ImageTest() {
-		
+//	
-		//now perform the image transform in Java 
+//		System.out.println("Whislte2Image test started");
-		double[] freqLimits = new double[] {2000., 20000.};
+//
-		double[] size = new double[] {680., 480.};
+//		/**
-		
+//		 * Test whether the Whistles2Image transform works properly
-		ArrayList<double[][]> whistleImageArr = new ArrayList<double[][]>();
+//		 */
-		whistleImageArr.add(whistleValues);
+//		String relMatPath  =	"./src/test/resources/rawDeepLearningClassifier/DelphinID/whistle_image_example.mat";
-		
+//
-		BufferedImage canvas = Whistles2Image.makeScatterImage(whistleImageArr, size, new double[]{50, 50. + 4.}, freqLimits,  6.);
+//		Path path = Paths.get(relMatPath);
-
+//	
-		double[][] imaged = new double[(int) size[0]][(int) size[1]];
+//		// Create MAT file with a scalar in a nested struct
-
+//		try {
-		float[] color = new float[3];
+//			MatFile matFile = Mat5.readFromFile(path.toString());
-		Raster raster = canvas.getData();
+//			Matrix array = matFile.getArray("tfvalues");
-		for (int i=0; i<imaged.length; i++) {
+//			
-			for (int j=0; j<imaged[0].length; j++) {
+//			//the values for the whistle detector.
-				color = raster.getPixel(i,j, color);
+//			double[][] whistleValues = DLMatFile.matrix2array(array);
-				imaged[i][j] = (255-color[0])/255.; //normalize
+//			
-			}
+////			//the image after compression 
-		}
+////			array = matFile.getArray("image1compressedgrayscale");
-		
+////			double[][] compressedWhistleImage = DLMatFile.matrix2array(array);
-		//create the model transforms
+////
-		ArrayList<DLTransform> modelTransforms = new ArrayList<DLTransform>();
+////			//the whistle2Image transform image
-		modelTransforms.add(new FreqTransform(DLTransformType.SPECFLIP));
+////			array = matFile.getArray("image1originalgrayscalenorm");
-//		modelTransforms.add(new FreqTransform(DLTransformType.SPECNORMALISE_MINIMAX));
+////			double[][] whislteImage = DLMatFile.matrix2array(array);
-		modelTransforms.add(new FreqTransform(DLTransformType.SPECRESIZE, new Number[] {Integer.valueOf(60), Integer.valueOf(80), SpecTransform.RESIZE_BICUBIC}));
+//			
-		modelTransforms.add(new FreqTransform(DLTransformType.GAUSSIAN_FILTER, new Number[] {Double.valueOf(0.5)}));
+//			//now perform the image transform in Java 
-
+//			double[] freqLimits = new double[] {2000., 20000.};
-		
+//			double[] size = new double[] {680., 480.};
-		SpecTransform specTransform = new SpecTransform(); 
+//			
-		specTransform.setSpecData(imaged);
+//			ArrayList<double[][]> whistleImageArr = new ArrayList<double[][]>();
-		specTransform.setSampleRate((float) (freqLimits[1]*2)); 
+//			whistleImageArr.add(whistleValues);
-
+//			
-		
+//			BufferedImage canvas = Whistles2Image.makeScatterImage(whistleImageArr, size, new double[]{50, 50. + 4.}, freqLimits,  10.);
-		//set the spec transform
+//
-		((FreqTransform) modelTransforms.get(0)).setSpecTransfrom(specTransform);
+//			double[][] imaged = new double[(int) size[0]][(int) size[1]];
-
+//
-		//process all the transforms. 
+//			float[] color = new float[3];
-		DLTransform transform = modelTransforms.get(0); 
+//			Raster raster = canvas.getData();
-		for (int i =0; i<modelTransforms.size(); i++) {
+//			for (int i=0; i<imaged.length; i++) {
-			transform = modelTransforms.get(i).transformData(transform); 
+//				for (int j=0; j<imaged[0].length; j++) {
-		}
+//					color = raster.getPixel(i,j, color);
-		
+//					imaged[i][imaged[0].length-j-1] = (255-color[0])/255.; //normalize
-		double[][] transformedData2 = ((FreqTransform) transform).getSpecTransfrom().getTransformedData(); 
+//				}
-		
+//			}
-		return transformedData2;
+//			
-	}
+//			//imaged = PamArrayUtils.transposeMatrix(imaged);
+//
+//			//create the model transforms
+//			ArrayList<DLTransform> modelTransforms = new ArrayList<DLTransform>();
+////			modelTransforms.add(new FreqTransform(DLTransformType.SPECFLIP));
+////			modelTransforms.add(new FreqTransform(DLTransformType.SPECNORMALISE_MINIMAX));
+//			modelTransforms.add(new FreqTransform(DLTransformType.SPECRESIZE, new Number[] {Integer.valueOf(60), Integer.valueOf(80), SpecTransform.RESIZE_BICUBIC}));
+//			modelTransforms.add(new FreqTransform(DLTransformType.GAUSSIAN_FILTER, new Number[] {Double.valueOf(0.5)}));
+//
+//			
+//			SpecTransform specTransform = new SpecTransform(); 
+//			specTransform.setSpecData(imaged);
+//			specTransform.setSampleRate((float) (freqLimits[1]*2)); 
+//
+//			
+//			//set the spec transform
+//			((FreqTransform) modelTransforms.get(0)).setSpecTransfrom(specTransform);
+//
+//			//process all the transforms. 
+//			DLTransform transform = modelTransforms.get(0); 
+//			for (int i =0; i<modelTransforms.size(); i++) {
+//				transform = modelTransforms.get(i).transformData(transform); 
+//			}
+//			
+//			double[][] transformedData2 = ((FreqTransform) transform).getSpecTransfrom().getTransformedData(); 
+//			
+////			Bilinear interpolation1 = new Bilinear(JamArr.doubleToFloat(transformedData2));
+////			Bicubic interpolation2 = new Bicubic(JamArr.doubleToFloat(imaged));
+////
+////			System.out.println("Len input: " + imaged.length);
+////
+////			float[][] resizeArr = interpolation2.resize(Integer.valueOf(80), Integer.valueOf(60));
+////			
+////			System.out.println("Len resize: " + resizeArr.length);
+//
+//			System.out.println("Size Java: " + transformedData2.length + " x " + transformedData2[0].length);
+//
+//			//now save this image to a MATFILE
+//			// Create MAT file with a scalar in a nested struct
+//			MatFile matFileWrite = Mat5.newMatFile()
+//			    .addArray("image1originalgrayscalenorm",DLMatFile.array2Matrix(imaged))
+//			    .addArray("imagecompressedgrayscalenorm",DLMatFile.array2Matrix(transformedData2));
+////		    	.addArray("imagecompressedgrayscalenorm_nearest",DLMatFile.array2Matrix(JamArr.floatToDouble(resizeArr)));
+//
+//			// Serialize to disk using default configurations
+//			Mat5.writeToFile(matFileWrite, "/Users/au671271/MATLAB-Drive/MATLAB/PAMGUARD/deep_learning/delphinID/whistle_image_example_java.mat");
+//			
+//			System.out.println("Whislte2Image test finished");
+//			
+//		} catch (IOException e) {
+//			// TODO Auto-generated catch block
+//			e.printStackTrace();
+//			assertEquals(false, false);
+//		}
+//
+//	}
 	
 	@Test
-	public void whistle2ImageTest() {
+	public static void modelInputTest() {
 		
-		System.out.println("Whislte2Image test started");
+		//test whether a single segment gives the correct answer. 
+		boolean result = rawDeepLearningClassifier.dlClassification.delphinID.DelphinIDTest.testDelphinIDImage(null);
 		
-		/**
+		assertTrue(result); 
-		 * Test whether the Whistles2Image transform works properly
-		 */
-		String relMatPath  =	"./src/test/resources/rawDeepLearningClassifier/DelphinID/whistle_image_example.mat";
-
-		Path path = Paths.get(relMatPath);
-	
-		// Create MAT file with a scalar in a nested struct
-		try {
-			MatFile matFile = Mat5.readFromFile(path.toString());
-			Matrix array = matFile.getArray("tfvalues");
-			
-			//the values for the whistle detector.
-			double[][] whistleValues = DLMatFile.matrix2array(array);
-			
-//			//the image after compression 
-//			array = matFile.getArray("image1compressedgrayscale");
-//			double[][] compressedWhistleImage = DLMatFile.matrix2array(array);
-//
-//			//the whistle2Image transform image
-//			array = matFile.getArray("image1originalgrayscalenorm");
-//			double[][] whislteImage = DLMatFile.matrix2array(array);
-			
-			//now perform the image transform in Java 
-			double[] freqLimits = new double[] {2000., 20000.};
-			double[] size = new double[] {680., 480.};
-			
-			ArrayList<double[][]> whistleImageArr = new ArrayList<double[][]>();
-			whistleImageArr.add(whistleValues);
-			
-			BufferedImage canvas = Whistles2Image.makeScatterImage(whistleImageArr, size, new double[]{50, 50. + 4.}, freqLimits,  10.);
-
-			double[][] imaged = new double[(int) size[0]][(int) size[1]];
-
-			float[] color = new float[3];
-			Raster raster = canvas.getData();
-			for (int i=0; i<imaged.length; i++) {
-				for (int j=0; j<imaged[0].length; j++) {
-					color = raster.getPixel(i,j, color);
-					imaged[i][j] = (255-color[0])/255.; //normalize
-				}
-			}
-			
-			//create the model transforms
-			ArrayList<DLTransform> modelTransforms = new ArrayList<DLTransform>();
-			modelTransforms.add(new FreqTransform(DLTransformType.SPECFLIP));
-//			modelTransforms.add(new FreqTransform(DLTransformType.SPECNORMALISE_MINIMAX));
-			modelTransforms.add(new FreqTransform(DLTransformType.SPECRESIZE, new Number[] {Integer.valueOf(60), Integer.valueOf(80), SpecTransform.RESIZE_BICUBIC}));
-			modelTransforms.add(new FreqTransform(DLTransformType.GAUSSIAN_FILTER, new Number[] {Double.valueOf(0.5)}));
-
-			
-			SpecTransform specTransform = new SpecTransform(); 
-			specTransform.setSpecData(imaged);
-			specTransform.setSampleRate((float) (freqLimits[1]*2)); 
-
-			
-			//set the spec transform
-			((FreqTransform) modelTransforms.get(0)).setSpecTransfrom(specTransform);
-
-			//process all the transforms. 
-			DLTransform transform = modelTransforms.get(0); 
-			for (int i =0; i<modelTransforms.size(); i++) {
-				transform = modelTransforms.get(i).transformData(transform); 
-			}
-			
-			double[][] transformedData2 = ((FreqTransform) transform).getSpecTransfrom().getTransformedData(); 
-			
-//			Bilinear interpolation1 = new Bilinear(JamArr.doubleToFloat(transformedData2));
-//			Bicubic interpolation2 = new Bicubic(JamArr.doubleToFloat(imaged));
-//
-//			System.out.println("Len input: " + imaged.length);
-//
-//			float[][] resizeArr = interpolation2.resize(Integer.valueOf(80), Integer.valueOf(60));
-//			
-//			System.out.println("Len resize: " + resizeArr.length);
-
-			System.out.println("Size Java: " + transformedData2.length + " x " + transformedData2[0].length);
-
-			//now save this image to a MATFILE
-			// Create MAT file with a scalar in a nested struct
-			MatFile matFileWrite = Mat5.newMatFile()
-			    .addArray("image1originalgrayscalenorm",DLMatFile.array2Matrix(imaged))
-			    .addArray("imagecompressedgrayscalenorm",DLMatFile.array2Matrix(transformedData2));
-//		    	.addArray("imagecompressedgrayscalenorm_nearest",DLMatFile.array2Matrix(JamArr.floatToDouble(resizeArr)));
-
-			// Serialize to disk using default configurations
-			Mat5.writeToFile(matFileWrite, "C:\\Users\\Jamie Macaulay\\MATLAB Drive\\MATLAB\\PAMGUARD\\deep_learning\\delphinID\\whistle_image_example_java.mat");
-			
-			System.out.println("Whislte2Image test finished");
-			
-		} catch (IOException e) {
-			// TODO Auto-generated catch block
-			e.printStackTrace();
-			assertEquals(false, false);
 	}
 
 }
-	
-	@Test
-	public void modelInputTest() {
-		
-		System.out.println("DelphinID mode test start");
-
-		//ttest the model
-		String modelPath = "D:/Dropbox/PAMGuard_dev/Deep_Learning/delphinID/testencounter415/whistle_model_2/whistle_4s_415.zip";
-		
-		String relMatPath  =	"./src/test/resources/rawDeepLearningClassifier/DelphinID/whistle_image_example.mat";
-
-		try {
-			
-			//load the model
-			SimpleArchiveModel model = new SimpleArchiveModel(new File(modelPath));
-			
-
-			Path path = Paths.get(relMatPath);
-		
-			// Create MAT file with a scalar in a nested struct
-				MatFile matFile = Mat5.readFromFile(path.toString());
-				Matrix array = matFile.getArray("tfvalues");
-				
-				//the values for the whistle detector.
-				double[][] whistleValues = DLMatFile.matrix2array(array);
-				
-				//the image after compression 
-				array = matFile.getArray("image1compressedgrayscalenorm");
-				double[][] compressedWhistleImage = DLMatFile.matrix2array(array);
-				
-				//the raw whistle frequency time scatter values
-				array = matFile.getArray("tfvalues");
-				//the values for the whistle detector.
-				double[][] pamguardWhistleImage = 	whistleScatter2Image(whistleValues);
-	
-				//System.out.println("Size python: " + compressedWhistleImage.length + " x " + compressedWhistleImage[0].length);
-				
-				float[][][] input = new float[1][][];
-				input[0] =  JamArr.doubleToFloat(compressedWhistleImage);
-			
-				System.out.println("Model output: ");
-				float[] outputPython = model.runModel(input);
-				
-				input[0] =   JamArr.doubleToFloat(JamArr.transposeMatrix(pamguardWhistleImage));
-				
-				System.out.println("Size Java: " + 	input[0].length + " x " + 	input[0][0].length);
-
-				//a bit ugly but works.
-//				transformedData2 = JamArr.transposeMatrix(transformedData2);
-				
-				
-				float[] outputJava = model.runModel(input);
-				
-				for (int i=0; i<outputPython.length; i++) {
-					System.out.println(String.format("Output Python: %.4f Java: %.4f",outputPython[i],outputJava[i] )); 
-				}
-
-				MatFile matFileWrite = Mat5.newMatFile()
-					    .addArray("imagePython",DLMatFile.array2Matrix(compressedWhistleImage))
-					    .addArray("imageJava",DLMatFile.array2Matrix(JamArr.transposeMatrix(pamguardWhistleImage)));
-				Mat5.writeToFile(matFileWrite, "C:\\Users\\Jamie Macaulay\\MATLAB Drive\\MATLAB\\PAMGUARD\\deep_learning\\delphinID\\whistle_image_inout_test.mat");
-
-//			JamArr.printArray(output);
-			
-		} catch (MalformedModelException | IOException e) {
-			// TODO Auto-generated catch block
-			e.printStackTrace();
-		} 
-		
-		System.out.println("DelphinID mode test end");
-
-	}
-
-
-}

src/test/resources/rawDeepLearningClassifier/DelphinID/ReadMe.txt

@@ -0,0 +1 @@
+Dde_415_s10_SI20120620_171333_d042_50.mat contains a Python image (whistle_image_gray_python) and the whistle contours (tfcontours) for a single segment. The whistle contours can be opened in Java to check that the and converted to an image then compared to the Python image.