Updates to deep learning module to handle individual clicks

readme.md

@@ -1,22 +1,22 @@
 # PAMGuard
 
 ## Introduction
-PAMGuard is a bioacoustics analysis program designed for use in real time research contexts and for the processing of large datasets. PAMGuard provides users access to a suite of state-of-the-art auotmated analysis algorithms alongside displays for visualisation data and a comprehensive data management systems. 
+PAMGuard is a bioacoustics analysis program designed for use in real time research contexts and for the processing of large datasets. PAMGuard provides users access to a suite of state-of-the-art automated analysis algorithms alongside displays for visualisation data and a comprehensive data management system. 
 
 ## Why do we need PAMGuard?
-PAMGuard fufills two main requirements within marine bioacoustics
+PAMGuard fulfils two main requirements within marine bioacoustics
 
-1) **Real time operation**: Almost all PAMGuard features and modules work in real time - this allows scientists and industry to detect, classify and localise the position of animals in real time on a standard consumer laptop, enabling mitigation and research surveys without expensive bespoke software solutions and the transparncy of open source software.  
+1) **Real time operation**: Almost all PAMGuard features and modules work in real time - this allows scientists and industry to detect, classify and localise the position of animals in real time on a standard consumer laptop, enabling mitigation and research surveys without expensive bespoke software solutions and the transparency of open-source software.  
 
-2) **Processing and visuslisation of large datasets**: Off-the-shelf autonomous PAM devices, large scale surveys involving drifters, towed arrays, glidders, bottom mounted devices and real time continuous monitoring system all generate huge volumes of data whcih requires automated analysis approaches. PAMGuard allows the processing of very large passive acoustic datasets using automated algorothms and crucially provides comprehensive visualisation tools for a manual analyst to check the results.
+2) **Processing and visualisation of large datasets**: Off-the-shelf autonomous PAM devices, large scale surveys involving drifters, towed arrays, glidders, bottom mounted devices and real time continuous monitoring system all generate huge volumes of data whcih requires automated analysis approaches. PAMGuard allows the processing of very large passive acoustic datasets using automated algorothms and crucially provides comprehensive visualisation tools for a manual analyst to check the results.
 
 ## Installation
-PAMGuard is available on Windows and can be downloaded from the [PAMGuard website](www.pamguard.org). Note that we are considering MacOS and Linux installers but they are not available at this time. 
+PAMGuard is available on Windows and can be downloaded from the [PAMGuard website](www.pamguard.org). Note that we are considering MacOS and Linux installers, but they are not available at this time. 
 
 ## Quick start tutorial
-PAMGuard is a modular program with two modes; real-time and viewer. Typically a user will start with real-time model, either in the field collecting data or post processing sound files from a recorder. Once data are processed, users move on to viewer mode where data can be explored and further processed. 
+PAMGuard is a modular program with two modes: real-time and viewer. Typically, a user will start with real-time model, either in the field collecting data or post processing sound files from a recorder. Once data are processed, users move on to viewer mode where data can be explored and further processed. 
 
-Upon opening PAMGuard for the first time you are greeted with a blank screen. You must add a series of modules to create the desired acosutic workflow - this is referred to as PAMGuard *data model*. For example if processing sound files then first add the Sound Acquisition module **_File->Add Modules->Sound Processing->Sound Acquisition_**. Then add the desired detection algorothms e.g.  **_File->Add Modules->Detector->Click Detectors_**. Some modules (such as the click detector) have their own bespoke displays, others are added to more generalised displays. For example, the whistle and moan detector module shows detections on a spectrgram display, time base display, map etc.. First add a new tab using  **_File->Add Modules->Displays->User Display**. Click on the user display tab and then from the top menu select **_User display-> New Spectrogram_**. Right click on the added spectrgram and select whistle and moan contours to show whistle detections overlaid on the raw spectrgram. 
+Upon opening PAMGuard for the first time you are greeted with a blank screen. You must add a series of modules to create the desired acoustic workflow - this is referred to as PAMGuard *data model*. For example, if processing sound files then first add the Sound Acquisition module **_File->Add Modules->Sound Processing->Sound Acquisition_**. Then add the desired detection algorothms e.g.  **_File->Add Modules->Detector->Click Detectors_**. Some modules (such as the click detector) have their own bespoke displays, others are added to more generalised displays. For example, the whistle and moan detector module shows detections on a spectrgram display, time base display, map etc.. First add a new tab using  **_File->Add Modules->Displays->User Display**. Click on the user display tab and then from the top menu select **_User display-> New Spectrogram_**. Right click on the added spectrgram and select whistle and moan contours to show whistle detections overlaid on the raw spectrgram. 
 
 Make sure to add the database and binary file storage modules **_File->Add Modules->Utilities->..._**) to save data then press the run button (red button) and data will process. PAMGuard can handle huge datasets so running might take hours or even days. Progress is shown on the bottom of the screen. 
 
@@ -24,10 +24,10 @@ Once the data has run, open PAMGuard viewer mode. Select the database you used t
 
 ## Features
 ### Hardware integration
-PAMGuard connects with hardware such as various GPS and AIS systems and a multitude of different sound cards (e.g. [National Instruments](www.ni.com) devices, [SAIL DAQ cards](www.smruconsulting.com/contact-us), almost all ASIO sound cards and standard computer sound cards) for real time data collection and processing. PAMGuard also works with some very bespoke hardware such as [DIFAR Sonobuoys]();
+PAMGuard connects with hardware such as various GPS and AIS systems and a multitude of different sound cards (e.g. [National Instruments] (www.ni.com) devices, [SAIL DAQ cards](www.smruconsulting.com/contact-us), almost all ASIO sound cards and standard computer sound cards) for real time data collection and processing. PAMGuard also works with some very bespoke hardware such as [DIFAR Sonobuoys] ();
 
 ### Real time operation
-PAMGuard takes advanatge of multi-core processors to run multiple signal processing automatic analysis algorithms in real time to detect whales, dolphins, bats etc. Data are shown in different displayes, including interactive spectrograms and maps. You might be using PAMGuard for simply viewing a spectrgram and making recordings or running deep learning algorithms for multiple species and loclaising the results to view locations on a map. Whatever acosutic workflow a user creates, PAMGuard can run it in real time. 
+PAMGuard takes advantage of multi-core processors to run multiple signal processing automatic analysis algorithms in real time to detect whales, dolphins, bats etc. Data are shown in different displays, including interactive spectrograms and maps. You might be using PAMGuard for simply viewing a spectrogram and making recordings or running deep learning algorithms for multiple species and localising the results to view locations on a map. Whatever acoustic workflow a user creates, PAMGuard can run it in real time. 
 
 ### Support for compressed audio
 PAMGuard supports processing audio data from standard files (e.g. wav, aif) and also compressed files (e.g. .flac and .sud). Notew that sud files are created on SoundTraps widely used marine recorders and can be read by PAMGuard without decompressing - PAMGuard will automtically import click detections if present in sud files. PAMGuard also supports importing detection data from CPODs and FPODs. 
@@ -36,7 +36,7 @@ PAMGuard supports processing audio data from standard files (e.g. wav, aif) and
 PAMGuard is designed to collect/process data from large acosutic datasets. PAMGuard stores data in an SQLite databases and "Binary" files. The database stores important metadata such as when data has been processed and some low volume data streams such as GPS. Binary files are not human readbale but efficient to access - PAMGuard stores detection data (e.g. clicks, whistles, noise, etc) in these files. this allows PAMGuard to rapidly access data from large datasets. Data from binary files can be viewed in PAMGuard viewer mode or can be exported to MATLAB using the PAMGuard-MATLAB library or the exported to R using the R PAMBinaries package. 
 
 ### Access to detection and classification algorithms
-PAMGuard allows users to inegrate automated detection and classification algorithms directly into their acosutic workflow. There are a multitude of differwent algorothms to choose from, including a basic click detector, whislte and moan detector, GPL detector, click train detectors and many others. The idea behind PAMGuard is allow researchers to access open source state-of-the-art algorithms devleoped within the scientific community - if you want to contribute and get your algorithm into PAMGuard get in touch. 
+PAMGuard allows users to inegrate automated detection and classification algorithms directly into their acosutic workflow. There are a multitude of differwent algorothms to choose from, including a basic click detector, whislte and moan detector, GPL detector, click train detectors and many others. The idea behind PAMGuard is allow researchers to access open-source state-of-the-art algorithms devleoped within the scientific community - if you want to contribute and get your algorithm into PAMGuard get in touch. 
 
 ### Localisation
 PAMGuard has a mutltude of different options for acoustic loclaisation. There's a comprehesnive beam forming module for beam forming arrays, a large aperture localiser for 3D loclaisation and target motion analysis for towed hydrophone arrays. 
@@ -60,7 +60,7 @@ PAMGuard allows users to run their own deep learning models using the deep learn
 PAMGuard Integrates with Tethys database. Users can export processed PAMGuard data to a Tethys database seamlessly; this ifeature is great for large scale projects or organisatiosn with long term datasets. 
 
 ## Feature roadmap
-There's lots of features we would like to add to PAMGuard. If you want to add a feature you can either code it up yourself in Java and submit a pull request or get in touch with us to discuss how to it might be integrated. Some smaller features might be in our roadmap anyway but larger features usually require funding. Some features we are thinking about (but do not necassarily have time for yet) are;
+There's lots of features we would like to add to PAMGuard. If you want to add a feature you can either code it up yourself in Java and submit a pull request or get in touch with us to discuss how to it might be integrated. Some smaller features might be in our roadmap anyway but larger features usually require funding. Some features we are thinking about (but do not necassarily have time for yet) are.
 
 * Support for decidecade noise bands (base 10 filter bank) in noise band monitor to meet Euopean standards
 * Capabaility to export data directly from PAMGaurd e.g. as MAT files (in progress).
@@ -75,5 +75,5 @@ If you are a PAMGuard developer, you should clone and branch this repository and
 
 PAMGuard uses Maven as build tool. 
 
-# Organisation and License
-PAMGuard is open source under an MIT license. It is currently primarily managed by the Sea Mammal Research Unit within the [University of St Andrews](https://www.st-andrews.ac.uk/). Please get in touch if you have any questions. 
+## Organisation and License
+PAMGuard is open source under an MIT license. It is currently primarily managed by the Sea Mammal Research Unit within the [University of St Andrews](https://www.st-andrews.ac.uk/). Please get in touch if you have any questions.

src/pamViewFX/symbol/PeakFreqOptionsPane.java

@@ -153,7 +153,7 @@ public class PeakFreqOptionsPane extends StandardSymbolModifierPane  {
 
 			super.setParams();
 			
-			//important to have here because the super.setParams set this bak to false. 
+			//important to have here because the super.setParams set this back to false. 
 			setParams = true; 
 
 //			StandardSymbolOptions standardSymbolOptions = (StandardSymbolOptions) getSymbolModifier().getSymbolChooser().getSymbolOptions();

src/rawDeepLearningClassifier/dlClassification/genericModel/DLModelWorker.java

@@ -53,7 +53,7 @@ public abstract class DLModelWorker<T> {
 	public float[][][] dataUnits2ModelInput(ArrayList<? extends PamDataUnit> dataUnits, float sampleRate, int iChan){
 		
 		@SuppressWarnings("unchecked")
-		ArrayList<GroupedRawData> rawDataUnits = ( ArrayList<GroupedRawData>) dataUnits;
+		ArrayList<GroupedRawData> rawDataUnits = (ArrayList<GroupedRawData>) dataUnits;
 		 
 		//the number of chunks. 
 		int numChunks = rawDataUnits.size(); 

src/rawDeepLearningClassifier/dlClassification/genericModel/GenericModelWorker.java

@@ -45,7 +45,7 @@ public class GenericModelWorker extends DLModelWorker<StandardPrediction> {
 				waveStack[i] = transformedDataStack[i][0]; 
 			}
 			
-			//System.out.println("RUN GENERIC MODEL WAVE: " + waveStack.length +  "  " + waveStack[0].length +  " " + waveStack[0][0]);
+			System.out.println("RUN GENERIC MODEL WAVE: " + waveStack.length +  "  " + waveStack[0].length +  " " + waveStack[0][0]);
 			results =  getModel().runModel(waveStack);
 		}
 //		System.out.println("GENERIC MODEL RESULTS: " + (results== null ? null : results.length));

src/rawDeepLearningClassifier/layoutFX/DLSettingsPane.java

@@ -3,12 +3,14 @@ package rawDeepLearningClassifier.layoutFX;
 import java.util.ArrayList;
 
 import org.controlsfx.control.PopOver;
+import org.controlsfx.control.ToggleSwitch;
 
 import PamController.PamGUIManager;
 import PamController.SettingsPane;
 import PamDetection.RawDataUnit;
 import PamView.dialog.warn.WarnOnce;
 import PamguardMVC.PamDataBlock;
+import PamguardMVC.PamRawDataBlock;
 import PamguardMVC.dataSelector.NullDataSelectorCreator;
 import clickDetector.ClickDetection;
 import clipgenerator.ClipDataUnit;
@@ -116,6 +118,10 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 
 	private DLModelSelectPane modelSelectPane;
 
+	private PamToggleSwitch segEnableSwitch;
+
+	private PamGridPane segmenterGridPane;
+
 
 
 	public DLSettingsPane(DLControl dlControl){
@@ -178,11 +184,26 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 		vBox.getChildren().add(createDataSelectorPane());
 
 
-		// the segmentation params
-		Label label = new Label("Segmentation"); 
-		PamGuiManagerFX.titleFont2style(label); 
+		// the segmentation parameters
+		Label segLabel = new Label("Segmentation"); 
+		PamGuiManagerFX.titleFont2style(segLabel); 
 
-		vBox.getChildren().add(label);
+
+		segEnableSwitch = new PamToggleSwitch("Enable");
+		segEnableSwitch.selectedProperty().addListener((obsVal, oldVal, newVal)->{
+			enableControls(); 
+		});
+		segEnableSwitch.setVisible(false); //set visible by default
+		segEnableSwitch.setAlignment(Pos.CENTER_RIGHT);
+		
+
+		//segmentation can have an option to disable for certain input datablocks
+		PamBorderPane segmenterPane = new PamBorderPane();
+		segmenterPane.setLeft(segLabel);
+		segmenterPane.setRight(segEnableSwitch);
+
+		//add to the main pane
+		vBox.getChildren().add(segmenterPane);
 
 		windowLength = new PamSpinner<Integer>(0, Integer.MAX_VALUE, 10,  10000); 
 		windowLength.getStyleClass().add(Spinner.STYLE_CLASS_SPLIT_ARROWS_HORIZONTAL);
@@ -211,7 +232,7 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 			hopLength.getValueFactory().setValue(Math.round(windowLength.getValue()/2));
 		});
 
-		PamGridPane segmenterGridPane = new PamGridPane(); 
+		segmenterGridPane = new PamGridPane(); 
 		segmenterGridPane.setHgap(5);
 
 		ColumnConstraints col1 = new ColumnConstraints();
@@ -244,35 +265,16 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 
 		vBox.getChildren().add(label2);
 
-		/**
-		 * Pane which allows users to select a model type. 
-		 */
+		//Pane which allows users to select a model from a file, link or default model. 
 		modelSelectPane = new DLModelSelectPane(this); 
 
-		//		//add the possible deep learning models. 
-		//		dlModelBox= new ComboBox<String>();
-		//		for (int i=0; i<dlControl.getDLModels().size(); i++) {
-		//			dlModelBox.getItems().add(dlControl.getDLModels().get(i).getName()); 
-		//		}
-		//		dlModelBox.prefWidthProperty().bind(vBox.widthProperty());
-		//
-		//		dlModelBox.setOnAction((action)->{
-		//			setClassifierPane(); 
-		//			if (mainPane!=null) {
-		//				if (mainPane.getScene().getWindow() instanceof Stage) {
-		//					Stage stage = (Stage) mainPane.getScene().getWindow();
-		//					stage.sizeToScene();
-		//				}
-		//			}
-		//			//this.dlControl.getAnnotationType().getSymbolModifier(symbolChooser).
-		//		});
-		//
-		//		vBox.getChildren().add(dlModelBox);
-
+		//create pane which shows the classifier settings
 		classifierPane = new PamBorderPane(); 
 
-		vBox.getChildren().addAll(modelSelectPane, classifierPane);
+		//bump this in case no settings
+		segEnableSwitch.setSelected(true);
 
+		vBox.getChildren().addAll(modelSelectPane, classifierPane);
 
 		return vBox; 
 	}
@@ -354,6 +356,20 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 		}
 
 		dataSelectorButton.setDisable(!dataSelectorCheckBox.isSelected());
+
+		boolean segEnable = true; //should we enable segmenter controls
+		if (sourcePane.getSource() instanceof PamRawDataBlock) {
+			//if a raw data block then we always enable segmentation no matter what
+			segEnable=true;
+			segEnableSwitch.setVisible(false);
+		}
+		else {
+			segEnable = segEnableSwitch.isSelected();
+			segEnableSwitch.setVisible(true);
+		}
+
+		segmenterGridPane.setDisable(!segEnable);	
+		infoLabel.setDisable(!segEnable);	
 	}
 
 
@@ -380,11 +396,11 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 	 * Set the classifier pane. 
 	 */
 	protected void setClassifierPane() {
-		
-		
+
+
 		//set the classifier Pane.class 
 		if (modelSelectPane.currentClassifierModel!=null && modelSelectPane.currentClassifierModel.getModelUI()!=null) {
-			
+
 			classifierPane.setCenter(modelSelectPane.currentClassifierModel.getModelUI().getSettingsPane().getContentNode()); 
 
 			if (modelSelectPane.currentClassifierModel!=null) {
@@ -456,7 +472,7 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 		//		}
 
 		currParams.useDataSelector = dataSelectorCheckBox.isSelected(); 
-		
+
 		if (dlControl.getDataSelector()!=null) {
 			dlControl.getDataSelector().getDialogPaneFX().getParams(true); 
 		}
@@ -471,6 +487,8 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 
 		currParams.modelURI = this.modelSelectPane.currentSelectedFile; 
 
+		currParams.enableSegmentation =	segEnableSwitch.isSelected();
+
 		return currParams;
 	}
 
@@ -555,10 +573,12 @@ public class DLSettingsPane  extends SettingsPane<RawDLParams>{
 
 		setClassifierPane(); 
 
-		enableControls(); 
-
 		setSegInfoLabel();
 
+		segEnableSwitch.setSelected(currParams.enableSegmentation);
+
+		enableControls(); 
+
 		//		//set up the model and the custom pane if necessary.  
 		this.modelSelectPane.loadNewModel(currParams.modelURI); 
 		//this.modelSelectPane.updatePathLabel(); 

src/rawDeepLearningClassifier/segmenter/GroupedRawData.java

@@ -114,6 +114,15 @@ public class GroupedRawData extends PamDataUnit implements PamDetection, Cloneab
 	public double[][] getRawData() {
 		return rawData;
 	}
+	
+
+	/**
+	 * Set the raw data grouped by channel.
+	 * @param the raw acoustic data to set
+	 */
+	public void setRawData(double[][] rawData) {
+		 this.rawData=rawData;
+	}
 
 	/**
 	 * Get the current pointer for rawData.

src/rawDeepLearningClassifier/segmenter/SegmenterProcess.java

@@ -6,7 +6,6 @@ import java.util.Arrays;
 
 import PamController.PamController;
 import PamDetection.RawDataUnit;
-import PamUtils.PamArrayUtils;
 import PamUtils.PamUtils;
 import PamView.GroupedSourceParameters;
 import PamView.PamDetectionOverlayGraphics;
@@ -493,10 +492,13 @@ public class SegmenterProcess extends PamProcess {
 							rawDataChunk[i], chans[i], dlControl.getDLParams().rawSampleSize, dlControl.getDLParams().sampleHop, true);
 				}
 				else {
-					//send the whole data chunk to the deep learning unit
+//					//send the whole data chunk to the deep learning unit
 					newRawData(pamDataUnit,
 							rawDataChunk[i], chans[i], 	rawDataChunk[i].length, rawDataChunk[i].length, true);
+//					currentRawChunks[i] = new GroupedRawData(pamDataUnit.getTimeMilliseconds(), getSourceParams().getGroupChannels(i), 
+//							pamDataUnit.getStartSample(), 	rawDataChunk[i].length, 	rawDataChunk[i].length); 
 				}
+				
 			//the way that the newRawdata works is it waits for the next chunk and copies all relevant bits
 			//from previous chunks into segments. This is fine for continuous data but means that chunks of data
 			//don't get their last hop...
@@ -659,8 +661,8 @@ public class SegmenterProcess extends PamProcess {
 
 						//add the hop from the current grouped raw data unit to the new grouped raw data unit 
 						//					System.out.println("Pointer to copy from: "  + (currentRawChunks[i].rawData[groupChan].length - dlControl.getDLParams().sampleHop )); 
-						int overFlow2 = nextRawChunks[i][j].copyRawData(lastRawDataChunk.rawData[groupChan], lastRawDataChunk.rawData[groupChan].length - getBackSmapleHop()  , 
-								getBackSmapleHop() , groupChan);
+						int overFlow2 = nextRawChunks[i][j].copyRawData(lastRawDataChunk.rawData[groupChan], lastRawDataChunk.rawData[groupChan].length - getBackSmapleHop(rawSampleSize, rawSampleHop)  , 
+								getBackSmapleHop(rawSampleSize, rawSampleHop) , groupChan);
 
 						//					System.arraycopy(currentRawChunks[i].rawData[groupChan], currentRawChunks[i].rawData[groupChan].length - dlControl.getDLParams().sampleHop,
 						//							nextRawChunks[i].rawData[groupChan], 0, dlControl.getDLParams().sampleHop); 
@@ -734,11 +736,12 @@ public class SegmenterProcess extends PamProcess {
 
 		//add some extra metadata to the chunks 
 		packageSegmenterDataUnit(currentRawChunks[i]); 
-		//System.out.println("Segmenter process: Save current segments to datablock: " + currentRawChunks[i].getParentDataUnit().getUID()); 
+		System.out.println("Segmenter process: Save current segments to datablock: " + currentRawChunks[i].getParentDataUnit().getUID() + " " + i + currentRawChunks[i].getRawData()[0][0]); 
 
 		//send the raw data unit off to be classified!
 
 		this.segmenterDataBlock.addPamData(currentRawChunks[i]);
+		
 
 		if (nextRawChunks[i]!=null) {
 			int n = nextRawChunks[i].length-1; 
@@ -771,8 +774,9 @@ public class SegmenterProcess extends PamProcess {
 	}
 
 
-	private int getBackSmapleHop() {
-		return dlControl.getDLParams().rawSampleSize - dlControl.getDLParams().sampleHop; 
+	private int getBackSmapleHop(int segSize, int segHop) {
+		return segSize-segHop;
+//		return dlControl.getDLParams().rawSampleSize - dlControl.getDLParams().sampleHop; 
 	}
 
 	//	/***TODO - hand small windows***/

src/test/rawDeepLearningClassifier/ClickDLTest.java

@@ -1,31 +1,132 @@
 package test.rawDeepLearningClassifier;
 
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+
+import org.jamdev.jdl4pam.transforms.DLTransformsFactory;
+import org.jamdev.jdl4pam.transforms.DLTransfromParams;
+import org.jamdev.jdl4pam.transforms.SimpleTransformParams;
+import org.jamdev.jdl4pam.transforms.DLTransform.DLTransformType;
 import org.junit.jupiter.api.Test;
 
+import PamDetection.RawDataUnit;
+import PamUtils.PamArrayUtils;
+import PamguardMVC.RawDataHolder;
+import matchedTemplateClassifer.MatchTemplate;
 import rawDeepLearningClassifier.dlClassification.genericModel.GenericModelParams;
 import rawDeepLearningClassifier.dlClassification.genericModel.GenericModelWorker;
+import rawDeepLearningClassifier.dlClassification.genericModel.StandardPrediction;
+import rawDeepLearningClassifier.segmenter.GroupedRawData;
+import us.hebi.matlab.mat.format.Mat5;
+import us.hebi.matlab.mat.format.Mat5File;
+import us.hebi.matlab.mat.types.Matrix;
+import us.hebi.matlab.mat.types.Struct;
 
 public class ClickDLTest {
 
 	
 	@Test
 	public void clickDLTest() {
+		
+		float SAMPLE_RATE = 500000;
 		//relative paths to the resource folders.
 		System.out.println("*****Click classification Deep Learning*****"); 
 
-		//relative paths to the resource folders.
-		String relModelPath  =	"D:/Dropbox/PAMGuard_dev/Deep_Learning/click_classifier_Thomas/best_model/saved_model.pb";
+		//relative paths to the resource folders.		
+		String relModelPath  =	"./src/test/resources/rawDeepLearningClassifier/Generic/risso_click/best_model/saved_model.pb";
+		String clicksPath  =	"./src/test/resources/rawDeepLearningClassifier/Generic/risso_click/clicks.mat";
+
+		Path path = Paths.get(relModelPath);
 
 
 		GenericModelWorker genericModelWorker = new GenericModelWorker(); 
 
 		GenericModelParams genericModelParams = new GenericModelParams(); 
-
-		genericModelParams.modelPath = relModelPath;
-		
-		genericModelWorker.prepModel(genericModelParams, null);
+		genericModelParams.modelPath =  path.toAbsolutePath().normalize().toString();
 
 	
+		//create the transforms. 
+		ArrayList<DLTransfromParams> dlTransformParamsArr = new ArrayList<DLTransfromParams>();
+
+		dlTransformParamsArr.add(new SimpleTransformParams(DLTransformType.NORMALISE_WAV)); 
+				
+		genericModelParams.dlTransfromParams = dlTransformParamsArr;
+		genericModelParams.dlTransfroms = DLTransformsFactory.makeDLTransforms((ArrayList<DLTransfromParams>)genericModelParams.dlTransfromParams); 
+		
+		
+		//create the clicks. 
+		path = Paths.get(clicksPath);
+		ArrayList<GroupedRawData> clicks = importClicks(path.toAbsolutePath().normalize().toString(),  SAMPLE_RATE); 
+		
+		//prep the model
+		genericModelWorker.prepModel(genericModelParams, null);
+		
+		
+		ArrayList<GroupedRawData> groupedData = new ArrayList<GroupedRawData>();
+		
+		int i=0;
+		
+		float prediction = 0; 
+
+		groupedData.add(clicks.get(i)); //TODO for loop
+
+		System.out.println("Waveform input: " + groupedData.get(i).getRawData().length + " " + groupedData.get(i).getRawData()[0].length);
+		
+		ArrayList<StandardPrediction> genericPrediction = genericModelWorker.runModel(groupedData,SAMPLE_RATE, 0);		
+
+		float[] output = genericPrediction.get(0).getPrediction();
+		
+		System.out.println(String.format("Click %d Predicted output: %.2f true output: %.2f passed: %b", clicks.get(i).getUID(),
+				output[0], prediction, output[0]>prediction*0.9 && output[0]<prediction*1.1)); 
+
+		
+	}
+	
+	/**
+	 * Import a bunch of clicks from a .mat file
+	 */
+	public static ArrayList<GroupedRawData> importClicks(String filePath, float sR) {
+
+		try {
+			 Mat5File mfr = Mat5.readFromFile(filePath);
+
+			//		//get array of a name "my_array" from file
+			Struct mlArrayRetrived = mfr.getStruct( "clickpreds" );
+
+			int numClicks= mlArrayRetrived.getNumCols();
+			ArrayList<GroupedRawData> clicks = new ArrayList<GroupedRawData>(numClicks); 
+
+			GroupedRawData clickData;
+			for (int i=0; i<numClicks; i++) {
+				Matrix clickWav= mlArrayRetrived.get("wave", i);
+				
+				double[][] clickwaveform= PamArrayUtils.matrix2array(clickWav);
+				
+				clickwaveform = PamArrayUtils.transposeMatrix(clickwaveform);
+				//System.out.println("click: " + click[0].length + " num: " + numClicks);
+				
+				Matrix clickUID= mlArrayRetrived.get("UID", i);
+				Matrix clickmillis= mlArrayRetrived.get("millis", i);
+				Matrix channelMap= mlArrayRetrived.get("channelMap", i);
+				Matrix startSample= mlArrayRetrived.get("startSample", i);
+				Matrix sampleDuration= mlArrayRetrived.get("sampleDuration", i);
+
+				clickData = new GroupedRawData(clickmillis.getLong(0), channelMap.getInt(0), startSample.getLong(0), sampleDuration.getLong(0), sampleDuration.getInt(0));
+				clickData.setUID(clickUID.getLong(0));
+				clickData.setRawData(clickwaveform);
+				
+				clicks.add(clickData); 
+			}
+			
+			return clicks; 
+		} 
+		catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+			return null; 
+		}
+
 	}
 	
 }