B. Prepare data
In order to have uniform format and to check for potential issue, a script is run to quality check the data and - if there are no problems - to rename the Time Folders and move to folders into their proper season.
This is done with the following jupyter notebook:
$ dataOrganization/B_nameTimeFolders.ipynb
Input¶
Some input needs to be defined by the user so the script can work properly. You can find this under the cell define input :
- folderName = folder that holds the new Location Folders that need to be sorted into the organized data
This should be the same one you specified in A: checking for duplicates
How it works¶
The scripts will do quality checks for all folders within your given folder that have a coordinates in the name with the format {1-2}digits.{6}digitsN {1-2}digits.{6}digitsE such as 6.229627N_23.945689E. If the coordinate format is not correct, no further quality checks will be made. It is common that there are not 6 digits after the dot, which is done if the last digits are all 0. So adding 0s would remedy the issue.
If it has a coordinate in the name it will check differnt attributes. If none of the following checks throw an error, the Time Folder is renames to its proper start and end date. If there is an issue, the Time Folder needs to be renamed manually: 1. Does the Location Folder have time subfolders? Sometimes the images are directly in the Location Folder. In this case make a new folder and put the images inside. 2. Does a Time Folder have images? It will five a warning if there is a folder with no images. Check if what is in this folder and if it is needed. 3. For each Time Folder, the script reads the metadata of the first and last imagetimestamp to define the start and end date. 4. Before renaming the folders, it will check some time formatting of the session. 1. If the end date is before the start date. This happens either when there is a timeshift or the camera trap session goes over new years. The script will then suggest a proper start and end date to you after going through all metadata. It is likely a timeshift if the suggested start date is several years before the end date and is on the 1st of January. Then the Location Folder should be marked in the name with time_shift. Sometimes the dates can be inferred with the time frames of the other folders, but that is not always possible. If you see that the suggested dates are someting like "05.12.2021 to 31.01.2022" it likely just goes through the new year and can be renamed manually with the given suggested dates. 2. It checks the length of the camera trap session. Usually, camera traps run for less than half a year. If it goes for longer, it will give you a warning to check what the case is. Maybe the camera actually ran for a very long time, more often however, more that one camera trap session at the same location is in the same folder. Eg. The Camera was set up from January - March and then again from August - December. If we treat it as one session we assume monitoring from January to December. Therefore any continous monitoring session should get its own Location Folder that are then numbered (order numbering by start date of each session). So Location_Folder_A with a setup as above should be changed to Location_Folder_A_1 (Jan - Mar) and Location_Folder_A_2 (Aug - Dec).
As a summary: The script checks each coordinate folder indicated by "checking folder: Location Folder Name". If below that no warnings are printed before the next Location Folder is checked, the Location Folder had no issues and all Time Folders were renamed. This can then be moved into the organized data. If there is a warning for a Location Folder, it is printed right below "checking folder: Location Folder Name". Resolve the issues as explained above. Rerun the script until no more warning is shown for the folder you want to move. I prefer working one Locaton Folder at a time. Where I resolve all issues regarding that folder,rerun the script to check that it worked, move it to its proper season and then rerun the script again to check for the next folder.
moving the data¶
the safest way to move the folders is by executing the cells below Moving the Location Folders. Here, you need to give the name of the season you wish to move the folder into, and folders are the previously quality checked and approved folders that properly go into the given season. The proper season is identified as described at the beginnig. ONLY the start date of the first Time Folder of the Location Folder is relevant, nothing else.
If any folders do not have coordinates, they are not moved into the organized data. They need to be double checked by an expert (Thierry) who may be able to figure out where they are located.