Skip to content

C. Organize data

This step is to be done after the folders have been organized. To make sure, that there are no issues with the data and no mistakes were made during organization, several checks are run with the following jupyter notebook:

$ dataOrganization/C_checkDataStructure.ipynb

Input

Some input needs to be defined by the user so the script can work properly. You can find this under the cell Main Script :

  1. images = folder that holds all the organized data
  2. skipSeasons = If skipSeasons is an empty List = [], then the formatting for all seasons will be checked. However, when actively fixing issues, you know that the formatting of some seasons is ok. You can then give all the seasons that are ok as a list of strings such as ["2011_dry_season", "2012_wet_season"]. Then then format checking of these seasons will be skipped which will make it go faster. I recommend to make a check of all seasons after you have finished fixing issues.

How it works

It will go through all folders in a season and print out any issues it finds:

  1. coordinate formatting. Same formatting needed as above. This should not really come up as we fix this issue in the scripts before. But if not, adjust coordinates to the proper format.
  2. Checking of the time subfolders (most issues will be here). Folders with time_shift in the name will be exempt from these checks:
  3. it checks if the naming of the time folder is a proper date - if not check the issue and fix the naming
  4. it uses the first date of the first time subfolder to check if the Location Folder in the right season, and if it is not in the correct season it suggests the season it should be in. Move the folder there, be sure it is not at undocumented time_shift folder.
  5. Checks if a time folder has more images that its previous folder, usually camera traps take 1000 pictures and then save the images in a new folder. Therefore, if a folder thats later in time has more pictures, it suggests that the camera was set up a second time. This is not always the case and needs to be checked. If the second folder is actually a continued records of the first (eg. battery was switched but there is no time gap in the monitoring) you add _continued to the name of the second time folder to indicate that they belong to the same session. But there are also cases where 2 camera traps set up at the same time at the same location (time overlaps but view should be different) or the battery ran out and was only replaces a while later (e.g. 7 days later) then the folders should be split into different sessions with the numbering system explained above.
  6. A similar check as above is done, but not with number of images but to check for time gaps. If 2 time folders are more than 2 days apart, it will say that folders are not continuous in time. This may be se same case as above and needs to be manually checked if splitting of the time folders into different sessions is needed.
  7. it checks, if a folder with the same coordinates is within the same season. This ignores foldes that follow the proper numbering system. It can happen that the same folder was named differently in different backups and the duplicate was therefore not found in step A. Check the same things for the found duplicates as you did in step A.

The data is "clean" if there are no warnings or errors displayed. This step needs to be complete before the following steps can be excuted.