Skip to content

A. Check for duplicates

As the data has several copies, it was sometimes unclear wether a specific folder was already organized or not. Therefore, the first step is to check wether the data we wish to add to the organization is not already there. This is done with the jupyter notebook:

$ dataOrganization/A_findDuplicateFolders.ipynb

Input

Some input needs to be defined by the user so the script can work properly. You can find this under the cell define input :

  1. organizedFilesPath = folder where the already organized camera trap images are
  2. unorganizedFilesPath = folder that holds the new Location Folders that need to be sorted into the organized data

How it works

The scripts checks if a folder with the same name already exists. It will ignore common addatives like "time_shift" in the name. If an organized and unorganized folder are found to be duplicates of each other, the scripts checks if they 1. Have the same number of Time Folders and 2. The same number of images. If that is the case, the script compares every 100th image between the 2 folders by pixels to see if they are identical. If all these checks are true, it will delete the folder that is in the unorganizedFilesPath.

If any of the above are false. It will print the name of the unorganized and organized folder giving you information on the number of Time Folders and images for both. From there on you will have to manually inspect if the data is the same or not. Common cases are:

  • the number of Time folders is identical and number images is almost identical:
  • sometimes I have deleted a few images because they either had technical issues, where taken by accident a few days before or after the camera session to test if the camera works or other similar reasons.
  • Decision: They are the same, unorganized data folder can be deleted
  • the number of Time folders is different but same number of images:
  • Sometimes data is differently organized and folders where merged. I take a lot of time to check if the bigger folder is a merge of the smaller ones and checking the time range written in the folder names.
  • Decision: This strongly depends on what you find. If they bigger one is a merge of the smaller folders then the unorganized can be deleted. But they may also be different sessions from the same locations.
  • the number of Time Folders and Images are different:
  • Sometimes different sessions (recordings that are not continuous in time) are in the same unorganized folder and we only have organized a single session yet. Carefully check with the time range which data is already organized and which not. Decision: See what data is missing in the organzed data and add it. If it is not a continuous effort make a second location folder and add the data there.