Skip to content

E. update Database

this script checks for cohesion between the organized data and the database

This is done using the following python script:

$ dataOrganization/E_checkDatabaseCohesion.py

Input

This scripts needs access to the database, so check that you have the following information:

for database connection:

DB_USER = user name for the database with read and write access
DB_PASS = password for the database user
DB_HOST = "localhost"
DB_NAME = "chinkoCamTrapData"
  1. seasonFolders = season folders that you want to check cohesion for, eg. when you organize data from 2023 you will not need to check for choesion in other years.
  2. dataPath = path to the directory containing the organized data
  3. outputPath = path to the directory where the output will be saved

How it works

The script will go through all the season folders and check if the information in the database is consistent with the organized data. It will check for the following issues:

Sessions:

  1. checks if all sessions exist in the database and if all database sessions exist in the organized data. It will print out any missing sessions to the terminal and specify if they are missing in the database or in the organized data.

  2. add any sessions that are in the database but not in the organized data and should be deleted from the database to a list variable called oldSessions. The naming convention for the sessions in the list is year_season/session_name. This list is then used to delete the session, along with all the images and tags connected to it.

  3. add any sessions that are in the organized data but not in the database and should be added to the database to a list variable called missingSessions. The naming convention for the sessions in the list is year_season/session_name. This list is then used to add the session where it extracts all data directly from the folder, along with all the images and tags connected to it.

Images:

  1. checks if all that all images in the organized data exist in the database and if all images in the database exist in the organized data. For every session where there is a missmatch, it will print to the terminal how many images are missmached and generate a file missingImagesinDatabase_season_sessionName.csv or missingImagesinFolder_season_sessionName.csv in the output folder. The files will contain the names of the missing images and will only be generated if there are missmatches. Note that this may take a while to run as it checks every image in the organized data and the database for the specific session.

  2. adds images that are in the organized data but not in the database to the database. It will take the information from the previously generated csv files. In the function there is the option of deleteErrorFiles for image files that are not machine readable and throw errors. If this argument is set to False, it will simply skip the images and NOT delete them. If it is set to True, it will prompt the user to confirm deletion of the images from the organized data.