ConvertAndMerge
This is a basic example of me attempting to merge selected classes from multiple datasets into one to improve detection. In this example I am building and smoke and fire dataset to try and detect fires in valley where I live. Less a step by step and more my thinking and steps I took to sort what might get the best results since adjustments will need made for the data used to get the best results for that dataset. Hopefully this gives you ideas of how to build your own dataset and understand why you probably should build one instead of using someone else's. Also how things are sometimes counterintuitive to what you might think a change will do.
Note the python command may also python3 depending on your setup
Note too I'm adding tests and debug info to tools I as see the need so format of results may not match what you see in the current version of a tool
See Lessons learned at the bottom if you just want a summary.
See Keep improving for ideas on how to improve your dataset after you model is deployed.
First I gathered up a set of images from my cams to test with and as a starting point I did a train and test with them.
rem only needed to tee log to screen and file
PATH=C:\cygwin\bin;%PATH%
rem vars that may change
rem model name
set MNAME=fire
rem folder name which is used as default project name
set MFOLD=deepstack
rem name of docker container being used to test with
set DNAME=tester
rem change to trainer folder so imports work
cd ../deepstack-trainer/
del /S /Q train-runs\%MFOLD%\%MNAME%
rem run trainer
rem See https://securitycam101.rmrr42.com/2021/12/deepstack-training.html for info about options
del ..\deepstack\*.cache
python3 train.py --name %MNAME% --epochs 600 --exist-ok --dataset-path "../%MFOLD%" 2>&1 | tee %MNAME%.train.log
rem copy best model to MyModels
copy C:\DeepStackWS\deepstack-trainer\train-runs\%MFOLD%\%MNAME%\weights\best.pt C:\DeepStack\MyModels\%MNAME%.pt
rem restart container to pickup new model
docker restart %DNAME%
rem run tests
cd ../deepstack
python3 trainTest.py %MNAME% 2>&1 | tee %MNAME%.trainTest.log
pause
Which did OK given the small number of samples with min_confidence = 0.50
For images in:../RMRR.fire/train/
fire found 35.0 of 42.0 expected objects with an average confidence of 0.7513892102857143 min:0.5149792 max:0.87116677 adjusted avg:0.6261576752380952
smoke found 53.0 of 53.0 expected objects with an average confidence of 0.8004761194339624 min:0.53082126 max:0.92523134 adjusted avg:0.8004761194339624
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
clouds found 20.0 of 27.0 expected objects with an average confidence of 0.8333023735000001 min:0.5936424 max:0.90549916 adjusted avg:0.6172610174074075
plane_night found 1.0 of 2.0 expected objects with an average confidence of 0.66921365 min:0.66921365 max:0.66921365 adjusted avg:0.334606825
For images in:../RMRR.fire/test/
fire found 1.0 of 2.0 expected objects with an average confidence of 0.5553619 min:0.5553619 max:0.5553619 adjusted avg:0.27768095
smoke found 3.0 of 4.0 expected objects with an average confidence of 0.7383358066666666 min:0.61496925 max:0.8233214 adjusted avg:0.553751855
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
clouds found 1.0 of 1.0 expected objects with an average confidence of 0.65458566 min:0.65458566 max:0.65458566 adjusted avg:0.65458566
plane_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
Found 114.0 of 131.0 expected objects
Of 357 tests
Ran:357
Skipped:0
Passed:340
Warnings:0
Failed:17
Ran 356 trainTest tests in 0:00:11.178449
First make things less complex by moving the files in the images and labels folders under the train and test folders to their parents.
The next issue was they are only labeled 0=Not WildFire and 1=WildFire. Plus a quick check of the files and it appeared they ought to be labeled smoke and fire and I was not seeing any fire ones.
Plus there is no classes.txt file so the first thing I needed to do is make one. To keep it simple I used:
smoke
noSmoke
for trainPath/classes.txt. Then ran python chkClasses.py to see how many fire pics there are. And it comfirms no "WildFire" (noSmoke) pics are labeled
In ./../WildFire_Smoke_Dataset_YOLO/train/ from mapping files
0:smoke used 516.0 times.
1:noSmoke used 0.0 times.
In ./../WildFire_Smoke_Dataset_YOLO/train/../test from mapping files
0:smoke used 74.0 times.
1:noSmoke used 0.0 times.
Ran 1182 classes tests in 0:00:00.083001
We can then run training on them. If you are using my local training with docker DeepStack setup you can set up a script like this to train and test in one go.
rem only needed to tee log to screen and file
PATH=C:\cygwin\bin;%PATH%
rem vars that may change
rem model name
set MNAME=WildFire_Smoke_Dataset_YOLO
rem folder name which is used default project name
set MFOLD=WildFire_Smoke_Dataset_YOLO
rem name of docker container being used to test with
set DNAME=tester
rem change to trainer folder so imports work
cd ../deepstack-trainer/
rem run trainer
rem See https://securitycam101.rmrr42.com/2021/12/deepstack-training.html for info about options
python3 train.py --name %MNAME% --exist-ok --dataset-path "../%MFOLD%" 2>&1 | tee %MNAME%.train.out
rem copy best model to MyModels
copy C:\DeepStackWS\deepstack-trainer\train-runs\%MFOLD%\%MNAME%\weights\best.pt C:\DeepStack\MyModels\%MNAME%.pt
rem restart container to pickup new model
docker restart %DNAME%
rem run tests
cd ../deepstack
python3 trainTest.py %MNAME%
python3 trainTest.py %MNAME% ../RMRR.fire
pause
Sadly the results are not great with min_confidence on just 0.50
For images in:../WildFire_Smoke_Dataset_YOLO/train/
smoke found 513.0 of 516.0 expected times with an average confidence of 0.8350199973489278 min:0.5339119 max:0.9529293 adjusted avg:0.8301652299224805
noSmoke found 0.0 of 0.0 expected times with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
For images in:../WildFire_Smoke_Dataset_YOLO/train/../test/
smoke found 70.0 of 74.0 expected times with an average confidence of 0.8335191768571428 min:0.5964276 max:0.9340932 adjusted avg:0.7884640862162161
noSmoke found 0.0 of 0.0 expected times with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
Of 2952 tests
Ran:2952
Skipped:0
Passed:2945
Warnings:0
Failed:7
Ran 2951 trainTest tests in 0:01:34.884582
For images in:..\RMRR.fire\train/
fire found 0.0 of 42.0 expected times with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0.0
smoke found 1.0 of 53.0 expected times with an average confidence of 0.8056533 min:0.8056533 max:0.8056533 adjusted avg:0.015201005660377359
For images in:..\RMRR.fire\train/../test/
fire found 0.0 of 2.0 expected times with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0.0
smoke found 0.0 of 4.0 expected times with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0.0
Of 357 tests
Ran:357
Skipped:0
Passed:227
Warnings:0
Failed:130
This would imply that dataset is pretty much useless for detecting smoke in my valley camera views and is only OK for its own data. Plus I would expect better for a published set testing against its own files.
Flipping it and testing the WildFire_Smoke_Dataset_YOLO images against the RMRR.fire model does relatively better.
For images in:../WildFire_Smoke_Dataset_YOLO\train/
smoke found 42.0 of 516.0 expected objects with an average confidence of 0.667017940952381 min:0.51102 max:0.91914463 adjusted avg:0.05429215798449613
noSmoke found 0.0 of 0.0 expected objects with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
For images in:../WildFire_Smoke_Dataset_YOLO\train/../test/
smoke found 9.0 of 74.0 expected objects with an average confidence of 0.69011008 min:0.50798607 max:0.8675983 adjusted avg:0.08393230702702703
noSmoke found 0.0 of 0.0 expected objects with an average confidence of 0 min:1.0 max:0.0 adjusted avg:0
clouds found 35.0 of 0.0 expected objects with an average confidence of 0.7237117559999997 min:0.56235784 max:0.92387784 adjusted avg:0
Found 86.0 of 590.0 expected objects
Of 2952 tests
Ran:2952
Skipped:0
Passed:2413
Warnings:29
Failed:539
Ran 2951 trainTest tests in 0:01:43.260316
So adding part of the WildFire_Smoke_Dataset_YOLO set to the RMRR.fire set seems the way forward.
We'll want to move all train, test and valid folder image and txt files we want into one folder to simplify things. Since they are already labeled I'll stick them in the folder pointed to by labeled to avoid the need to change the config.py vars. Of course you could run against each folder separately but why?
That gives me:
In labeled/ from mapping files
0:smoke used 737.0 times.
1:noSmoke used 0.0 times.
I next did a quick inspection of the images to remove those that did not look like a fit:
- Removed night shots since my ColorVus see color 24/7. Cranked down to 1/3 second exposure they do not even need the moon to be up to see in color.
- Removed pixelated images. Not sure what is being blocked but pixelating seems to mess with detection and training. Cropping or a black box would be better which I could do but in this case the smoke is so far off it did not seem worth it.
- Removed images where I could not make out the smoke in Windows file manager in extra large icon view on my 40 inch monitor. I'm wanting alerts of people burning brush or lots of trash within about a mile of me (about how far the valley ridge is from me), not chimney smoke or fires well beyond the ridge.
That took me down to about half of the images
In labeled/ from mapping files
0:smoke used 371.0 times.
1:noSmoke used 0.0 times.
***Note it checks the train folder as well to let you know what you should leave even if not used in labeled since we will merge with train at some point. So I removed noSmoke from the labeled/classes.txt and ran python chkClasses.py 0 to merge labeled into trainPath without dropping any classes.
If you wanted to remove all the classes only used in the labeled folder pics at least 2 times you would run python chkClasses.py 2. Note It remaps only the files in labeled folder so not class names that might be place holders for training are lost.
Then quick check of the files in trainPath to see if any might need tweaking or adding of objects with labelImg
That gives me:
In train/ from mapping files
0:fire used 42.0 times.
1:smoke used 416.0 times.
2:clouds used 377.0 times.
3:plane_night used 2.0 times.
4:helicopter used 6.0 times.
5:plane used 3.0 times.
6:bird used 5.0 times.
In train/../test from mapping files
0:fire used 2.0 times.
1:smoke used 4.0 times.
2:clouds used 1.0 times.
3:plane_night used 0.0 times.
4:helicopter used 0.0 times.
5:plane used 0.0 times.
6:bird used 0.0 times.
Obviously I added a few objects as I was checking. I also dropped a few images I missed in the quick scan.
Doing a train and test on this merged set yields:
For images in:train/ with min_confidence set to 50.00%
fire found 39.0 of 42.0 expected objects with an average confidence of 78.41% min:53.28% max:91.37% adjusted avg:72.81%
smoke found 416.0 of 416.0 expected objects with an average confidence of 93.10% min:50.89% max:98.10% adjusted avg:93.10%
clouds found 379.0 of 377.0 expected objects with an average confidence of 89.75% min:52.43% max:97.96% adjusted avg:90.22%
plane_night found 0.0 of 2.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 91.94% min:91.30% max:92.36% adjusted avg:91.94%
plane found 3.0 of 3.0 expected objects with an average confidence of 61.19% min:51.44% max:77.14% adjusted avg:61.19%
bird found 4.0 of 5.0 expected objects with an average confidence of 70.31% min:60.10% max:80.15% adjusted avg:56.25%
For images in:train/../test/ with min_confidence set to 50.00%
fire found 2.0 of 2.0 expected objects with an average confidence of 73.54% min:73.01% max:74.07% adjusted avg:73.54%
smoke found 4.0 of 4.0 expected objects with an average confidence of 82.10% min:68.31% max:91.70% adjusted avg:82.10%
clouds found 3.0 of 1.0 expected objects with an average confidence of 95.38% min:93.38% max:96.44% adjusted avg:286.15%
plane_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 856.0 of 858.0 expected objects
Of 2552 tests
Ran:2552
Skipped:0
Passed:2543
Warnings:7
Failed:9
Ran 2551 trainTest tests in 0:01:09.796748
As a test I copied the rest of the RMRR.fire train data to test and reran train and test to see if that helped. At first glance it looks worse with 14 more fails but I duped 54 images from train into test which caused 153 more objects to be detected.
For images in:train/ with min_confidence set to 50.00%
fire found 36.0 of 42.0 expected objects with an average confidence of 77.95% min:57.64% max:91.15% adjusted avg:66.81%
smoke found 417.0 of 416.0 expected objects with an average confidence of 92.13% min:53.67% max:97.96% adjusted avg:92.35%
clouds found 377.0 of 377.0 expected objects with an average confidence of 87.38% min:51.41% max:97.51% adjusted avg:87.38%
plane_night found 0.0 of 2.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 91.82% min:90.44% max:93.39% adjusted avg:91.82%
plane found 2.0 of 3.0 expected objects with an average confidence of 68.82% min:60.30% max:77.34% adjusted avg:45.88%
bird found 5.0 of 5.0 expected objects with an average confidence of 62.18% min:52.26% max:68.48% adjusted avg:62.18%
For images in:train/../test/ with min_confidence set to 50.00%
fire found 38.0 of 44.0 expected objects with an average confidence of 77.24% min:57.64% max:91.15% adjusted avg:66.70%
smoke found 55.0 of 56.0 expected objects with an average confidence of 89.34% min:62.76% max:96.74% adjusted avg:87.74%
clouds found 58.0 of 56.0 expected objects with an average confidence of 94.60% min:84.01% max:96.72% adjusted avg:97.98%
plane_night found 0.0 of 2.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 2.0 of 2.0 expected objects with an average confidence of 68.82% min:60.30% max:77.34% adjusted avg:68.82%
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 996.0 of 1011.0 expected objects
Of 2917 tests
Ran:2917
Skipped:0
Passed:2894
Warnings:7
Failed:23
Ran 2916 trainTest tests in 0:01:24.165949
That leaves 22 filed to be sorted. Marked up images and map files that match the found items are in the debugPath folder. So if you point labelImg at the debugPath folder you can get a great idea of what was expect versus found and further tweak the maps in debugPath which have already been updated to match the found items data. For example
you can see the small cloud was not found so can probably be left out the training map. But in this example
It missed seeing the fire so I tried using a larger box.
Once all done I copied the mapping files into the train and test folders (no worries if the image is not there since tools run against image lists not map lists) and rerun train and test.
That sorted 19 of the 23 fails and 6 of 7 warnings.
For images in:train/ with min_confidence set to 50.00%
fire found 42.0 of 42.0 expected objects with an average confidence of 78.24% min:51.44% max:89.00% adjusted avg:78.24%
smoke found 417.0 of 417.0 expected objects with an average confidence of 93.13% min:50.65% max:98.24% adjusted avg:93.13%
clouds found 377.0 of 377.0 expected objects with an average confidence of 90.50% min:51.91% max:97.66% adjusted avg:90.50%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 73.51% min:73.51% max:73.51% adjusted avg:73.51%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 92.43% min:91.32% max:93.93% adjusted avg:92.43%
plane found 0.0 of 1.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
bird found 4.0 of 4.0 expected objects with an average confidence of 84.99% min:81.30% max:91.58% adjusted avg:84.99%
For images in:train/../test/ with min_confidence set to 50.00%
fire found 43.0 of 44.0 expected objects with an average confidence of 78.21% min:51.44% max:89.00% adjusted avg:76.44%
smoke found 55.0 of 55.0 expected objects with an average confidence of 90.63% min:73.16% max:97.00% adjusted avg:90.63%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.23% min:83.55% max:97.09% adjusted avg:95.23%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 73.51% min:73.51% max:73.51% adjusted avg:73.51%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 1.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1004.0 of 1007.0 expected objects in 474 image files
Of 2905 tests
Ran:2905
Skipped:0
Passed:2901
Warnings:1
Failed:4
Ran 2904 trainTest tests in 0:01:22.718020
Of the 4 files in the debugPath folder only one looked like it might need a tweak. (It missed a small fire.) The rest of the misses and adds look fine, so one more round of update maps, train and test.
But that made things worse making me wish I'd made a backup. I'll know better going forward.
For images in:train/ with min_confidence set to 50.00%
fire found 42.0 of 42.0 expected objects with an average confidence of 79.38% min:61.25% max:89.80% adjusted avg:79.38%
smoke found 416.0 of 417.0 expected objects with an average confidence of 92.96% min:60.52% max:98.18% adjusted avg:92.74%
clouds found 378.0 of 377.0 expected objects with an average confidence of 89.34% min:50.56% max:97.82% adjusted avg:89.58%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 70.40% min:70.40% max:70.40% adjusted avg:70.40%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 91.96% min:87.78% max:93.49% adjusted avg:91.96%
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 4.0 of 4.0 expected objects with an average confidence of 70.74% min:66.95% max:75.91% adjusted avg:70.74%
For images in:train/../test/ with min_confidence set to 50.00%
fire found 43.0 of 44.0 expected objects with an average confidence of 78.71% min:50.73% max:89.80% adjusted avg:76.92%
smoke found 55.0 of 55.0 expected objects with an average confidence of 87.63% min:53.34% max:96.88% adjusted avg:87.63%
clouds found 58.0 of 58.0 expected objects with an average confidence of 93.90% min:65.78% max:96.96% adjusted avg:93.90%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 70.40% min:70.40% max:70.40% adjusted avg:70.40%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1004.0 of 1005.0 expected objects in 474 image files
Of 2903 tests
Ran:2903
Skipped:0
Passed:2898
Warnings:4
Failed:5
Ran 2902 trainTest tests in 0:01:23.262812
If I had a backup I'd revert to it here. Since I did not I tried another round of update maps, train and test. which made things even worse but I could revert this time. That one pic where fire is always missed is in test but not train so I added it to train to see if that helped and it did.
For images in:train/ with min_confidence set to 50.00%
fire found 43.0 of 43.0 expected objects with an average confidence of 72.87% min:51.41% max:86.02% adjusted avg:72.87%
smoke found 418.0 of 418.0 expected objects with an average confidence of 92.69% min:60.72% max:98.00% adjusted avg:92.69%
clouds found 382.0 of 378.0 expected objects with an average confidence of 90.26% min:53.95% max:97.22% adjusted avg:91.22%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 73.80% min:73.80% max:73.80% adjusted avg:73.80%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 93.83% min:93.11% max:94.73% adjusted avg:93.83%
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 4.0 of 4.0 expected objects with an average confidence of 76.67% min:69.59% max:81.38% adjusted avg:76.67%
For images in:train/../test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 71.89% min:51.01% max:86.02% adjusted avg:71.89%
smoke found 55.0 of 55.0 expected objects with an average confidence of 89.75% min:60.72% max:96.67% adjusted avg:89.75%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.23% min:84.31% max:96.89% adjusted avg:95.23%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 73.80% min:73.80% max:73.80% adjusted avg:73.80%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1012.0 of 1008.0 expected objects in 475 image files
Of 2910 tests
Ran:2910
Skipped:0
Passed:2909
Warnings:5
Failed:1
Ran 2909 trainTest tests in 0:01:19.197277
All of my files now work. The one fail is it only because overlapping smoke labels in the expect are detected as one. All the warns are unlabeled clouds the are there. So best yet. Made a back up of train and test folders and on to next dataset.
The FireNet files and mapping are in separate folders and divided between train and test groups.
To simplify and make process more generic I wanted to move or copy them all into one folder pointed to by newPicPath
newPicPath = "./new/" in this example.
As a quick check I ran python quickLabel.py to see what the fire model at this point could fine in the 502 images of the dataset. It found objects in only 41 of them and only one was right. Not really all that surprising given only a couple images are similar to the small hillside fires and smoke plumes I'm wanting to detect. I might note it was mistaking people's pants as smoke which kind of points out the need to have a model that includes all the object types you expect to see in it to reduce false positives.
Should you want to create a model with FireNet, it is labeled with PascalVOC XML files so they need to be converted. For this I wrote a quick util [pascalvoc2yolo](https://github.com/avatar42/deepstack/pascalvoc2yolo.py. Just run python pascalvoc2yolo.py and it will put move the images with PascalVOC XML maps in newPicPath into folder point to by labeled
These are all unlabeled / mapped so the first thing I did was to run python quickLabel.py to see what the fire model to see do a quick check of the 14761 images in this dataset
These look much better. 1979 files with objects.
0:fire used 5.0 times.
1:smoke used 1774.0 times.
2:clouds used 204.0 times.
Though most of the clouds are smoke and it missed fire a lot of the time. Plus it seems most of the smoke pictures are Photoshopped by overlaying a few images of smoke in random places on sequential frames from videos of non fire locations. Plus those smoke all seem to have hard flat bottoms and some even hard flat tops and or sides to them which makes me worry about using them. Some are even made with corrupted images, probably in an automated process. So I removed them all.
I did leave 11 images of close up fires I probably should not have. I also labeled some fire instances in the training pics to help improve the fire detection. The results
For images in:train/ with min_confidence set to 50.00%
fire found 252.0 of 253.0 expected objects with an average confidence of 88.33% min:50.07% max:97.51% adjusted avg:87.98%
smoke found 600.0 of 598.0 expected objects with an average confidence of 94.37% min:54.29% max:97.95% adjusted avg:94.69%
clouds found 390.0 of 389.0 expected objects with an average confidence of 89.62% min:50.86% max:97.98% adjusted avg:89.85%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 78.29% min:78.29% max:78.29% adjusted avg:78.29%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 91.24% min:89.54% max:92.28% adjusted avg:91.24%
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 4.0 of 4.0 expected objects with an average confidence of 76.52% min:57.45% max:89.81% adjusted avg:76.52%
For images in:train/../test/ with min_confidence set to 50.00%
fire found 42.0 of 44.0 expected objects with an average confidence of 81.86% min:50.07% max:93.80% adjusted avg:78.14%
smoke found 53.0 of 55.0 expected objects with an average confidence of 92.85% min:69.00% max:96.88% adjusted avg:89.48%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.90% min:90.44% max:97.24% adjusted avg:95.90%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 78.29% min:78.29% max:78.29% adjusted avg:78.29%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1407.0 of 1409.0 expected objects in 634 image files
Of 3947 tests
Ran:3947
Skipped:0
Passed:3938
Warnings:7
Failed:9
Ran 3946 trainTest tests in 0:01:48.015724
As a "general" tool it might look a bit better but for the images in train, which are actual images from my cameras, it did worse losing both fire and smoke detections. So as a first attempt at tweaking I removed the 11 images of close up fires and reran train and test.
For images in:train/ with min_confidence set to 50.00%
fire found 211.0 of 212.0 expected objects with an average confidence of 87.71% min:53.03% max:96.88% adjusted avg:87.29%
smoke found 598.0 of 597.0 expected objects with an average confidence of 94.51% min:51.88% max:98.47% adjusted avg:94.67%
clouds found 384.0 of 384.0 expected objects with an average confidence of 88.97% min:52.58% max:98.07% adjusted avg:88.97%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 85.38% min:85.38% max:85.38% adjusted avg:85.38%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 87.71% min:85.26% max:89.94% adjusted avg:87.71%
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 3.0 of 4.0 expected objects with an average confidence of 74.40% min:55.93% max:84.37% adjusted avg:55.80%
For images in:train/../test/ with min_confidence set to 50.00%
fire found 42.0 of 44.0 expected objects with an average confidence of 83.13% min:63.27% max:92.26% adjusted avg:79.35%
smoke found 55.0 of 55.0 expected objects with an average confidence of 92.58% min:78.79% max:96.65% adjusted avg:92.58%
clouds found 59.0 of 58.0 expected objects with an average confidence of 95.82% min:55.73% max:97.92% adjusted avg:97.47%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 85.38% min:85.38% max:85.38% adjusted avg:85.38%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1360.0 of 1362.0 expected objects in 623 image files
Of 3856 tests
Ran:3856
Skipped:0
Passed:3848
Warnings:6
Failed:8
Ran 3855 trainTest tests in 0:01:45.829078
Oddly this seems to have helped smoke detection but not fire.
Since this is a Tensorflow set with generated training to run against real test images and to confirm my worry that the Photoshopped images are not good for training I created a training set from the forest-fire\train-smoke images to run against my test images using python quickLabel.py and the above fire model and tweaking the result.
quickLabel labeled 3643 of the 12631 images which left a lot. So I grabbed a random sampling of 208 the unlabeled to try create a model able to quickLabel the rest. This yielded
For images in:train/ with min_confidence set to 50.00%
fire found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
smoke found 3099.0 of 3120.0 expected objects with an average confidence of 83.36% min:50.03% max:92.01% adjusted avg:82.79%
clouds found 795.0 of 787.0 expected objects with an average confidence of 87.34% min:50.07% max:93.57% adjusted avg:88.22%
plane_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
For images in:train/../test/ with min_confidence set to 50.00%
fire found 0.0 of 44.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
smoke found 0.0 of 55.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
clouds found 11.0 of 58.0 expected objects with an average confidence of 62.72% min:50.67% max:79.12% adjusted avg:11.90%
plane_night found 0.0 of 1.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 3905.0 of 4065.0 expected objects in 3909 image files
Of 19687 tests
Ran:19687
Skipped:0
Passed:19464
Warnings:43
Failed:223
Ran 19686 trainTest tests in 0:09:39.416419
Again it seems to do well detecting training images but nothing else. Still for grins I used that to label the remaining files with quickLabel and it seemed to go well
Of 8778 pics in new/
Ignored dup files:0
Labeled:8167 objects in 7837 files.
Unlabeled:941 files.
Failures:0
I then moved my test folder to valid, made the forest-fire\test_big the test set and reran train and test one more time to see how well it worked against its own test files and the improved model against my set. Note this training took over 13 hours on my 4K video editing workstation which is about 9 times faster than Google Colab.
For images in:train/ with min_confidence set to 50.00%
fire found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
smoke found 10662.0 of 10683.0 expected objects with an average confidence of 93.48% min:50.18% max:97.82% adjusted avg:93.30%
clouds found 1363.0 of 1391.0 expected objects with an average confidence of 93.90% min:50.02% max:97.93% adjusted avg:92.01%
For images in:train/../test/../valid/ with min_confidence set to 50.00%
fire found 0.0 of 44.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
smoke found 21.0 of 55.0 expected objects with an average confidence of 78.22% min:50.37% max:92.65% adjusted avg:29.86%
clouds found 8.0 of 58.0 expected objects with an average confidence of 69.71% min:57.29% max:87.97% adjusted avg:9.61%
plane_night found 0.0 of 1.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 12054.0 of 12232.0 expected objects in 13753 image files
Of 59202 tests
Ran:59202
Skipped:0
Passed:58992
Warnings:2040
Failed:210
Ran 59201 trainTest tests in 0:28:47.049682
Better but still not good. Note however almost all those 1391 clouds in train ought to be smoke. Stick their smoke test set into new and running quickLabel yielded
Of 2794 pics in new/
Ignored dup files:0
Labeled:918 objects in 883 files.
Unlabeled:1911 files.
Failures:0
Though ~ 1600 of those 1911 are negative test images with no smoke so it should not have found any.
So all in all this set does not really seem to help even if I limit it. So before moving on I restored my last best dataset and did a train and test to make sure it matches my earlier results.
For images in:test/ with min_confidence set to 50.00%
fire found 42.0 of 44.0 expected objects with an average confidence of 71.41% min:53.34% max:81.47% adjusted avg:68.16%
smoke found 53.0 of 55.0 expected objects with an average confidence of 86.21% min:50.85% max:95.10% adjusted avg:83.07%
clouds found 58.0 of 58.0 expected objects with an average confidence of 92.30% min:73.21% max:95.16% adjusted avg:92.30%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 74.66% min:74.66% max:74.66% adjusted avg:74.66%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
So something is not right. I removed deepstack*.cache and that got me close but still not back to where I was. Guess I need to add the cache files to the backup next time.
For images in:train/ with min_confidence set to 50.00%
fire found 43.0 of 43.0 expected objects with an average confidence of 78.05% min:61.64% max:89.18% adjusted avg:78.05%
smoke found 416.0 of 418.0 expected objects with an average confidence of 93.13% min:72.99% max:98.22% adjusted avg:92.68%
clouds found 381.0 of 378.0 expected objects with an average confidence of 90.07% min:52.07% max:98.54% adjusted avg:90.79%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 78.78% min:78.78% max:78.78% adjusted avg:78.78%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 93.29% min:89.34% max:95.72% adjusted avg:93.29%
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 4.0 of 4.0 expected objects with an average confidence of 75.62% min:67.61% max:81.90% adjusted avg:75.62%
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 77.74% min:61.64% max:89.18% adjusted avg:77.74%
smoke found 54.0 of 55.0 expected objects with an average confidence of 90.84% min:75.46% max:96.99% adjusted avg:89.19%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.69% min:87.37% max:97.35% adjusted avg:95.69%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 78.78% min:78.78% max:78.78% adjusted avg:78.78%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1008.0 of 1008.0 expected objects in 475 image files
Of 2910 tests
Ran:2910
Skipped:0
Passed:2907
Warnings:3
Failed:3
Ran 2909 trainTest tests in 0:01:24.513788
Next I removed deepstack*.cache and deepstack-trainer\train-runs\deepstack\fire which indicates maybe I should backup the debugPath folder too.
For images in:train/ with min_confidence set to 50.00%
fire found 43.0 of 43.0 expected objects with an average confidence of 75.94% min:56.23% max:90.73% adjusted avg:75.94%
smoke found 416.0 of 418.0 expected objects with an average confidence of 93.28% min:64.67% max:98.36% adjusted avg:92.83%
clouds found 378.0 of 378.0 expected objects with an average confidence of 90.60% min:53.82% max:97.81% adjusted avg:90.60%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 78.88% min:78.88% max:78.88% adjusted avg:78.88%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 92.04% min:89.09% max:95.23% adjusted avg:92.04%
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 4.0 of 4.0 expected objects with an average confidence of 78.46% min:76.63% max:79.36% adjusted avg:78.46%
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 75.28% min:56.23% max:90.73% adjusted avg:75.28%
smoke found 54.0 of 55.0 expected objects with an average confidence of 91.46% min:80.12% max:97.05% adjusted avg:89.80%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.45% min:87.56% max:97.03% adjusted avg:95.45%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 78.88% min:78.88% max:78.88% adjusted avg:78.88%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1005.0 of 1008.0 expected objects in 475 image files
Of 2910 tests
Ran:2910
Skipped:0
Passed:2904
Warnings:3
Failed:6
Ran 2909 trainTest tests in 0:01:20.774026
Note is shows 6 fails but seems to show only 3 smoke objects undetected. If we look at trainTest.results.txt
For images in:train/ with min_confidence set to 0.5
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['clouds', 'clouds', 'smoke'] in ck0k9ghqt7a8l0944mcvy0jsx_jpeg.rf.5b55501a403c70b4dba61727046660b5.jpg
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in ck0kdgfrn7loi0944bc5njxd7_jpeg.rf.e77b9a3d458cca0a65c750a0e39b2036.jpg
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in ck0khu2xz94nj070165dkfirk_jpeg.rf.d511cbdb4c1162cf08d710561abef2b3.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['clouds', 'smoke', 'clouds'] in ck0kkg80v5x900794hcgsb4e1_jpeg.rf.c14e4f9813670fcb8bc4bc762fb0afa8.jpg
Of 4 expected ['smoke', 'clouds', 'clouds', 'clouds'] found objects, found:3 objects ['clouds', 'clouds', 'smoke'] in ck0l9vmuparhg0838oe20j0bd_jpeg.rf.ef939ad40b43a4c66054c5da14275db8.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['clouds', 'clouds', 'smoke'] in ck0lzyygqj00t0944qnwlwjf2_jpeg.rf.fbbbc2b4e6b9dd5b8242be555a897ed4.jpg
Of 2 expected ['smoke', 'smoke'] found objects, found:1 objects ['smoke'] in ck0uk75x5ysls0721e5a9j891_jpeg.rf.00d7fd8503e1e0e6a66294f3db79b346.jpg
Of 5 expected ['plane_night', 'smoke', 'smoke', 'clouds', 'fire'] found objects, found:4 objects ['fire', 'plane_night', 'smoke', 'clouds'] in HV418.20220111_195547068.30.jpg
it is apparent that the diff in lost clouds is hidden by extra clouds found. Reviewing the images in debug shows that only the missing smoke in HV418.20220111_195547068.30.jpg is of any significance and given its somewhat vague smoke vs cloud nature I can let slide for now. So backup everything and on to next set.
Again most of the pics here are of fairly close fires and or smoke. From what I learned above it looks like only 8 of the 100 images might be of use to me.
So I copied them to newPicPath folder and with the same config.py settings ran python quickLabel.py on them. This yielded the no so promising result of
Of 8 pics in new/
Ignored dup files:0
Labeled:3 objects in 2 files.
Unlabeled:6 files.
Failures:0
I started looking at the unlabeled first with labelImg Note set "Change Save dir" to labeled folder then set "Open Dir" to unlabeled folder. This is so it sees and mods the right classes.txt file. After I labeled smoke and fire in the images I moved them to labeled
I again verified and tweaked as needed the labels in labeled with labelImg.
Running python chkClasses.py showed
In labeled/ from mapping files
0:fire used 6.0 times.
1:smoke used 11.0 times.
2:clouds used 1.0 times.
Note the one cloud object was the only one I did not need to tweak / change.
Now I ran train and test to see if any of these helped.
For images in:train/ with min_confidence set to 50.00%
fire found 47.0 of 49.0 expected objects with an average confidence of 74.77% min:51.76% max:94.54% adjusted avg:71.72%
smoke found 428.0 of 429.0 expected objects with an average confidence of 91.29% min:55.10% max:97.58% adjusted avg:91.08%
clouds found 388.0 of 379.0 expected objects with an average confidence of 88.48% min:50.89% max:97.49% adjusted avg:90.58%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 51.44% min:51.44% max:51.44% adjusted avg:51.44%
helicopter found 6.0 of 6.0 expected objects with an average confidence of 89.14% min:88.22% max:90.10% adjusted avg:89.14%
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 2.0 of 4.0 expected objects with an average confidence of 74.27% min:72.45% max:76.09% adjusted avg:37.13%
For images in:test/ with min_confidence set to 50.00%
fire found 43.0 of 44.0 expected objects with an average confidence of 72.83% min:51.76% max:85.05% adjusted avg:71.18%
smoke found 54.0 of 55.0 expected objects with an average confidence of 89.06% min:64.72% max:96.19% adjusted avg:87.44%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.14% min:78.47% max:97.36% adjusted avg:95.14%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 51.44% min:51.44% max:51.44% adjusted avg:51.44%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1028.0 of 1026.0 expected objects in 483 image files
Of 2960 tests
Ran:2960
Skipped:0
Passed:2952
Warnings:10
Failed:8
Ran 2959 trainTest tests in 0:01:21.983182
So again it looks like these were not help and even made things worse. Two of the new images used to train for fire had nothing detected in the images.
For me the simpliest way to do this was to use Cygwin to run a shell script. Mine looks like this
##
## Usage: copyNew.sh &ALERT_PATH &MEMO
## where &ALERT_PATH is like DAH412.20210731_143912.4034343.3.jpg
## and &MEMO is like person:93%,dog:81
##
#set -x
PATH="/usr/bin"
export FLAGGED=/cygdrive/c/_dea/odrive/GD.video/cams/DeepStackWS/data/new
export ALERTS=/cygdrive/e/BlueIris/Alerts
echo "$*" >> /cygdrive/c/DeepStack/newLog.out
obj=`echo $2 | cut -f1 -d':'`
echo "(${obj})" >> /cygdrive/c/DeepStack/newLog.out 2>&1
mkdir -p ${FLAGGED}/${obj} >> /cygdrive/c/DeepStack/newLog.out 2>&1
clip=`echo $1 | cut -f1-2 -d'.'`
dat=`ls -1tr ${ALERTS}/${clip}*.dat | tail -1`
if [ -r "$dat" ]
then
pic=`echo $dat | sed -e 's/.dat/.jpg/g'`
#/usr/bin/cp ${ALERTS}/$1 ${FLAGGED}/${obj} >> /cygdrive/c/DeepStack/newLog.out 2>&1
/usr/bin/cp ${pic} ${FLAGGED}/${obj} >> /cygdrive/c/DeepStack/newLog.out 2>&1
else
# dat is missing for some reason
/usr/bin/cp ${ALERTS}/${clip}*.jpg ${FLAGGED}/${obj} >> /cygdrive/c/DeepStack/newLog.out 2>&1
fi
ls -latr ${FLAGGED}/${obj} | tail -5
Running on all servers they save to a folder synced to cloud storage which is then down synced to my workstation
Then I set Blue Iris up to run the script if any objects I wanted to save were detected by basic / built in dectection or my custom model. RMRR in this example.
mouse,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,person,pig,raccoon,coyote,squirrel,bunny,cat_black,cat_grey,cat_orange,cat_tort,cat_calico,cow,dog_black_lab,dog_pyr,deer,opossum,US-plate,mercedesbenz,amazon,prime,smile,ups,fedex,Dave,box_nutrisystem,Ryder,kia,budget,US-mail,truck_trash,ACDI
C:\cygwin\bin\sh.exe
-x C:\DeepStack\copyNew.sh &ALERT_PATH &MEMO
Note you want to be sure to turn off burn labels. They may confuse training and or testing.
Note you can leave off Custom models if it is in your "MyModels" folder pointed to in your Blue Iris AI config.
To keep things going on the alerts I'm also setting up an alert action to copy alerts images (without burn in marks Note some versions of Blue Iris appear to not save images without them.) to a folder matching the first object found so I can add any mislabeled images with corrected maps to the next training round. More useful for models more frequent things than brush fires.
Import by putting into newPath and run Quick Label to label them.
I pulled 115 images from past videos of fireworks and fly overs and bumped the stats to
For images in:train/ with min_confidence set to 50.00%
fire found 84.0 of 87.0 expected objects with an average confidence of 83.54% min:52.45% max:93.80% adjusted avg:80.66%
smoke found 486.0 of 489.0 expected objects with an average confidence of 93.80% min:70.67% max:98.02% adjusted avg:93.22%
clouds found 498.0 of 497.0 expected objects with an average confidence of 92.09% min:52.23% max:98.43% adjusted avg:92.27%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 84.61% min:69.52% max:93.53% adjusted avg:84.61%
helicopter found 15.0 of 15.0 expected objects with an average confidence of 92.19% min:87.84% max:94.90% adjusted avg:92.19%
plane found 9.0 of 33.0 expected objects with an average confidence of 59.26% min:50.67% max:73.70% adjusted avg:16.16%
bird found 6.0 of 6.0 expected objects with an average confidence of 81.55% min:59.58% max:93.88% adjusted avg:81.55%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 86.82% min:80.02% max:91.41% adjusted avg:86.82%
For images in:test/ with min_confidence set to 50.00%
fire found 42.0 of 44.0 expected objects with an average confidence of 82.36% min:52.45% max:93.68% adjusted avg:78.62%
smoke found 55.0 of 55.0 expected objects with an average confidence of 92.62% min:81.73% max:97.65% adjusted avg:92.62%
clouds found 58.0 of 58.0 expected objects with an average confidence of 96.49% min:87.59% max:98.34% adjusted avg:96.49%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 93.53% min:93.53% max:93.53% adjusted avg:93.53%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1269.0 of 1300.0 expected objects in 588 image files
Of 3654 tests
Ran:3654
Skipped:0
Passed:3620
Warnings:3
Failed:34
Ran 3653 trainTest tests in 0:01:48.951070
After reviewing the debug pics there was only one fire I thought it should have caught so excepted the rest of the suggestions and reran (after backing up of course) train and test. And it looks real good.
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in ck0kdgfrn7loi0944bc5njxd7_jpeg.rf.e77b9a3d458cca0a65c750a0e39b2036.jpg
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in ck0khu2xz94nj070165dkfirk_jpeg.rf.d511cbdb4c1162cf08d710561abef2b3.jpg
Of 4 expected ['smoke', 'clouds', 'clouds', 'clouds'] found objects, found:3 objects ['clouds', 'clouds', 'smoke'] in ck0m0ch9ugnna07940o8x989j_jpeg.rf.0101cdb46a16b3fde020710836b4af0b.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV411.0900-1059.01_15_44_09.Still006.jpg
Of 4 expected ['fire', 'smoke', 'smoke', 'clouds'] found objects, found:5 objects ['smoke', 'smoke', 'fire', 'smoke', 'clouds'] in HV418.0841-1105.10_48_04_06.Still026.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 83.29% min:53.16% max:92.35% adjusted avg:83.29%
smoke found 488.0 of 486.0 expected objects with an average confidence of 93.21% min:60.87% max:97.80% adjusted avg:93.59%
clouds found 495.0 of 498.0 expected objects with an average confidence of 92.08% min:53.44% max:97.99% adjusted avg:91.52%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 79.26% min:62.04% max:86.73% adjusted avg:79.26%
helicopter found 15.0 of 15.0 expected objects with an average confidence of 92.47% min:88.86% max:96.38% adjusted avg:92.47%
plane found 9.0 of 9.0 expected objects with an average confidence of 75.97% min:59.87% max:86.01% adjusted avg:75.97%
bird found 6.0 of 6.0 expected objects with an average confidence of 82.27% min:66.83% max:90.51% adjusted avg:82.27%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 85.41% min:80.87% max:87.84% adjusted avg:85.41%
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 83.74% min:63.82% max:92.35% adjusted avg:83.74%
smoke found 55.0 of 55.0 expected objects with an average confidence of 92.42% min:79.03% max:97.18% adjusted avg:92.42%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.74% min:90.09% max:97.99% adjusted avg:95.74%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 83.21% min:83.21% max:83.21% adjusted avg:83.21%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1272.0 of 1273.0 expected objects in 589 image files
Of 3627 tests
Ran:3627
Skipped:0
Passed:3624
Warnings:3
Failed:3
Ran 3626 trainTest tests in 0:01:38.346210
The missing are only in the training images and the extras are all in the test images. Plus all seem correct. Contrary to what you might think, if I took these suggestions it actually made things worse.
For images in:train/ with min_confidence set to 50.00%
fire found 83.0 of 86.0 expected objects with an average confidence of 81.19% min:58.07% max:93.46% adjusted avg:78.36%
smoke found 489.0 of 488.0 expected objects with an average confidence of 89.90% min:50.80% max:97.33% adjusted avg:90.08%
clouds found 497.0 of 495.0 expected objects with an average confidence of 87.87% min:52.87% max:96.87% adjusted avg:88.22%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 78.09% min:55.97% max:86.16% adjusted avg:78.09%
helicopter found 17.0 of 17.0 expected objects with an average confidence of 88.26% min:72.92% max:92.68% adjusted avg:88.26%
plane found 8.0 of 9.0 expected objects with an average confidence of 74.15% min:67.02% max:85.80% adjusted avg:65.91%
bird found 4.0 of 4.0 expected objects with an average confidence of 77.59% min:72.78% max:82.70% adjusted avg:77.59%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 84.93% min:80.26% max:88.43% adjusted avg:84.93%
For images in:test/ with min_confidence set to 50.00%
fire found 42.0 of 44.0 expected objects with an average confidence of 80.98% min:58.28% max:93.46% adjusted avg:77.30%
smoke found 54.0 of 55.0 expected objects with an average confidence of 88.79% min:56.08% max:94.93% adjusted avg:87.18%
clouds found 61.0 of 61.0 expected objects with an average confidence of 93.61% min:83.57% max:96.43% adjusted avg:93.61%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 86.16% min:86.16% max:86.16% adjusted avg:86.16%
helicopter found 1.0 of 1.0 expected objects with an average confidence of 90.53% min:90.53% max:90.53% adjusted avg:90.53%
plane found 1.0 of 1.0 expected objects with an average confidence of 68.07% min:68.07% max:68.07% adjusted avg:68.07%
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 1.0 of 1.0 expected objects with an average confidence of 88.43% min:88.43% max:88.43% adjusted avg:88.43%
Found 1274.0 of 1278.0 expected objects in 592 image files
Of 3644 tests
Ran:3644
Skipped:0
Passed:3633
Warnings:8
Failed:11
Ran 3643 trainTest tests in 0:01:41.131741
Note I went to copy some images from train to test to ensure I had at least one of each and noticed a couple helicopters had been marked birds so I fixed those as well. Though it does not seem to effect the model I wanted it to be right but that meant I needed to revert and do that again to revert just the other changes.
For images in:train/ with min_confidence set to 50.00%
fire found 83.0 of 86.0 expected objects with an average confidence of 82.69% min:54.64% max:92.35% adjusted avg:79.81%
smoke found 489.0 of 486.0 expected objects with an average confidence of 90.04% min:50.10% max:96.47% adjusted avg:90.60%
clouds found 482.0 of 498.0 expected objects with an average confidence of 88.81% min:50.39% max:97.15% adjusted avg:85.96%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 82.24% min:70.42% max:90.28% adjusted avg:82.24%
helicopter found 18.0 of 17.0 expected objects with an average confidence of 86.87% min:51.57% max:93.09% adjusted avg:91.98%
plane found 7.0 of 9.0 expected objects with an average confidence of 68.71% min:50.14% max:79.52% adjusted avg:53.44%
bird found 3.0 of 4.0 expected objects with an average confidence of 65.78% min:57.40% max:73.88% adjusted avg:49.34%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 82.55% min:76.56% max:87.79% adjusted avg:82.55%
For images in:test/ with min_confidence set to 50.00%
fire found 43.0 of 44.0 expected objects with an average confidence of 82.80% min:54.64% max:92.35% adjusted avg:80.92%
smoke found 54.0 of 55.0 expected objects with an average confidence of 88.95% min:58.07% max:94.92% adjusted avg:87.33%
clouds found 63.0 of 62.0 expected objects with an average confidence of 93.47% min:82.15% max:96.28% adjusted avg:94.98%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 90.28% min:90.28% max:90.28% adjusted avg:90.28%
helicopter found 1.0 of 1.0 expected objects with an average confidence of 90.85% min:90.85% max:90.85% adjusted avg:90.85%
plane found 0.0 of 1.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0.00%
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 2.0 of 2.0 expected objects with an average confidence of 86.80% min:85.84% max:87.76% adjusted avg:86.80%
Found 1261.0 of 1281.0 expected objects in 592 image files
Of 3651 tests
Ran:3651
Skipped:0
Passed:3620
Warnings:11
Failed:31
Ran 3650 trainTest tests in 0:01:42.733645
Despite restoring train, test, debug.pics and deepstack-trainer\train-runs\deepstack\fire plus both sets of cache files things seem to have gotten worse. So to double check I restored again without changes other than wiping deepstack-trainer\train-runs\deepstack\fire and things got even better.
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 83.56% min:54.49% max:94.48% adjusted avg:83.56%
smoke found 488.0 of 486.0 expected objects with an average confidence of 92.17% min:50.45% max:98.16% adjusted avg:92.55%
clouds found 498.0 of 498.0 expected objects with an average confidence of 90.74% min:52.57% max:98.18% adjusted avg:90.74%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 81.23% min:71.15% max:90.40% adjusted avg:81.23%
helicopter found 15.0 of 15.0 expected objects with an average confidence of 90.06% min:82.51% max:94.77% adjusted avg:90.06%
plane found 8.0 of 9.0 expected objects with an average confidence of 77.85% min:68.55% max:88.46% adjusted avg:69.20%
bird found 6.0 of 6.0 expected objects with an average confidence of 79.76% min:70.14% max:90.69% adjusted avg:79.76%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 83.67% min:77.95% max:87.32% adjusted avg:83.67%
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 83.42% min:54.49% max:94.48% adjusted avg:83.42%
smoke found 56.0 of 55.0 expected objects with an average confidence of 91.22% min:54.04% max:96.42% adjusted avg:92.88%
clouds found 58.0 of 58.0 expected objects with an average confidence of 96.00% min:87.89% max:97.85% adjusted avg:96.00%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 90.40% min:90.40% max:90.40% adjusted avg:90.40%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Found 1275.0 of 1273.0 expected objects in 589 image files
Of 3627 tests
Ran:3627
Skipped:0
Passed:3625
Warnings:5
Failed:2
Ran 3626 trainTest tests in 0:01:39.367666
Once more I edited the 2 helicopter image maps and reran to see the effect.
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in ck0khu2xz94nj070165dkfirk_jpeg.rf.d511cbdb4c1162cf08d710561abef2b3.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['clouds', 'smoke', 'clouds'] in ck0kkgwvk9hhg0701edmbvr7r_jpeg.rf.72b0cbcbdd84a2f967e4d56a8a994552.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['smoke', 'clouds', 'smoke'] in ck0u98ba1xzpe0721rgjjjun5_jpeg.rf.c8cc83fd4689caf4aff86fee001b8a16.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['smoke', 'clouds', 'smoke'] in ck0ukkr2f8vml0848uyh1iw50_jpeg.rf.22c704f3f83a1947293534fb1cda1482.jpg
Of 4 expected ['fire', 'smoke', 'smoke', 'clouds'] found objects, found:5 objects ['smoke', 'smoke', 'fire', 'smoke', 'clouds'] in HV411.20220111_201302223.67.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:2 objects ['smoke', 'smoke'] in HV440.0859-1200.10_37_00_11.Still013.jpg
Of 4 expected ['clouds', 'smoke', 'plane_night', 'fire'] found objects, found:3 objects ['plane_night', 'smoke', 'clouds'] in HV444.20211127_234941_1.00_05_07_0.Still011.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 85.0 of 86.0 expected objects with an average confidence of 83.90% min:57.57% max:93.49% adjusted avg:82.93%
smoke found 490.0 of 486.0 expected objects with an average confidence of 93.60% min:50.64% max:98.21% adjusted avg:94.37%
clouds found 497.0 of 498.0 expected objects with an average confidence of 91.60% min:50.61% max:98.14% adjusted avg:91.41%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 83.88% min:68.73% max:92.35% adjusted avg:83.88%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 92.38% min:84.25% max:95.18% adjusted avg:92.38%
plane found 9.0 of 9.0 expected objects with an average confidence of 73.84% min:56.47% max:86.33% adjusted avg:73.84%
bird found 5.0 of 5.0 expected objects with an average confidence of 80.03% min:66.45% max:86.01% adjusted avg:80.03%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 85.87% min:83.08% max:87.42% adjusted avg:85.87%
Of 4 expected ['fire', 'smoke', 'smoke', 'clouds'] found objects, found:5 objects ['smoke', 'smoke', 'fire', 'smoke', 'clouds'] in HV411.20220111_201302223.67.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:1 objects ['clouds'] in HV440.20211102_104104882.4.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 82.36% min:51.63% max:93.49% adjusted avg:82.36%
smoke found 55.0 of 55.0 expected objects with an average confidence of 92.18% min:66.32% max:96.94% adjusted avg:92.18%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.63% min:84.83% max:97.79% adjusted avg:95.63%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 92.35% min:92.35% max:92.35% adjusted avg:92.35%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 3627 tests
Ran:3627
Skipped:0
Passed:3623
Warnings:6
Failed:4
And it got worse again! And for for fire which was not even in the images I edited maps for. Obviously the caches are altering the model each run even if nothing changes. To confirm this I ran again without changing anything and saw
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in ck0khu2xz94nj070165dkfirk_jpeg.rf.d511cbdb4c1162cf08d710561abef2b3.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['clouds', 'clouds', 'smoke'] in ck0lzyygqj00t0944qnwlwjf2_jpeg.rf.fbbbc2b4e6b9dd5b8242be555a897ed4.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV411.0900-1059.01_15_44_09.Still006.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 86.02% min:60.93% max:95.18% adjusted avg:86.02%
smoke found 487.0 of 486.0 expected objects with an average confidence of 93.65% min:58.42% max:97.79% adjusted avg:93.84%
clouds found 498.0 of 498.0 expected objects with an average confidence of 92.80% min:60.35% max:98.30% adjusted avg:92.80%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 83.92% min:70.03% max:91.22% adjusted avg:83.92%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 92.15% min:83.24% max:95.69% adjusted avg:92.15%
plane found 9.0 of 9.0 expected objects with an average confidence of 80.43% min:74.06% max:86.44% adjusted avg:80.43%
bird found 5.0 of 5.0 expected objects with an average confidence of 86.90% min:76.44% max:92.38% adjusted avg:86.90%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 87.40% min:84.43% max:91.12% adjusted avg:87.40%
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 84.93% min:61.63% max:95.18% adjusted avg:84.93%
smoke found 55.0 of 55.0 expected objects with an average confidence of 92.65% min:75.77% max:97.33% adjusted avg:92.65%
clouds found 58.0 of 58.0 expected objects with an average confidence of 96.32% min:89.00% max:98.26% adjusted avg:96.32%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 91.22% min:91.22% max:91.22% adjusted avg:91.22%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 3627 tests
Ran:3627
Skipped:0
Passed:3626
Warnings:3
Failed:1
Ran it again (with no changes) and it got WAY worse.
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in ck0kdgfrn7loi0944bc5njxd7_jpeg.rf.e77b9a3d458cca0a65c750a0e39b2036.jpg
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in ck0khu2xz94nj070165dkfirk_jpeg.rf.d511cbdb4c1162cf08d710561abef2b3.jpg
Of 3 expected ['smoke', 'clouds', 'clouds'] found objects, found:2 objects ['smoke', 'clouds'] in ck0nd5toq009k0804lhknr6zt_jpeg.rf.7dbcdd37e0a5ddbbb3a5c9351577ae1f.jpg
Of 2 expected ['smoke', 'smoke'] found objects, found:1 objects ['smoke'] in ck0uk75x5ysls0721e5a9j891_jpeg.rf.00d7fd8503e1e0e6a66294f3db79b346.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV411.0900-1059.01_15_44_09.Still006.jpg
Of 4 expected ['fire', 'smoke', 'smoke', 'clouds'] found objects, found:5 objects ['smoke', 'smoke', 'fire', 'smoke', 'clouds'] in HV418.0841-1105.10_48_04_06.Still026.jpg
Of 3 expected ['fire', 'smoke', 'clouds'] found objects, found:2 objects ['smoke', 'clouds'] in HV418.20220111_195451734.28.jpg
Of 3 expected ['smoke', 'clouds', 'fire'] found objects, found:2 objects ['smoke', 'clouds'] in HV418.20220111_195505901.29.jpg
Of 3 expected ['fire', 'smoke', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in HV418.20220111_200446318.36.jpg
Of 3 expected ['plane', 'plane', 'clouds'] found objects, found:2 objects ['plane', 'clouds'] in HV419.20200422_190631_1.00_00_13_12.Still005.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 83.0 of 86.0 expected objects with an average confidence of 82.05% min:51.74% max:92.98% adjusted avg:79.19%
smoke found 487.0 of 486.0 expected objects with an average confidence of 93.13% min:51.24% max:97.71% adjusted avg:93.32%
clouds found 495.0 of 498.0 expected objects with an average confidence of 91.58% min:58.68% max:98.03% adjusted avg:91.03%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 80.31% min:59.83% max:91.03% adjusted avg:80.31%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 91.77% min:85.21% max:94.40% adjusted avg:91.77%
plane found 8.0 of 9.0 expected objects with an average confidence of 76.27% min:65.37% max:83.45% adjusted avg:67.79%
bird found 5.0 of 5.0 expected objects with an average confidence of 78.36% min:66.55% max:90.29% adjusted avg:78.36%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 85.35% min:80.22% max:89.57% adjusted avg:85.35%
Of 3 expected ['fire', 'smoke', 'clouds'] found objects, found:2 objects ['smoke', 'clouds'] in HV418.20220111_195451734.28.jpg
Of 3 expected ['smoke', 'clouds', 'fire'] found objects, found:2 objects ['smoke', 'clouds'] in HV418.20220111_195505901.29.jpg
Of 3 expected ['fire', 'smoke', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in HV418.20220111_200446318.36.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 41.0 of 44.0 expected objects with an average confidence of 81.26% min:51.74% max:92.98% adjusted avg:75.72%
smoke found 55.0 of 55.0 expected objects with an average confidence of 91.22% min:57.83% max:96.54% adjusted avg:91.22%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.88% min:87.90% max:97.98% adjusted avg:95.88%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 91.03% min:91.03% max:91.03% adjusted avg:91.03%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 3627 tests
Ran:3627
Skipped:0
Passed:3616
Warnings:3
Failed:11
One more time just to see and it is better again but not as good as the 2nd run.
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['clouds', 'clouds', 'smoke'] in ck0kmf5q58yik0944gdwwrqnt_jpeg.rf.acd14651f0f7c3762be88a81ee82e1a2.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:4 objects ['clouds', 'clouds', 'clouds', 'smoke'] in ck0nculry62mq0944n824zvzk_jpeg.rf.de8abeaa0043fe7af64f470b761f71ee.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['clouds', 'clouds', 'smoke'] in ck0nfobvt4msi07944oj7bzx6_jpeg.rf.75aeb95413f2c1471c857ff8b104fc1c.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['clouds', 'smoke', 'clouds'] in ck0t4rrmsnjuy0944xt22ilg4_jpeg.rf.a8491d83b504197be76828935e6ea28e.jpg
Of 4 expected ['clouds', 'fire', 'fire', 'smoke'] found objects, found:3 objects ['fire', 'smoke', 'clouds'] in HV418.0841-1105.09_42_42_09.Still007.jpg
Of 3 expected ['fire', 'smoke', 'clouds'] found objects, found:2 objects ['smoke', 'clouds'] in HV418.20220111_195845897.33.jpg
Of 3 expected ['fire', 'smoke', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in HV418.20220111_200446318.36.jpg
Of 2 expected ['clouds', 'plane'] found objects, found:3 objects ['plane', 'plane', 'clouds'] in HV419.20200422_190631_1.00_00_17_13.Still007.jpg
Of 4 expected ['clouds', 'smoke', 'plane_night', 'fire'] found objects, found:3 objects ['plane_night', 'smoke', 'clouds'] in HV444.20211127_234941_1.00_05_07_0.Still011.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 82.0 of 86.0 expected objects with an average confidence of 83.63% min:58.37% max:95.75% adjusted avg:79.74%
smoke found 486.0 of 486.0 expected objects with an average confidence of 93.39% min:58.63% max:98.02% adjusted avg:93.39%
clouds found 503.0 of 498.0 expected objects with an average confidence of 91.75% min:50.04% max:98.47% adjusted avg:92.68%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 81.77% min:63.64% max:90.80% adjusted avg:81.77%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 92.79% min:85.69% max:95.66% adjusted avg:92.79%
plane found 10.0 of 9.0 expected objects with an average confidence of 72.85% min:55.40% max:83.67% adjusted avg:80.95%
bird found 5.0 of 5.0 expected objects with an average confidence of 75.74% min:55.56% max:88.42% adjusted avg:75.74%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 86.99% min:79.80% max:89.95% adjusted avg:86.99%
Of 3 expected ['fire', 'smoke', 'clouds'] found objects, found:2 objects ['smoke', 'clouds'] in HV418.20220111_195845897.33.jpg
Of 3 expected ['fire', 'smoke', 'clouds'] found objects, found:2 objects ['clouds', 'smoke'] in HV418.20220111_200446318.36.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV440.20211102_104104882.4.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 42.0 of 44.0 expected objects with an average confidence of 84.12% min:59.39% max:95.75% adjusted avg:80.30%
smoke found 56.0 of 55.0 expected objects with an average confidence of 91.13% min:60.82% max:96.12% adjusted avg:92.79%
clouds found 58.0 of 58.0 expected objects with an average confidence of 96.57% min:86.75% max:98.12% adjusted avg:96.57%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 90.80% min:90.80% max:90.80% adjusted avg:90.80%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 3627 tests
Ran:3627
Skipped:0
Passed:3621
Warnings:7
Failed:6
So even runs seem better.
Of 2 expected ['smoke', 'clouds'] found objects, found:1 objects ['smoke'] in ck0kmf5q58yik0944gdwwrqnt_jpeg.rf.acd14651f0f7c3762be88a81ee82e1a2.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:1 objects ['smoke'] in ck0lxhudct3ac0a46446v80ie_jpeg.rf.e82cf40e85c95c941bb2f1e6b79fd863.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 87.80% min:53.71% max:95.74% adjusted avg:87.80%
smoke found 486.0 of 486.0 expected objects with an average confidence of 95.43% min:82.82% max:98.73% adjusted avg:95.43%
clouds found 496.0 of 498.0 expected objects with an average confidence of 94.93% min:53.38% max:98.66% adjusted avg:94.55%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 86.54% min:77.49% max:93.15% adjusted avg:86.54%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 94.44% min:90.44% max:96.97% adjusted avg:94.44%
plane found 9.0 of 9.0 expected objects with an average confidence of 84.15% min:76.56% max:89.13% adjusted avg:84.15%
bird found 5.0 of 5.0 expected objects with an average confidence of 84.57% min:77.94% max:91.59% adjusted avg:84.57%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 91.04% min:88.47% max:93.39% adjusted avg:91.04%
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV440.20211102_104104882.4.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 87.25% min:65.48% max:95.74% adjusted avg:87.25%
smoke found 56.0 of 55.0 expected objects with an average confidence of 93.26% min:53.16% max:97.53% adjusted avg:94.95%
clouds found 58.0 of 58.0 expected objects with an average confidence of 97.31% min:83.82% max:98.59% adjusted avg:97.31%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 93.15% min:93.15% max:93.15% adjusted avg:93.15%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 3627 tests
Ran:3627
Skipped:0
Passed:3625
Warnings:2
Failed:2
So I wiped the caches again and reran and got
Of 2 expected ['smoke', 'clouds'] found objects, found:1 objects ['smoke'] in ck0kmf5q58yik0944gdwwrqnt_jpeg.rf.acd14651f0f7c3762be88a81ee82e1a2.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV411.0900-1059.01_15_44_09.Still006.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 87.83% min:58.04% max:96.82% adjusted avg:87.83%
smoke found 487.0 of 486.0 expected objects with an average confidence of 95.36% min:54.80% max:98.84% adjusted avg:95.56%
clouds found 497.0 of 498.0 expected objects with an average confidence of 94.49% min:57.25% max:98.49% adjusted avg:94.30%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 86.81% min:77.06% max:94.71% adjusted avg:86.81%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 94.81% min:91.53% max:97.30% adjusted avg:94.81%
plane found 9.0 of 9.0 expected objects with an average confidence of 80.03% min:64.91% max:87.77% adjusted avg:80.03%
bird found 5.0 of 5.0 expected objects with an average confidence of 85.10% min:77.70% max:92.54% adjusted avg:85.10%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 90.61% min:86.66% max:92.29% adjusted avg:90.61%
Of 3 expected ['fire', 'clouds', 'smoke'] found objects, found:2 objects ['fire', 'clouds'] in HV418.20220111_200015328.34.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 86.29% min:58.04% max:96.82% adjusted avg:86.29%
smoke found 54.0 of 55.0 expected objects with an average confidence of 94.04% min:59.64% max:97.96% adjusted avg:92.33%
clouds found 58.0 of 58.0 expected objects with an average confidence of 97.06% min:86.15% max:98.14% adjusted avg:97.06%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 94.71% min:94.71% max:94.71% adjusted avg:94.71%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 3627 tests
Ran:3627
Skipped:0
Passed:3625
Warnings:2
Failed:2
A third time I got
Of 5 expected ['smoke', 'clouds', 'clouds', 'clouds', 'clouds'] found objects, found:6 objects ['clouds', 'clouds', 'clouds', 'clouds', 'clouds', 'smoke'] in ck0nfz8fe60yr0838az9x331y_jpeg.rf.152d851a0f20d6f573a9f6f250204a9c.jpg
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV411.0900-1059.01_15_44_09.Still006.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 88.09% min:57.16% max:97.06% adjusted avg:88.09%
smoke found 487.0 of 486.0 expected objects with an average confidence of 95.08% min:54.83% max:98.42% adjusted avg:95.28%
clouds found 499.0 of 498.0 expected objects with an average confidence of 94.53% min:50.25% max:98.67% adjusted avg:94.72%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 85.81% min:72.08% max:92.18% adjusted avg:85.81%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 94.29% min:89.15% max:97.07% adjusted avg:94.29%
plane found 9.0 of 9.0 expected objects with an average confidence of 82.57% min:74.86% max:88.94% adjusted avg:82.57%
bird found 5.0 of 5.0 expected objects with an average confidence of 86.15% min:77.37% max:91.23% adjusted avg:86.15%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 90.81% min:88.95% max:93.09% adjusted avg:90.81%
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV440.20211102_104104882.4.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 87.34% min:61.72% max:97.06% adjusted avg:87.34%
smoke found 56.0 of 55.0 expected objects with an average confidence of 94.19% min:64.52% max:96.97% adjusted avg:95.90%
clouds found 58.0 of 58.0 expected objects with an average confidence of 97.37% min:88.89% max:98.47% adjusted avg:97.37%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 92.18% min:92.18% max:92.18% adjusted avg:92.18%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 3627 tests
Ran:3627
Skipped:0
Passed:3627
Warnings:4
Failed:0
So 600 epochs and wiping caches helps but the random nature of the samples means it is still not repeatable.
On my high powered video editing station 3000 epochs took about 6.5 hours which would be about 2 days on Google's Colab so this is only sort of practical on a high end PC but I had to see if it made a measurable diff So I tried it.
Of 4 expected ['fire', 'smoke', 'smoke', 'clouds'] found objects, found:5 objects ['smoke', 'smoke', 'fire', 'smoke', 'clouds'] in HV411.20220111_201302223.67.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 82.32% min:55.18% max:91.34% adjusted avg:82.32%
smoke found 56.0 of 55.0 expected objects with an average confidence of 90.96% min:64.37% max:95.31% adjusted avg:92.61%
clouds found 58.0 of 58.0 expected objects with an average confidence of 94.32% min:60.78% max:96.48% adjusted avg:94.32%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 88.09% min:88.09% max:88.09% adjusted avg:88.09%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 2 expected ['smoke', 'clouds'] found objects, found:1 objects ['smoke'] in ck0kmf5q58yik0944gdwwrqnt_jpeg.rf.acd14651f0f7c3762be88a81ee82e1a2.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:1 objects ['smoke'] in ck0lxhudct3ac0a46446v80ie_jpeg.rf.e82cf40e85c95c941bb2f1e6b79fd863.jpg
Of 2 expected ['smoke', 'clouds'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV411.0900-1059.00_44_51_01.Still003.jpg
Of 4 expected ['fire', 'smoke', 'smoke', 'clouds'] found objects, found:5 objects ['smoke', 'smoke', 'fire', 'smoke', 'clouds'] in HV411.20220111_201302223.67.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 83.84% min:56.05% max:91.36% adjusted avg:83.84%
smoke found 488.0 of 486.0 expected objects with an average confidence of 92.47% min:54.33% max:96.50% adjusted avg:92.85%
clouds found 496.0 of 498.0 expected objects with an average confidence of 92.31% min:55.26% max:96.83% adjusted avg:91.94%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 81.24% min:72.36% max:88.09% adjusted avg:81.24%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 88.80% min:85.44% max:91.60% adjusted avg:88.80%
plane found 9.0 of 9.0 expected objects with an average confidence of 72.55% min:63.45% max:80.17% adjusted avg:72.55%
bird found 5.0 of 5.0 expected objects with an average confidence of 79.53% min:72.76% max:83.18% adjusted avg:79.53%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 83.54% min:77.53% max:86.02% adjusted avg:83.54%
Of 3245 tests
Ran:3245
Skipped:0
Passed:3243
Warnings:2
Failed:2
Just to see if this made the process repeatable I ran with a wiped caches and 3000 epochs again and still got diff results though the best yet so I published it as v0.2
Of 2 expected ['clouds', 'smoke'] found objects, found:3 objects ['smoke', 'smoke', 'clouds'] in HV411.0900-1059.01_15_44_09.Still006.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 86.35% min:68.02% max:92.27% adjusted avg:86.35%
smoke found 487.0 of 486.0 expected objects with an average confidence of 93.50% min:69.21% max:97.22% adjusted avg:93.69%
clouds found 498.0 of 498.0 expected objects with an average confidence of 93.79% min:69.22% max:96.90% adjusted avg:93.79%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 81.63% min:68.68% max:88.45% adjusted avg:81.63%
helicopter found 16.0 of 16.0 expected objects with an average confidence of 90.65% min:86.90% max:93.40% adjusted avg:90.65%
plane found 9.0 of 9.0 expected objects with an average confidence of 76.91% min:69.00% max:83.26% adjusted avg:76.91%
bird found 5.0 of 5.0 expected objects with an average confidence of 81.35% min:77.83% max:84.61% adjusted avg:81.35%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 85.57% min:81.97% max:88.26% adjusted avg:85.57%
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 85.31% min:63.50% max:92.06% adjusted avg:85.31%
smoke found 55.0 of 55.0 expected objects with an average confidence of 91.50% min:55.09% max:96.04% adjusted avg:91.50%
clouds found 58.0 of 58.0 expected objects with an average confidence of 95.73% min:89.41% max:96.84% adjusted avg:95.73%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 88.45% min:88.45% max:88.45% adjusted avg:88.45%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
Of 3627 tests
Ran:3627
Skipped:0
Passed:3627
Warnings:2
Failed:0
Note the extra warn above is an image in test without a map.
So I added the missing map and relabeled the helicopter marked as a bird. Then reran with 600 epochs and got
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 87.41% min:59.90% max:96.42% adjusted avg:87.41%
smoke found 486.0 of 486.0 expected objects with an average confidence of 95.41% min:82.90% max:98.32% adjusted avg:95.41%
clouds found 498.0 of 498.0 expected objects with an average confidence of 94.99% min:76.19% max:98.50% adjusted avg:94.99%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 87.84% min:76.21% max:94.06% adjusted avg:87.84%
helicopter found 17.0 of 17.0 expected objects with an average confidence of 94.40% min:89.29% max:96.36% adjusted avg:94.40%
plane found 9.0 of 9.0 expected objects with an average confidence of 80.72% min:72.27% max:85.90% adjusted avg:80.72%
bird found 4.0 of 4.0 expected objects with an average confidence of 79.66% min:67.89% max:87.37% adjusted avg:79.66%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 90.66% min:87.33% max:93.83% adjusted avg:90.66%
Of 3 expected ['fire', 'clouds', 'smoke'] found objects, found:2 objects ['fire', 'clouds'] in HV418.20220111_200015328.34.jpg
Of 2 expected ['helicopter_night', 'clouds'] found objects, found:3 objects ['clouds', 'helicopter_night', 'clouds'] in HV444.20211127_234941_1.00_04_09_0.Still001.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 86.80% min:59.90% max:96.42% adjusted avg:86.80%
smoke found 54.0 of 55.0 expected objects with an average confidence of 94.81% min:81.97% max:97.21% adjusted avg:93.08%
clouds found 60.0 of 59.0 expected objects with an average confidence of 97.35% min:91.00% max:98.40% adjusted avg:99.00%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 94.06% min:94.06% max:94.06% adjusted avg:94.06%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 1.0 of 1.0 expected objects with an average confidence of 92.25% min:92.25% max:92.25% adjusted avg:92.25%
Of 3633 tests
Ran:3633
Skipped:0
Passed:3632
Warnings:1
Failed:1
Which really does not look that bad for my use but a second try was even better. I checked it in as v0.3
Of 2 expected ['plane', 'clouds'] found objects, found:3 objects ['plane', 'plane', 'clouds'] in HV419.20200422_190631_1.00_00_11_07.Still002.jpg
For images in:train/ with min_confidence set to 50.00%
fire found 86.0 of 86.0 expected objects with an average confidence of 88.18% min:68.66% max:96.92% adjusted avg:88.18%
smoke found 486.0 of 486.0 expected objects with an average confidence of 95.54% min:82.90% max:98.86% adjusted avg:95.54%
clouds found 498.0 of 498.0 expected objects with an average confidence of 94.53% min:59.26% max:98.52% adjusted avg:94.53%
plane_night found 7.0 of 7.0 expected objects with an average confidence of 86.99% min:77.73% max:92.76% adjusted avg:86.99%
helicopter found 17.0 of 17.0 expected objects with an average confidence of 94.12% min:88.60% max:96.24% adjusted avg:94.12%
plane found 10.0 of 9.0 expected objects with an average confidence of 79.16% min:51.25% max:88.13% adjusted avg:87.96%
bird found 4.0 of 4.0 expected objects with an average confidence of 84.32% min:75.91% max:89.59% adjusted avg:84.32%
helicopter_night found 8.0 of 8.0 expected objects with an average confidence of 90.29% min:84.03% max:93.76% adjusted avg:90.29%
Of 2 expected ['helicopter_night', 'clouds'] found objects, found:3 objects ['clouds', 'helicopter_night', 'clouds'] in HV444.20211127_234941_1.00_04_09_0.Still001.jpg
For images in:test/ with min_confidence set to 50.00%
fire found 44.0 of 44.0 expected objects with an average confidence of 87.19% min:68.66% max:96.92% adjusted avg:87.19%
smoke found 55.0 of 55.0 expected objects with an average confidence of 94.16% min:55.72% max:97.41% adjusted avg:94.16%
clouds found 60.0 of 59.0 expected objects with an average confidence of 97.04% min:90.31% max:98.30% adjusted avg:98.69%
plane_night found 1.0 of 1.0 expected objects with an average confidence of 92.76% min:92.76% max:92.76% adjusted avg:92.76%
helicopter found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
plane found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
bird found 0.0 of 0.0 expected objects with an average confidence of 0 min:100.00% max:0.00% adjusted avg:0
helicopter_night found 1.0 of 1.0 expected objects with an average confidence of 93.19% min:93.19% max:93.19% adjusted avg:93.19%
Of 3633 tests
Ran:3633
Skipped:0
Passed:3633
Warnings:2
Failed:0
These looked possibly useful for fire detection from security cams.
Note one image in "non-fire" is of a fire. Most are forest shots. Many with people and a few with smoke.
In the end I decided I might use the people shots in another model but there seemed little to gain from including any in my fire training set.
- You want images that closely match what you expect to detect. Both objects and views of them.
- Be mindful of things you might not care about that might be seen. They will probably cause false positives unless you train for them as well.
- Try to avoid nightvision images unless nightvision detection is your primary goal.
Each epoch gets tested against your test images so they ought to be:
- All or mostly images from the cams you intend to use the model with.
- Have at least one good image with a fullly visible, full color object in view.
- Have at least one "edge case" (an image that is about as bad as you would except as passing). This may mean partly covered and or blurry.
- If multiple angles are expected to be detected then at least a sample of some of those should be included.
- Being able to detect type of dog, for instance, is nice but requires more images than just training with all of them for dog and may increase with false postives for dogs being seen as non dogs or non dogs seen as a dog breed.
For instance trying to seperate cars from vans by labeling new images correctly, make sure any car labels in your previous set are labeled correctly.
trainTest will catch many of these and show them in debugPath but there is not guarentee it will see them all. In the example above a few helicopters were mislabeled and I only caught them when I noticed they were listed in debugPath\bird.lst.txt.
Before making changes and doing another train and test backup up your train and test folders as well as the model pt, train.cache and test.cache files. Even reversing your changes might not get you back to where you were. See Revert to backup above.
Rerunning training will get you different test results. See Add more images from your cameras regularly above for an example of how this can complicate fine tuning your model.
Also it seems path stored in them are relative to the cache. For example I generally work in my deepstack folder. If I try and train passing the ../RMRR.fire folder it will read the cache files in RMRR.fire but look for files in the deepstack/train folder where it was originally trained from.
See So how about 3000 epochs and wiping caches? above.
Even 3000 is not enough for a repeatable process with my 533 training and 57 test images. 600 does not look too bad for a compromise but I would still suggest running training and test at least twice, saving backups inbetween and then keeping the best of the runs.
Even if capitalized in the classes.txt file DeepStack will return them as all lowercase which might cause a disconnect with some tools that do not do a case ignore compare. Best to just keep them all lowercase.
When verifing the quickLabel labeled files it is often more important to fix labels than to add unlabeled objects. If you do not have a half way decent model to start with you will have a lot of mislabeled and not labeled objects. For a first training path it might save time to fix, train, fix and retrain than to try and get it right in one fix and train. trainTest will create suggested mapping files for any image maps that do not contain the objects, more, less or diff as expected. The only exceptions to this would be object for which a class ID has not been created or only a few examples exist. Run python chkClasses.py to check only the classes you want are used in the labeled folder map files.