teach it about Moths or Shells or ..? >> No, we have no way of incrementally training this neural net classifier. Can we retrain a new net regularly? >> We could train every month or so, if we had millions of training samples. >> We didn’t.
store feature vectors for all images • Compute feature vector for query image • Calculate distance from query vector to every other vector • Pick the top 20 • Suggest the classes of these images to the user
Sometimes, it was like magic with spot on suggestions Images were immediately indexed and ‘learned’ from Even a single example image could help classify new images
grew linearly, we had to keep increasing RAM Sometimes, it seemed like it had learned nothing at all It had two related problems: • It didn’t know where to look ◦ it didn’t differentiate between subject & background • It didn’t know what was important ◦ It was too sensitive to differences in size or rotation
based storage & query system • Two stage query process to scale better ◦ Hamming distance query across binary hashes of vectors (pg_similarity) ◦ Euclidean distance across first N Hamming results (Cube) We researched image segmentation to remove background interference
solution to all Machine Learning problems” - half of hackernews “TensorFlow™ is an open source software library for high performance numerical computation. … allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. … it comes with strong support for machine learning and deep learning ...” - tensorflow.org
is an open source software library for high performance numerical computation. … allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. … it comes with strong support for machine learning and deep learning ...” - tensorflow.org
to use Tensorflow and Google’s Inception net. We finally started training fine-tuning a neural net of our own. We created a specialised net just for Butterflies & Moths (leps) • ~ 3000 classes, 20-1000 images per class • ~ 24 hours training time with a GPU • Original images with original crops. No segmentation. • ~ 70% accuracy
the categories of the classification results to display category suggestions with the most similar image • Using feature vectors from the same inception net But, This caused a further challenge when we needed to update the inception net to add more classes - the feature vectors were no longer comparable We ended up with two inception nets; one for classification that was updated regularly and one for feature extraction that remained static
neural net passes were required Users could only benefit from the specialised net when using the Leps app. When uploading a moth image on main Fieldguide we still had to use the slower, less accurate, less satisfying search method. We were now maintaining two parallel systems; one using Caffe, one using Tensorflow.
level Inception network to classify into broad Nature categories. This then directed the request towards the speciality Leps net or a narrowed similarity search. • We used this same Inception net to extract features instead of the generic Caffenet. ◦ Now everything was under one Tensorflow framework. New challenges: • Latency increased now that up to three neural net passes were required!
relative speed of the Leps net. They wanted more! We got more data, lots more data! Could we scale this fine-tuned approach to a much wider set of categories?
for everything with >100,000 classes? • Seems near impossible. No published research beyond 1000-10,000 classes. • Train many specialised Inception nets? • Possible for a limited number of nets but very costly in terms of training time + resources and inference resource requirements.
Why is the solution to every problem another neural net!? Maybe everything looks like a nail because all we’ve got are hammers! What if we go traditional and split the feature extraction step from the classification step? A CNN like Inception is mostly a feature extractor with a small classifier layer added on top. Can we train an Inception feature extractor specialised for nature? Can we train thousands of ‘tiny’ classifiers to classify parts of the category tree using these feature vectors?
feature extractor specialised for nature? > Yes we can. Trained as a classifier for ~5k Genus categories from across nature. Can we train a hierarchical ensemble of thousands of ‘tiny’ classifiers to classify parts of the category tree using these feature vectors? > Yes, from sklearn.linear_model import SGDClassifier
to retrain any neural networks. Training smaller classifiers was fast and ‘easy’. It was very fast. It used a fraction of the memory. It gave best effort hierarchical results.
as we hoped. Accuracy on Leps was significantly lower than with the speciality Inception net. This approach is less robust to noise and mislabeled data.
and automatic curation tools to help improve our training sets. We currently have a hybrid approach with a hierarchical ensemble of classifiers and a few speciality Inception nets for special areas of interest As the data quality improves & our accuracy improves we hope to move away from the speciality approach.
& machine learning fields are doing amazing open work. The best libraries & frameworks are all in Python :) CV may start out as ‘plug+play’ but you quickly end up deep down the rabbit hole. A lot of ‘glue’ and pipeline code is required between the various components Scaling CV is hard, and expensive. Neural net orchestration becomes a thing :( Training still needs human supervision & manual intervention
constant challenges Alp Güler & Grant Van Horn for computer vision & machine learning advice & guidance Taylan & my colleagues on the Fieldguide team @ Hipo
fergalwalsh
shinout
bkuhlmann
seike460
ttskch
urmot
manfredsteyer
coconala_engineer
palkan
myfinder
teamlab
nagainaganawa
nearme_tech
sferik
eileencodes
tenderlove
chriscoyier
carmenintech
sstephenson
myddelton
geeforr
aarron
aki_iinuma
tanoku
![Fergal Walsh PyConIE 2018 Questions? Me: @fergalwalsh, [email protected] Hipo: @hipolabs](https://files.speakerdeck.com/presentations/c16d3082ba3c4f4da4c8d55433b85365/slide_84.jpg){kind=link}