Optical Music Recognition: Applications on Mobile

SPIN Fellows OMR talk

Transcript

1. OPTICAL MUSIC RECOGNITION APPLICATIONS on MOBILE A PROJECT BY robert cheung WITH MENTORSHIP OF michael welge & colleen bushell
2. THE PROJECT §  motivations •  Musicians have too much paper sheet music •  Rehearsing is impractical with PDFs on tablets §  the idea •  It’s the digital age! Let’s keep all of our music in one organized place •  Instead of PDFs, let’s use a more versatile data structure §  the plan •  Develop a way to convert printed music into more useful abstraction •  Display music on tablets dynamically!
3. STEP 1 – FIND THE STAFFS §  y-projection •  2-D Bitmap è 1-D Array •  Allows us to summarize the y composition •  Peaks indicate staff lines §  measure & partition •  Measure groups of five peaks & space between those groups •  Partition the image in those intervals •  These are staffs 2 0 4 1
4. STEP 2 – PARTITION §  iterate •  For each of the staffs identified in Step 1, do the following §  x-projection •  Summarize x composition of the image over the height of current staff only •  We call these ‘Blobs’ §  measure & partition •  Record the position of each Blob and send them to Step 3
5. STEP 3 – FILTER & PARTITION §  y-projection on each blob §  clean o" sta" lines and note stems •  These get in the way of us finding the relevant pieces we need •  Clear staff line spikes •  Clear Note stems (tricky!) •  Left with only the main components of music §  Pass o" all partitions to classification algorithm
6. STEP 4 – CLASSIFY §  artificial neural networks •  Multiple-to-one, all or nothing •  “If an input of a neuron is repeatedly and persistently causing the neuron to fire, a metabolic change happens in the synapse of that particular input to reduce its resistance.” •  ANN works similarly •  Start with naïve set of weights (random) •  Systematically adjust weights at each step so that the desired outcome is achieved •  Do this for all elements in the training set •  The state of the ANN can always be saved!
7. STEP 0 – PREPROCESS §  none of that works if the image isn’t what we expect! •  We need to pre-process non-ideal images before our classification methods are useful §  assume perfectly horizontal sta" lines •  Fix with Fast Fourier Transform §  assume monochrome •  Fix with bit shifting
8. DEMONSTRATION TIME enough TALK, let’s see if it WORKS
9. FINAL THOUGHTS & FUTURE PLANS §  what i’ve learned •  Plan for everything! Projections and ANNs don’t do some things very well. §  where it’s at •  Working on ‘good’ images. Will need work before we can do ‘good’ camera photos. Even more work for practical use. §  where it’s going •  Built from open-source building open source. Pay it forward!