Slide 1
Slide 1 text
Natural Language Framework
Daiki Matsudate, Freelancer
#WWDC18 WWDCΰϦΰϦΩϟονΞοϓձ
Slide 2
Slide 2 text
Daiki Matsudate
@d_date
Slide 3
Slide 3 text
Note
• WWDC 2018Ͱެද͞Εͨbeta൛Ͱఏڙ͞Ε͍ͯΔAPIͷใͰߏͯ͠
͍·͢
• ࠓޙͷbeta releaseͰAPIͷڍಈ͕มΘΔ͔͠Ε·ͤΜɻࢀর͢Δࡍ͝
ҙ͍ͩ͘͞
Slide 4
Slide 4 text
NaturalLanguage.framework
Slide 5
Slide 5 text
Natural Language
Intelligence
Linguistics Machine Learning
Language Identification
Tokenization
Part of Speech
Lemmatization
Named Entity Recognition
Word Sentence Paragraph
Natural
Language Input
NaturalLanguage.framework
WWDC18 session 713 Introducing Natural Language
Slide 6
Slide 6 text
• For processing Natural Language
• NLTokenizer
• NLLanguageRecognizer
• NLTagger
• NLModel
NaturalLanguage.framework
Slide 7
Slide 7 text
NLTokenizer
Tokenize text into word, sentence, paragraph or document
νϟμʔ৯͍ͨʂ
Slide 8
Slide 8 text
NLTokenizer
Tokenize text into word
νϟμʔ৯͍ͨʂ
νϟμʔ ৯ ͍ͨ
let tokenizer = NLTokenizer(unit: .word)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 9
Slide 9 text
NLTokenizer
Tokenize text into word
let tokenizer = NLTokenizer(unit: .word)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 10
Slide 10 text
NLTokenizer
Tokenize text into word
let tokenizer = NLTokenizer(unit: .word)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 11
Slide 11 text
NLTokenizer
Tokenize text into sentence
let tokenizer = NLTokenizer(unit: .sentence)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 12
Slide 12 text
NLTokenizer
Tokenize text into sentence
νϟμʔ৯͍ͨʂ
νϟμʔ৯͍ͨ
let tokenizer = NLTokenizer(unit: .sentence)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 13
Slide 13 text
NLTokenizer
Tokenize text into sentence
͋ͷΠʔϋτʔϰΥͷ͖͢ͱ͓ͬͨ෩ɺՆͰఈʹྫྷͨ͞Λͭ੨͍ͦΒɺ͏͍ͭ͘͠Ͱ০ΒΕͨϞϦʔΦࢢɺ߫֎ͷ͗Β͗
Βͻ͔Δͷɻ
ɹ·ͨͦͷͳ͔Ͱ͍ͬ͠ΐʹͳͬͨͨ͘͞ΜͷͻͱͨͪɺϑΝθʔϩͱϩβʔϩɺ༽ࣂͷϛʔϩɺإͷ͍͜Ͳͨͪɺओ
ͷςʔϞɺࢁೣത࢜ͷϘʔΨϯτɾσετΡύʔΰͳͲɺ͍·͜ͷ҉͍ڊ͖ͳੴͷݐͷͳ͔Ͱߟ͍͑ͯΔͱɺΈΜͳΉ͔͠෩
ͷͳ͔͍ͭ͠੨͍ݬ౯ͷΑ͏ʹࢥΘΕ·͢ɻͰɺΘ͍͔ͨͭ͘͠ͷখ͞ͳΈͩ͠Λ͚ͭͳ͕Βɺ͔ͣ͠ʹ͋ͷͷΠʔϋτʔ
ϰΥͷޒ݄͔Βे݄·ͰΛॻ͖͚ͭ·͠ΐ͏ɻ
let tokenizer = NLTokenizer(unit: .sentence)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 14
Slide 14 text
NLTokenizer
Tokenize text into sentence
͋ͷΠʔϋτʔϰΥͷ͖͢ͱ͓ͬͨ෩ɺՆͰఈʹྫྷͨ͞Λͭ੨͍ͦΒɺ͏͍ͭ͘͠Ͱ০ΒΕͨϞϦʔΦࢢɺ߫֎ͷ͗Β͗
Βͻ͔Δͷɻ
ɹ·ͨͦͷͳ͔Ͱ͍ͬ͠ΐʹͳͬͨͨ͘͞ΜͷͻͱͨͪɺϑΝθʔϩͱϩβʔϩɺ༽ࣂͷϛʔϩɺإͷ͍͜Ͳͨͪɺओ
ͷςʔϞɺࢁೣത࢜ͷϘʔΨϯτɾσετΡύʔΰͳͲɺ͍·͜ͷ҉͍ڊ͖ͳੴͷݐͷͳ͔Ͱߟ͍͑ͯΔͱɺΈΜͳΉ͔͠෩
ͷͳ͔͍ͭ͠੨͍ݬ౯ͷΑ͏ʹࢥΘΕ·͢ɻͰɺΘ͍͔ͨͭ͘͠ͷখ͞ͳΈͩ͠Λ͚ͭͳ͕Βɺ͔ͣ͠ʹ͋ͷͷΠʔϋτʔ
ϰΥͷޒ݄͔Βे݄·ͰΛॻ͖͚ͭ·͠ΐ͏ɻ
let tokenizer = NLTokenizer(unit: .sentence)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 15
Slide 15 text
NLTokenizer
Tokenize text into paragraph
͋ͷΠʔϋτʔϰΥͷ͖͢ͱ͓ͬͨ෩ɺՆͰఈʹྫྷͨ͞Λͭ੨͍ͦΒɺ͏͍ͭ͘͠Ͱ০ΒΕͨϞϦʔΦࢢɺ߫֎ͷ͗Β͗
Βͻ͔Δͷɻ
ɹ·ͨͦͷͳ͔Ͱ͍ͬ͠ΐʹͳͬͨͨ͘͞ΜͷͻͱͨͪɺϑΝθʔϩͱϩβʔϩɺ༽ࣂͷϛʔϩɺإͷ͍͜Ͳͨͪɺओ
ͷςʔϞɺࢁೣത࢜ͷϘʔΨϯτɾσετΡύʔΰͳͲɺ͍·͜ͷ҉͍ڊ͖ͳੴͷݐͷͳ͔Ͱߟ͍͑ͯΔͱɺΈΜͳΉ͔͠෩
ͷͳ͔͍ͭ͠੨͍ݬ౯ͷΑ͏ʹࢥΘΕ·͢ɻͰɺΘ͍͔ͨͭ͘͠ͷখ͞ͳΈͩ͠Λ͚ͭͳ͕Βɺ͔ͣ͠ʹ͋ͷͷΠʔϋτʔ
ϰΥͷޒ݄͔Βे݄·ͰΛॻ͖͚ͭ·͠ΐ͏ɻ
let tokenizer = NLTokenizer(unit: .paragraph)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 16
Slide 16 text
NLTokenizer
Tokenize text into document
͋ͷΠʔϋτʔϰΥͷ͖͢ͱ͓ͬͨ෩ɺՆͰఈʹྫྷͨ͞Λͭ੨͍ͦΒɺ͏͍ͭ͘͠Ͱ০ΒΕͨϞϦʔΦࢢɺ߫֎ͷ͗Β͗
Βͻ͔Δͷɻ
ɹ·ͨͦͷͳ͔Ͱ͍ͬ͠ΐʹͳͬͨͨ͘͞ΜͷͻͱͨͪɺϑΝθʔϩͱϩβʔϩɺ༽ࣂͷϛʔϩɺإͷ͍͜Ͳͨͪɺओ
ͷςʔϞɺࢁೣത࢜ͷϘʔΨϯτɾσετΡύʔΰͳͲɺ͍·͜ͷ҉͍ڊ͖ͳੴͷݐͷͳ͔Ͱߟ͍͑ͯΔͱɺΈΜͳΉ͔͠෩
ͷͳ͔͍ͭ͠੨͍ݬ౯ͷΑ͏ʹࢥΘΕ·͢ɻͰɺΘ͍͔ͨͭ͘͠ͷখ͞ͳΈͩ͠Λ͚ͭͳ͕Βɺ͔ͣ͠ʹ͋ͷͷΠʔϋτʔ
ϰΥͷޒ݄͔Βे݄·ͰΛॻ͖͚ͭ·͠ΐ͏ɻ
let tokenizer = NLTokenizer(unit: .document)
tokenizer.string = text
tokenizer.tokens(for: text.startIndex..
Slide 17
Slide 17 text
NLLanguageRecognizer
Automatically identify the language of text
νϟμʔ৯͍ͨʂ
Slide 18
Slide 18 text
νϟμʔ৯͍ͨʂ
NLLanguage.japanese
let recognizer = NLLanguageRecognizer()
recognizer.processString("νϟμʔ৯͍ͨʂ")
print(recognizer.dominantLanguage!) // ja
NLLanguageRecognizer
Automatically identify the language of text
Slide 19
Slide 19 text
let recognizer = NLLanguageRecognizer()
recognizer.processString("νϟμʔ৯͍ͨʂ")
print(recognizer.dominantLanguage!) // ja
NLLanguageRecognizer
Automatically identify the language of text
Slide 20
Slide 20 text
let recognizer = NLLanguageRecognizer()
recognizer.processString(“Lorem ipsum dolor sit amet”)
recognizer.languageHints = [.english: 0.3, .portuguese: 0.1]
print(recognizer.dominantLanguage!) // en
NLLanguageRecognizer
Specify language hints with factor
Specify language hints
[NLLanguage : Double]
Check dominant language
NLLanguage.english
Slide 21
Slide 21 text
let recognizer = NLLanguageRecognizer()
recognizer.processString(“Lorem ipsum dolor sit amet”)
recognizer.languageHints = [.french: 0.3, .english: 0.00001]
print(recognizer.dominantLanguage!) // en
NLLanguageRecognizer
Specify language hints with factor
Specify language hints
[NLLanguage : Double]
Check dominant language
NLLanguage.french
Slide 22
Slide 22 text
let recognizer = NLLanguageRecognizer()
recognizer.processString(“Lorem ipsum dolor sit amet”)
recognizer.languageHints = [.french: 0.3, .english: 0.1]
print(recognizer.dominantLanguage!) // en
NLLanguageRecognizer
Specify language hints with factor
Specify language probabilities
[NLLanguage : Double]
Check dominant language
NLLanguage.english
Slide 23
Slide 23 text
let recognizer = NLLanguageRecognizer()
recognizer.processString(“νϟμʔ৯͍ͨʂ”)
recognizer.languageHints = [.french: 0.3, .english: 0.1]
print(recognizer.dominantLanguage!) // en
NLLanguageRecognizer
Specify language hints with factor
Specify language probabilities
[NLLanguage : Double]
Check dominant language
NLLanguage.japanese
Slide 24
Slide 24 text
let recognizer = NLLanguageRecognizer()
recognizer.processString(“Lorem ipsum dolor sit amet”)
let hyphotheses = recognizer.languageHypotheses(withMaximum: 2)
print(hyphotheses)
NLLanguageRecognizer
Hypotheses languages
Hypotheses with maximum
language count
Hypotheses says probably this is English
// [pt: 0.32105109095573425, en: 0.4647941291332245]
Slide 25
Slide 25 text
NLTagger
Analyzes natural language text
νϟμʔ৯͍ͨʂ
let text = "νϟμʔ৯͍ͨʂ"
let tagger = NLTagger(tagSchemes: [.lexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 26
Slide 26 text
NLTagger
Analyzes natural language text
νϟμʔ৯͍ͨʂ
let text = "νϟμʔ৯͍ͨʂ"
let tagger = NLTagger(tagSchemes: [.lexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 27
Slide 27 text
NLTagger
Analyzes natural language text
Chowder time
let text = “$IPXEFSUJNF”
let tagger = NLTagger(tagSchemes: [.lexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 28
Slide 28 text
NLTagger
Analyzes natural language text
let text = “Chowder time”
let tagger = NLTagger(tagSchemes: [.lexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 29
Slide 29 text
NLTagger
Analyzes natural language text
let text = “Chowder time”
let tagger = NLTagger(tagSchemes: [.lexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 30
Slide 30 text
NLTagger
Analyzes natural language text - Lexical Class
let text = “I could not go WWDC in this year”
let tagger = NLTagger(tagSchemes: [.lexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 31
Slide 31 text
NLTagger
Analyzes natural language text - Lemma
let text = “I could not go WWDC in this year”
let tagger = NLTagger(tagSchemes: [.lemma])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 32
Slide 32 text
NLTagger
Analyzes natural language text - Name type
let text = “I could not go WWDC in this year”
let tagger = NLTagger(tagSchemes: [.nametype])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 33
Slide 33 text
NLTagger
Analyzes natural language text - Name type or lexicalClass
let text = “I could not go WWDC in this year”
let tagger = NLTagger(tagSchemes: [.nameTypeOrLexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 34
Slide 34 text
NSLinguisticTagger in iOS 11
Slide 35
Slide 35 text
NSLinguisticTagger
Analyzes natural language text - Name type or lexicalClass
let text = “I could not go WWDC in this year”
let tagger = NSLinguisticTagger(tagSchemes: [.nameTypeOrLexicalClass])
tagger.string = text
tagger.tags(in: NSRange(string: text)!, unit: .word, scheme: .lexicalClass)
.forEach { (tag, range) in
if let tag = tag {
print(tag.rawValue, text[range])
}
}
I in
could not go WWDC this year
Pronoun Noun
Verb Verb
OrganizationName
Adverb Preposition Determinator
Slide 36
Slide 36 text
NSLinguisticTagger
Analyzes natural language text - Name type or lexicalClass
let text = “I could not go WWDC in this year”
let tagger = NSLinguisticTagger(tagSchemes: [.nameTypeOrLexicalClass])
tagger.string = text
tagger.tags(in: NSRange(string: text)!, unit: .word, scheme: .lexicalClass)
.forEach { (tag, range) in
if let tag = tag {
print(tag.rawValue, text[range])
}
}
I in
could not go WWDC this year
Pronoun Noun
Verb Verb
OrganizationName
Adverb Preposition Determinator
Same?
Slide 37
Slide 37 text
Compare NLTagger and NSLinguisticTagger
func testNLTagIsEqualToNSLinguisticTag() {
let tSchemes: [NLTagScheme] =
[.language, .lemma, .lexicalClass, .nameTypeOrLexicalClass, .nameType, .tokenType, .script]
let lSchemes: [NSLinguisticTagScheme] =
[.language, .lemma, .lexicalClass, .nameTypeOrLexicalClass, .nameType, .tokenType, .script]
let tTagger = NLTagger(tagSchemes: tSchemes)
let lTagger = NSLinguisticTagger(tagSchemes: lSchemes)
zip(tSchemes, lSchemes).forEach { (tScheme, lScheme) in
print("---- \(tScheme.rawValue) ----")
XCTAssertEqual(tScheme.rawValue, lScheme.rawValue)
let tags = tTagger.tags(text: text, unit: .word, scheme: tScheme, options:
[.omitPunctuation, .omitWhitespace])
let lTags = lTagger.tags(text: text, unit: .word, scheme: lScheme, options:
[.omitPunctuation, .omitWhitespace])
zip(tags, lTags).forEach({ (tTag, lTag) in
print(tTag.0.rawValue, lTag.0.rawValue, tTag.0.rawValue == lTag.0.rawValue)
XCTAssertEqual(tTag.0.rawValue, lTag.0.rawValue)
XCTAssertEqual(text[tTag.1], text[Range(lTag.1, in: text)!])
let text = "I could not go WWDC in this year."
Slide 38
Slide 38 text
Compare NLTagger and NSLinguisticTagger
func testNLTagIsEqualToNSLinguisticTag() {
let tSchemes: [NLTagScheme] =
[.language, .lemma, .lexicalClass, .nameTypeOrLexicalClass, .nameType, .tokenType, .script]
let lSchemes: [NSLinguisticTagScheme] =
[.language, .lemma, .lexicalClass, .nameTypeOrLexicalClass, .nameType, .tokenType, .script]
let tTagger = NLTagger(tagSchemes: tSchemes)
let lTagger = NSLinguisticTagger(tagSchemes: lSchemes)
zip(tSchemes, lSchemes).forEach { (tScheme, lScheme) in
print("---- \(tScheme.rawValue) ----")
XCTAssertEqual(tScheme.rawValue, lScheme.rawValue)
let tags = tTagger.tags(text: text, unit: .word, scheme: tScheme, options:
[.omitPunctuation, .omitWhitespace])
let lTags = lTagger.tags(text: text, unit: .word, scheme: lScheme, options:
[.omitPunctuation, .omitWhitespace])
zip(tags, lTags).forEach({ (tTag, lTag) in
print(tTag.0.rawValue, lTag.0.rawValue, tTag.0.rawValue == lTag.0.rawValue)
XCTAssertEqual(tTag.0.rawValue, lTag.0.rawValue)
XCTAssertEqual(text[tTag.1], text[Range(lTag.1, in: text)!])
let text = "I could not go WWDC in this year."
Slide 39
Slide 39 text
Compare NLTagger and NSLinguisticTagger
token NLTagger NSLinguisticTagger
I I I
ca nil “”
n’t not not
Slide 40
Slide 40 text
Contractions recognition
Actural “I / ca / n’t / go / WWDC / in / this / year."
Ideal “I / can’t / go / WWDC / in / this / year."
Slide 41
Slide 41 text
Actural “I / can’t / go / WWDC / in / this / year."
Ideal “I / can’t / go / WWDC / in / this / year."
Contractions recognition
.joiningContractions
Slide 42
Slide 42 text
NLTagger with joining contractions
let text = “I could not go WWDC in this year”
let tagger = NLTagger(tagSchemes: [.nameTypeOrLexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 43
Slide 43 text
NLTagger with joining contractions
let text = “I could not go WWDC in this year”
let tagger = NLTagger(tagSchemes: [.nameTypeOrLexicalClass])
tagger.string = text
tagger.tags(in: text.startIndex..
Slide 44
Slide 44 text
NLModel
A custom model trained to classify or tag natural language text.
Create ML
User Data NLModel NLTagger
Slide 45
Slide 45 text
NLModel
A custom model trained to classify or tag natural language text.
WIP
Waiting for sample from Apple
Introducing Create ML WWDC18
Slide 46
Slide 46 text
• Natural Language
• Tokenization … NLTokenizer can separate text into tokens
• Language Recognition
… NLLanguageRecognizer can detect the language in text
• Tagging … NLTagger can tag token of lexical class, name type and so on
• NLTagger and NSLinguisticTagger have same behavior, but only NLTagger can
join contractions into one token
• Custom model … NLModel can be used for tagging by setting to tagger. Also,
use model independently.
Summary
Slide 47
Slide 47 text
Introducing Natural Language WWDC18
Natural Language Processing and your apps WWDC17
Introducing Core ML WWDC17
Core ML in depth WWDC17
Introducing Create ML WWDC18
What’s new in Core ML, Part 1 WWDC18
iOS 11 Programming, ୈ3ষ PEAKS
Resources
Slide 48
Slide 48 text
WWDC18