Recognition (1/3) Sum -> Sum • [+-] Product (0) the number where the item starts ┌───┬───┬───┬───┬───┬───┬───┬───┬───┐ │ 1 │ + │ ( │ 2 │ * │ 3 │ - │ 4 │ ) │ └───┴───┴───┴───┴───┴───┴───┴───┴───┘ S0 S1 S2 Completed if: Sum -> Sum [+-] Product • (0) S9 how much the item has been parsed
Prediction: add rule of the next non-terminal to the current set o Scan: move fat dot forward, add this item into the next set. Thursday, July 3, 2025 8 Earley Recognition (2/3) S0 initialized with • start rule, • (0), • leading fat dot predicted S1 scanned ┌───┬───┬───┬───┬───┬───┬───┬───┬───┐ │ 1 │ + │ ( │ 2 │ * │ 3 │ - │ 4 │ ) │ └───┴───┴───┴───┴───┴───┴───┴───┴───┘
Rules are in the Chomsky Norm Form Thursday, July 3, 2025 10 CYK Recognition Illust. S → PRP VP NP → PRP PP NP → DT NNS VP → VBD PRP VP → VBD NP VP → VP PP PP → IN NP l=5 l=4 l=3 l=2 l=1 l=0 PRP VBD NN PRP IN DT NNS I saw him with the binoculars
i+3 Thursday, July 3, 2025 16 1. chart[1..n, 1..n, 1..V] = False 2. for p = 1 to n: 3. for rule A->w_p in rules: 4. chart[1, p, A] := True 5. for l = 2 .. n: 6. for p = 1 .. n - l + 1: 7. for s = 1 .. l - 1: 8. for rule A->BC : 9. chart[l, p, A] = 10. chart [l, p, A] or 11. chart[s, p, B] and 12. chart[l-s, p+s, c] 13. 14. return chart[n, 1, S] The ||| cat is jumping The cat ||| is jumping The cat is ||| jumping
⊗: associative o CYK recognition is ({True, False}, OR, AND, True, False) oInside Algorithm o (ℝ⩾0 ⋃{+∞}, +, *, 0, 1) o for all marginal tree weights oState Transition Equation: Thursday, July 3, 2025 17 Semiring-based Parsing Ak i = B C j πA→BCBj i Ck j+1
Rules, with less observation o A mixed idea of language without pragmatics in the ESL education o Explain the human language ability as suggested by generative grammar oLearning grammar rules from data o Usually, the grammar formalization is assumed beforehand oSelected Works o DIORA (NAACL 2019) o C-PCFG (ACL 2019) and TD-PCFG (NAACL 2021) o Perturb-and-Parse (ACL 2019) o R2D2 (ACL 2021) Thursday, July 3, 2025 19 Grammar Induction
domain oLatent tree parsers o produce representations for all internal nodes o each generated with a soft weighting over all possible sub-trees o requires sentence level annotations for training (usually labels for downstream tasks, such as NLI) oPrevious works o predict trees not aligned with known treebanks o no mechanism to model phrases, requiring a complex procedure to extract syntactic structures (such as ON-LSTM) Thursday, July 3, 2025 22 DIORA: Motivations
o Ill-behaved optimization landscape o Overly strict independence assumptions of PCFGs oSuccessful approaches resort to o carefully-crafted auxiliary objectives o priors or non-parametric models o manually engineered features oThey propose to o parameterizing PCFG with neural networks makes it possible to induce linguistically meaningful grammars by simply optimizing log-likelihood o to incorporate side information is straight-forward Thursday, July 3, 2025 25 C-PCFG: Motivations
is adopted not because its adequacy but its tractability. o C-PCFG is a restricted version of some higher-order PCFG Thursday, July 3, 2025 27 C-PCFG: Models (2)
30 non-terminals and 60 pre-terminals oMore symbols are important: o Dividing PTB categories into subtypes improves parsing o Increasing the number of hidden states is helpful for learning latent variables Thursday, July 3, 2025 30 TD-PCFG: Motivations
o where U is row-normalized, and V, W are column-normalized Thursday, July 3, 2025 31 TD-PCFG: Methods T = d l=1 T(l) T(l) ijk = u(l) i · v(l) j · w(l) k Ak i = B C j πA→BCBj i Ck j+1 Sik = U j (V T Sij ) (WT Sjk ), U ∈ Rn×d, V, W ∈ Rm×d
in size and domain oLinguistic structures trained for downstream tasks o provide an inductive bias specifying structures o not making any assumptions regarding what the structures represent osample global structures in a differentiable way Thursday, July 3, 2025 37 Perturb-and-Parse: Motivations
perturbation o Softmax instead of argmax (in Eisner), though the output T is not valid dependency trees anymore, but a soft selection of arcs instead. Thursday, July 3, 2025 38 Perturb-and-Parse: Methods
oPretrained-LMs o has fixed depth and requires positional embeddings o do not explicitly reflect the hierarchical structures oFully differentiable CKY o is O(N3) and hard to scale up oContributions: o Recursive Transformers learn both representations and structures o an efficient optimization algorithm O(n) to scale up o an effective training objective Thursday, July 3, 2025 42 R2D2: Motivations
artificial o targeted and specialized o not exactly equivalent to NL o Analysis of NL by FL is barking up the wrong tree o Semantic Parsing knows little semantic o semantic doesn't have to be composed o at least, semantic without pragmatics is not meaning o Assuming o semantic parsing is enough o within application domains, ad-hoc process requirements Thursday, July 3, 2025 47 Meaning/Mind Formal Language Natural Language Formal Semantic Parsing Model Theory Understanding
to semantics, syntactic trees to semantics) o CCG / SCFG / HRG / AM Algebra oMapping as the probabilistic model: (2010-) o Log-linear models/hybrid trees/generative models o Agenda-based Parsing/Float Parser/Transition-based Parsers o Neural Nets oMapping with pattern templates(by intermediate repr.): (2014-) o Paraphrasing o Factored CCG/Sketches/Intermediate Grammar oAlignment is found useful again (2020-) Thursday, July 3, 2025 49 Mapping-centric Perspective
the smallest state NP NP PP NP of the capital the smallest state NP NP PP NP of the population the largest city (a) (b) (c) Training Testing capital:c population:i argmin state size $1 $1 $1 Isomorphic Semantic SQL SELECT POPULATION FROM CITY WHERE CITY_NAME = ( SELECT CAPITAL FROM STATE WHERE AREA = ( SELECT MIN( STATE1.AREA ) FROM STATE AS STATE1 ) ) ; Thursday, July 3, 2025 50 Compositional Generalization
a homomorphism between algebra oSyntactic algebra o o latent and learnt from data oSemantic algebra o M=<M,G> o by enumerating all available semantic primitives and operations oHomomorphism mapping Thursday, July 3, 2025 61 AlgeRecom: Motivations L =< L, (fγ )γ ∈ Γ >, fγ : Lk → L
independently o can be better at learning context-insensitive rules. oSemantic parses as graphs haven't been tested on systematic generalization oExisting methods (e.g. SpanBasedSP) raise complexities and rigidities against seq2seq models. oLatent Alignment o inferred with an MAP algorithm involving minimum cost bipartite matching problems with the Hungarian algorithm Thursday, July 3, 2025 66 LAGr: Motivations
oThe graph o with M = L * N nodes arranged in L layers o indicating node labels and edge labels Thursday, July 3, 2025 67 LAGr: Formalization Γa Γa = (z, ξ), z ∈ V M n , ξ ∈ V M×M e
by permuting the columns of a latent aligned graph 𝛤a . oThe permutation denoted as a o aj is the index of column in 𝛤a that becomes the j-th column of 𝛤na oThen the (intractable) model is Thursday, July 3, 2025 68 LAGr: Weakly-supervised Formalization
zxteloiv
asuna_jp
matsumoto_r
verypluming
.png){kind=avatar}
satai
datascientistsociety
signer
mickey_kubo
hkefka385
lunsophia
yumulab
hargon24
yeseniaperezcruz
thoeni
sugarenia
destraynor
maltzj
pauljervisheath
bryan
chriscoyier
csswizardry
bkeepers
eileencodes
mraible
