How to conduct research and write papers Hiroshi Kajino IBM Research - Tokyo

Self introduction Hiroshi Kajino • 2016/3 PhD degree from Univ. Tokyo • 2016/4- IBM Research – Tokyo Research interests: Those that look like useless but are actually useful • Molecular optimization • Reinforcement learning • Spiking neural network 2

Today’s talk is based on my past experience 3

Paper is a means to share research with the community Why sharing? • To inform that I am a genius • To interest others in what I am interested in • To solve the community’s issues 4

Paper is a means to share research with the community Why sharing? • To inform that I am a genius • To interest others in what I am interested in • To solve the community’s issues What does the community feel? • 🤔🤔🤔 • Interesting and will commit to the research • Glad that the issue has been solved 5

Paper is a means to share research with the community Why sharing? • To inform that I am a genius • To interest others in what I am interested in • To solve the community’s issues What does the community feel? • 🤔🤔🤔 • Interesting and will commit to the research • Glad that the issue has been solved 6 • Need to follow a protocol for sharing • Need to consider how the community feels

More aligned to the protocol, more likely the paper gets accepted (so, I recommend to learn the protocol) 7 Papers submitted to top conferences Papers that do not follow the protocol Papers that follow the protocol but do not really address what the community wants to address Papers that follow the protocol and address what the community wants to address ※Personal experience Today’s target: To learn the protocol

We have to satisfy all of the conditions of a good paper Conditions of a good paper • Clear problem setting • Problem is worth solving • Clearly identify the cause of the problem • Propose a method that addresses the cause • Validate the method addresses the problem • Paper is logically structured • Follow the standard writing practices etc… Lacking even one of them is fatal • Ambiguous problem • Problem is not well motivated • Little discussion on the cause of the problem • Cause and method are not linked • No clear message in the empirical studies • Logic is broken • Hard to read 8

Research question “Something significant” does not necessarily satisfy the readers Do: Discuss with those who have real problems • Chat at conferences • Chat with those who are not in ML community but are close to it • Discuss with clients Do: Try by oneself and find issues • Play with the math in papers • Implement existing methods to find issues Existing methods often do not work well Don’t: Think about research questions w/o any trial-and-error Don’t: Think about “something significant enough to be published as a paper” • It just considers the amount of significance, and may lead to a problem no one is interested in • This is very typical for those who find difficulty in finding research questions • You should try to find real problems by trial- and-error. 9

Related work Systematize the existing studies and clarify how your research can contribute to the system Do: Survey when thinking about research topics Most of the problems can be alleviated by existing methods 1. If encountering existing studies • Play with them to find other problems • If losing interest, think about something different 2. No related work More survey! 3. There exist relevant existing studies, but your idea can contribute to the literature Go to the next step Do: Systematize the existing studies Related work ＝ Community where you market your research ＝ Potential reviewers • Find key papers of the topic • Organize existing studies into chronological order and classify them by method. • State how your idea can contribute to the community Don’t: Find differences with existing studies only • We have to understand the community 10

Related work Systematize the existing studies and clarify how your research can contribute to the system Don’t: Random survey on relatively new papers • We have to understand the history of the community • We cannot see whether the problem is a real problem in the community, without understanding the community Don’t: Survey when writing a paper • Risk of rework 11

Solution Modify the problem so that it can be solved / Leave it for a while Do: If it is difficult to solve, modify the problem • Too abstract problem is difficult to solve Ex) Learn from fewer data • Ask other researchers for help (Solving the problem is the most important) Don’t: Stick to unsolvable problems • There are infinitely many problems to be solved but we have finite time; leave it for a while • Recall the problem sometimes and try to solve it sometimes 12

Empirical studies / Proof Validate whether the proposed method can address the problem Do: Set appropriate assumptions and theoretical statements • If assumptions are not realistic, no theory may be better than a theory with them • Doubt common assumptions used in the community Do: Design empirical studies carefully • Set the purposes of the experiments • Choose appropriate methods to be compared • Design appropriate metrics for evaluation • These must be aligned with the purposes Don’t: Follow the experiments of the existing work w/o consideration • Existing experiments are often not appropriately designed • Purposes of the experiments should be closely related to your research question Don’t: Skip necessary experiments • Reviewers always say ”more experiments” • If the experiments are well designed, we can defence it 13

Writing Logic behind the research has to be flawless / write paragraph by paragraph Do: Consider the logic of the research • Must be thoroughly logical • No inconsistency • No logical leaps • If encountering any trouble, fix it asap Do: Write paragraph by paragraph • 1 message per paragraph • Paragraphs are easy to be edit to construct a logical paper Don’t: Pause to write • Writing allows us to be aware of leaps • Short time until deadline Don’t: Not disclosing a draft to others • Only a sufficiently logical person can tell whether a text is logical or not • Ask him/her to check the logic • Many cannot write logical texts without training 14

Writing Standard structure of a paragraph Structure of a paragraph • Topic sentence: Summarize this paragraph by one sentence • Supporting details: Support the topic sentence • Closing sentence: Paraphrase the topic sentence • Closing sentence will be read after a reader understands the supporting details • Should be a little bit different from the topic sentence 15

Tips on paper writing Do: Organization first, details second • Detailed comments are not helpful initially • Focus on the structure of the paper first • Discuss about the structure with co-authors first • Given an initial draft, they may be tempted to check the details • Ask for suggestions on the structure • A well-structured paper is easy to be presented even in a few minutes Do: Revise, revise, revise! Five to ten times Don’t: Just enumerate papers in related work • Related work is a section to discuss how your work contributes to the existing literature • Differences with existing studies don’t explain the literature • Existing studies have to be organized • I put much effort on writing related work 16

Communication with reviewers When reviews are out, one often says… • Random reviews 😡 • Reviewers don’t understand my reseach 😭 • Off the point 🤯 • Review is too short 🤬 Are these reviews attributed to reviewers only? There is a room for improvement! Reviewers are… • Randomly chosen from the community • Sometimes experts on the topic, the other times knowledgable on the topic • Some like math, the others dislike math • Some can spend much time for reviewing, the others cannot We should remind these when writing a paper 17

Communication with reviewers Do: Say thank you to reviewers • Reviewers are sampled from the researchers, for whom the paper is written • Reviews are always informative • Since they spend their time, we should appreciate it • A short review may indicate that the paper is not interesting and/or the title and the abstract fail to summarize the paper, which leads to bad matching between the paper and reviewers • A misunderstood review may indicate that your writing can be improved Do: Report any issue to SPC/AC • Misunderstanding is attributed to writing, which shold be improved • However, we have to correct the misunderstanding and let the reviewer understand correctly • If not corrected, report to SPC/AC that the reviewer is misunderstanding Don’t: Curse the reviews Reviews are always important to improve your paper 18

Summary Introduce my personal philosophy on research • This philosophy is suited to those who want to address others’ problems • This may not fit the others There are other philosophies • One may pursue his/her own interest • One may wish to increase his/her own value Anything will be ok as long as it is consistent Always try to improve the way of research • Purpose of writing a paper can change gradually • I am still trying to improve my way of research 19

Panels Q. Good research idea vs. bad research idea • A necessary condition for a good research idea is that you are enthusiastic about it • A good research idea is targeted at a specific “client” • Client can be myself • “I get excited about the research” • Have to clarify why you get excited about it Q. Tech-oriented vs. objective-oriented research • Objective-oriented research is straightforward • Tech-oriented one is ok if you can interest the readers • Tech-oriented research is popular in some research areas • Clarify why it is worth investigating • Tech-oriented research should have some objective 20

Panels Q. Tips on schedule management • Initial draft should be completed 2 weeks before the deadline • Start writing a paper 1 month before DL • Write down the schedule plan • Initial plan can be just a guess • Update the plan continuously • Understand one’s own capacity • Work hard 💪 💪 💪 Q. Tips on mentoring • Discuss overview first, details second • Consideration • Provide the context and reasons behind suggestions • Do not pretend to know • Recall mentee’s work before meetings • Decompose a complex task into simpler ones 21