Showing posts with label conference. Show all posts
Showing posts with label conference. Show all posts

How to write a high quality review?


As a researcher, either you are pursuing a PhD. or an experienced researcher, we need to review conference or journal papers for the research community time to time. For example, your supervisor might assign you as a sub-reviewer to review paper(s) of your interests/expertise, or you might get invited more and more as you grow as an independent researcher in your field after your PhD.

Therefore, as a researcher in your research community, and since the review process pushes the research community moving forward, it is our responsibility to provide quality, constructive and not offensive reviews. Despite the importance of providing high quality reviews, unfortunately, we normally are not trained on this task during the PhD or afterwards.

Recently, professor Noah Smith from University of Washington gave a podcast which provides some fantastic advice on how to review a conference paper for the research community. Below, I summarize the main take-home messages from this podcast, as well as advice from other resources which might be helpful to reseachers who would like to write a high quality reviews.

Content

A review has three important parts:
  1. A brief summary of the paper
  2. Major pros and cons
  3. Minor things/comments/corrections
Let's discuss each part in the following in detail.

Brief Summary of the Paper

The first section provides a brief summary of the paper for others, e.g., an area chair who is extremely busy. This sectionn should:
  • say as it is, (what's new and contributions claimed by authors etc.)
  • that is, not judgement for this section
  • make it easy for others to get the idea of the paper quickly

Major Pros & Cons

Then we can move on to the pros & cons. Remember to give pros! of the paper. Even you do not like the paper, we should always analyze pros of a paper just as for finding cons of it.
  • particularly, we need to encourage (early) researchers keep going
  • there should be pros of a paper, e.g., the question itself is good and challenging

Even when we write cons of a paper, we still need to be constructive and positive. 
  • always give actionable details
  • e.g., instead of the method is confusing, talk about what equations or details are confusing

This section is main part of the review and needs to take into account of many key questions. 

For example, we can consider some of the questions below which include the ones from Professor Emery Berger (Computer Science at the University of Massachusetts Amherst):

Problem
  • Is the paper well-motivated? 
  • What problem does it address, and is it an important problem?
Quality
  • Is the submission technically sound? 
  • Are claims well supported by theoretical analysis or experimental results? Does the paper credibly support its claimed contributions? 
  • Is this a complete piece of work or work in progress? 
  • Does the paper describe something that has actually been implemented? If so, has it been evaluated properly? Is it publicly available so that these results can be verified? Are the results in the paper able to be reproduced easily?
  • Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
Clarity
  • Is the submission clearly written? Is the paper sufficiently clear that most venue attendees will be able to read and understand it? 
  • Is it well organized? (If not, please make constructive suggestions for improving its clarity.) 
  • Does it adequately inform the reader? (Note: a superbly written paper provides enough information for an expert reader to reproduce its results.)
  • What is the intuition behind certain choices?
Originality
  • What are the paper’s key insights? 
  • What are the paper’s key scientific and technical contributions? 
  • What did you learn from the paper? 
  • Does the paper significantly advance the state of the art or break new ground? 
  • Are the tasks or methods new? 
  • Is the work a novel combination of well-known techniques? 
  • Is it clear how this work differs from previous contributions? 
  • Is related work adequately cited? Does the paper clearly establish its context with respect to prior work? Does it discuss prior work accurately and completely? Are comparisons with previous work clear and explicit? 
  • Incremental approach could be challenged to accept, check what contribution is left after removing others.
Significance
  • Are the results important? 
  • Are others (researchers or practitioners) likely to use the ideas or build on them? 
  • Does the submission address a difficult task in a better way than previous work? 
  • Does it advance the state of the art in a demonstrable way? 
  • Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
  • What impact is this paper likely to have (on theory & practice)? Is the work of broad appeal and interest to the research community?




Minor Things/Comments/Corrections

This section can provide some other minor issues, such as grammer errors. Please note that these minor things are not enough to reject a paper though.




About Reviewing Process

Prof. Noah Smith also discussed about his personal process of reviewing a paper, which is useful to me to adopt for future reviews.
  • quick scan (introduction, figures, conclusions, reference list)
    • where it is positioned in the literature?
    • what kind of paper it is? (position paper? theoretical one? system one?)
  • top-down reading with red pen
  • come back later with my review notes
  • any unclear things can be asked as a reviewer
    • e.g., the explaination/notation/prove is confusing or did the preprocessing also applied to baselines etc.
    • note that it's not criticizing, it is just want to encourage the paper make clear on the next version


Summary

To sum up, in this post we discussed the structure of review, and the review process and a couple of things can be considered or checked while reviewing a paper. 

Finally, we should keep in mind that the reviews that we write should
  • help the authors to improve the paper 
  • be what we expect to see in the next version of the paper

Did you have experience or any thought or advice which might be helpful for writing a high quality review? Leave a comment to share it with others:)


Related resources

No comments:
Labels: , , , , , ,

CIKM2016 Travel Report

I attended CIKM2016, Indianapolis, USA from 23-28th October. This is the first time I attended a IR conference. It has several hundreds of attendees and it was surprising to see so many Chinese attendees from China and USA.



The conference has almost 1000 submissions (so huge number of submissions...), and both full and short tracks have around 23% acceptance rate.


The word cloud reflects hot keywords in the accepted papers, which includes deep learning and search etc.

Keynotes:

There were three keynote speakers from big companies such as MS, Google.

1. Toward Data-Driven Education
Rakesh Agrawal (Data Insights Laboratories)

2. Personalized Search: Potential and Pitfalls
Susan Dumais (Microsoft Research)

3. A Personal Perspective and Retrospective on Web Search Technology
Andrei Broder (Google Research)


The second keynote was interesting for me as the keynote speaker talked about personalization in the context of search, and mentioned the User Modeling(UM) aspect. 

Susan talked two types of UM (or user profiles), one is local profile which can be stored in PC, and the other one is cloud profile. The local profile is good considering user privacy as the profile is in the local PC for personalization, i.e., the ranking results will be personalized based on the local profile of a user. However, it suffers from in efficiency, e.g., due to the lack of portability, it will be hard to reuse it if the user changes the working environment  (e.g., change PC). Personal score was carried out by content matching as well as features based on interaction history. 

She also talked about evaluation of personalization alternatives, offline and online ones. As we can expect, the former one is safe to exploit many different alternatives. On the other hand, the later one (e.g., A/B testing) has more accurate evaluation, and with some challenges. Explicit feedback from users (e.g., asking users about the personalization is good or not) can be a good indicator, however, it also might change the user search behavior. On the other hand, implicit feedback could be noisy.

Another interesting point is personalization can also provide interesting items (serendipity)...

Tutorials

There were eight tutorials and I chose the tutorial: "Data-Driven Behavioral Analytics: Observations, Representations and Models", which was given by Dr. Meng Jiang and Dr. Jiawei Han

 The tutorial is about human behavior analytics, which is one of the six disruptive research areas defined by Department of Defense. They introduced many models incorporating different factors of social network information into traditional #RecSys approaches such as Matrix Factorization (MF). 

Sessions

I also attended #RecSys session. As this conference is about IR, there were many models introduced by different problems, and MF seems like the dominated one. Surprisingly, none of the first authors came to present those papers. 

Maybe due to the venue?, there were many people didn't come either for presenting or taking CIKM cup awards...:). It was a good experience to attend the first IR conference for me, and the next CIKM(2017) will be at Singapore.

No comments:
Labels: , , , , , ,

SEMANTiCS 2016 Travel Report

Day-1: Tutorials & Workshops

I attended the afternoon session about Knowledge Engineering track using PoolParty from Semantic Web Company. I'm interested in how those Semantic Technologies being used in different enterprises, and what kind of solutions they need for soloving what kinds of problems. There were many industrial participants in Europe including Springer etc. As a researcher working closely on Semantic Technologies,  Some told they are already using PoolParty and some were attending for better understanding of using Semantic Technologies in Enterprise scenarios, and most of the cases were wondering about integration of heterogeneous data sources, taxonomies and ontologies.



Day-2: Main conference

Stats: This year's conference received 85 submissions with 18 full papers(21.2%) and 8 short papers.

The first keynote: "Linked data experience at Springer Nature" by Michele Pasin

Dr. Michele talked about a summary of Springer's experience with Linked Data & Semantic Technologies for enterprise metadata management at large scale. He also introduced scigraph.com - a upcoming LD platform: one place for their all linked data efforts towards liked science data.





The second keynote: "The semantics of human network" by Marie Wallace, IBM

 Marie from IBM shared their experience of using human network which generated by their enterprise social networks using IBM connections for different applications and services. She stressed that capturing human context at a global level, which is happening thanks to the social networks and IoT enabled world, is really important to help human digital experience.



These social dashboards for each employee shows different factors such as activity, reaction etc. of your personal social status and can also provide some recommendations for your improvements in different aspects.

I presented my full paper: "Exploring Dynamics and Semantics of User Interests for User Modeling on Twitter for Link Recommendations" in the Knowledge Discovery session. It is impressive to see the room was full of audiences and had interesting discussion with some audiences. This work also won the best paper award at #semanticsconf.





Day-3: Main conference

 The first keynote: "Learning with Memory Embeddings and its Application in the Digitalization of Healthcare" by Volker Tresp from SIEMENS

 He talked about mapping of the knowledge graph to a tensor representation whose entries are predicted by models using latent representations of generalized entities, and extension of this approach for medical decision processes.





The second keynote: "Enriching Content with User Data and Semantic Information" by Cathy Dolbear from Oxford Press

She talked about combining human-authored semantic information with semantic tags and taxonomy classifications automatically extracted from our content. She also introduce the Oxford Global Languages project, which links lexical information from multiple global and also digitally under-represented  languages such as isiZulu and Urdu in a triple store.

It was a wonderful event which can meet industry people who are dealing with real-world problems with Semantic Technologies, as well as academic researchers. Hope to attend the conference again in the future:)


No comments:
Labels: , , , , , , , ,

UMAP2016 Travel Report

This week, I attended the 24th Conference on User Modeling, Adaptation and Personalization (UMAP 2016). This year, it held in conjunction with Hypertext Conference sharing some sessions (e.g., Doctoral Consortium, Keynote speakers). Overall, there were around 130 participants for the conference. This year, the conference received 123 submissions with a 28 % acceptance rate. 


A major change in this year was the presentation format. Different from previous years, we present 13 mins (long), 8 mins (short) with a poster session to receive more audiences and discussions.
  • Keynote Speakers:

The first keynote speaker was Hossein Derakhshan: Killing the Hyperlink, Killing the Web: the Shift from Library-Internet to Television-Internet.


The speaker is an Iranian-Canadian blogger who was imprisoned in Tehran from November 2008 to November 2014. He is credited with starting the blogging revolution in Iran and is called the father of Persian blogging by many journalists.



 Some impressive phrases during the speech:

 - Many internet users in Brazil and India think Facebook is the Internet
- With 150 "likes", Facebook can know better about you than your parents, with 300 likes, the service can know better you than your spouse

 The second speaker Lada Adamic, who is leading the Product Science group within Facebook's Data Science Team.

The speaker described three large-scale analyses of re-share cascades on Facebook, which were performed in aggregate using de-identified data.






Summaries of the speech:



- Cascades grow

- Cascades recur

- Cascades evolve


The third speaker Sandra Carberry, who is one of the founders of the User Modeling research area at the first woskshop in Maria Laach, 1986, gave a talk on "User Modeling: the Past, the Present and the Future".

--------------------------------------------------------------------------------------------------------------------------

 I was there to present a short paper, a doctoral consortium paper and an extended abstract.
  • Short Paper



  • Doctoral Consortium 

 In the Doctoral Consortium, each student was assigned an expert in your topic. Tsvika Kuflik, who is on the editorial board of UMUAI, was my mentor during the conference and offered many constructive feedbacks about my thesis. 

  •  Extended Abstract
This preliminary work describes a first step of user modeling using different fields of LinkedIn profiles to investigate which field of LinkedIn profiles can be helpful for user modeling in the context of MOOC recommendations.



 Many audiences asked about data collection. We used Google Customized Search Engine to search the LinkedIn website using a specific keyword like "coursera" to filter out LinkedIn profiles containing Coursera courses. For the details about the dataset, you can check the post here.

 -----------------------------------------------------------------------------------------------------------------------

Impressively, the proceedings of UMAP 2016 have been available during the conference.


No comments:
Labels: , , ,

ACM SAC 2016 Travel Report

From the 4th to the 8th of April I had the pleasure to participate the 31st ACM Symposium on Applied Computing (ACM SAC), which was held in beautiful city Pisa, Italy. I was there to present my full paper "Measuring Semantic Distance for Linked Open Data-enabled Recommender Systems" and to participate the Student Research Competition sponsored by Microsoft.

This year, there were over 500+ registrations from 59 countries at this conference. There were 37 tracks and the overall acceptance rate for this year is 24%.

Keynotes:

There were two keynotes given by John Mylopoulos and Marco Conti, respectively. The first keynote is about the requirements problem in Software Engineering and the second keynote is about "From MANET to people-centric computing and communications.


Semantic Web Track:

There were two sessions with eight papers for Semantic Web Track where three of the participants two of the participants from our institute. Pasquale Minervini presented "Leveraging the Schema in Latent Factor Models for Knowledge Graph Completion" and another college Feng Gao presented "QoS-Aware Adaptation for Complex Event Service" in another (SOA) track.


Social Network and Media Analysis Track (SONAMA):

One of the papers in this track I'm interested in was "Inferring Semantic Interest Profiles from Twitter Followees: Does Twitter Know Better than Your Friends?" from Christoph Besel, University of Passau, Germany, which is related to my work. Although many previous works focused on using tweets for inferring user interest profiles, they used the alternative source (followees) to retrieve user interest profiles, which are based on the tendency that more and more users are consuming feeds instead of producing content on the social networks.

Student Research Competition (SRC):

I also participated in SAC SRC and went through 2nd round (top-5 list) and it was a good opportunity to compete across different disciplines. Congrats to all top-3 winners! 

Lunch

Banquet

What would make the conference better?

It would be better to have a Twitter channel to communicate and disseminate activities during the conference. Next year, it will be in Morocco and hope I could attend again:).

---------------------------------------------------------------------------------------------------------
Update after the conference:

 The proceedings are available from June, 2016

No comments:
Labels: , , , , , , , , ,

Review of 2015

Research

The year 2015 is finished and I've been in Ireland and Insight for 1 year and 6 months. Doing research here gives me quite different experience than previous research experience, and it poses good opportunities as well as challenges for myself.

Independent:  

You have to grow up and be able to do your research (not projects) with your ideas and opinions, and conduct experiments by your own. I remembered the seminar I participated at the beginning of the PhD journey and the speaker described our academic supervisor as advisor since it is more appropriate. That means our advisor is who giving advices for your research but not who telling you every step you should move forward, and usually our supervisors also too busy to do so. 

 At first, I could not start own research and conduct experiment by myself, and there was always uncertainty about myself and I realized the way I've been trained always was "supervised" by others. It reminds me the time in South Korea when I was a master student as well as an employee in a company where I received a lot of things to-do every day from senior members. In contrast, I did not receive any call here, and all communication has been done through emails which is still a surprising fact for me. Thanks to God, even I have a lot of improvements to achieve, I have started the research, with advice from my supervisor.

 I started to recognize the statements from (So you want to do a PhD from Open University) that a PhD is confirming your "research independence", i.e., you have to demonstrate that:

  1. Ability to do research by yourself, rather than simply doing what your supervisor tells you
  2. Awareness of where your work fits in relation to the discipline, and what it contributes to the discipline
  3. Mature overview of the discipline

Insight Centre: 


There have been many changes for Insight@Galway which was formally well known as DERI. Our former director Prof. Steffan Decker moved to Germany and we have new director Prof. Dietrich Rebholz-Schuhmann. Interestingly, many researchers, including PhD students, Postdocs moved to Germany as well. There are many career paths for graduates from here including academic positions as well as industry ones, or even some of them start running own startups etc.


Conference

After several attempts for conferences, I've published two full papers in JIST2015 and SAC2016, and I found that it is really important to publish or try to publish your results in any conference or journal to get started, and get feedbacks from the experts. In my previous experience, I've been recommended do not to read and present a conference paper for a seminar during previous studies. However, here, one thing I love is that top conferences have the same importance to top journals. There is an interesting article to read if you have the same wondering: https://homes.cs.washington.edu/~mernst/advice/conferences-vs-journals.html
At the end of the year, I submitted a paper to ESWC2016 which has very interested tutorials for me http://2016.eswc-conferences.org/program/workshops-tutorials and hope I will have an opportunity to attend it:). Another conference I'd like to participate is UMAP2016 which is also highly related to my research. So... Fingers crossed for the upcoming new year.


No comments:
Labels: , , , , , , , , , , , , ,

JIST2015 Travel Report

This is the first travel report for my first participation at a conference for presenting my paper. The Joint International Semantic Technology Conference is a merged conference of ASWC (Asian Semantic Web Conference) and CSWC (Chinese Semantic Web Conference). This year, the conference consists of main conference as well as an industrial forum and a data challenge. The majority of the participants come from China and Japan and the committees expects more participants from the USA and Europe. Overall, there were around 100 participants for the conference and the industrial forum.

 Keynote speakers:
There were two keynote speakers and many invited talks during the conference and the industrial forum. Speakers include Prof. Frank van Harmelen, Hong-gee Kim from SNU Korea, Ming Zhou from Microsoft Asia, Prof. Jeff Pan and many industrial participants from giant companies such as Baidu, Alibaba, China Mobile etc.

 Prof. Frank van Harmelen talked about the research efforts on Semantic Web till now and the near future and several projects that they are currently working on for understanding the Semantic Web.
  • 2000-2010: how to build it
  • 2010-2015: how to use it
  • next decades: how to understand it 

To this end, he talked about How to build a WoD observatory and their solution named - LOD Laundromat: clean your dirty triples: crawl from registries (CKAN), which consists of 38 billion triples and counting!

Look back the research papers on Semantic Web community, there were a majority of papers optimized in DBpedia last 5 years. This is an especially bad idea since Semantic Web!= DBpedia... They try to use some analytical tools on top of the LOD Laundromat (LODLab, LOTUS) re-run experiments on larger datasets (100+) which was collected by the WoD observatory.

Even at an early stage, there are some interesting findings such as "Hotspots: Most of the dataset is not used in many queries (in terms of datasets)", if we can identify it, we can optimize for these queries and local sub-graph works well for answering queries (no explanation yet - theory)

So the agenda should be changed: not building things, but why those things work so well!


Industry forum: 
It was very impressive that there were around 10 companies participated in the forum to present their work on using Semantic Technologies in their companies. The most impressive talk for me was given by Jie Bao from Memect Inc., who was working at RPI and MIT before. He pointed out the difference of research in Europe and the USA:
  • based on project vs. business
  • conference: ISWC vs. semantics
He stressed that even Semantic Web community is often talking about schema.org for a good example of SW, it is a project and not a business. There are little start ups survived at present even we expected a new semantic search engine, semantic agent etc..

 And other slides on start-up companies with Semantic Web...


















Data Challenge:
I think this session is one of the most beneficial ones for me. I personally participated the data challenge on entity type prediction and placed 4th among 13 teams. Other top-5 participants came from Peking University, Chinese Academy of Sciences, Fujitsu R&D and Dongnan University. Learnt a lot about NLP and Machine Learning and I need to improve NLP skills and ML skills in the near future.

JIST2015 data challenge from GUANGYUAN PIAO




Editors' session:
Since my paper "Computing the Semantic Similarity of Resources in DBpedia for Recommendation Purposes" for the main conference was one of the Best Paper Candidates, I had the chance to meet journal editors to get some feedbacks about the work. I had many useful comments from them and had a lot of work to do to improve/strengthen it.



It was a good experience to attend the conference and the JIST2016 will be at Singapore:)

 ---------------------------------------------------------------------------------------------------------
Update after the conference:

 The proceedings are available from March, 2016. This is a very slow publication of post proceeding compared to other conferences such as UMAP. The proceeding of UMAP was available during the conference while JIST2015 proceeding had not been available until March, 2016. Both were published by Springer, what's the difference??

 Previous JIST conferences have external proceedings for posters and workshops in CEUR but not this time. May be it depends on the organizing committees and it is better to check it before you submit any workshop or poster papers in a conference.

 Although the conference has a line up of great speakers and programs, it would be better to disseminate the activities of the conference in global channel like Twitter (instead of using the localized social network) as it is a "international" conference. 





No comments:
Labels: , , , , ,
Subscribe to: Posts (Atom)