Showing posts with label Research. Show all posts
Showing posts with label Research. Show all posts

Huawei Data Science Interview Questions


This post is about the interview experience for a data scientist position at Huawei IRC (Irish Research Centre). As part of Huawei Europe, IRC mainly focuses on other markets except Asian ones such as China, South Korea, and Japan, and is trying to solve business problems such as Ad optimization on those markets.

Regarding the job post, an agency contacted me regarding the position and I only went through one hiring manager round and stopped. So this post is mainly about the interview questions with the hiring manager (HM). 


The HM round is a 1 HR interview consists of the following structure.

  • 5 min: introduction of each
  • 10 min: presentation about your previous experience
  • 15 min: Q&A regarding the specifications about the job post
  • 25 min: code pairing (Leetcode-like, messed up this)
  • 5 min: ask any questions regarding the role


Some questions mentioned during the interview are:
  • Some questions related to the presentation.
  • What is your contribution/role in a specific project? I think it is mainly trying to figure out your part exactly if there are many team members or authors in a paper.
  • What is the current state-of-the-art for the problem?
  • What are some impressive papers you have read recently regarding the specific area of the job post?
  • Regarding the project of job post related questions, e.g., if you do xxx., how will you approach or provide solutions


Overall, the interview process is well organized, and the interviewer is kind despite of the messing up for the code paring section. 



No comments:
Labels: , , , , , , , ,

Generating stunning images arts paintings with your creativities using Artificial Intelligent (Stable Diffusion, DALL E)

The recent AI news, which catches social media attention is absolutely those AI (Artificial Intelligent) models generating stunning images based on your given text FOR FREE. And as soon as those models open to public, creative individuals have been experimenting with the different text input to generate creative art images, paintings, and all other different sorts of interesting images.


 



One such kind of model is Stable Diffusion, which has been made to be freely accessible to the public on this August. The web application can be accessed on Hugging Face. 

Another popular model is DALL*E from OpenAI, which also has been made available to everyone on this September recently without to be added and queued in a waiting list. Once you sign up, you can immediately get 50 credits (1 credit can generate 1 image based on your given text), and every month you can get 15 credits for now.






In the following, I also share some images generated while I'm playing with the Stable Diffusion and DALL E models. First, I got the text from the Stable Diffusion application page which is "A pikachu fine dining with a view to the Eiffel Tower" and generated images using those two different AI models. Personally and my Kido also love the first ones generated by the DALL E model.

Text: A pikachu fine dining with a view to the Eiffel Tower

DALL E results


Stable Diffusion results


Next, we can look at the images generated with a text of "Wildflowers, grassy field, autumn rhythm, watercolor" from the DALL E website, again, using those two models mentioned above. Those generated images clearly look different based on the same text input.


Text: Wildflowers, grassy field, autumn rhythm, watercolor

DALL E results

Stable Diffusion results


It is interesting to see the creativity and new images that can be generated by those models and their potential for public usage. And there are many debates and worries going on regarding those AI generated arts. It would be interesting to see how those creative tools will impact our lives in the near future with more interesting and creative applications. 


Finally, close up with some more images while I'm playing with my Kido using those AI models.

DALL E: A cute pokemon eletric type 

A pikachu with armour detaild 4k, 8k high resolution

Stable Diffusion: A cute pokemon eletric type 

DALL E: A cute girl with big eyes and long hair, detailed, realistic, 4k, 8k, HD, high resolution



"A cute girl with big eyes and long hair, detailed, realistic, 4k, 8k, HD, high resolution" from the Stable Diffusion.





No comments:
Labels: , , , , , , ,

Yahoo Research Scientiest Interview Questions



The interview process is structured as follows:

  • HR screening interview (30 min)
  • Manager/Priciple Scientist screening interview (1 hr)
  • Research Talk/Presentation (1 hr)
  • Technical Interview (1 hr) x 4 panels

HR screening interview is straighforward. Didn't ask those behavior questions, instead, it was more focused on describing the role and expectations and skills required, and your status for joining if succeed, e.g., visa status, when to start etc.

Manager or principle (in my case) screening interview lasted for 1 hour. Basically technical questions and research related questions based on your resume. Questions might related to their projects ongoing or related to your projects or papers before. For example, in my case,
  • what activation fuctions you use for multilabel classification?
  • what are the evaluation metrics you have used in your research paper? what is the difference between mAP and nDCG (when to use which?)

Then they will invite you for a rearch talk - 1 hr session - with a group of audience in the large research team in addition to your interview panels (4-5 people). The talk topic can be anything such as your thesis topic or about your recent papers.

Afterwards, there will be 4 rounds of technical interviews (1 hr for each) with interview panel members. Each member will ask technical questions based on which research area your position belonging to. In case of ML, questions can be quite vary such as:
  • what's the statistical assumption behind logistic regresstion?
  • what's the relationship between dropout and l2 regularization?
  • why we don't initialize weights of parameters as zeros?
  • what optimizers have you used and describe those
In addition to those technical questions, the interview session also includes open questions regarding their previous or ongoing projects, such as:
  • how to design entity recommendation system from start to end?
  • how to design a system to recommend trending topics? how to extract trending terms?


Overall, the interview process is well arranged with the HR and the feedback loop is quick fast, and the panel members were quite nice and friendly. 

No comments:
Labels: , , , , , ,

Recommender Systems: Online Business Metrics

Image by Freepik

 

Content

  • What is CTR?
  • What is CVR?
  • What is CPM and eCPM?
  • What is RPM?
  • What is GMV?


CTR (Click Through Rate)

CTR indicates the ratio showing how often people who see your ad or free product listing end up clicking it. Click-through rate (CTR) can be used to gauge how well your keywords and ads, and free listings, are performing.

CTR is the number of clicks that your ad receives divided by the number of times your ad is shown: clicks ÷ impressions = CTR. For example, if you had 5 clicks and 100 impressions, then your CTR would be 5%.

CTP (Click Through Probability)

CTP sometimes used as CTR measured as below:
# of unique visitors who click/# of unique visitors to page


CVR (Conversion Rate)

CVR, or conversion rate, in in-app advertising is the percentage of users who saw an app-install ad, clicked on it, and converted through some pre-specified action. CVR tells app advertisers how many users their ad converted.

Different from the CTR task, the samples available for this task is sparse and usually around 1% of the samples in the CTR task.

CPM (Cost Per Thousand/Mile) and eCPM

Cost per thousand (CPM), also called cost per mille, is a marketing term used to denote the price of 1,000 advertisement impressions on one web page. If a website publisher charges \$2.00 CPM, that means an advertiser must pay \$2.00 for every 1,000 impressions of its ad.

On the other hand, eCPM is concerned with the revenue generated by a thousand impressions. In marketing, CPM is crucial since it helps advertisers determine which platforms are viable for publishing ads. However, the effective cost per thousand ad impressions indicates the publisher’s ad revenue.

RPM (Revenue Per Mile)

Revenue per mille (RPM) is the estimated earnings that accrue for every 1000 impressions received (in Latin, mille means thousand), a commonly used measurement in radio, television, newspaper, magazine, out-of-home, and online advertising.
No comments:
Labels: , , , , , , , ,

Wikidata problems to be soloved

Wikidata is a collaboratively edited multilingual knowledge graph hosted by the Wikimedia Foundation, which is founded in late 2012, 8 years ago. At the time of writing, Wikidata currently contains 93,647,881 items. 1,422,596,352 edits have been made since the project launch.


Wikidata recenly has growing interests with more dedicated events such as Wikidata Workshop @ISWC or Wiki Workshop @WWW. A recent systematic literature review by Marçal Mora-Cantallops et al. - "A systematic literature review on Wikidata" - also summarizes the research efforts in the context of Wikidata nicely.


Keynote by Lydia Pintscher at the Wikidata Workshop 2020, co-located with ISWC 2020


Lydia Pintscher who is a product manager of Wikidata has pointed out critical challenges despite of promising progress over the last 10 years at the Wikidata Workshop 2020, co-located with ISWC 2020, inlcuding the following points:

Reliability and growth

Q: How can we make Wikidata's data openly available for querying at scale?

Dream: Free and open graph DB with fast updates and response times for a lot of data


Data quality

Q: How can we measure the quality of our datamore accurately?

Looking at factors such as accuracy, objectivity, completeness

Dream: Reliable automated quality assessment at scale


Q: How can we reliably find and address issues in the data we have?

Vandalism and misinformation, outdated data, ontology problems

Dream: Automated identification and verification tools; 

Tools for finding and suggesting fixes for ontology issues


Language and Culture coverage

Q: Where do we stand?

Dream: Privacy-respecting way of understanding the make-up of the editor community; 

Clarity on conflicts and toll usage around knowledge discovery

Good understanding and communication around the gaps and biases in the data

No comments:
Labels: , , , , , , , , ,

MIT-Harvard CINCS / Hamilton Institute Seminar: Inclusive Search and Recommendations from Pinterest by Dr Nadia Fawaz

From https://medium.com/pinterest-engineering/powering-inclusive-search-recommendations-with-our-new-visual-skin-tone-model-1d3ba6eeffc7


The topic of this week's MIT-Harvard CINCS  / Hamilton Institute Seminar was about "Inclusive Search and Recommendations" from Dr Nadia Fawaz, Pinterest.


Nadia Fawaz is a research scientiest and tech lead at Pinterest, and she gave an interesting talk on inclusive search and recommendations with interesting examples of this line of efforts at Pinterest when they are building their ML-based search and recommendation systems.


Why this problem is important? It has been clear that in addition to the benefits of ML systems for a wide range of domains, there are also some critical problems have been noticed such as our learned language models could be biased (e.g., "Man is to Doctor as Woman is to Nurse"). This is mainly due to the training data we collected and used to train a ML system is biased. Without considering those biases, ML loop will enhance the bias instead of eliminating it. Nadia Fawaz mentioned during her talk that bias mainly comes from demographic features such as age, skintone, gender, etc., and the talk is focused on Pinterest's efforts to build inclusive search and recommendations with skintone as an example.


Example of Pinterest inclusive AI solution for skintone [medium article]. Motivated by a top request from Pinners where they want to feel represented in the product, they built the first version of skin tone ranges, an inclusive search feature, in 2018. This aims to provide more inclusive inspirations to be recommended in search as well as allow Pinners to choose ranges for recommendations/search results. Some important aspects in terms of building inclusive service are such as:

  • data balanced across a wide range of groups (e.g., a wide range of skin tones)
  • error analysis should be tailed down to each group (so that the system does not perform well on only some specific groups while performing poor on other groups)
  • improve fairness and reduce potential bias in other ML models (e.g., incorporating fairness into objective functions)
  • also as one might expect, to achieve the goal of inclusive ML services, multidisciplinary efforts and a lot of labeling works required from domain experts.

Interestingly, at the time of writing this post, there is a comprehensive survey on "Fairness on Ranking" available on Arxiv which is highly relevant to the same topic discussed in this post.


No comments:
Labels: , , , , , , , ,

User Modeling

 



What is User Modeling?

User modeling is the subdivision of human–computer interaction (HCI) which describes the process of building up and modifying a conceptual understanding of the user. The main goal of user modeling is customization and adaptation of systems to the user's specific needs. The system needs to "say the 'right' thing at the 'right' time in the 'right' way [1].

A user model is a (data) structure that is used to capture certain characteristics about an individual user, and a user profile is the actual representation in a given user model. The process of obtaining the user profile is called user modeling [2].


What are some research resources as a starting point?

On top of the definitions regarding user modeling, depending on what kind of characteristics we are focusing on,  there has been many research going on in different domains such as social media and e-learning. 

In the following, we list some of the tutorials and surveys that might be a good starting point based on different kinds of characteristics and domains you might be interested in. If you find other surveys or tutorials that are interesting and important but missing from the list, you can leave a comment on that.

Surveys      


Tutorials


References

  1. Fischer, Gerhard (2001), "User Modeling in Human-Computer Interaction": Fischer, Gerhard (2001), "User Modeling in Human-Computer Interaction", User Modeling and User-Adapted Interaction 11: 65–86. doi:10.1023/A:1011145532042 ↩︎
  2. Piao, Guangyuan; Breslin, John G. (2018). "Inferring User Interests in Microblogging Social Networks: A Survey". User Modeling and User-Adapted Interaction (UMUAI): 55. arXiv:1712.07691. doi:10.1007/s11257-018-9207-8↩︎
No comments:
Labels: , , , ,

Differences between Knolwedge Base, Knowledge Graphs, and Ontology



The terms "knowledge base" and "knowledge graphs" have gained a lot of popularity recent years, especially after Google's introduction about the Google Knowledge Graph. However, those terms have been used interchangeably and there has been lacking a good definition for distinguishing those terms. 

Recently, the SEMANTiCS paper Towards a Definition of Knowledge Graphs from Ehrlinger, Lisa [2] provides a quite comprehensive analysis on those terms and a good definition about knowledge graph to define the distinction and relationships between those terms, which are quite useful.


"The knowledge base is a dataset with formal semantics that can contain different kinds of knowledge, for example, rules, facts, axioms, definitions, statements, and primitives" [1]



 "A knowledge graph acquires and integrates information into an ontology (or knowledge base) and applies a reasoner to derive new knowledge."


More recently,   Auer, Sören et al. relaxed the definition a little bit by any method instead of a reasoner when deriving new knowledge. That is, 


 "A knowledge graph acquires and integrates information into an ontology (or knowledge base) and applies a reasoner or other computaitonal methods to derive new knowledge."


This definition aligns with the assumption that a knowledge graph is somehow superior and more complex than a knowledge base (e.g., an ontology) because it applies a reasoning engine to generate new knowledge and integrates one or more information sources. Consequently, a manually created knowledge graph that does not support integration aspects is a plain knowledge base or knowledge-based system if it provides reasoning capabilities. 

It is also interesting to note for me that an ontology consists not only of classes and properties (e.g., owl:ObjectProperty and owl:DatatypeProperty), but can also hold instances (i.e., the population of the ontology). 


[1]. J. Davies, R. Studer, and P. Warren. Semantic Web Technologies: Trends and Research in Ontology-based Systems. John Wiley & Sons, 2006.

[2]. Ehrlinger, Lisa and Wolfram Wöß. Towards a Definition of Knowledge Graphs. SEMANTiCS conference, 2016.

[3]. Auer, Sören et al. Towards a knowledge graph for science, International Conference on Web Intelligence, Mining and Semantics, 2018.

No comments:
Labels: , , , , ,

Industry vs Academic Research

No comments:
Labels: , , , ,

Machine Learning for Communication Systems and Networks Summer School Report

From 1st to 3rd September, I attended to the "Machine Learning (ML) for Communication Systems and Networks" summer school, which is hosted by CONNECT research centre in Trinity. The summer school focuses on research topics, ethics, and innovation perspectives on use cases of ML for communication systems and networks. It is really good opportunity to learn about communication systems use cases given I'm more familiar with ML after my PhD. One of the best things about this summer school for me is the mixture of speakers from different backgrounds such as from academia, startups, journalists, etc.

This post provides an overview and summary of the summer school, which might be of interests for those who have missed the summer school or who will be interested in participating the summer school in the future. An overview of the program is in the following figure.



Day-1


The first day consists of 4 lectures, and started from Irene Macaluso (CONNECT) to give a nice overview of different ML approaches such as unsupervised learning, supervised learning, and reinforcement learning, and some research topics related to applying these approaches in the context of communication systems and networks. An interesting takeaway as a person very new to the research area is that the real-world telecommunication data is difficult to gather and to be used for ML/deep learning (DL) approaches. As a result, many research has been done by using simulation data.

The second lecture is more about the topics related to ethics when we are conducting ML research in general. The speakers introduced the Artificial Intelligence and ML Ethics Toolkit v.01, which is a set of questions should be assessed when doing your ML research with data related to your research.

The third lecture is about traffic analysis using ML/DL from Paul Patras (Univ. of Edinburgh), which is much more related to the current topics I'm working on. It is interesting to see they have introduced so many DL models for traffic forecasting such as STN, CloudLSTM, etc. An interesting work is ZipNet-GAN, which trys to get fine-grained traffic measurements from coarse ones (e.g., measurements aggregated every 10 minutes). The problem of ML research for communication systems and networks is the lack of public datasests. In this regard, some of the research papers have used public datasets "A multi-source dataset of urban life in the city of Milan and the Province of Trentino" and some from network operators which cannot be disclosed.

Finally, the last lecture is from Dermot Casey, a venture leader at NDRC for helping start-ups to grow. Take one step back from talking about all the ML/DL hypes at the moment, the main messages from this talk is focusing on the problem and customers for innovation.








Day-2


The second day is started by Panayotis Mertikopoulos (CNFR) who introduced online learning and optimization approaches with their use cases in the context of wireless communication systems, ranging from channel selection and adaptive routing to rate minimization in a multi-user MIMO network.

In the second lecture, James Little from NCI and ThinkSmarter who has been successful in many commercialized AI products such as Intucell (acquired by Cisco) for self-optimizing networks, talked about AI/ML from the commercialization point of view. This reminds me Andrew Ng said "Enough Papers, Let's Build AI Now!".  James Little mentioned that everything about AI/ML is good, but we need to think about use cases that match customers' thoughts.
AI has initial appeal, but must match the customers way of thinking

Another interesting talk is from Michaela Blott from XILINX research, where her research is mainly focusing on making hardware (e.g., Field Programmable Gate Arrays (FPGAs)) more efficient for training and inference of recent DL models.

Finally, the last talk of the second day was from Claire O'Connor, which focuses on how to communicate your scientific research with non-technical audiences. For example, a good way of doing it is "go with a use case that targets the audience". Some good practices can be trying to answer following questions or summerizing your research in an easy way:

  • What are you working on?
  • What will be the outcome of your work?
  • Boil your story down to a tweet





Day-3


The final day of the summer school consists of three research talks. The first one is from Marco Di Renzo (CNRS) who talked about Model-based or AI-based or hybrid approach for solving problems in networks and communications such as optimal resource allocation problem. Model-based denotes that we approximate the wireless network (WN) model, and apply optimization for optimal resource allocation. In contrast, AI-based denotes we use live measurement data from real system and use ML/DL to solve the problem. The main drawback of the first approach is that the difference between the WN model and the real systems, and that of the second approach is difficult to get real-system data. It is interesting to see transfer learning in this context where they use WN model to simulate and get enough data for training neural network model, and transferring the model with target domain which is the real system with a small amount of data to refine the model compared to training with only target domain data from scratch.

The second talk is from Yong Li (Tsinghua Univ.) with respect to learning users' mobility patterns with the collaboration between telecommunication companies such as China Telecom and tech companies such as Tencent. Despite the big data they got from these companies, it still has many challenges with respect to low quality such as

  • separate data 
  • low resolution (e.g., district or city level data)
  • part of data (only one telco given we are interested in the whole population's mobility)
The talk covered how to resolve these research challenges. After projecting this low quality data to high-quality one, a lot of interesting research has been done with respect to the application of the data such as predicting the move of an individual or crowd.

The final talk of the summer school is given by Jakob Hoydis (Bell Labs) with respect to DL for pysical layer. He talked about the idea of end-to-end learning through autoencoders. By interpreting a communications system as an autoencoder, they develop a fundamental new way to think about communications system design as an end-to-end reconstruction task that seeks to jointly optimize transmitter and receiver components in a single process. He also raised an interesting question with respect to ML/DL in networks and communications: when will the ImageNet moment for networks and communications be arrived. ML/DL which has been explored recent years especially started by computer vision (CV) research, we usally say ImageNet moment has been arrived for CV in 2015. And recently in NLP, one of my previous labmate Sebastian Ruder (now Research Scientist at DeepMind) also has a popular article saying that NLP's ImageNet moment has arrived.

SO WHEN WILL BE THE IMAGENET MOMENT FOR NETWORKS AND COMMUNICATIONS? ONLY TIME WILL TELL！

Summary

ML/DL has been widely adopted in at least academic research in communication systems and networks with a wide range of topics but not limited to:

  • traffic analysis, forcasting
  • adaptive rounting, channel selection
  • resource allocation
  • user mobility analysis
  • end-to-end communication system
  • ...
Real-world data is difficult to obtain for ML/DL, or even got the data, the quality is low in terms of requirements. Research data usually comes from following sources:

  • simulation (can get enough data to some extent, the difference gap from real data)
  • real-data (direct collaboration with telco operators, cannot be disclosed)
  • the combination of above

Some topics such as transfer learning, DL for enhancing data quality can help in this regard. 

Sooner or later, ImageNet moment for communication systems and networks will be arrived


No comments:
Labels: , , , , , , , ,

A Gentle Introduction of Important Terms in Time Series Analysis

Time-series data is everywhere such as sensor streams from Internet of Things with respect to traffic to the stock prices of each company in finance domain. As I recently started delving into time-series data, I found it is interesting and there are many new terms and methods to learn.

In the following, we overview important things and terms before delving into details of time-series analysis and forecasting.

Despite the data is relatively simpler compared to unstructed data such as images or natural langues, time-series analysis is very challenging.

First of all, the predictability of a time-series depends on those checklists below:

  • how well we understand the factors that contribute to the forecasting
  • how much data is available
  • whether the forecasted results impact the thing we would like to forecast
Secondly, we have to make it clear with end users who will use the forecast with respect to:
  • what should be forecasted?
  • what is the forecasting horizon? (e.g., forecasting 1 month in advance or 6 months in advance)
  • how frequently are forecasts required? 

Important Terms



Variable of interest is the variable we want to forecast.

Predictors are other variables used for predicting/forecasting the variable of interest.





Trends exist if there is a long-term increase or decrease in the data.

Seasonality exists when a time series is affected by seasonal factors such as the time of the year or the day of the week. Seasonality is always of a fixed and known frequency.

Cyclic occurs when the data exhibit rises and falls that are not of a fixed frequency.

Stationary: A stationary time series is one whose properties do not depend on the time at which the series is observed. This means that time series with trends, or with seasonality, are not stationary. (How to do stationary test in Python? KPSS statistical test)

Differencing is a technique to compute the differences between consecutive observations (e.g., $y_t'=y_t - y_{t-1}$) and usually is being used for making a non-stationary time series to be a stationary time series. Differencing can help

  • stabilise the mean of a time series by removing changes in the level of a time series, and
  • therefore eliminating (or reducing) trend and seasonality.

Sometimes, second-order differencing is required to make the time series to become stationary.

$y_t'=y_t' - y_{t-1}'$



No comments:
Labels: , , , , , , , ,

연구주제를 뽑는 방법/패턴들

소프트웨어 디자인 패턴처럼 연구주제 디자인 하는 패턴이 있다면 얼마나 좋을까? 모든 연구하는 사람들이라면 공감할만 할 주제인거 같다. 박사과정에 있던 프로젝트를 따와야 하던 연구소에서 향후 연구주제를 정하든 말이다.

UCSD의 Philip Guo교수님이 이와 관련해서 연구주제 디자인 패턴들이라는 유익한 글을 올렸다. 짧고 굵게 정리해보자면...


  • Seinfeld: 문제가 눈앞에 보이지만 누구도 터치하지 않았던 것들, 예를 들자면 Philip교수님은 많은 연구가 젊은 세대의 프로그래밍을 배우는 문제에 대해 연구했지만 나이대가 높은 층에 대해서는 연구가 거의 없어서 해당 문제를 제자들과 연구해보기로 했다.

  • Inside-out: 모든 사람들이 당연하다고 생각하는 문제에 대한 방법을 뒤집어 생각했을때 가능한 방안이 있는지 혹은 재미있는 연구주제가 있는지 보는 것.

  • All the things: 한가지 아이디어를 극한으로 밀어붙이는 것. 예를 들자면 코딩 디버깅하고 visualize하는 툴들은 대부분 일부분 변수정보를 보여주지만 모든 것을 보여주면 어떨까 하고 가정해보았다.

  • Best of both worlds: 두가지 현존하는 것들을 서로의 장점으로 각자의 단점을 보완하는 방법.

  • Back to the future: 기존의 아주 오래된 아이디어들을 재탄생 시키는 것.

  • Horizontal transfer: 다른 영역에 있는 좋은 아이디어를 자신의 연구영역에 응용하는 것.

  • Vertical transfer: 좀 이해가 잘 안가는 패턴이긴 하지만 낮은 레벨의 아이디를 높은 레벨의 목적에 도달하게 끔 하는 것.

  • Mr. Beast: 다른 사람들이 절대로 안 할 긴 시간이 필요한 연구를 하는 것.

  • Force of nature: 다른 사람들이 쉽게 못하는 기술과 인프라가 필요한 긴 시간이 필요한 프로젝트를 견지 하고 키우고 논문을 쌓아가는 것

  • Nelson-haha!: 다른 사람들이 할 수 없는 연구를 하는 것. 기업의 연구소 MSR이나 Huawei research나 구글 연구소 같은 곳에서 인턴하거나 연구원으로 있을 때 할 수 있는 연구는 Academic에서 할 수 없는 것들일 경우가 많다. 데이터 공유는 대부분 안되고 그 스케일 자체가 다르니까... 그래서 haha!라고 이름 졋다는...


[1]: Research Design Patterns - https://www.phrasemix.com/phrases/whats-the-deal-with-something
No comments:
Labels: , , , , , , ,

How to write a high quality review?


As a researcher, either you are pursuing a PhD. or an experienced researcher, we need to review conference or journal papers for the research community time to time. For example, your supervisor might assign you as a sub-reviewer to review paper(s) of your interests/expertise, or you might get invited more and more as you grow as an independent researcher in your field after your PhD.

Therefore, as a researcher in your research community, and since the review process pushes the research community moving forward, it is our responsibility to provide quality, constructive and not offensive reviews. Despite the importance of providing high quality reviews, unfortunately, we normally are not trained on this task during the PhD or afterwards.

Recently, professor Noah Smith from University of Washington gave a podcast which provides some fantastic advice on how to review a conference paper for the research community. Below, I summarize the main take-home messages from this podcast, as well as advice from other resources which might be helpful to reseachers who would like to write a high quality reviews.

Content

A review has three important parts:
  1. A brief summary of the paper
  2. Major pros and cons
  3. Minor things/comments/corrections
Let's discuss each part in the following in detail.

Brief Summary of the Paper

The first section provides a brief summary of the paper for others, e.g., an area chair who is extremely busy. This sectionn should:
  • say as it is, (what's new and contributions claimed by authors etc.)
  • that is, not judgement for this section
  • make it easy for others to get the idea of the paper quickly

Major Pros & Cons

Then we can move on to the pros & cons. Remember to give pros! of the paper. Even you do not like the paper, we should always analyze pros of a paper just as for finding cons of it.
  • particularly, we need to encourage (early) researchers keep going
  • there should be pros of a paper, e.g., the question itself is good and challenging

Even when we write cons of a paper, we still need to be constructive and positive. 
  • always give actionable details
  • e.g., instead of the method is confusing, talk about what equations or details are confusing

This section is main part of the review and needs to take into account of many key questions. 

For example, we can consider some of the questions below which include the ones from Professor Emery Berger (Computer Science at the University of Massachusetts Amherst):

Problem
  • Is the paper well-motivated? 
  • What problem does it address, and is it an important problem?
Quality
  • Is the submission technically sound? 
  • Are claims well supported by theoretical analysis or experimental results? Does the paper credibly support its claimed contributions? 
  • Is this a complete piece of work or work in progress? 
  • Does the paper describe something that has actually been implemented? If so, has it been evaluated properly? Is it publicly available so that these results can be verified? Are the results in the paper able to be reproduced easily?
  • Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
Clarity
  • Is the submission clearly written? Is the paper sufficiently clear that most venue attendees will be able to read and understand it? 
  • Is it well organized? (If not, please make constructive suggestions for improving its clarity.) 
  • Does it adequately inform the reader? (Note: a superbly written paper provides enough information for an expert reader to reproduce its results.)
  • What is the intuition behind certain choices?
Originality
  • What are the paper’s key insights? 
  • What are the paper’s key scientific and technical contributions? 
  • What did you learn from the paper? 
  • Does the paper significantly advance the state of the art or break new ground? 
  • Are the tasks or methods new? 
  • Is the work a novel combination of well-known techniques? 
  • Is it clear how this work differs from previous contributions? 
  • Is related work adequately cited? Does the paper clearly establish its context with respect to prior work? Does it discuss prior work accurately and completely? Are comparisons with previous work clear and explicit? 
  • Incremental approach could be challenged to accept, check what contribution is left after removing others.
Significance
  • Are the results important? 
  • Are others (researchers or practitioners) likely to use the ideas or build on them? 
  • Does the submission address a difficult task in a better way than previous work? 
  • Does it advance the state of the art in a demonstrable way? 
  • Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
  • What impact is this paper likely to have (on theory & practice)? Is the work of broad appeal and interest to the research community?




Minor Things/Comments/Corrections

This section can provide some other minor issues, such as grammer errors. Please note that these minor things are not enough to reject a paper though.




About Reviewing Process

Prof. Noah Smith also discussed about his personal process of reviewing a paper, which is useful to me to adopt for future reviews.
  • quick scan (introduction, figures, conclusions, reference list)
    • where it is positioned in the literature?
    • what kind of paper it is? (position paper? theoretical one? system one?)
  • top-down reading with red pen
  • come back later with my review notes
  • any unclear things can be asked as a reviewer
    • e.g., the explaination/notation/prove is confusing or did the preprocessing also applied to baselines etc.
    • note that it's not criticizing, it is just want to encourage the paper make clear on the next version


Summary

To sum up, in this post we discussed the structure of review, and the review process and a couple of things can be considered or checked while reviewing a paper. 

Finally, we should keep in mind that the reviews that we write should
  • help the authors to improve the paper 
  • be what we expect to see in the next version of the paper

Did you have experience or any thought or advice which might be helpful for writing a high quality review? Leave a comment to share it with others:)


Related resources

No comments:
Labels: , , , , , ,
Subscribe to: Posts (Atom)