The Research System – Its problems and what we can do about it

This time I would like to talk about our current research system. With research system I mean the framework in which research is conducted at the beginning of the 21st century. In my opinion there are many different problems with the current system, some of which lead to false incentives and ultimately corrupt science. I will start with a discussion of what I think are the main issues with our research system. After that I will talk about some ways in which we researchers could actively influence and change this system into a better one. At the end I will present some ideas of how I personally want to cope with this flawed system in the future. I can already tell you at this point that I am not leaving academia. At least not now.

The problems of our research system

There exist many blog posts and articles about the problems of the research system. Most researchers are aware of these issues or even frequently experience them. The huge problem is that nobody is doing anything about it. With our passive behavior we are gambling the reputation of science. This needs to change.

We are conducting research within a framework that consists for the most part of non-democratic structures. There are huge differences with respect to the power and influence of different researchers within this framework, which cannot be simply explained by the quality of their work. There are third parties involved, mainly in the publication process, which impede our research work, mostly because of financial reasons. We decide whether a work is worth publishing or not using a system which cannot work for very simple game theoretical considerations. And finally, we measure the quality of research and the success of researchers using certain flawed indicators. This all together leads to many false incentives for all the parties involved in that process. This might not be a problem in other situations, but here we deal with persons which presumably belong to the smartest in our society. And therefore it is of greatest importance that we work within a system which gives the right incentives in order to perform good research.

Indicators

To a large extend we measure quality and success based on indicators. The quality of a publication is measured by the number of citations it gets. The quality of journals is measured by the impact factor, which is the average of the number of citations from publications within this journal. Our own success is measured by the number of publications we publish and the number of citations our publications receive. Researchgate uses indicators called impact points and rg scores and google scholar uses indicators called h-index and i10-index to measure our success.

The problem with all these indicators is that they do not accurately reflect the quality of our work. At the very best there is a slightly positive correlation. Let me give you an example. At google scholar I have 163 citations, an h-index of 8 and an i10-index of 7, at researchgate I have 6.85 impact points and an rg-score of 8.28. Another researcher has 8150 citations, an h-index of 34 and an i10-index of 57 at google scholar and 72.06 impact point and an rg-score of 28.95 at researchgate. Comparing these indicators, it seems that this researcher is performing much better research than me. What if I tell you that this researcher is Zong Woo Geem, whose career is based on his “invention” of the dubious harmony search algorithm and whose academic fraud was recently exposed by myself (see my previous blog post for more details: The Harmony Search Algorithm – My personal experience with this “novel” metaheuristic)?

If these indicators do not accurately reflect the quality of our work, why are we actually using them to measure the quality of our work? Instead of conducting high quality research, the system gives incentives to optimize indicators. And such a system obviously leads to a research output which improves the indicators and not necessarily to a research output of high quality. This is definitely not a feature we want to have in a research system.

Peer reviewing

We use peer reviewing to decide what is getting published in journals and what is getting accepted for conferences. There are several studies which show that peer reviewing does not really work as intended. For example, some studies show that results of peer reviewing are extremely stochastic. Therefore a small number of reviewers, which is commonly used for journal articles and conference submissions, cannot be used to adequately estimate the quality of a given work. Let me give you another argument for why peer reviewing does not work. For this purpose, lets look at it from a game theoretical point of view (I am not aware of any such treatment in literature, please let me know if you know any such sources). So lets assume that you are asked to review some manuscript. What are your incentives to do a good job? Apart from any ethical reasons or issues with your conscience, I cannot think of such incentives. But are there incentives for not making a good job? In my opinion there are a lot of such incentives. It makes sense not to invest too much time for the review, since in this way you could better devote your time to other parts of your research work, maybe something which might help you with your own career. It also makes sense to be biased with your assessment of the manuscript. It is very likely that the authors of the manuscript work in the same field as you do. If they are friends you might want to be very positive in your review, or if the manuscript contains serious flaws, you might want to prevent them from harm. If they are competitors you could do the exact opposite, you might want to be very negative with your review, or if the manuscript contains serious flaws, you might want to suggest publication in order to criticize the authors later. Game theoretically speaking, this leads to a kind of dilemma situation between behaving ethically correct and doing a good job on the one side and behaving egoistic and possibly doing a bad job on purpose on the other side.

Some of those issues are thoroughly covered and well-known. Still we heavily rely on peer reviewing to assess the quality of our work. Most researchers agree with me, that peer reviewing is flawed, but then they justify its usage based on the fact that there is no better method available. It hurts to hear such an argument from researchers, who are supposed to change things and to facilitate progress. At certain times there was no better system known than slavery or monarchy. People slowly understood that these systems were flawed. Instead of keeping them, because no alternatives were available, they were actually looking for alternatives (By the way, this should also trigger us to think critically about the political and economical system in which we are living). This is what we should do as well. Instead of relying on an approach, which we all know is seriously flawed, we should try to find alternatives. I will come back to this issue later in this blog post.

Publishers

The role of publishing companies in today’s research system is kind of awkward. They run a big business with huge margins for profit, but their benefits to research are not clear at all. I would even go one step further and say that they are actively impeding research work. Lets have a closer look at that.

It is, and always was, of great importance to make research results available to others. In the past this has been done in the form of letters, journals or books. Here publishers played a key role. For them the whole thing was a business with financial incentives. They were providing a certain service which was important for the process of communicating research results. They were not necessarily interested in promoting excellent research, but since they provided the research community with a service which had a certain value, this seems completely fair in my opinion.

This situation has changed dramatically due to the Internet and the personal computer. Using modern editing tools on personal computers, we are actually preparing high quality papers (here I mean high quality with respect to typesetting and related issues), which are basically ready for publishing. Additionally, the Internet allows us to easily make our research results instantaneously available to the wide public. But instead of that we are still using those old structures that have developed in the past, and we still publish our results with the help of publishing companies.

The crucial question here is, which kind of service are those companies actually providing to the research community? I personally do not see any kind of service they are actually providing, except their existing structures (which are probably not even a service in that sense). In my field we submit so-called manuscripts to conferences and journals. In the original meaning, “[a] manuscript is any document written by hand or typewritten, as opposed to being mechanically printed or reproduced in some automated way.” What an irony, nowadays our manuscripts are basically pdf documents which are ready for publishing. That means, the only thing these publishing companies are doing, is, apart from some minor things, to take our documents and to print them or just to put them online. For obvious reasons, we do not need printed versions anymore in the 21st century. That means they are taking our documents and putting them in the Internet. Is that really a service? And on top of that they make huge profits with that system.

And it is getting even worse. Using these kind of obsolete structures, they actively impede research. First of all, results could be made available much faster without using the “services” of publishing companies. But the real problem is, that the publishing companies are aware of the fact that they are actually providing little or no service. For this reason they try to make us dependent on these obsolete structures in order to continue with their antiquated business model. This happens on many different levels. For example, in the form of so-called “top” or “high impact” journals, by enforcing us to rely on indicators, or by “sponsoring” research. Another point is that their profits rely heavily on the number of publications they are able to sell back, for example to universities. This gives them a financial incentive to publish vast quantities of papers and they try to impose this incentive on us researchers.

The whole point is, we do not need these sort of publishing companies anymore. Using personal computers and the Internet, we should be able to easily communicate our research results without the help of any third party involved.

Power structures

The current research system relies on several non-democratic structures and on processes which are far from transparent. In this environment, it is extremely difficult, to obtain some idealistic purpose such as good research. At this point I would like to briefly discuss two of these issues, the typical career path in academia and the power of researchers that are journal editors or part of a conference committee.

The typical career in academia consists of something like finishing your studies, doing a PhD, working a few years as a PostDoc and then trying to somehow get a professor position or quit at some point in between. This is a very inflexible career path which comes at the cost of many inherent problems. Apart from that, it is commonly assumed that as a PhD student you are basically thriving on the ideas of your supervisor and that as a PostDoc you slowly start with independent research, still under the supervision of a professor. While in reality, some PhD students are already doing great independent research and their supervisors are simply exploiting them (in order to optimize their own indicators). I am doing independent research since years, I have been exploited, and still I have to prove that I am able to conduct independent research. This whole career path is strongly intertwined with the indicators I have discussed before. In order to advance in your career, you have to optimize your indicators. Nobody really cares about your potential as a researcher. There are definitely many brilliant researchers who manage to make a successful career in academia, but due to our increasing focus on indicators, there is also an increasing amount of researchers which make a successful career because they are better in optimizing their indicators or simply because they know how to play the game. And this creates a vicious circle.

The other point is that certain researchers are equipped with an immense power in the current research system. Journal editors and committee members of conferences have the power to shape the direction of research. They are those people who decide what gets published and what does not get published. The problem is not that there are people who have such a power, the problem is that they are selected in a highly non-democratic way and that they are able to execute their power in a non-transparent way. No wonder that we are supposed to do excessive networking at conferences instead of research.

How can we change this situation?

The current state of the research system is not acceptable. But it will not change if we just hope that those parties who profit from this system will alter their behavior and act in a more idealistic way. If we want to change something, we have to do it ourselves. The reason why I still want to continue in academia is, that I somehow hope it is possible to change things. If it turns out during the next years that it is not possible to change anything, then you will find me among those researchers which simply quit and leave academia (maybe with another of those infamous blog posts). Maybe I will find another way to pursue my passion of research, but I will definitely not become some indicator optimizer.

So, how can we change the situation? This is a complex and difficult task and I would like to invite everyone who managed to read that far, to discuss this with me (by email, by twitter @dennisweyland, or by commenting on this blog post). I have three suggestions that I would like to share here.

The first suggestion is to stop playing the game. Forget about your indicators, stop reviewing manuscripts for the big publishing companies without compensation, and stop submitting your manuscripts to journals run by the big publishing companies. If we just continue as before without leaving our comfort zone, it is very unlikely that anything will change. Question everything about the research system and draw your own conclusions. If we all try that, I am sure change will come.

My second suggestion is to develop in a collaborative effort an alternative publishing model based on the Internet. I have a pretty clear vision of what I would consider an ideal publishing model and I think I might write a separate blog post about this topic in the near future. But it is important that we shape the system in a way which suits all of us.

The last suggestion is simply to create a union for researchers. In this way we can organize ourselves and obtain some political power in order to change things as project funding or publishing policies on the political level. The structures that are imposed on us are not the structures we want and maybe we can change certain things in this way.

My personal research policies

In order to give you some examples of how we could change our own behavior regarding the current research system, I would like to discuss some changes of my own research policies.

As I said, for the moment I will try to continue in academia. Unfortunately, I am not in a position to drastically change policies on a higher level. But there are some small things which I can do (and which basically everyone could do). First of all, I will not participate anymore in the peer reviewing process under the current conditions. Well, peer reviewing gives me a certain power over my peers and in theory I could use it for the overall good. But this is exactly one of the many problems with peer reviewing. It is a non-transparent system which empowers the reviewers over their peers. You might say that my behavior is not fair, because other researchers will still review my manuscripts. I can assure you that this will not be the case, because I also plan to publish my research results in an alternative way. I will put them on the Internet, freely available for everyone and open to discourse. This does not mean that I believe that my work is free from defects and does not need any sort of reviewing. If there is something wrong with my papers or there are reasons to change parts of them, I will be more than happy to do it.

Conclusions

The current research system is fundamentally flawed. The different problems are extremely intertwined and facilitate an environment which is diametrically opposed to conducting good research. It will require our collaborative effort to change things. This is definitely not an easy task, but I still hope it is somehow possible. Please let me know (by email, by twitter @dennisweyland, or by commenting on this blog post) what you think about this whole issue.

2 thoughts on “The Research System – Its problems and what we can do about it”

  1. I read through your earlier post, it sounds good that you wrote something formal about the sketchiness of that research area.

    Did you see the work of Levitt mentioned here? http://alexandreafonso.me/2013/11/21/how-academia-resembles-a-drug-gang/

    A lot of academia is different people struggling for attention. I am not too jaded because in my experience, well-respected people with tenured positions at top universities are indeed deserving of it; by and large they seem hard-working, caring, and intelligent, notwithstanding that we all have our own biases. In less respected institutes though I think it is more true that you end up with people who tend to toot their own horn too much.

    I think it’s important not to lose sight of your own work, that what you do yourself is more important than any crappiness of other parts of the “system,” because it’s not a really a “system” in the first place. It’s not a designed or integrated whole. Think of it as a loose affiliation of publishers and academians. And note that some schools may hire anyone they need to teach without necessarily endorsing them as a top researcher. I do think that overall, over time, everything gets filtered to the level where they belong, even if it takes longer for some things than others.

Leave a Reply to Ian Murray Cancel reply

Your email address will not be published. Required fields are marked *