This is a 3 part blog series.
Research Impact Part 1: Moving Away from tracking authors’ articles.
I am not going to mentioned the company we went with. The primary reason for this is because I am trying to write this as broadly as possible so that it applies to anybody who is considering this type of endeavor, not the nitty gritty of a specific software. While there is always room for improvement I am happy with what we chose and I am very happy with the support we have received upon implementing it. If you are interested in learning more about the specific products we chose, email me and I will answer those questions.
As I mentioned in Part 1, there are a lot of products out there, Converis, InCites, Profiles, Pure, Plum, etc. After looking at several products we ended up choosing two products by the same vendor. The two products allowed us to upload HR data so that articles would be automatically sorted and indexed by author AND department, and it also included article level metrics that were more informative than just the journal impact factor.
There were a few major points that we had have in our system. I recommend creating your list or requirements BEFORE you start contacting vendors because it easy to get caught up in all of the cool things their products can do which may or may not be compatible with your needs. For example, you don’t want to get excited about new dishwasher that has a new wash cycle that gets your dishes so clean it could wash the white off of them when that model only comes with large handle that blocks your silverware drawer making it necessary to always open the dishwasher before opening the silverware drawer (or completely re-design your kitchen). So have your must haves ahead of time.
Our must haves:
Automation – That sounds stupid but there are some systems that are more automated than others. All require some human touching even after implementation. Think about how much time you want/can spend on the system once it is all up and running.
Institutional organization structure – It must be able to organize published articles from all of our employees by department and institute. (Institute has several departments within it.) This was a requirement because Administration wants to know the authorial output of each department and institute and annual performance review time. So we need to click on Urology and see the papers written by people in Urology. Do you need to track secondary appointments? Be careful that can be a long dark rabbit hole to go down.
Impact – While almost all of the products we looked at had some type of article impact number/indicator we needed to communicate with Administration as to the one that THEY wanted and felt the most comfortable with. This is VERY important. There are about as meany methods of measurement as there are digits in pi. Our Administration is very traditional, so that required us to look at product that used well established metrics that have been around for many years that our people were familiar with.
Things we didn’t need:
Repository system – Currently our institution has no interest in hosting a repository of the papers written by authors. Obviously this could change, and if it does then it requires a fresh new look at things.
Author submission – Authors are not reliable providers of the citations they publish. We had 20+ years of experience with this. Some authors don’t have the time to upload anything. Some authors add citations to their weight loss article in Ladies Home Journal. Other authors have citations that say “in press” from 5 years ago. Your data out is only as good as the data you put in, and we needed tight control over the data. So we didn’t want author submission. If it was a feature, it had to be something that could be turned off.
ORCID – That sounds odd. We actually need ORCID. Everybody needs ORCID. Until there is a mandate that requires an author to provide their ORCID number upon publication then ORCID will just be something “nice to have.” Even in a heavy research and publishing institution, ORCID is still something of a novelty. We did not want something that was overly built on ORCID.
Panacea systems – Many of these products track everything under the sun. They track grants, funding, etc. There are systems that track the entire research life cycle, from the sparkle in that researcher’s eye to the mature cited paper and everything in between. Like many institutions we have a various systems (some home grown) that track a lot of things that the “all in one” systems track. Unless you have buy in across the institution to change every part of the research IT process then an all in one system may be overkill.
Lessons we learned:
HR or organizational data is messy – Unfortunately this is not unique to us. I have heard from people at several large institutions to learn that this type of data is often not clean. What do I mean by that? Assuming HR will let you have an HR dump of all employees (they are often very reluctant to do this) you might discover that there is missing or duplicate information. You might find out that several people’s secondary appointment is an entire hospital (not a department). You might find that HR data doesn’t include graduate students. We had to piece together our data with several institutional systems and we created a python script to strip, clean, and organize the data into the format that they vendor used.
Updated Paragraph 5/19/17
Regarding HR uploads….. Think very carefully about if you want your entire HR list of employees added into the system. That could be a lot of unnecessary data. Do you need/want people in environmental services, security, IT, etc. in your system? Do you only want doctors and researchers? What about nurses, PA’s, medical students, residents, and allied health who publish? You need to sit down and figure who you want to track and how you are going to get that list of people.
Comparisons – You have to be very mindful if you use one of these systems to compare your institution with another. If your administration is competitive and likes to see how they are ranked in their disciplines or overall, they are going to ask you to use the product to compare themselves against their peers. Most of the products we looked at could compare different institutions, disciplines, and people. But you must do this carefully. For example: You cannot compare a large research hospital system with university hospital system. Even though they are peers, the university system includes many more researchers and disciplines that can skew the results. While you can compare disciplines or subjects, you cannot compare departments. One institution’s cardiovascular department may include pediatric cardiology while another may not.